Feb 242017
 

Firefox + HTML5 + XP logo1.) Introduction & Explanation

This is one thing that has brought to me by two users ([SK1] on [Voodooalert]German flag and [liquidLD] who talked to me about this on IRC), and because I got a bit pissed off by it myself, I decided to look into the matter. Basically, HTML5 video on Windows XP / XP x64. But not just with webm (VP8/VP9), but also with H.264/AVC. Let’s face it, a lot of videos on the web rely on H.264 and sometimes you simply can’t watch certain videos or you won’t get all the available resolutions. Of course you could just rely on Adobe Flash, but since Google basically took over with their Pepperflash plugin and their Chrome browser no longer supports XP, it’s not the best move either. Especially when you think about Adobes’ history with critical security loopholes in Flash. HTML5 is just much, much safer, and free as well, and Firefox still supports XP.

Note that this guide is thus based on Firefox exclusively. Anything starting with version 47 should work, official support came in 49, and I’ll be using the current version, 51.0.1 at the time of writing.

So, why doesn’t it “just work” in the first place? It did a few years back, right? Because H.264 playback relies on a DRM plugin, on Linux it would be the Google Widevine plugin, on Windows it’s the Adobe Primetime plugin. So yes, Firefox does support DRM out of the box. But even if content isn’t signed and encrypted, the browser still relies on those plugins to play H.264. And the problem is, that Adobe found some problems with that plugin on XP, so they disabled support on the platform. Their version 17 plugin is still being rolled out with the browser however, and it is binary-compatible with XP, so let’s show you how to re-enable it!

2.) Making it work

On Windows XP and XP x64, the plugin should reside in the folder:

%USERPROFILE%\Application Data\Mozilla\Firefox\Profiles\<your profile folder>\gmp-eme-adobe\17\

That folder should contain the files eme-adobe.dll, eme-adobe.info and eme-adobe.voucher. If it doesn’t (maybe because you have a DRM-free version of Firefox), just create the folder structure yourself, get the necessary files from [here] and place them in that folder.

Having the files present won’t enable Adobe Primetime for you however as you can see on about:plugins (Note: The Cisco stuff you can see there is just for WebRTC, so it’s unusable for HTML5 <video>), we still need to tweak a few things on the about:config page of Firefox. Look for the following properties and set them to the values shown below. If a property doesn’t exist yet – media.gmp-eme-adobe.forceSupported most likely won’t – just create them yourself, all of them are boolean properties and all of them need to be set to true:

media.gmp-eme-adobe.enabled		true
media.gmp-eme-adobe.forceSupported	true
media.gmp-eme-adobe.visible		true
media.gmp.decoder.enabled		true
media.eme.enabled			true
media.mediasource.mp4.enabled		true
media.mp4.enabled			true

After making those changes, you’ll need to restart Firefox. Now you might already be good to go, but on some configurations, about:plugins might show something like this:

HTML5+H.264 on Firefox not yet working

Adobe Primetime seems enabled, but there is no file information? So it’s not actually loading the eme-adobe.dll yet (click to enlarge)

If that happens, open your preferences menu on the top right, click on “Add-ons”, then “Plugins” or just go to about:addons. What you should be seeing is this:

 

However, if Adobe Primetime shows a notice saying that it’s going to be “installed shortly”, forget it. Just do it manually on the plugins’ options page you can see on the right image. To do so, click “Check for Updates”. The warning should be gone momentarily. After that, re-check about:plugins, and you should be getting this:

Adobe Primetime fully enabled

Adobe Primetime fully enabled (click to enlarge)

3.) Testing

Now you can do a quick check on the [Youtube HTML5 page], and it should confirm that everything’s working:

Youtube confirming full HTML5 video support

Youtube confirming full HTML5 video support including H.264 and Media Source Extensions (click to enlarge)

With MSE, even Javascript players (like the Flowworks player) bytestreaming H.264 to Firefox should work! Of course, that’s not very thorough. What you’d want is a real playback test, since you can never be sure what you’re getting on Youtube without a bit of extra work. Decent playback tests are currently available on [Quirksmode], and it should look like this:

Firefox playing HTML5 H.264/AVC video in Firefox on Windows XP x64

Firefox playing HTML5 H.264/AVC video on Windows XP x64 (click to enlarge)

With this, even stuff like Netflix works, because you’re getting not just H.264 playback, but also DRM support. Now, whether DRM support is a good thing or not… You’ll have to decide that for yourself. I’m not supportive of DRM content on the web, but if you want to view or listen to such content, you can!

Just one last word of warning though: Adobe has ended their support for XP with a reason, as the Primetime content decryption plugin has shown problems and instabilities on XP! I’ve been using this for about a week now, and I’ve had one case of a video getting stuck, which is a typical symptom of Primetime throwing up on you. Don’t worry though, Firefox won’t crash. Just move the video slider a bit or restart the video, and it’ll work again! You don’t even need to restart the browser, and such occurrences seem to be quite rare, so I’m fine with it.

There you go!

4.) Thanks

Big thanks fly out to [the guys at MSFN] who came up with all of this. I basically got 100% of my information from them, so thank you! You rock! :)

Update: If you update your version of Firefox to the latest and final 52.0 ESR (extended support release), the last version which will be officially supported until 09-2017 for XP, you might notice that Adobe Primetime just disappeared after the update. That’s because the installer may delete the property media.gmp-eme-adobe.visible from your prefs.js. To reenable it, you’ll have to manually recreate the boolean property and set it to true:

media.gmp-eme-adobe.visible		true

Restart Firefox after the change, and the plugin should reappear on about:plugins and about:addons!

Nov 242016
 

Broken Windows logo[1] I know what I should do if a system service on Microsoft Windows starts crashing of course; Fixing it is the way to go! But sometimes you simply can’t, because the component causing a certain instability can’t be swapped out or updated. Now Windows services do have a mechanism for monitoring and restarting a service upon failure, but it seems that only works if the system gets an actual error code back from the service upon termination. But it doesn’t seem to work (at least for me) if the service just dies abnormally. Windows recognizes the service has stopped somehow of course, but the restart procedure just doesn’t kick in.

So I thought I’d do it myself, programmatically. And it’s actually pretty easy. I solved this with VBScript, Windows Batch and Mark Russinovichs’ pslist plus grep. So the prerequisites are:

  • Microsoft Windows (well, huh..)
  • MS Windows Script(ing) Host / VBScript, Windows should come with this preinstalled since Windows 2000.
  • [pslist]
  • [grep][src] (grep is optional, I used GNU grep 2.5.4 in this case, licensed under the [GPLv3+])

Make sure the pstools and grep are within your %PATH%, so Windows can find those .exe files. If you don’t want to use grep, you can also use Microsofts’ own find command, if your version of Windows has it.

I divided this into two small scripts. Since the main part is Batch, it might be problematic if you run it at very short intervals, checking for the services’ status, because you get a command window popping up on the desktop. Since most users wouldn’t want that, another script acts as a launcher, hiding the cmd.exe window so it’s run fully in the background without disturbing any potential users or administrators. The launcher looks like this, in my case it’s meant to watch over an Apache web server:

  1. Set WshShell = CreateObject("WScript.Shell")
  2. WshShell.Run chr(34) & "C:\Server\Scripts\monitor-httpd.bat" & Chr(34), 0
  3. Set WshShell = Nothing

And that script C:\Server\Scripts\monitor-httpd.bat we’re launching looks like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| grep httpd`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

A version relying on Microsoft find instead of GNU grep could look like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| find /I "httpd"`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

To get a services’ exact name, just launch services.msc from Start \ Run or run the command net start on a cmd terminal.

As you can see, this greps “httpd” from the process list and pushes its output into %%I and finally into %HTTPDSTATUS%. We have to use a FOR /F for that, as Windows has no way of pushing command outputs from subshells into shell variables like UNIX has (like e.g. var=`command` or var=$(command)). Then we check for the status of that variable. If it’s not defined, then the process http.exe was nowhere to be found! In that case we restart the associated system service (needs proper permissions!). If the variable is defined, we do nothing but unsetting it, since we can assume the service is operating normally. Or at the very least it’s running. ;)

You can automate that by using the Windows task scheduler:

Scheduling an Apache web server "watchdog"

Scheduling an Apache web server “watchdog” (German Windows)

Create a Schedule to your liking and you’re done! If you can afford the affected service to be down for 5 minutes and no longer, just run it every 4 minutes or so.

The solution shown above can easily be adapted to monitor and restart any Windows service you have, as long as the service isn’t fundamentally broken so that it wouldn’t even start up anymore. Also, you can do a lot more, like sending notification eMails with a command line mailer like [blat] when crashes do occur. Of course, this is only useful for services that crash rarely. If it dies every few minutes, you should reaaally fix it instead of just pushing the restart button all the time… ;)

And that’s that!

[1] © Mar.0007. Original Version for desktopwallpapers4.me.

Aug 052016
 

VirtualDimension logoShort story: It’s [VirtualDimension].

Long story… It’s most definitely not what Microsoft added in Windows 10. Besides it being limited to Windows 10, it just sucks for a multitude of reasons. And there I was, having hopes for it as well… If you’ve ever used multiple desktops on a graphical, X11-based Linux or UNIX window manager / desktop environment, you’d know what I’m talking about. Usually, what you’d get on those systems, whether KDE, Gnome, Xfce4, LXDE, or whatever is just one small, configurable panel which allows you to control multiple virtual desktops. On my Gnome 2 on CentOS 6.8 Linux, it looks like this (others are very similar):

Virtual desktops on Gnome 2

Virtual desktops on Gnome 2

The leftmost desktop is my usual “Internet” environment, here I chat, read emails, browse the web for anything work-related and so on. The second one is a Linux distribution development desktop. Here I’m building a derivative of Klaus Knoppers’ [Knoppix] distro. Then comes the testing environment for said distribution on desktop #3. Usually some shells and one VMware Player instance. Next to that are two more VM desktops for software testing and for writing user guides for software installation on different operating systems. At the moment that’s MacOS X and a Windows XP x64 software build VM. Usually there’s also a Windows 7 one. One is empty (for arbitrary stuff), then comes the server administration desktop with 9 open shells, one for each server. And the last one is my private desktop with yet another web browser, and some shells for spawning screen sessions for running software compilations, encoding runs and the likes.

Now, I have a 30″ screen both at home and at work, resolution is 2560×1600. But it’s just never enough screen real estate. So I wanted well-integrated virtual desktops for Windows as well, but last time I tried out some software, I couldn’t find a good one. Recently, I tried again for some reason, like “let’s give this one last shot”. And I tried a lot of programs!

Among the software tested were [Dexpot], [Finestra], [VirtuaWin], [WindowsPager], Xilisoft [Multiple Desktops] and the Windows PowerToy predecessor of [Desktops 2.0] written by Windows Hacker Mark Russinovich and Bryce Cogswell. And finally, [VirtualDimension]. Some of those are free and open source software, others are not.

One of my primary requirements was compatibility to Windows XP x64. Of course it’d be nice if it worked on Windows 10 as well. But most of the above had important features missing or severely misbehaving on XP. Some were just very, very sluggish when switching desktops. Others had missing features to begin with, like previews on the desktop tiles. A blank desktop tile doesn’t help at all, as I need to see roughly what’s running where at a glance.

I’m not gonna make this a lengthy top list or anything, I’m just gonna show you what the software of my choice – VirtualDimension – could do for me, let’s look at the tiles first:

VirtualDimension on XP x64

VirtualDimension on XP x64

We’ll start with my good old XP x64 first. Here you can see my system tray, and Miranda being open. VirtualDimension cannot be embedded into the taskbar properly (damn), but it has an “always on top” feature. Since the contact list in my docked and always-launched Miranda doesn’t go all the way down, there is free and unused space there. Perfect for VirtualDimension! And since it’s always on top, it doesn’t disappear when clicking on Miranda for chatting.

Given the source code is definitely coming from a UNIX or Linux user (given he built it with GCC/Mingw), some features immediately ring a bell. Like “mouse warp”, where you switch desktops by moving your mouse to the border of the screen. I disabled that, don’t like it. But yeah, it’s there.

Important: While it doesn’t give you live window geometry previews, it does give you iconized previews, so you can always identify any desktop quickly by seeing what’s running there. The desktops can also be named, and there is an OSD that you can have pop up on you when switching, like so:

VirtualDimension OSD

OSD showing right after a desktop switch

In this case I had just switched to desktop #2, which is for A/V processing exclusively. This is just the top left part of the screen, where one of my eight transcoding shells was running a x265 benchmark prototype test. Color, display duration, transparency to mouse clicks on the OSD part, font and size are configurable.

Also, you can freely define keyboard shortcuts for switching desktops as well. I chose CTRL+Shift+Right as well as CTRL+Shift+Left for switching desktops and Alt+Right / Alt+Left for pushing a window to the next/previous desktop as those don’t conflict with other shortcuts I’m using.

What else can it do? Let’s right click on one of the icons in the preview tiles:

VirtualDimension iconized window right click

Clicking on a program icon in VirtualDimensions’ preview tiles gives you this menu

The first five options from the top are global ones. However, the ones below are specific to the icon you right-clicked. With “Activate”, you’d switch to the target desktop and put focus on that programs’ window. The others are pretty self-explanatory as well I guess. We also get a graceful “Close” option, and a brutal “Kill” option that’s equivalent to murdering the process in task manager. Maybe useful since it’s faster that way.

And if we click on the free area?

VirtualDimension, right click on the free area of a preview tile

Right-clicking on the free area of a preview tile gives you a list of all programs on that desktop.

Ok, not sure how useful that is, but at least it may help with identifying the windows on a desktop in more detail, as you get the window titles here. For my encoding shells I could get very quick glance at the progress, but not exactly in great detail. So the helpfulness of this is limited.

What else?

Well, it’s extremely fast! That’s one major plus for VirtualDimension, as several of the other solutions (open source ones as well) were abysmally slow, at least on XP x64. But damn, VirtualDimension just flies! And its memory footprint is minimal. I saw less than 12MiB of consumption here. Even if you add a truckload of Desktops (there seems to be no upper limit), it just won’t slow down unless you spawn like 50 CPU intensive processes all over the place killing your CPU or maxing out your RAM. But that wouldn’t have been VirtualDimensions fault then. Its memory footprint will linearly grow by spawning more desktops, so with eight you may see around 20MiB. Still neat.

And what’s bad about it?

Well, sometimes, if you have a lot of windows on one desktop, the icons are’t cut off in the right spot at the bottom of the preview tile, so they overflow just a little bit. Just a cosmetic issue. Also, you should maybe deactivate the shell integration. With this, VirtualDimension hooks itself into all windows (such a DLL hook means entering another processes’ memory area). With that, you can get its functions via right clicking on a windows’ title bar, like on UNIX.

Nice, but dangerous! This can trigger anti-cheat systems in online games, because they really don’t like you stepping into their processes’ memory areas! That’s what cheating tools do to modify a games’ parameters on the fly as well. You don’t wanna be banned because of such a thing!

In my case, I managed to lock myself out of Mechwarrior Online because of this. I wasn’t banned, but the login process wouldn’t even let me launch its window. Disabling the feature, launching MWO, then re-enabling it and trying to log in caused a pretty abnormal process termination:

Mechwarrior Online really doesn't like VirtualDimensions' shell integration feature

Mechwarrior Online really doesn’t like VirtualDimensions’ shell integration feature! And no, there was no “update available”… (click to enlarge)

There is an exception list for this feature, and I added all of MWOs’ .exe files to it, to no avail. Better to stay away from this one.

Now, well, this otherwise beautiful piece of software was dropped by its developer around 2005. About the time my XP x64 came out. Latest alpha build is from some time in 2006. So this is ancient! It even supports Windows 98 and NT 4.0, I mean… So, how about Windows 10 then? I mean, Windows 10 doesn’t even have a GDI UI anymore, this is like one completely different world. Since I do have a Windows 10 machine (yeah, ew), let’s check it out:

VirtualDimension on Windows 10

VirtualDimension on Windows 10 – hey, it really works!?

Miranda seemingly can’t dock properly on Windows 10 anymore. It kinda… floats near the desktop border when docked. It’s strange and it wastes space, but well, I don’t know how to fix that yet. But anyway, I embedded VirtualDimension into Miranda (by just moving the window there, removing its title bar and resizing it properly again). And guess what?

It just works™!

I launched some Metro / Modern UI apps in a window as well, and while those aren’t shown in the preview tiles, they can be controlled with keyboard shortcuts, just like regular windows. Also, it’s just as blazing fast as it is on XP. Ah, and… yeah, it actually does work on all 64-bit x86 Windows versions it seems! It’s amazing, but an ancient piece of 32-bit software that does alter a Windows systems’ usage pattern quite fundamentally still works fine on Windows 10 64-bit. I gotta say, I’m pretty relieved, because Windows 10s’ own solution just sucks – where is my live preview? – and I don’t want to change my usage paradigm too much when switching operating systems (even from Linux/UNIX to Windows and back).

Some of the other solutions like Dexpot or Finestra may be faster on Windows 10 then they are on a just half-supported XP x64, but nah, don’t need them.

VirtualDimension is as perfect as it gets, despite its age! Or maybe because of it?

Still, anyone interested in picking up that [VirtualDimension project on SourceForge] and in continuing its development? ;) I guess I can’t touch that code, would probably just mess everything up. Ah, it’s C++ by the way…

A few things could use some fixing, like the icon overflowing issue, Modern UI window detection, certain, rare windows being sticky on all Desktops even if nobody told them to do so (Miranda, X-Chat DCC windows) and that exclusion list for the shell integration, which doesn’t hook into all windows properly when active either anyway.

Would be so nice if somebody could continue working this! :)

Until that happens (I know, it never will), I’ll just continue using v0.95 alpha. ;)

Jul 272016
 

x264 logoSince I’ve been doing a bit of Anime batch video transcoding with x264 and x265 in the last few months, I thought I’d document this for myself here. My goal was to loop over an arbitrary amount of episodes and just batch-transcode them all at once. And that on three different operating systems: Windows (XP x64), Linux (CentOS 6.8 x86_64) and FreeBSD 10.3 UNIX, x86_64. Since I’ve started to split the work across multiple machines, I lost track of what was where and which machine finished what, and when.

So I thought, why not let the loop send me a small notification email upon completion? And that’s what I did. On Linux and UNIX this relies on the bash shell and the mailx command. Please note that I’m talking about [Heirloom mailx], not some other mail program by the same name! I’m mentioning this, because there is a different default mailx on FreeBSD, that won’t work for this. That’s why I put alias mailx='/usr/local/bin/mailx' in my ~/.bash_profile on that OS after installing the right program to make it the default for my user.

On Windows, my loops depend on my own [colorecho] command (you can replace that with cmds’ ECHO if you want) as well as the command line mailer [blat]. Note that, if you need to use SSL/TLS encryption when mailing, blat can’t do that. A suitable replacement could be [mailsend]. Please note, that mailsend does not work on Windows XP however.

In the x265 case, avconv (from the [libav] package) is required on all platforms. You can get my build for Windows [here]. If you don’t like it, the wide-spread [ffmpeg] can be a suitable drop-in replacement.

Now, when setting up blat on Windows, make sure to run blat -help first, and learn the details about blat -install. You need to run that with certain parameters to set it up for your SMTP mail server. For whatever reason, blat reads some of that data from the registry (ew…), and blat -install will set that up for you.

Typically, when I transcode, I do so on the elementary streams rather than .mkv files directly. So I’d loop through some source files and extract the needed streams. Let’s say we have “A series – episode 01.mkv” and some more, all the way up to “A series – episode 13.mkv”, then, assuming track #0 is the video stream…

On Windows:

FOR %I IN (01,02,03,04,05,06,07,08,09,10,11,12,13) DO mkvextract tracks "A series - episode %I.mkv" ^
 0:%I\video.h264

On Linux/UNIX:

for i in {01..13}; do mkvextract tracks "A series - episode $i.mkv" 0:$i/video.h264; done

mkvextract will create the non-existing subfolder for us, and a x264 transcoding loop would then look like this on Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps ^
 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f ^
 -2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats %I\v.stats -t 2 --no-fast-pskip ^
 --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o %I\pass1.264 & colorecho "Pass 1 ^
 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & ^
 "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps 24000/1001 --preset veryslow --tune animation ^
 --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats ^
 %I\v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o ^
 %I\pass2.264 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10) & echo !SERIES! ^
 transcoding complete | blat - -t "myself@another.mailhost.com" -c "myself@mailhost.com" -s "x264 ^
 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET SERIES="

Note that I always write all the iteration out in full here. That’s because cmd can’t do loops with leading zeroes in the iterator. The reason for this is that those source files usually have them in their lower episode numbers. If it wasn’t 01,02, … ,12,13, but 1,2, … ,12,13 instead, you could do FOR /L %I IN (1,1,13) DO. But this isn’t possible in my case. Even if elements need alphanumeric names like here,  FOR %I IN (01,02,03,special1,special2,ova1,ova2) DO, you still won’t need that syntax on Linux/UNIX because the bash can have iterator groups like for i in {{01..13},special1,special2,ova1,ova2}; do. Makes me despise the cmd once more. ;)

Edit:

Ah, according to [this], you can actually do something like cmd /V /C "FOR /L %I IN (1,1,13) DO (SET "fI=00%I" & echo "!fI!:~-2")", holy shit. It actually works and gives you leading zeroes. :~-2 for 2 digits, :~-3 for three. Expand fI for more in this example. I mean, what is this even? Some number formatting magic? I probably don’t even wanna know… Couldn’t find any way of having several groups for the iterator however. Meh. Still don’t like it.

So, well, it’s like this on Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do nice -n19 x264 --fps \
24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f \
-2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats $i/v.stats -t 2 --no-fast-pskip \
--cqm flat --non-deterministic --demuxer lavf $i/video.h264 -o $i/pass1.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && nice -n19 \
x264 --fps 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid \
normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats $i/v.stats -t 2 --no-fast-pskip --cqm flat \
--non-deterministic --demuxer lavf $i/video.h264 -o $i/pass2.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x264 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

The variable $MACHINE or %MACHINE%/!MACHINE! specifies the machines’ host name. This will be noted in the email, so I know which machine just completed something. $EPNUM – or %EPNUM%/!EPNUM! on Windows – is used for periodic updates on the shell. The output would be like “Pass 1 done for Episode 07/13 of AnimeX” in green on Windows and bold red on Linux/UNIX (just change the color to your liking).

Finally, $SERIES aka %SERIES%/!SERIES! would be the series’ name. So say, the UNIX machine named “BEAST” above is done with this loop. The email would come with the subject line “x264 notification from BEAST” and would read “AnimeX transcoding complete” in plain text. That’s all.

Please note, that cmd batch on Windows is extremely creepy. Every whitespace (especially the leading ones when doing multi-line like this for display) needs to be exactly where it is. The same goes for double quotes where you might think they aren’t needed. They are! Also, this needs delayed variable expansion once again, which is why we see variables like !EPNUM! instead of %EPNUM% and why it’s called in a subshell by running cmd /V /C.

On Linux/UNIX we don’t need to rely on some specific API like cmds’ SetConsoleTextAttribute() to print colors, as most terminals understand ANSI color codes.

And this is what it looks like for x265:

Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO avconv -r 24000/1001 -i %I\video.h264 -f yuv4mpegpipe ^
 -pix_fmt yuv420p -r 24000/1001 - 2>NUL | "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 ^
 --fps 24000/1001 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate ^
 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 ^
 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
 --pass 1 --slow-firstpass --stats %I\v.stats --sar 1 --range full -o %I\pass1.h265 & colorecho ^
 "Pass 1 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & avconv -r 24000/1001 -i ^
 %I\video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>;NUL | ^
 "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 --fps 24000/1001 -p veryslow --pmode ^
 --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 ^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2 --stats %I\v.stats --sar ^
 1 --range full -o %I\pass2.h265 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" ^
 10) & echo !SERIES! transcoding complete | blat - -t "myself@another.mailhost.com" -c ^
 "myself@mailhost.com" -s "x265 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET ^
 SERIES="

Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do avconv -r 24000/1001 -i \
$i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice -19 x265 - --y4m \
-D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 \
--rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq \
5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 \
--slow-firstpass --stats $i/v.stats --sar 1 --range full -o $i/pass1.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && avconv -r \
24000/1001 -i $i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice \
-19 x265 - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid \
--bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd \
1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 \
--pass 2 --stats $i/v.stats --sar 1 --range full -o $i/pass2.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x265 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

And that’s it. The loops for audio transcoding are simpler, as that part is so fast, it doesn’t need email notifications. Runs for minutes rather than days. When all is done, I’d usually fire up the MKVToolnix GUI, and prepare a mux for the first episode. There is a nice “copy command line to clipboard” function there when you click on “Muxing” after everything is set up. With that I can build another loop that muxes everything to final .mkv files. On Windows that part is more complicated if you want Unicode support, so I needed to create input files by using a generator I wrote in Perl for that, but that’s for another day… :)

Oh, and if you wanna ssh into your Linux or UNIX boxes from afar to check on your transcoders, consider launching them on a GNU screen] session. It’s immensely useful! Too bad it won’t work on the Windows cmd. :(

Jul 152016
 

H.265/HEVC logoSince I’ve started to release x265 binaries for Windows XP / XP x64 (and above), I’ve started to take interest in how it’s performance has developed over time. When looking back on x264, we can see that the encoders’ performance has always gradually improved with more and faster assembly code, optimizations for new CPUs’ instruction set extensions (think AMD FMA4, SSE4a or Intel AVX/AVX2) and so on. Over many years x264 has become a ton faster, as you can also see when comparing an old x264 v0.107.1745 (white results) with newer ones (red results) for the same CPUs in the [x264 benchmark].

So, x265 is much newer, but still, it’s been around for a few years now – since 2013 to be more precise – so I’d like to take a look at its performance trends using the options I typically apply for transcoding animated content. You can find the encoding settings and other information about the test below.

Note that I don’t compile every revision of x265, so I can compare only a few, namely the ones I chose to [release] periodically, excluding the new 2.0+2, which I haven’t released yet (waiting a bit more for a release). Here we go:

x265 performance trend across versions (2017-01-26)

x265 performance trend across versions, state 2017-01-26 (click to enlarge)

previous charts
x265 performance trends across versions, 2016-07-15

x265 performance trends across versions, state 2016-07-15

Note that this runtime graph is inverted, in the sense that “higher = better”. Version 1.7 was the first which could actually encode using the options of my choice, older versions don’t understand all of them and are not comparable in my case because of that. We start off with pretty bad performance and then we get an ample boost with the earlier 1.9 versions.

From 1.9+15 to 1.9+141 we can see a gradual performance increase, as expected from continued development and optimization. Naturally, 8-bits per color channel is fastest, as it makes use of a lot of 8-bit arithmetic internally. For higher bit depths (10 and 12 bits per channel, or 30 and 36 bits per pixel respectively) the internal arithmetic is boosted to a 16 bit precision, resulting in better outputs for finer gradients. This costs performance of course.

As expected, 10 and 12 bit runtimes are relatively close in terms of speed with 8 bit being quite far ahead.

Now the most surprising thing is the nose dive we take from 1.9+141 to 1.9+200. I have no idea what happened here?! Did they fix a major bug? Or did some performance-critical options change in the --preset veryslow that I’m using? I have no idea. But to put is in numbers easier to understand: For 8 bit, my test encode went from a runtime of 30:39.312 (1.9+141) to 41:20.296 (1.9+200)! Ah, the time format being MM:SS.sss, M = minutes, S = seconds and s = milliseconds. That’s almost -35%! It’s less pronounced for the higher bit depths with -25% for 10 bits and -24% for 12 bits, but that’s still significant. Maybe I shouldn’t have deleted the output data, might have been useful to look at the H.265 stream headers.

Well, from there on out we continue to see a gradual performance increase again, in a steady and stable fashion. But what happened between 1.9+141 and 1.9+200? I don’t know. Something major must have changed, I just don’t know what it was exactly…

Maybe somebody can enlighten me a bit[1], so here are the options used, right from the benchmarking script:

expand/collapse source code
  1. @ECHO OFF
  2.  
  3. :: ###################################################################################
  4. :: #                                                                                 #
  5. :: #  Please note, that command lines passed to timethis.exe may not be longer than  #
  6. :: #  1024 characters including expanded variables, so we need to keep it short      #
  7. :: #  enough. Some of the options passed to x265 may already be superflous due to    #
  8. :: #  being included in the veryslow preset, but I did not really double-check that. #
  9. :: #  Instead, I kept it just short enough by using shorter file names instead.      #
  10. :: #                                                                                 #
  11. :: #  Where to get the files:                                                        #
  12. :: #    * x265.exe     : http://wp.xin.at/x265-for-winxp-and-winxp-x64               #
  13. :: #    * avconv.exe   : http://wp.xin.at/x265-for-winxp-and-winxp-x64               #
  14. :: #    * colorecho.exe: http://wp.xin.at/archives/3064                              #
  15. :: #    * timethis.exe : https://support.microsoft.com/en-us/kb/927229               #
  16. :: #                                                                                 #
  17. :: #  GPL v3 license for this scripts code, colorecho.exe, parts of avconv.exe and   #
  18. :: #  the CygWin companion libraries for avconv.exe:                                 #
  19. :: #    * https://www.gnu.org/licenses/gpl-3.0.en.html                               #
  20. :: #                                                                                 #
  21. :: #  GPL v2 license for x265.exe:                                                   #
  22. :: #    * https://www.gnu.org/licenses/old-licenses/gpl-2.0.html                     #
  23. :: #                                                                                 #
  24. :: #  LGPL v3 license for other parts of avconv.exe:                                 #
  25. :: #    * https://www.gnu.org/licenses/lgpl-3.0.html                                 #
  26. :: #                                                                                 #
  27. :: #  NT Resource Kit license for timethis.exe:                                      #
  28. :: #    * https://enterprise.dejacode.com/license_library/Demo/ms-nt-resource-kit    #
  29. :: #                                                                                 #
  30. :: #  The x265 encoder  : http://x265.org                                            #
  31. :: #  The libav         : https://libav.org                                          #
  32. :: #  The Cygwin project: https://www.cygwin.com                                     #
  33. :: #                                                                                 #
  34. :: ###################################################################################
  35.  
  36. :: First round, 8 bits per channel color depth (MAIN profile), 8 bit arithmetic:
  37. FOR %%I IN (1.7-8b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  38.  ""Testing v%%I, 8 bits..."" 10 & .\timethis.exe ""echo x265 v%%I-8b & .\avconv.exe -r ^
  39.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  40.  .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  41.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  42.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  43.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  44.  --slow-firstpass --stats %%I-8b.stats --sar 1 --range full -o %%I-8b-p1.h265 2>NUL & ^
  45.  .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  46.  24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow ^
  47.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  48.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  49.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
  50.  --pass 2 --stats %%I-8b.stats --sar 1 --range full -o %%I-8b-p2.h265 2>NUL"" 1> ^
  51.  .\results-%%I-8b.txt 2>.\timethis-errorlog-%%I-8b.txt
  52.  
  53. :: Second round, 10 bits per channel color depth (MAIN10 profile), 16 bit arithmetic:
  54. FOR %%J IN (1.7-10b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  55.  ""Testing v%%J, 10 bits..."" 10 & .\timethis.exe ""echo x265 v%%J-10b & .\avconv.exe -r ^
  56.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  57.  .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  58.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  59.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  60.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  61.  --slow-firstpass --stats %%J-10b.stats --sar 1 --range full -o %%J-10b-p1.h265 2>NUL ^
  62.  & .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  63.  24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow ^
  64.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  65.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  66.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass ^
  67.  2 --stats %%J-10b.stats --sar 1 --range full -o %%J-10b-p2.h265 2>NUL"" 1> ^
  68.  .\results-%%J-10b.txt 2>.\timethis-errorlog-%%J-10b.txt
  69.  
  70. :: Second round, 12 bits per channel color depth (MAIN12 profile), 16 bit arithmetic:
  71. FOR %%K IN (1.7-12b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  72.  ""Testing v%%K, 12 bits"" 10 & .\timethis.exe ""echo x265 v%%K-12b & .\avconv.exe -r ^
  73.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  74.  .\x%%K.exe - --y4m -D 12 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  75.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  76.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  77.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  78.  --slow-firstpass --stats %%K-12b.stats --sar 1 --range full -o %%K-12b-p1.h265 2>NUL ^
  79.  & .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  80.  24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 -p veryslow ^
  81.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  82.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  83.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass ^
  84.  2 --stats %%K-12b.stats --sar 1 --range full -o %%K-12b-p2.h265 2>NUL"" 1> ^
  85.  .\results-%%K-12b.txt 2>.\timethis-errorlog-%%K-12b.txt
  86.  
  87. .\colorecho.exe "Benchmarks completed, cleaning up..." 10
  88.  
  89. :: Removing output and statistics files:
  90. del /Q *p*.h265 *stats*
  91.  
  92. .\colorecho.exe "All done, results are to be found in the results-*.txt files!" 10

So if I missed some critical changes that happened in between 1.9+141 and 1.9+200, please let me know! Oh, and here are the exact system specifications, in case it matters (it probably doesn’t):

  • Intel Xeon X5690 3.33GHz (SSE4.2 is max, no AVX), running at an all-core turbo of 3.6GHz
  • 24GB DDR-III/1066 10-10-10 2T
  • X58 chipset
  • Windows XP Professional x64 Edition SP2 with all Server 2003 R2 x64 updates

I guess I’ll keep doing the performance evaluations from here on out just to see how the encoder evolves over time, performance-wise… And maybe I’ll redo 1.9+141 and one of the newer versions and parse the stream headers to see if the effective encoding options differ anywhere after all. If yes, I’ll update this post!

Update 2017-01-27:

Three new x265 versions have been added to the performance trend analysis, 2.0+54, 2.1+60 and 2.2+22. Much to my dismay, there haven’t been any big developments when it comes to x265s’ performance on x86 machines over the last 6-8 months. Since 1.9+230 it’s pretty much linear, with only minimal variance.

So there have been changes and new options and all that, but it seems that the basic speed of the encoder in the --preset veryslow, implying --no-rskip has stayed the same. Maybe a Google Summer of Code performance challenge would do something for x265? As far as I can remember this worked miracles for x264 in the past as well, if my memory serves me right. Let’s see if we’ll get any significant improvements in the future…

[1] Thanks to “Particular” this [has been clarified] in the comments.

Apr 082016
 

H.265/HEVC logoPreviously, I have shown you [how to compile x265 on Windows] using Microsoft Visual Studio 2010 in a way that results in binaries compatible with Windows NT 5.1/5.2, or in other words: Windows XP, XP x64 and Windows Server 2003. And while that works for most purposes, today I’d like to show you how to build an actual multilib binary, that can handle all three color bit depths supported by x265, the standardized 8- and 10-bit (MAIN and MAIN10 profiles) as well as 12-bit (MAIN12 profile). With that, it’s all in one exe instead of three. As before though, multilib x265 is only supported on 64-Bit Windows. But first, once again…

1.) Giving you the binaries

There were a lot of improvements since the last version I published back in February of course, also performance-wise. So here’s the current version from Multicoreware for both 32-bit and 64-bit Windows, compiled with MSVC 2010 SP1 and yasm 1.3.0. This requires the Microsoft Visual C++ 2010 runtime to work, see previous article:

This time around, the binaries have been tested as well! On regular 32-bit Windows XP, only fundamental binary compatibility was tested. However, all versions, so the 32-Bit one and the 64-Bit multilib ones have been ran through a 2-pass ABR encoding test with output verification for 8-bit color depth (32- & 64-bit) as well as 10- and 12-bit color depths (64-bit only) on Windows XP Professional x64 Edition using the following command line (see previous post for details):

avconv -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL^ 
 | .\x265.exe - --y4m -D 10 --fps 24000/1001 -p veryslow^
 --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0^
 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1^
 --slow-firstpass --stats v.stats --sar 1 --range full -o pass1.h265 &amp; avconv^
 -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL^
 | .\x265.exe - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6^
 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75^
 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2^
 --stats v.stats --sar 1 --range full -o pass2.h265

Needless to say, they should work fine on Windows Vista/7/8/8.1/10/Server 2008/Server 2012/HS 2007/HS 2011 as well.

From time to time, I’ll release new binaries, so you might wanna check back every few months or so, if you’re interested. You can also request a build in the comments if you’re growing impatient and need a specific version more quickly because of some bugfix / feature improvement in x265.

2.) Compiling an XP/2003-compatible x265 multilib binary yourself

First, please look at the previous article I linked to in the beginning, point 2. You need the software prerequisites listed in 2a and you might still wish to read through 2b to understand some of the stuff better. You don’t need to actually run any of the commands shown there though.

Now, the multilib build is done a bit differently from the rest, as everything is scripted, so this is 100% command line work, no graphical cmake, no running the full Visual Studio IDE. Usually, with all software in place, sitting in the root directory of the x265 source tree, all you need to do is to go to build\vc10-x86_64\ and run ./multilib.bat. This won’t give us an XP/2003-compatible binary however, and the reason lies within the build script multilib.bat, here is the stock version:

expand/collapse source code (multilib.bat)
  1. @echo off
  2. if "%VS100COMNTOOLS%" == "" (
  3.   msg "%username%" "Visual Studio 10 not detected"
  4.   exit 1
  5. )
  6.  
  7. call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat"
  8.  
  9. @mkdir 12bit
  10. @mkdir 10bit
  11. @mkdir 8bit
  12.  
  13. @cd 12bit
  14. cmake -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON
  15. if exist x265.sln (
  16.   MSBuild /property:Configuration="Release" x265.sln
  17.   copy/y Release\x265-static.lib ..\8bit\x265-static-main12.lib
  18. )
  19.  
  20. @cd ..\10bit
  21. cmake -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF
  22. if exist x265.sln (
  23.   MSBuild /property:Configuration="Release" x265.sln
  24.   copy/y Release\x265-static.lib ..\8bit\x265-static-main10.lib
  25. )
  26.  
  27. @cd ..\8bit
  28. if not exist x265-static-main10.lib (
  29.   msg "%username%" "10bit build failed"
  30.   exit 1
  31. )
  32. if not exist x265-static-main12.lib (
  33.   msg "%username%" "12bit build failed"
  34.   exit 1
  35. )
  36. cmake -G "Visual Studio 10 Win64" ../../../source -DEXTRA_LIB="x265-static-main10.lib;x265-static-main12.lib" -DLINKED_10BIT=ON -DLINKED_12BIT=ON
  37. if exist x265.sln (
  38.   MSBuild /property:Configuration="Release" x265.sln
  39.   :: combine static libraries (ignore warnings caused by winxp.cpp hacks)
  40.   move Release\x265-static.lib x265-static-main.lib
  41.   LIB.EXE /ignore:4006 /ignore:4221 /OUT:Release\x265-static.lib x265-static-main.lib x265-static-main10.lib x265-static-main12.lib
  42. )
  43.  
  44. pause

So I took all the options from the files generated by the original cmake when doing the normal build, and added them to the script to ensure our output binaries would be XP-compatible. This is the fixed build script:

expand/collapse source code (multilib.bat, patched for XP/2003)
  1. @echo off
  2. if "%VS100COMNTOOLS%" == "" (
  3.   msg "%username%" "Visual Studio 10 not detected"
  4.   exit 1
  5. )
  6.  
  7. call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat"
  8.  
  9. @mkdir 12bit
  10. @mkdir 10bit
  11. @mkdir 8bit
  12.  
  13. @cd 12bit
  14. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DWINXP_SUPPORT=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON
  15. if exist x265.sln (
  16.   MSBuild /property:Configuration="Release" x265.sln
  17.   copy/y Release\x265-static.lib ..\8bit\x265-static-main12.lib
  18. )
  19.  
  20. @cd ..\10bit
  21. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DWINXP_SUPPORT=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF
  22. if exist x265.sln (
  23.   MSBuild /property:Configuration="Release" x265.sln
  24.   copy/y Release\x265-static.lib ..\8bit\x265-static-main10.lib
  25. )
  26.  
  27. @cd ..\8bit
  28. if not exist x265-static-main10.lib (
  29.   msg "%username%" "10bit build failed"
  30.   exit 1
  31. )
  32. if not exist x265-static-main12.lib (
  33.   msg "%username%" "12bit build failed"
  34.   exit 1
  35. )
  36. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DWINXP_SUPPORT=ON -DEXTRA_LIB="x265-static-main10.lib;x265-static-main12.lib" -DLINKED_10BIT=ON -DLINKED_12BIT=ON
  37. if exist x265.sln (
  38.   MSBuild /property:Configuration="Release" x265.sln
  39.   :: combine static libraries (ignore warnings caused by winxp.cpp hacks)
  40.   move Release\x265-static.lib x265-static-main.lib
  41.   LIB.EXE /ignore:4006 /ignore:4221 /OUT:Release\x265-static.lib x265-static-main.lib x265-static-main10.lib x265-static-main12.lib
  42. )
  43.  
  44. pause

You can just rename your original script for backup and put the fixed code in its place, build\vc10-x86_64\multilib.bat, then run it on the command line. If all the required tools are present, it will compile a 12-bit library, then a 10-bit library (both static) and finally an 8-bit binary that will have the other two libraries statically linked in. The final x265.exe can then be found in build\vc10-x86_64\8bit\Release\. To check whether it’s the real thing, look for the bitness by running .\x265.exe --version while sitting in that folder on the command line. You should see something like this:

x265 multilib binary

A x265 multilib binary shows that it’s “8-bit+10-bit+12-bit”

Per-color-channel bitness can be defined with x265s’ command line option -D. So that’d be -D 8, -D 10 or -D 12. Note that only 8- and 10-bit are part of the official Blu-Ray UHD/4k specification however.

3.) A side note

In case you’re new to this, you might not get why “8-bit” and “10-bit” etc. Aren’t color spaces supposed to be 16-bit, 24-bit, 32-bit etc.? Well, it seems that in the world of video processing, people don’t refer to whole color space bitness, but rather individual color channel bitness. So with three channels (red, green, blue for instance), you’d have 8/10/12 bits per channel, so that’s 24-, 30- and 36-bit total, or 16.7 million, 1 billion and 64 billion colors.

The more important part – and the reason why nobody encodes to 12-bit – is the internal arithmetic precision of x265 though (same applies to x264). At 8-bit color depth, arithmetic precision is also at 8-bits. When you hop over to 10-bit, you can’t use 8-bit operations and data types any longer, so everything is done at 16-bit precision. This makes the code slower, but also more efficient in preserving color gradients. Since 10-bit H.265/HEVC is officially a part of Blu-Ray UHD/4k, this would be the sweet spot, unless you’re dealing with devices too slow to play it.

Going to 12-bit won’t boost the precision further, it just gives you more colors, that most of today’s displays won’t be able to show anyway. Not much benefit.

So that’s that.

Have fun! :)

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.

Feb 232016
 

H.265/HEVC logoAfter [compiling] and running the x265 HEVC encoder, and after [looking at its quality] for animated content, here’s another little piece of information about my experiments with H.265/HEVC. And this time it’s the decoding part. Playing H.265-encoded videos on the PC is relatively easy. On Windows I tend to use [MPC-HC] for this, and on Linux/UNIX you can use [mplayer] or [VLC]. The newest versions of those players are all linked against a modern libav or ffmpeg library collection, so they can decode anything any H.265 encoder can throw at them.

The questions are: At what price? And: What about mobile devices?

H.265/HEVC is costly in terms of computation, and not just in the encoding stage. Decoding this stuff is hard as well. So I looked at two older Core 2 processors to see how they fare when decoding regular 10-bit H.264/AVC and the same content encoded as 10-bit H.265/HEVC, both times at the same bitrate of 3Mbit/s ABR. Again, the marvelous Anime movie “The Garden of Words” was used for this. The video player of my choice for playback was [MPC-HC] v1.7.10, rendering to a VMR9 surface.

On top of that, I can also provide some insight on how relatively modern Android devices will handle this (devices partly without a H.265 hardware decoder chip however!), all thanks to [Umlüx], who’s been willing to install the necessary Apps and run some tests! On Android, [MX Player] was used.

For the record, the encoding settings were like this for x264 (pass 1 & pass 2), …

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 3000 --aq-mode 1 -p 1 --slow-firstpass --stats v.stats -t 2 --no-fast-pskip --cqm flat
--non-deterministic

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 3000 --aq-mode 1 -p 2 --stats v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic

…and this for x265:

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 3000 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1
--slow-firstpass --stats v.stats --sar 1 --range full

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 3000 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2
--stats v.stats --sar 1 --range full

PC first:

Contender #1 is my old Sony TT45X subnotebook, and this is the processor inside:

CPU-Z on a Core 2 Duo SU9600

A Core 2 Duo SU9600 Penryn, which is an ULV processor at 1.6GHz, shown under all-core load, where it doesn’t boost to 1.83GHz any longer (And yeah, this is still WinXP+POSReady2009).

Contender #2 is my secondary workstation at work, that I recently upgraded with an SSD and a better processor we had lying around. It’s using this chip now:

CPU-Z on a Core 2 Quad Q9505

A Core 2 Quad Q9505 Yorkfield, which has been overclocked to a rock solid 3.4GHz.

Let’s throw some video at them! First, we’ll try some good old H.264 on the slow-clocked Core 2 Duo mobile chip:

Core 2 Duo playing the beginning of "The Garden of Words" as 3Mbit H.264/AVC

The Core 2 Duo SU9600 playing the beginning of “The Garden of Words” as 3Mbit H.264/AVC (on a German version of Windows).

We can see some high load there. Mind you, since this is 10-bit H.264, there is no GPU acceleration whatsoever, minus maybe the bicubic scaler, which is implemented as Pixel Shader 2.0 code. Everything else has to be done by the CPU and its SSE extensions. Let’s take a look at the new H.265 then:

Core 2 Duo playing the beginning of "The Garden of Words" as 3Mbit H.265/HEVC

Same thing as before, but with 3Mbit H.265/HEVC.

The beginning of the movie, which shows only some moving logo on mostly uniform background and then some static text does just fine. You can really see what’s happening on screen when looking at the usage curve above. As soon as the first serious video frame with lots of movement starts to fade in, the machine just falls on its face, as the bitrate is rising. Realtime playback can no longer be achieved, A/V synchronization is being lost and a ton of frames are being dropped, resulting in a horrible experience.

Thing is, even with all the frame drops, it’s still losing sync even and falling short of delivering those frames which are still being decoded with the proper timing. It’s just abysmal.

So let’s change our environment to a CPU with twice the cores and roughly twice the clock speed, while staying with the same architecture / instruction set, H.264 first:

Core 2 Quad playing the beginning of "The Garden of Words" as 3Mbit H.264/AVC

The quad-core Q9505 hardly has any trouble with H.264 at all. It’s completely smooth sailing.

And with the harder stuff:

Core 2 Quad playing the beginning of "The Garden of Words" as 3Mbit H.265/HEVC

We can see that the load has roughly doubled with H.265.

Now, H.265 is clearly putting some load even on the quad-core. I’m assuming that with higher bitrates as we would use for 4K/UHD material, this processor might be in trouble. For 1080p and bitrates up to maybe 6Mbit/s it should still be ok however. Of course, with the most modern graphics cards you can give even a Core 2 a companion device which can do H.265 decoding in hardware, like an NVidia GPU that can provide you with [PureVideo] feature sets E or even F, or maybe an AMD Radeon with  [UVD] level 6. But even then, 10-bit content might not be accelerated, so you need to be careful when choosing your GPU. Intel has recently added 10-bit support to some of their onboard GPUs (HD Graphics 5500 & 6000, Iris Graphics 6100) with driver v15.36.14.4080, nVidia added it starting with the Maxwell generation and AMD Radeons still don’t have it as far as I know.

Now, what about mobile devices? A fast enough PC can do H.265/HEVC even without hardware assist, at the very least in 1080p, and likely 4K as well, when we look at modern Core i3/i5/i7 and somewhat comparable AMD Athlon FX chips. How about multicore ARMv7/v8 chips with NEON instruction set extensions?

Let’s look at two Android devices, a Sony Xperia Tablet Z2 (without a H.265 hardware decoder) and a Sony Xperia Z5 phone:

The Tablet features the following CPU:

The Xperia Tablet Z2 uses a Qualcomm Snapdragon 801

The Xperia Tablet Z2 uses a Qualcomm Snapdragon 801 quad-core.[1]

That’s not exactly a slow chip, but an out-of-order execution pipeline with NEON extensions at a decent clock rate. And the phone:

The Xperia Z5 has a Qualcomm Snapdragon 810

The Xperia Z5 has a Qualcomm Snapdragon 810 in big.LITTLE configuration.[1]

So the phone has a very modern 64-bit ARMv8 big.LITTLE CPU setup with four faster out-of-order cores, and four slower, energy-efficient in-order cores. Optimally, all of them should be used as much as possible when throwing a seriously demanding task at the device. Let’s look at how it goes, but first on the older tablet with H.264 for starters:

The Xperia Tablet Z2 playing H.264/AVC.

The Z2 manages to play the H.264/AVC version without any stuttering, but just barely. It has to boost its clock speed all the way to the top to manage.[1]

You’re probably thinking “There’s no way this can work with H.265”, right? Well, you’d be correct:

With H.265, the Xperia Tablet Z2 stumbles.

With 3-Mbit H.265, the Xperia Tablet Z2 stumbles. Full clock speed, all cores under load, no chance to play demanding scenes without massive problems.[1]

It just bombs. Frame drops and stuttering, no way you’d want to watch anything when it goes like that. Some of the calmer scenes (=lower bitrate) still work, but lots of rain all over the frame and the Z2 is done for. So let’s move to the more modern hybrid-core processor of the Sony Xperia Z5 smartphone:

The Z5 playing the 10-bit H.264 on CPU only

The Z5 playing the 10-bit H.264 on CPU only. The octa-core big.LITTLE processor seems to load its fat cores mostly, which makes sense with demanding workloads.[1]

The big Cortex-A57 cores seem to be doing most of the work here, clocking at their maximum speed. Given we’re over 50% load with the most difficult scenes however, some threads seem to be pushed over to the smaller Cortex-A53 cores as well. In any case, the end result is that H.264 works smoothly throughout the movie. But still, load is high, and our only headroom left are some free, slow in-order cores… So, H.265?

The Z5 fails with 10-bit H.264 too. Where is my hardware decoder?!

The Z5 fails as well. While it can cope a little better, scenes like the above are just too much. Where is my hardware decoder?![1]

Something strange is happening now: H.265/HEVC hardware decoders should support this 10-bit H.265 file just fine. But for some reason, MX Player still falls back to software decoding, despite the player being up-to-date – something even a CPU as powerful as a Snapdragon 810 cannot handle for all parts of “The Garden of Words” at the given settings. The really demanding scenes will still fail to play back decently. CPU clock is also lowered a bit now, probably because of power and/or heat management.

My assumption would be that MX Player simply doesn’t support the HW decoder in this phone yet, which is a shame if it’s true. Another reason might be that I used some parameters and/or features of H.265 in this encode, that are not implemented in this chip. Whichever the case may be, the Snapdragon 810 alone cannot handle it either!

Update: After further research it turns out that almost no hardware accelerator supports Hi10p or Main-10 for H.265. In other words: No 10-bit decoding for H.265! It’s possible on NVidias Tegra K1 & X1 due to some CUDA hacks in MX Player, but nowhere else it seems. The upcoming Snapdragon 820 should however support it, and devices based on it should become available around March 2016:

Snapdragon 820s' Advancements

This is what Snapdragon 820 (MSM8996) should give us over Snapdragon 810 (MSM8994), including 10-bit H.265/HEVC decoding in hardware ([source]Russian flag)!

And this concludes my little performance analysis, after which I can say that either you need a relatively ok PC to use H.265, or a hardware decoder chip that works with your files and your software, if you’re targeting other platforms like hardware players or smartphones and tablets!

PS.: Thanks fly out to Umlüx for doing the Android tests!

[1] Images are © 2016 Umlüx, used with express permission.

Sep 022015
 

colorecho logoRecently, I am doing a lot of audio/video transcoding, and for that I’m using tools like mkvextract, mkvmerge, x264, avconv and fghaacenc on the command line. Mostly this is to get rid of old files that are either very inefficiently encoded – like DivX3 or MPEG-2 – or are just too large/high-bitrate for what the content’s worth. Since I do have large batches of relatively similar files, I wrote myself a few nice loops that would walk through an entire stack of videos and process all of it automatically. Because I want to see at first glance how far through a given video stack a script is, I wanted to output some colored notifications on the shell after each major processing step, telling me the current video number. That way, I could easily see how far the job had progressed. We’re talking processes here that take multiple days, so it makes sense.

Turns out that this was harder than expected. In the old DOS days, you would just load ANSI.SYS for that, and get ANSI color codes to work with. The same codes still work in modern UNIX terminals. But not within the Windows cmd for whatever reason. Now there is an interface for that kind of stuff, but seemingly no tools to use it. Like e.g. the echo command simply can’t make use of it.

Now there is project porting ANSI codes back to the cmd, and it’s called [ANSICON]. Basically just a DLL hooked into the cmd. Turns out I can’t use that, because I’m already using [clink] to make my cmd more UNIX-like and actually usable, which also is additional code hooked into the cmd, and the two confict with each other. More information about that [here].

So how the hell can I do this then? Turns out there is a batch script that can do it, but it’s a very outlandish hack that doesn’t seem to want to play nice with my shells if I do @echo off instead of echo off as well. I guess it’s clinks work breaking stuff here, but I’m not sure. Still, here is the code, as found by [Umlüx]:

expand/collapse source code
  1. ECHO OFF
  2. SETLOCAL EnableDelayedExpansion
  3.  
  4. FOR /F "tokens=1,2 delims=#" %%A IN ('"PROMPT #$H#$E# & ECHO ON & FOR %%b IN (1) DO REM"') DO (
  5.   SET "DEL=%%A"
  6. )
  7.  
  8. ECHO say the name of the colors, dont read
  9.  
  10. CALL :ColorText 0a "blue"
  11. CALL :ColorText 0C "green"
  12. CALL :ColorText 0b "red"
  13. ECHO.
  14. CALL :ColorText 19 "yellow"
  15. CALL :ColorText 2F "black"
  16. CALL :ColorText 4e "white"
  17.  
  18. GOTO :eof
  19.  
  20. :ColorText
  21.  
  22. ECHO OFF
  23.  
  24.  "%~2"
  25. findstr /v /a:%1 /R "^$" "%~2" NUL
  26. DEL "%~2" > NUL 2>&1
  27.  
  28. GOTO :eof

So since those two solutions didn’t work for me, what else could I try?

Then I had that crazy idea: Since the interface is there, why not write a little C program (actually ended up being C++ instead) that uses it? Since I was on Linux at the time, I tried to write it there and attempt my first cross-compile for Windows using a pre-built [MinGW-w64] compiler, that luckily just played along nicely on my CentOS 6.6 Linux. You can get pre-built binaries for 32-Bit Windows targets [here], and for 64-bit Windows targets [here]. Thing is, I know zero C++, so that took some time and it’s quite crappy, but here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include "windows.h"
  3.  
  4. // Set NT commandline color:
  5. void SetColor(int value) {
  6.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  7. }
  8.  
  9. int main(int argc, char *argv[]) {
  10.   SetColor(atoi(argv[2]));
  11.   std::cout << "\r\n" << argv[1] << "\r\n";
  12.   SetColor(7);
  13.   return 0;
  14. }

Update 2016-06-22: Since I wanted to work with wide character strings in recent days (so, non-ANSI Unicode characters), my colorecho command failed, because it couldn’t handle them at all. There was no wide character support. I decided to update the program to also work with that using Windows-typical UCS-2le/UTF-16, here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include <io.h>
  3. #include <fcntl.h>
  4. #include "windows.h"
  5.  
  6. // Set NT commandline color:
  7. void SetColor(int value) {
  8.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  9. }
  10.  
  11. // Unicode main
  12. int wmain(int argc, wchar_t *argv[]) {
  13.   SetColor(_wtoi(argv[2]));
  14.   _setmode(_fileno(stdout), _O_U16TEXT);      // Set stdout to UCS-2le/UTF-16
  15.   std::wcout << "\r\n" << argv[1] << "\r\n";  // Write unicode
  16.   SetColor(7);                                // Set color back to default
  17.   return 0;
  18. }

I also updated the download section and the screenshots further below. End of update.

This builds nicely using MinGW on Linux for 64-bit Windows targets. Like so: While sitting in the MinGW directory (I put my code there as echo.cpp as well, for simplicities’ sake), run: ./bin/x86_64-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe. If you want to target 32-bit Windows instead, you’d need to use the proper MinGW for that (it comes in a separate archive): ./bin/i686-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe

When building like that, you’d also need to deploy libstdc++-6.so and libgcc_s_seh-1.dll along with the EXE files though. The DLLs can be obtained from your local MinGW installation directory, subfolder ./x86_64-w64-mingw32/lib/ for the 64-bit or ./i686-w64-mingw32/lib/ for the 32-bit compiler. If you don’t want to do that and rather rely on Microsofts own C++ redistributables, you can also compile it with Microsoft VisualStudio Express, using its pre-prepared command line. You can find that here, if you have Microsofts VisualStudio installed – version 2010 in my case:

VC2010 command lines

The command lines of Visual Studio (click to enlarge)

Here, the “Visual Studio Command Prompt” is for a 32-bit build target, and “Visual Studio x64 Win64 Command Prompt” is for building 64-bit command line programs. Choose the appropriate one, then change into a directory where you have echo.cpp and run the following command: cl /EHsc /W4 /nologo /Fecolorecho.exe echo.cpp, giving you the executable colorecho.exe.

Alternatively, you can just download my pre-compiled versions here:

Update 2016-06-22: And here are the Unicode versions:

End of update.

Note that this tool does exactly what I need it to do, but it’ll likely not do exactly what you’d need it to do. Like e.g. the line breaks it adds before and after its output. That’s actually a job for the shells echo command to do, not some command line tool. But I just don’t care. So that’s why it’s basically a piece of crap for general use. The syntax is as follows, as shown for the 64-bit VC10 build:

colorecho-x86_64-vc10.exe "I am yellow!" 14

When run, it looks like this:

Colorecho running

colorecho invoked on the Windows cmd

So the first parameter is the string to be echoed, the second one is the color number. That number is 2-digit and can affect both the foreground and the background. Here the first 16 of them, which are foreground only:

  • 0: Black
  • 1: Dark blue
  • 2: Dark green
  • 3: Dark cyan
  • 4: Dark red
  • 5: Dark purple
  • 6: Dark yellow
  • 7: Light grey
  • 8: Dark grey
  • 9: Light blue
  • 10: Light green
  • 11: Light cyan
  • 12: Light red
  • 13: Light purple
  • 14: Light yellow
  • 15: White

If you go higher than that, you’ll also start changing the background colors and you’ll get different combinations of foreground and background colorization.

The background colors actually follow the same order, black, dark blue, dark green, etc. Numbers from 0..15 are on black background, numbers from 16..31 are on a dark blue background and so on. This makes up pretty much the same list:

  • 0..15: 16 foreground colors as listed above on the default black background, and
  • 16..31: on a dark blue background
  • 32..47: on a dark green background
  • 48..63: on a dark cyan background
  • 64..79: on a dark red background
  • 80..95: ona dark purple background
  • 96..111: on a dark yellow background
  • 112..127: on a light grey background
  • 128..143: on a dark grey background
  • 144..159: on a light blue background
  • 160..175: on a light green background
  • 176..191: on a light cyan background
  • 192..207: on a light red background
  • 208..223: on a light purple background
  • 224..239: on a light yellow background
  • 240..255: on a white background

Going over 255 will simply result in an overflow causing the system to start from the beginning, so 256 is equal to 0, 257 to 1, 260 to 4 and so on.

Update 2016-06-22: With the Unicode version, you can now even do stupid stuff like this:

colorecho unicode with DejaVu Sans Mono font on Windows cmd

It could be useful for some people? Maybe?

Note that this has been rendered using the DejaVu Sans Mono font, you can get it from [here]. Also, you need to tell the Windows cmd which fonts to allow (only monospaced TTFs work), you can read how to do that [here], it’s a relatively simple registry hack. I have yet to find a font that would work and also render CJK characters though, but with the fonts I myself have available at the moment, asian stuff won’t work, you’ll just get some boxes instead:

colorecho unicode CJK fail

Only characters which have proper glyphs for all desired codepoints in the font in use can be rendered, at least on Windows XP.

Note that this is not a limitation of colorecho, it does output the characters correctly, “ですね” in this case. It’s just that DejaVu Sans Mono has no font glyphs to actually draw them on screen. If you ever come across a neat cmd-terminal-compatible font that can render most symbols and CJK, please tell me in the comments, I’d be quite happy about that! End of update.

If somebody really wants to use this (which I doubt, but hey), but wishes it to do things in a more sane way, just request it in the comments. Or, if you can, just take the code, change and recompile it by yourself. You can get Microsofts VisualStudio Express – in the meantime called [VisualStudio Community] – free of charge anyway, and MinGW is free software to begin with.

Jul 152015
 

XP x64 is dead logo[1] Update 2: It’s not looking good. At all. I (and some other users) have been playing around with Windows Storage Server 2003 systems (see the first update below), and while the OEMs are supposed to uphold support for the system, it seems they just don’t care. Two distributions have been inspected, one from Dell and one from Hewlett Packard. The latter actually does feature an updating software, that will however ignore all newer Microsoft security fixes. It just doesn’t do anything useful at all it seems.

For now, I am not even sure whether Microsoft truly ships any updates to the OEMs for rollout to customers, or whether they’ve just abandoned WSS03 completely, making their lifecycle statements regarding the operating system an empty promise, as no OEM could patch the OS itself like Microsoft can. It looks like we’re not getting anywhere with this. If you are a Windows Storage Server 2003 operator and you know more about this, or if you actually do have access to WSS03 Windows updates from after 2015-07-14, please do let me know in the comments! Thank you.

Update: While I’m not completely sure about it, it might be the case that we can disregard the text below this paragraph. A user that goes by the name of [tagg] pointed me towards a comment in the RyanVM forums, where 5eraph is releasing his XP x64 update packs. It seems that a special version of Windows Server 2003 called the “Windows Storage Server 2003” is actually supported until [October of 2016], which could mean that XP x64 could get yet another extension to its lifetime. I’m currently inspecting the operating system and methods to extract full update installers out of it to see whether it can truly be done. If yes, there might be life in the old dog yet!


The dark ages have finally arrived – for XP x64 users, that is. Yesterday, Microsofts extended support for Windows Server 2003 has ended, and with it also my [unofficial XP x64 update] project. There won’t be any more updates for Windows XP Professional x64 Edition SP2 and its on-board components from this day on. And while the regular 32-bit version with its older NT 5.1 kernel will live on until 2019 due to the [POSReady2009 hack], XP x64 will not, as there is no 64-bit embedded POSReady version of XP naturally, given that point-of-service systems like ticketing machines or airport terminals don’t need 64-bit address spaces.

So, as said on the update page, if you’re going to keep using XP x64, you’re doing so at your own risk. And you should really be knowing what you’re doing too, or you may not only put yourself at risk, but also other users on the web, should your machine become compromised. But given that XP x64 users I’ve encountered recently are also often freaks and avid Linux/UNIX users as well, I think the danger is much lower than with 32-bit XP.

Well that’s it. From the Microsoft side of things, this is pretty much where XP x64 ends.

It’s a bit sad that there is no free kernel API extension to make newer software run on the old system, like there is for Win9x with [KernelEx]. Software built with modern Microsoft development environments (Visual Studio 2013+) and in certain cases older ones will likely work on XP less and less from now on. People will no longer care so much for linking against the old platform SDKs and compiling for the v110_xp platform target.

Several free software projects (like FileZilla to name just one of many) have already ceased to support NT5.x even before the end of Server 2003. It’s even worse for commercial applications of course. Then there are others which still care to keep the platform supported, like the x265/HEVC encoder or closed software like AnyDVD HD.

But one thing’s for sure.

From this day on, it’ll only get darker and colder for those who decide to stay on this ship!

[1] Original image is © 2015 Windows 8 Wallpapers

Jun 072015
 

Kung Fury: Street Rage logoSo after the release of that crazy crowdfunded (and free of charge) movie [Kung Fury], there is also a game! Now that was fast. Made by the Swedish game developers of [Hello There], the game is basically a clone of [One Finger Death Punch], as many gamers have already pointed out. Not that anybody seems to mind that – me included. It’s a superficially very simple 2-button street fighting game, where one button means “punch/kick/whatever to the left” and the other “punch/kick/whatever to the right”. Don’t let the seeming simplicity fool you though. There is more skill involved than you might think…

So let’s have a look at the intro of the game, which strongly resembles an 80s arcade machine style:

Kung Fury: Street Rage; That's our Hero!

Kung Fury: Street Rage; That’s our Hero!

So with the use of some Direct3D 9.0c shaders, the game simulates the look of an old CRT monitor, just like the arcade machines of old had! At the press of a button or after waiting for a bit we’re greeted with this:

Kung Fury: Street Rage - Insert Coin

Insert Coin!

Another button press and we can hear our virtual player throwing a coin into the machine, which gives us three lives (after being hit three times, we’d go down for good). And then, whenever any enemy approaches us from either side, you just press left or right to punch, kick, shoot or electrify the guy. It’s ok, they’re all Nazis anyway. We do this with our pals Barbarianna, Triceracop and Hackerman standing around in the background – all three as seen in the movie of course, just like all the enemies we’re beating up:

Kung Fury: Street Rage; Beat 'em up!

There are splatter effects even!

That screenshot is from the very beginning of the game, where we can only see our lowest-end Nazi foes. There are some Swedish Aryans too, which can take two hits, then that clone chick with Kung Fury essence infused into her, which needs a more advanced left/right combo to put down, and more. Like the kicking machine and the mysterious Ninja, all as seen in the movie. As long as you don’t miss too much (you have limited range) or get hit, you’ll build up a score bonus too. Not sure if there are more enemies than that, I haven’t really gotten that far yet.

Actually, I did reach a new High Score while doing those screenshots accidentally, leaving both chicks behind me, pretty neat:

Kung Fury: Street Rage; New High Score!

A new High Score!

Now Thor might still be doable, but Hackerman will be one tough nut to crack. I don’t think I’ll ever make it to the top though, the game is pretty damn hard. As it progresses, it starts speeding up more and more, and it’ll also throw more of the harder enemies at you, which will require quick reaction and sharp perception to get the combos right. “Just mash two buttons” may sound easy as said, but don’t underestimate it! Like with “One Finger Death Punch”, only the most skillful players will have a chance to reach the top!

When you’ve got enough, just press <Esc> (on the PC), and you’re asked whether you really want to quit. In an interesting way:

Kung Fury: Street Rage; Quitting the game.

Quitting/bluescreening the game.

If you confirm to quit the game, you’ll get another shader-based CRT effect thrown in:

Kung Fury: Street Rage; Quitting the game

It’s shutting down…

I haven’t really managed to play this for more than 5 minutes in a row, which sounds like very little, but this game is extremely fast-paced, so I can’t take much more in one go. ;) It’s quite a lot of fun though, and while not as sophisticated as “One Finger Death Punch”, it’s awesome in its own right, given the Kung Fury cheesiness, the CRT look and the chiptune-like soundtrack of the game.

The game is available in both paid and partially also free editions on several platforms now, and while I’ve read that the free versions do have ads, the paid ones definitely don’t, as I can vouch for on the PC platform at least. So here are the links:

  • [PC version] @ Steam for 1.99€ / $2.50. Supports >=Windows XP, >=MacOS X 10.6 and SteamOS plus regular Linux on x86_32/x86_64.
  • [PC+ARM version] @ Windows Store for $2.29. Supports Windows >=8.0 on x86_32/x86_64 and ARM architectures.
  • [Android version] @ Google Play for free or for 2.46€ with ads removed. Also available as a [separate APK file]. Supports Android >=2.0.1.
  • [iOS version] @ iTunes for free or for $1.99 with ads removed. Supports iOS >=6.0 on the iPhone 5/6, iPad and iPod Touch.

Now if you’ll excuse me, I’ll take another shot at number 3! :)