Nov 192016

FreeBSD GMABoost logoRecently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # 200 or # 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Nov 142016

HP/Compaq nx6310/nc6320 logoA good while back, I got a free notebook from [The_Plague]German flag, a HP/Compaq nx6310[1][2] which he kinda pulled out of the trash at his company. It’s not exactly “Thinkpad T23” material, but it’s a pretty solid, well-built machine with a good keyboard. I’ve been using the thing as an operating system testbed for a while (Linux, ReactOS, Haiku OS, OpenBSD, Dragonfly BSD, and finally: FreeBSD UNIX). After settling for FreeBSD the machine clearly showed its limitations though, the most problematic being imposed by the very low-end i940GML chipset. That one has limited the machine to a single processor core and a 533MHz data rate FSB.

I did give the machine a Core Duo T2450, but switching dual core on in the BIOS results in a lockup at POST time. Also, the chipset cannot use dual-channel DDR-II and limits the user to 2GiB of memory, making the use of a 64-bit processor rather pointless. Which turned out to be bad, because some code doesn’t even provide full functionality for 32-bit anymore, like x265, which dropped deep color support on 32-bit architectures.

But now, The_Plague pulled another one out of the trash, it’s basically the exact same machine, but a higher-end model, the nc6320. This one has an i945GM chipset, which means dual core support, FSB667 and 4GiB dual-channel RAM capability! It came with a Core 2 Duo T5600 @ 1.83GHz with 2MiB L2 cache. I ordered the largest possible chip for this box from ebay Hong Kong, so now it has a Core 2 Duo T7600 @ 2.33GHz with 4MiB L2 cache. Also, 2×2=4GiB of DDR-II/667 CL4 are on their way already, together with a 12-cell secondary monster battery!

And of course, FreeBSD UNIX again, in its brand new version 11.0-RELEASE:

HP/Compaq nc6320 running FreeBSD 11.0 UNIX

HP/Compaq nc6320 running FreeBSD 11.0 UNIX (click to enlarge)

The CPU upgrade is actually even noticeable when browsing the web, lots of resource-hungry Javascript and CSS3, you know. Luckily, Chromium supports hardware acceleration on the Intel GMA950 GPU on FreeBSD, as the OS comes with a kernel modesetting compliant driver for almost all integrated Intel graphics chips. It’s too slow to do the rasterization stage on the GPU, but it still helps.

Once again, it shall serve mostly as a meeting and sysadmin machine, with a little bit of private-use-fun added on top. Let’s have a look at the software! Oh and by the way, I decided to make the screenshots 8-bit .png images, so some of them will look a bit bad. But still better+smaller than JPEG for that purpose:

Running screenfetch on the nc6320

Running screenfetch on the nc6320 (click to enlarge)

$ screenfetch is showing us some details about the machine, which also makes it clear that everything is “Tokisaki Kurumi”-themed. Since there’s a lot of red color on that girls’ garments it seems at least somewhat fitting for a FreeBSD machine.

Chromium with FVD Speed Dial

Chromium with FVD Speed Dial (click to enlarge)

I’m a [Vivaldi] fan personally, but that browser isn’t available on any BSD yet, so I installed a few extensions to make Chromium work somewhat like Vivaldi; The most important part being the static FVD speed dial you can see above. What you can’t see here are the other extensions that followed it: AdBlockPlus and Ghostery. I hear there are better/faster solutions than ABP for ad blocking these days however, so maybe I’ll revise that.

IBM Lotus Notes via wine 1.8

IBM Lotus Notes 6.5.1 via 32-bit wine 1.8.4 (click to enlarge)

Also, for work I would sometimes need IBM Lotus Notes, as it’s our Universities’ groupware solution (think of that what you will). While I couldn’t get the Linux version to run, our Domino servers still accept connections from older clients, so it’s Lotus Notes 6.5.1 running under a 32-bit [wine], which is a solution IBM officially recommended for running the software on Linux/UNIX a few years ago. And yeah, it still works. And if you have Windows software wine can’t cope with?

XP x64 via VirtualBox on FreeBSD

XP x64 via VirtualBox on FreeBSD (click to enlarge)

For anything that wine can’t handle, the VirtualBox port kicks in, as we can see here. Together with the CPUs VT-x extension and the guest tools, virtualizing Windows on FreeBSD UNIX works relatively well. Not all features are there (like USB passthrough), but it works ok for me. Will need a Windows 7 VM as well I think.

More stuff:

Communicating on FreeBSD

Communicating on FreeBSD (parts are censored, click to enlarge)

One important part is communication! Luckily, there is a version of licq in the ports tree now. It builds well together with its Qt4 UI, so no complaints there. Hexchat for IRC access is also available, but the tricky part was Skype; Not that I really need it, but I wanted to have the linuxulator up and running as well! For those of you who don’t know what the “linuxulator” is: It’s a series of kernel modules that extend FreeBSDs kernel with parts of the Linux kernel interface. On top of that, you can pull parts of Fedora 10 or CentOS 6.8 or some CentOS 7 Linux userspace components from the package servers. Together with the kernel modules those form a kind of runtime environment for executing Linux programs – Skype 4.3 in this case! So I have both wine and linuxulator ready for action, and with it access to ICQ, Jabber, MSN, IRC and Skype. Now, what about multimedia?

Multimedia on FreeBSD

smplayer and xmms on FreeBSD, unfortunately the 8-bit color is a bit too noticeable for this screenshot, my apologies (click to enlarge)

This is a part where the upgraded processor also helps. Here we can see (s)mplayer play the last episode of the Anime Hanayamata in taxing 2.5Mbit H.265/HEVC encoding, paired with AAC-LC audio. The Core 2 Duo T5600 had some issues with this, but the faster T7600 shows now problems. Additionally, xmms is playing a Commodore 64 SID tune using libsidplay2 and the reSID engine. xmms comes with a lot of funny plugins from the FreeBSD ports tree for Gameboy tunes or NES tunes, but the C64 one you need to compile for yourself. Not too hard though, you can fetch libsidplay2 and reSID from packages beforehand to make things easier! What else?


ioquake3, a cleaned up version of the Quake III Arena source code, here in its 64-bit FreeBSD build (click to enlarge)

A pretty fun part: Playing the native Quake3 port [ioquake3] in 64-bit, for whenever you just need to shoot something to blow off some steam. ;) I have to say, I had to tweak it quite a bit to run fluently on the WVA 1400×1050 display of this book given the weak GMA950 GPU, but it runs “rather ok” now. ioquake3 is also available for Windows, OSX and Linux by the way, including a more advanced OpenGL 2 renderer, which gives users access to some advanced graphical effects. And if I get bored by that…

HakuNeko Manga ripper and qComicbook

HakuNeko Manga ripper and qComicbook showing some sweet girls love! (click to enlarge)

Once again, fixing up HakuNekos’ build system and C++ code to work with FreeBSD properly took some time. Unfortunately there is no port for it yet (and I’m too stupid/lazy to create one), so you have to fix it by hand. Lots of replacing sed invocations with gsed, find with gfind etc. and the OS #ifdef parts, which need to be changed in several .cpp files, here’s an example from MangaConnector.cpp:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

Something like that needs to turn into this to compile on FreeBSD, otherwise you’ll end up with a HakuNeko that can’t do shit (it’ll still compile and run, but like I said, it’d be devoid of function):

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

This is true for the latest version 1.4.1 as well. I guess the modifications should also apply to other operating systems by adding things like __OpenBSD__ or similar.

Now all that’s left is to wait for that massive 12C battery, the RAM capacity+speed upgrade and some FreeBSD case sticker that I ordered from [] (hint: That’s a referral URL, it’s supposed to give you some $5 coupon upon ordering, I hope it works). Upon my order, a small part was donated to the LLVM project – very fitting, given that I’ve used clang/llvm a lot to compile stuff on FreeBSD as of late. :)

FreeBSD case sticker (preview)

This is what it’s supposed to look like, and it’s going to replace the current Windows XP+Vista sticker

I hope it’ll look as good in real life! :) Ah, I think I’m gonna have a lot of fun with that old piece of junk. ;)

Ah, and thanks fly out to The_Plague, who saved this laptop from the trash bin and gave it to me for free! Prost!

Edit: And the memory is here, two G.Skill “performance” modules doing 4-4-4 latencies at 667MHz data rate, replacing a single Samsung module running 5-5-5. Now I was interested in how much going from single channel CL5 to dual channel CL4 would really affect performance. Let’s just say, it didn’t do too much for CPU processes. However, the effect on the integrated GMA950 GPU (using shared system memory!) was amazing. It seems the graphics chip was held back a lot by the memory interface! Let’s have a quick look at Quake III Arena performance using a quickly recorded demo just for this purpose (ioquake3 can’t play old Quake III Arena demos like the “001” demo):

  • ioquake3 1.36, single channel DDR-II/667 CL5:
  • 30.6fps
  • ioquake3 1.36, dual channel DDR-II/667 CL4:
  • 41.2fps

Roughly +35%!!

Tests were run three times, then three more times after a reboot. After that, an average was taken. For ioquake3 this wouldn’t even have been necessary though, as the results were extremely consistent. It’s amazing how much the added memory speed really affects the game engine! I rebooted and re-ran the tests several times because I couldn’t believe in that massive boost in performance, but it’s actually true and fully reproducible! This reminds me of how well modern AMD APU graphics chips scale with main memory speed and it explains why people were asking for quad-channel DDR4 on those Kaveri APU chips. Its built-in Radeons would’ve probably loved the added bandwidth!

I also kinda felt that browsing web sites got a lot more smooth using Chromium with most of its GPU acceleration turned on. So I tried the graphics-centric browser test [Motionmark] to put that to the test. Parts of the results were inconclusive, but let’s have a look first:

  • Motionmark 1.0 (medium screen profile), single channel DDR-II/667 CL5:
  • Overall result: 13.85 ±22.24%
  • Multiply: 119.26 ±2.95%
  • Canvas Arcs: 19.04 ±68.48%
  • Leaves: 3.00 ±133.33%
  • Paths: 85.30 ±6.57%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 1.76 ±5.22%
  • Images: 40.58 ±2.56%
  • Design: 18.89 ±8.00%
  • Suits: 24.00 ±37.50%
  • Motionmark 1.0 (medium screen profile), dual channel DDR-II/667 CL4:
  • Overall result: 22.47 ±15.93%
  • Multiply: 124.55 ±1.60%
  • Canvas Arcs: 26.00 ±138.46%
  • Leaves: 65.90 ±16.93%
  • Paths: 37.00 ±16.89%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 2.00 ±50.00%
  • Images: 41.58 ±3.59%
  • Design: 24.49 ±2.35%
  • Suits: 90.65 ±13.55%

Now first things first: This was just my first pick for any kind of graphics-heavy browser benchmark. I thought I needed something that would make the browser do a lot of stuff on the GPU, given that hardware acceleration was almost fully enabled on FreeBSD UNIX + Chromium + GMA950. However, after repeated runs it showed that the variance was just far too high on the following tests: Leaves, Paths, Suits. Those would also mess up the overall score. The ones that showed consistent performance were: Multiply, Canvas Arcs, Canvas Lines, Focus, Images, Design, so we should focus on those. Well, not all of those tests show promising results (Multiply, Canvas Lines), but some clearly do. It seems my feeling that parts of CSS3 etc. had gotten faster after the memory upgrade was spot-on!

Not bad, not bad at all! And tomorrow morning, the [x264 benchmark] will also have finished, showing how much a classic CPU-heavy task would profit from that upgrade (probably not much, but we’ll see tomorrow).

Edit 2: And here is the rest. Like I thought, the memory upgrade had only minimal impact on CPU performance:

  • x264 benchmark, single channel DDR-II/667 CL5:
  • Runtime: 04:40:08.621
  • x264 benchmark, dual channel DDR-II/667 CL4:
  • Runtime: 04:38:23.851

So yeah it’s faster. But only by a meager +0.62%. Completely negligible. But it’s still a good upgrade given the GPU performance boost and the fact that I can now use more memory for virtual machines. :)

Ah, and here’s the 12-cell ultra capacity battery, which gives me a total of 18 cells in conjunction with the 6-cell primary battery:

Nice hardware actually, you can check it’s charge (roughly) with a button and a 4-LED display, and it has it’s own charging plug. What surprised me most though was this:

$ hwstat | grep -i -e "serial number" -i -e battery
[ACPI Battery (sysctl)]
        Serial number:                  00411 2006/10/12
        Serial number:                  00001 2016/07/29

That probably explains how a still sealed battery could come with a ~25% pre-charge. Manufactured in July 2016, wow. And that for a notebook that’s 10 years old? Ok, it’s an aftermarket battery by [GRS], but that’s just damn fine still! With that I’ll surely have enough battery runtime to make it through longer meetings as well! :)

Edit 3: And today I used the notebook for a sysadmin task, helping our lead developer in debugging a weird problem in a Java-based student exam submission and evaluation system of ours at work. I suspected that the new CuPPIX (=KNOPPIX derivative) distribution I built for this was to blame, but it turned out to be a faulty Java library handling MySQL database access, hence crashing our server software under high parallel loads. In any case, I had the nc6320 with me during the entire morning up until 12:30 or so, walking away with a total charge of 49% left after the developer had fixed the problem. Not stellar given a total of 18 cells, but definitely good enough for me! :)

Edit 4: And my FreeBSD sticker from unixstickers is finally here! They even gave me a bunch of random free stickers to go with it! I gave those to some colleagues for their kids. ;) And here it is:

FreeBSD sticker from

There was a Windows Vista/XP sticker before, now it shows some UNIX love! (click to enlarge)

The sticker shows some pretty good quality as well, nice stuff! :)

Sep 072016

TeamViewer on Linux logoI’m not exactly a big fan of TeamViewer, since you’ll never know what’s going to happen with that traffic of yours, so I prefer VNC over SSH instead. A few weeks ago I got TeamViewer access to a remote workstation machine for the purpose of processing A/V files however. Basically, it was about video and audio transcoding on said machine.

Since the stream meta data (like the language of an audio stream) wasn’t always there, I wanted to check it by playing back the files remotely in foobar2000 or MPC-HC. TeamViewer does offer a feature to relay the audio from a remote machine to your local box, as long as the remote server has some kind of soundcard / sound chip installed. I was using TeamViewer 11 – the newest version at the time of writing – to connect from CentOS 6.8 Linux to a Windows 7 Professional machine. Playing back audio yielded nothing but silence though.

Now, TeamViewer is actually not native Linux software. Both its Linux and MacOS X versions come with a bundled Wine 1.6 distribution preconfigured to run the 32-bit TeamViewer Windows binary. It was thus logical to assume that the configuration of TeamViewers’ built-in Wine was broken. This may happen in cases where you upgrade TeamViewer from previous releases (which is what I had done, 7 -> 8 -> 9 -> 11).

There are a multitude of proposed solutions to fix this, and since none of them worked for me as-is, I’d like to add my own to the mix. The first useful hint came from [here]. You absolutely need a working system-wide Wine setup for this. I already had one that I needed for work anyway, namely Wine 1.8.6 from the [EPEL] repository, configured using [winetricks]. We’re going to take some files from that installation and essentially replace TeamViewers’ own Wine with the one distributed by EPEL.

So I had TeamViewer 11 installed in /opt/teamviewer/ and some important configuration files for it in ~/.local/share/teamviewer11/ and ~/.config/teamviewer/. First, we backup the wine files of TeamViewer and replace them with the platform ones (the paths may vary depending on your Linux distribution, but the file names should not):

# mv /opt/teamviewer/tv_bin/wine/bin/wine /opt/teamviewer/tv_bin/wine/bin/wine.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wineserver /opt/teamviewer/tv_bin/wine/bin/wineserver.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/wine-preloader.BACKUP
# mv /opt/teamviewer/tv_bin/wine/lib/ /opt/teamviewer/tv_bin/wine/lib/
# mv /opt/teamviewer/tv_bin/wine/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/wine.BACKUP/
# cp /usr/bin/wine /usr/bin/wineserver /usr/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/
# cp /usr/lib/ /opt/teamviewer/tv_bin/wine/lib/
# cp -r /usr/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/

This will replace all the binaries and libraries, in my case shoving Wine 1.8.6 underneath TeamViewer. This isn’t all that’s needed however. We’ll also need the system registry hive of your working Wine installation (with sound). That should be stored in ~/.wine/system.reg! Let’s replace TeamViewers’ own hive with this one:

$ mv ~/.local/share/teamviewer11/system.reg ~/.local/share/teamviewer11/system.reg.BACKUP
$ cp ~/.wine/system.reg ~/.local/share/teamviewer11/

Ok, and the final part is adding the proper Linux audio backend to this Wines’ configuration. That part is stored in ~/.wine/user.reg. Replacing the whole file didn’t work for me though, as TeamViewer would crash upon launch, probably missing some keys from its own user.reg. So, let’s just edit its file instead, open ~/.local/share/teamviewer11/system.reg with your favorite text editor and add the following line in a proper location (it’s sorted alphabetically):

[Software\\Wine\\Drivers\\winepulse.drv] 1473239241

The corresponding file should be found within TeamViewers’ replaced Wine distribution now by the way, in my case it’s /opt/teamviewer/tv_bin/wine/lib/wine/fakedlls/winepulse.drv.

Now, run the TeamViewer profile updater (Some people say it’s required to make this work, it wasn’t for me, but it didn’t hurt either): $ /opt/teamviewer/tv_bin/TeamViewer --update-profile and then its’ Wine configuration: $ /opt/teamviewer/tv_bin/TeamViewer --winecfg. After that, you should be greeted with this:

TeamViewer 11 running its own winecfg

TeamViewer 11 running its own copy of winecfg.

Before the modifications, the configuration window would show “None” as the driver, without any way to change it. So no audio, whereas we have Pulseaudio now. Press “Test Sound” if you want to check whether it truly works. I haven’t tested the ALSA backend by the way. In my case, as soon as the registry was fixed, Wine just autoselected Pulseaudio, which is fine for me.

Now launch TeamViewer and check out the audio options in this submenu:

TeamViewer preferences

The TeamViewer 11 preferences can be found here.

It should look like this:

TeamViewers audio options

Make sure “Play computer sounds and music” is checked! (click to enlarge)

Now, after having connected and logged in, you may also wish to verify the conference audio settings in TeamViewers’ top menu:

TeamViewer 11 conference audio settings

TeamViewer 11 conference audio settings, make sure “Computer sound” is checked!

When you play a sound file on the remote computer, you should hear it on your local one as well. With that, I can finally test the audio files I’m supposed to use on that remote machine for their actual language (which is a rather important detail) where meta data isn’t available.

This seems to be a problem of TeamViewers installation / update procedure which hasn’t been addressed for several major released now. I presume just removing all traces of TeamViewer and installing it from scratch might also do the trick, but I didn’t try it for myself.

Ah, and one more thing: If you can’t launch TeamViewer on CentOS 6.x because you’re getting the following error…

teamviewerd error

TeamViewer Daemon not running…

…forget about the solutions on the web on top of what this message is telling you. TeamViewer 11 uses a systemd-style script for launching its daemon on Linux now, and that won’t do on SysV init systems. Just become root and launch the crap manually: # /opt/teamviewer/tv_bin/teamviewerd &, then press <CTRL>+<d> and it works!

Let’s hope that daemon isn’t doing anything evil while running as root. :roll:

Aug 282016

KERNEL_DATA_INPAGE_ERROR logoHere is how a responsible system administrator should handle downtimes and replacements of faulty hardware: Give advance notice to all users and make sure to give everybody enough time to prepare for services going offline, if possible. Specify a precise time window which is as convenient as possible for most users. Also, explain the exact technical reasons in words as simple as possible.

How I handled the replacement of XINs’ system hard disk? See that nice blue logo on the top left side? KERNEL_DATA_INPAGE_ERROR, bugcheck code 0x0000007a. And [it isn’t the first of its kind either], last one was a KERNEL_STACK_INPAGE_ERROR, clearly disk related given that the disk had logged controller errors as well as unrecoverable dead sectors. And NO, that one wasn’t the first one too. :roll: So yeah, I rebooted the [monster], and decided that it’s too much of a pain in the ass to fix it and hoped (=told myself while in denial) that it would just live on happily ever after! Clearly in ignorance of the obvious problem, just so I could walk over to my workstation and continue to watch some Anime and have a few cold ones in peace…

So, my apologies for being lazy in a slightly dangerous way this time. Well, it’s not like there aren’t any system backups or anything, but still. In the end, it caused an unannounced and unplanned downtime 3½ hours long. This still shouldn’t hurt XINs’ >=99% yearly availability, but it clearly wasn’t the right way to deal with it either…

Well, it’s fixed now, because this time I got a bit nervous and pissed off as well. Thanks to [Umlüx], the XIN server is now running a factory-new HP/Compaq 15000rpm 68p LVD/SE SCSI drive, essentially a Seagate Cheetah 15k.3. As I am writing this the drive has only 2.9h of power on time accumulated. Pretty nice to find such pristine hardware!

Thanks do however also fly out to [Grindhavoc]German flag and [lommodore]German flag from [Voodooalert]German flag, who also kindly provided a few drives, of which some were quite usable. They’re in store now, for when the current HP drive starts behaving badly.

Now, let’s hope it was just the disk and no Controller / cabling problem on top of that, but it looks like this should be it for now. One less thing to worry about as well. ;)

Aug 232016

UnrealIRCd logoOne of the services I’ve been running on for years now has been the IRC server UnrealIRCd. It’s available for Linux, UNIX and also Windows, so it’s a pretty neat choice I think. A few days ago however, a user had notified me, that his client couldn’t connect when using SSL/TLS encryption after an update of the software. I’m pretty sure this was due to the OpenSSL developers disabling the SSL v3 protocol by default. So his client only had TLS and my old UnrealIRCd 3.x only had SSL v3 => handshake failure:

error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

So what now? Just shoving a newer SSL library under my IRC server wouldn’t work in a stable fashion. So far, the only software I have ever seen which can be “magically” upgraded to modern protocols and ciphers this way was the Gene6 FTP server. All the way from OpenSSL 0.9.6 to 1.0.2. No idea how they did it.

Two options: Have users recompile their libraries and clients to enable SSL v3 (yeah, as if…), or try and backport a current (=2016-07-28) UnrealIRCd 4 to my server. One that supports both modern TLS v2 with modern ciphers as well as good old SSL v3, so legacy clients may connect in an encrypted fashion as well.

Why backport? Because it’s freaking Windows 2000 (and no, newer versions do *not* work), and UnrealIRCd dropped support for that, so I absolutely needed to recompile the server and several libraries it depends on. Now that was one wild ride for a user like me, I’m telling you.

Ah yes, this isn’t exactly a good step-by-step guide or anything, so in case you just wanna grab the files, scroll all the way down! If you want to know a few of the details… I don’t even remember all the things I did, but let’s see…


Here’s what you need:

  1. The Microsoft [Visual C++ 2008 runtime SP1 redistributable package] (only on the system where the server is supposed to run, not on the build system)
  2. Microsoft VisualStudio 2008 (I guess 2010 also works, as long as you have the v90 toolset available)
  3. Perl. I used [Strawberry Perl 5.24].
  4. The latest UnrealIRCd [dev package]. It’s for UnrealIRCd v3.4, but that doesn’t matter.
  5. The UnrealIRCd [source code]. I used the current/bugfixed version 4.0.5 for this build.
  6. A precompiled version of pcre2 supporting Windows 2000, I only found one eligible one [here]. (I failed to recompile/relink pcre2 properly, even with the version from the dev package :( )
  7. The stock [tre 0.8.0 library] source code, because it supports VS2008. The version shipped with the dev package doesn’t.
  8. The latest [OpenSSL library] source code, it’ll serve as a replacement for the older one shipped with the dev package.

If you cannot obtain Visual Studio 2008 via any (legal!) means, that’d probably mean you’re out of luck though. Luckily, I got all versions from Microsofts MSDNAA / DreamSpark program, but if you’re stuck on something like VS2012, 2013 or 2015, I cannot help you. Maybe this can still work out, but you’ll still need the 2008 version to get the v90 toolset (I guess, not an expert here…)


There are quite a few, but here are the ones that I still remember:

1.) Additional headers are required to link some of the software, there are free ones available. You can grab them [here]. Put them into the VC\include\ subdirectory of your Visual Studio 2008 installation folder. On top of those two, inttypes.h and stdint.h you’ll also need unistd.h, but that one’s easy: Just make a copy of io.h in that same folder and rename that copy to unistd.h and you’re done.

2.) First, cURL-SSL was built with the nmake options ENABLE_IPV6=no and ENABLE_IDN=no set. IPv6 support on Windows 2000 does exist by using an [experimental update], but it’s function calls are different than with Microsofts’ final version, so it’s unusable by most software. Also, IDN support is only available [for Windows XP and later], so internationalized domain names using non-ASCII characters don’t work. UnrealIRCd is to be linked against this version.

3.) tre replaced with latest stock tre 0.8.0 and recompiled, UnrealIRCd is to be linked against this build.

4.) Before building OpenSSL, it may need modifications to its makefile ms\ntdll.mak, which is generated by the ms\do_nasm step described in OpenSSLs INSTALL.W32, depending on your requirements. It is here where you can enable older, weaker ciphers and the older SSL v3/v2 protocols. Enable these deprecated version only if you absolutely need them!

Look for line 21 (Note, that the ^ line breaks aren’t in the file originally, it’s all in one line. I just added them here for readability purposes):

  1. CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^

You could replace this with the following, allowing weak ciphers and SSL v3, but not SSL v2 for example:

  1. #CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^
  8. CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^

Compile as shown in the documentation, and install somewhere.

5.) Before UnrealIRCd can use the new version of OpenSSL it may need modifications to match the ones patched into the OpenSSL makefile. By default, it will also block stuff like SSL v3. Enter its source tree and open ssl\ssl.c, then locate lines 245 and 321, which will look like this:

  1. SSL_CTX_set_options(ctx_server, SSL_OP_NO_SSLv3);

Just comment that out:

  1. /** SSL_CTX_set_options(ctx_server, SSL_OP_NO_SSLv3); **/

If you enabled SSLv2 as well and want the IRC server to be able to use it, do the same for lines 244 and 320, look for this…

  1. SSL_CTX_set_options(ctx_client, SSL_OP_NO_SSLv2);

…and comment it out again:

  1. /** SSL_CTX_set_options(ctx_client, SSL_OP_NO_SSLv2); **/

Now compile and link as shown in the UnrealIRCd documentation. Like the developers I’d recomment assembling a proper command line for this, as editing the makefile all the time can be cumbersome, especially if you’re running into trouble along the way.

What else?

Some of the VS project files may be preconfigured for platform toolsets you don’t have (like v100, v110, etc.) or may be set to produce a Debug build by default. Make sure you’re using only the v90 toolset and produce only Release builds. To learn how, check out the Visual Studio documentation online. It’s not that hard for the stuff you need to build with the GUI.

And here is the file:

Note that I may have done something horribly wrong along the way with this, because it really works only on Windows 2000. This is not how it should be. But launching it on a newer operating system yields something like this:

UnrealIRCd runtime error on anything greater than or equal to Windows XP

Yeah… umm… riiight…

And after pressing OK, this:

UnrealIRCd runtime error on anything greater than or equal to Windows XP #2


I searched for those errors on the web for a little, but couldn’t find anything that would’ve told me why it breaks like this on “modern” operating systems, yet still works on Windows 2000. Oh, the build system was XP x64 by the way. Well, it doesn’t really matter, the standard build of the developers works on XP+ anyway, and this works only on Windows 2000. Mission accomplished in any case.

In this incarnation, the server can support SSL v3 as well as TLS v1.2 protocols and supports the following ciphers:


The necessary tools for creating an SSL/TLS certificate and for installing a Windows service for the server are also included (openssl.exe, unrealsvc.exe).


UnrealIRCd and the software it was linked against in this case is released under the following licenses:

Any modifications to any of the software packages above as posted on this page are hereby licensed under the same license as the original software before modifications were applied. When downloading any unmodified source code, you’ll have to patch it yourself before building for a Windows 2000 platform target.

And what now?

Well, I guess my server supports IRC+TLS for all modern clients now, so yay! ;) URLs are the same as before: [irc+ssl://] with SSL v3/TLS v1.2 or [irc://] if you wish to connect without any encryption enabled, all plain text.

Aug 052016

VirtualDimension logoShort story: It’s [VirtualDimension].

Long story… It’s most definitely not what Microsoft added in Windows 10. Besides it being limited to Windows 10, it just sucks for a multitude of reasons. And there I was, having hopes for it as well… If you’ve ever used multiple desktops on a graphical, X11-based Linux or UNIX window manager / desktop environment, you’d know what I’m talking about. Usually, what you’d get on those systems, whether KDE, Gnome, Xfce4, LXDE, or whatever is just one small, configurable panel which allows you to control multiple virtual desktops. On my Gnome 2 on CentOS 6.8 Linux, it looks like this (others are very similar):

Virtual desktops on Gnome 2

Virtual desktops on Gnome 2

The leftmost desktop is my usual “Internet” environment, here I chat, read emails, browse the web for anything work-related and so on. The second one is a Linux distribution development desktop. Here I’m building a derivative of Klaus Knoppers’ [Knoppix] distro. Then comes the testing environment for said distribution on desktop #3. Usually some shells and one VMware Player instance. Next to that are two more VM desktops for software testing and for writing user guides for software installation on different operating systems. At the moment that’s MacOS X and a Windows XP x64 software build VM. Usually there’s also a Windows 7 one. One is empty (for arbitrary stuff), then comes the server administration desktop with 9 open shells, one for each server. And the last one is my private desktop with yet another web browser, and some shells for spawning screen sessions for running software compilations, encoding runs and the likes.

Now, I have a 30″ screen both at home and at work, resolution is 2560×1600. But it’s just never enough screen real estate. So I wanted well-integrated virtual desktops for Windows as well, but last time I tried out some software, I couldn’t find a good one. Recently, I tried again for some reason, like “let’s give this one last shot”. And I tried a lot of programs!

Among the software tested were [Dexpot], [Finestra], [VirtuaWin], [WindowsPager], Xilisoft [Multiple Desktops] and the Windows PowerToy predecessor of [Desktops 2.0] written by Windows Hacker Mark Russinovich and Bryce Cogswell. And finally, [VirtualDimension]. Some of those are free and open source software, others are not.

One of my primary requirements was compatibility to Windows XP x64. Of course it’d be nice if it worked on Windows 10 as well. But most of the above had important features missing or were severely misbehaving on XP. Some were just very, very sluggish when switching desktops. Others had missing features to begin with, like previews on the desktop tiles. A blank desktop tile doesn’t help at all, as I need to see roughly what’s running where at a glance.

I’m not gonna make this a lengthy top list or anything, I’m just gonna show you what the software of my choice – VirtualDimension – could do for me, let’s look at the tiles first:

VirtualDimension on XP x64

VirtualDimension on XP x64

We’ll start with my good old XP x64 first. Here you can see my system tray, and Miranda being open. VirtualDimension cannot be embedded into the taskbar properly (damn), but it has an “always on top” feature. Since the contact list in my docked and always-launched Miranda doesn’t go all the way down, there is free and unused space there. Perfect for VirtualDimension! And since it’s always on top, it doesn’t disappear when clicking on Miranda for chatting.

Given the source code is definitely coming from a UNIX or Linux user (given he built it with GCC/Mingw), some features immediately ring a bell. Like “mouse warp”, where you switch desktops by moving your mouse to the border of the screen. I disabled that, don’t like it. But yeah, it’s there.

Important: While it doesn’t give you live window geometry previews, it does give you iconized previews, so you can always identify any desktop quickly by seeing what’s running there. The desktops can also be named, and there is an OSD that you can have pop up on you when switching, like so:

VirtualDimension OSD

OSD showing right after a desktop switch

In this case I had just switched to desktop #2, which is for A/V processing exclusively. This is just the top left part of the screen, where one of my eight transcoding shells was running a x265 benchmark prototype test. Color, display duration, transparency to mouse clicks on the OSD part, font and size are configurable.

Also, you can freely define keyboard shortcuts for switching desktops as well. I chose CTRL+Shift+Right as well as CTRL+Shift+Left for switching desktops and Alt+Right / Alt+Left for pushing a window to the next/previous desktop as those don’t conflict with other shortcuts I’m using.

What else can it do? Let’s right click on one of the icons in the preview tiles:

VirtualDimension iconized window right click

Clicking on a program icon in VirtualDimensions’ preview tiles gives you this menu

The first five options from the top are global ones. However, the ones below are specific to the icon you right-clicked. With “Activate”, you’d switch to the target desktop and put focus on that programs’ window. The others are pretty self-explanatory as well I guess. We also get a graceful “Close” option, and a brutal “Kill” option that’s equivalent to murdering the process in task manager. Maybe useful since it’s faster that way.

And if we click on the free area?

VirtualDimension, right click on the free area of a preview tile

Right-clicking on the free area of a preview tile gives you a list of all programs on that desktop.

Ok, not sure how useful that is, but at least it may help with identifying the windows on a desktop in more detail, as you get the window titles here. For my encoding shells I could get very quick glance at the progress, but not exactly in great detail. So the helpfulness of this is limited.

What else?

Well, it’s extremely fast! That’s one major plus for VirtualDimension, as several of the other solutions (open source ones as well) were abysmally slow, at least on XP x64. But damn, VirtualDimension just flies! And its memory footprint is minimal. I saw less than 12MiB of consumption here. Even if you add a truckload of Desktops (there seems to be no upper limit), it just won’t slow down unless you spawn like 50 CPU intensive processes all over the place killing your CPU or maxing out your RAM. But that wouldn’t have been VirtualDimensions fault then. Its memory footprint will linearly grow by spawning more desktops, so with eight you may see around 20MiB. Still neat.

And what’s bad about it?

Well, sometimes, if you have a lot of windows on one desktop, the icons are’t cut off in the right spot at the bottom of the preview tile, so they overflow just a little bit. Just a cosmetic issue. Also, you should maybe deactivate the shell integration. With this, VirtualDimension hooks itself into all windows (such a DLL hook means entering another processes’ memory area). With that, you can get its functions via right clicking on a windows’ title bar, like on UNIX.

Nice, but dangerous! This can trigger anti-cheat systems in online games, because they really don’t like you stepping into their processes’ memory areas! That’s what cheating tools do to modify a games’ parameters on the fly as well. You don’t wanna be banned because of such a thing!

In my case, I managed to lock myself out of Mechwarrior Online because of this. I wasn’t banned, but the login process wouldn’t even let me launch its window. Disabling the feature, launching MWO, then re-enabling it and trying to log in caused a pretty abnormal process termination:

Mechwarrior Online really doesn't like VirtualDimensions' shell integration feature

Mechwarrior Online really doesn’t like VirtualDimensions’ shell integration feature! And no, there was no “update available”… (click to enlarge)

There is an exception list for this feature, and I added all of MWOs’ .exe files to it, to no avail. Better to stay away from this one.

Now, well, this otherwise beautiful piece of software was dropped by its developer around 2005. About the time my XP x64 came out. Latest alpha build is from some time in 2006. So this is ancient! It even supports Windows 98 and NT 4.0, I mean… So, how about Windows 10 then? I mean, Windows 10 doesn’t even have a GDI UI anymore, this is like one completely different world. Since I do have a Windows 10 machine (yeah, ew), let’s check it out:

VirtualDimension on Windows 10

VirtualDimension on Windows 10 – hey, it really works!?

Miranda seemingly can’t dock properly on Windows 10 anymore. It kinda… floats near the desktop border when docked. It’s strange and it wastes space, but well, I don’t know how to fix that yet. But anyway, I embedded VirtualDimension into Miranda (by just moving the window there, removing its title bar and resizing it properly again). And guess what?

It just works™!

I launched some Metro / Modern UI apps in a window as well, and while those aren’t shown in the preview tiles, they can be controlled with keyboard shortcuts, just like regular windows. Also, it’s just as blazing fast as it is on XP. Ah, and… yeah, it actually does work on all 64-bit x86 Windows versions it seems! It’s amazing, but an ancient piece of 32-bit software that does alter a Windows systems’ usage pattern quite fundamentally still works fine on Windows 10 64-bit. I gotta say, I’m pretty relieved, because Windows 10s’ own solution just sucks – where is my live preview? – and I don’t want to change my usage paradigm too much when switching operating systems (even from Linux/UNIX to Windows and back).

Some of the other solutions like Dexpot or Finestra may be faster on Windows 10 than they are on a just half-supported XP x64, but nah, with this tool, I don’t need them anymore, no matter the operating system.

VirtualDimension is as perfect as it gets, despite its age! Or maybe because of it?

Still, anyone interested in picking up that [VirtualDimension project on SourceForge] and in continuing its development? ;) I guess I can’t touch that code, would probably just mess everything up. Ah, it’s C++ by the way…

A few things could use some fixing, like the icon overflowing issue, Modern UI window detection, certain, rare windows being sticky on all Desktops even if nobody told them to do so (Miranda, X-Chat DCC windows) and that exclusion list for the shell integration, which doesn’t hook into all windows properly when active either anyway.

Would be so nice if somebody could continue working this! :)

Until that happens (I know, it never will), I’ll just continue using v0.95 alpha. ;)

Aug 032016

xmms logoYeah, I’m still using the good old xmms v1 on Linux and UNIX. Now, the boxes I used to play music on were all 32-bit x86 so far. Just some old hardware. Recently I tried to play some of my [AAC-LC] files on my 64-bit Linux workstation, and to my surprise, it failed to play the file, giving me the error Pulse coding not allowed in short blocks when started from the shell (just so I can read the error output).

I thought this had to have something to do with my file and not the player, so I tried .aac files with ADTS headers instead of audio packed into an mp4/m4a container. The result was the same. Also, the encoding (SBR/VBR or even HE-AAC, which sucks for music anyway) didn’t make a difference. Then I found [this post] on the Gentoo forums, showing that the problem was really the architecture. 32-bit builds wouldn’t fail, but the source code of the [libfaad2] decoding library of xmms’ mp4 plugin wasn’t ready for 64-bit.

xmms’ xmms_mp4Plugin-0.4 comes with a pretty old libfaad2, 2.0 or something. I will show you how to upgrade that to version 2.7, which is fixed for x86_64 (Readily fixed source code is also provided at the bottom). We’ll also fix up other parts of that plugin so it can compile using more modern versions of GCC and clang, and my test platform for this would be a rather conservative CentOS 6.8 Linux. First, get the source code:

I’m assuming you already have xmms installed, otherwise obtain version 1.2.11 from [here]!

Step 1, libfaad2:

Unpack both archives from above, then enter the mp4 plugins’ source directory. You’ll find a libfaad2/ directory in there. Delete or move it and all of its contents. From the faad2 source tree, copy the directory libfaad/ from there to libfaad2/ in the plugins’ directory, replacing the deleted one. Now that’s the source, but we also need the updated headers so that the xmms plugin can link against the newer libfaad2. To do that, copy the two files include/faad.h and include/neaacdec.h from the faad2 2.7 source tree to the directory include/ in the plugin source tree. Overwrite faad.h if prompted.

Step 2, libmp4v2:

Also, the bundled libmp4v2 of the mp4 plugin is very old and broken on modern systems due to code issues. Let’s fix them, so back to the mp4 plugin source tree. We need to fix some invalid pure specifiers in the following files: libmp4v2/mp4property.h, libmp4v2/mp4property.cpp, libmp4v2/rtphint.h and libmp4v2/rtphint.cpp.

To do so, open them in a text editor and search and replace all occurences of = NULL with = 0. This will fix all assignments and comparisons that would otherwise break. When using vi or vim, you can do :%s/=\sNULL/= 0/g for this.

On top of that, we need to fix a invalid const char* to char* conversion in libmp4v2/rtphint.cpp as well. Open it, and hop to line number 325, you’ll see this:

  1. char* pSlash = strchr(pRtpMap, '/');

Replace it with this:

  1. const char* pSlash = strchr(pRtpMap, '/');

And we’re done!

Now, in the mp4 plugins’ source root directory, run $ ./bootstrap && ./configure. Hopefully, no errors will occur. If all is ok, run $ make. Given you have xmms 1.2.11 installed, it should compile and link fine now. Last step: # make install.

This has been tested with my current GCC 4.4.7 platform compiler as well as GCC 4.9.3 on CentOS 6.8 Linux. Also, it has been tested with clang 3.4.1 on FreeBSD 10.3 UNIX, also x86_64. Please note that FreeBSD 10.3 needs extensive modifications to its build tools as well, so I can’t provide fixed source for this. However, packages on FreeBSD have already been fixed in that regard, so you can just # pkg install xmms-faad2 and it’s done anyway. This is likely the case for several modern Linux distros as well.

Let’s play:

xmms playing AAC-LC Freebsd 10.3 UNIX on x86_64

Yeah, it works. In this case on FreeBSD.

And the shell would say:

2-MPEG-4 AAC Low Complexity profile
MP4 - 2 channels @ 44100 Hz

Perfect! And here is the [fixed source code] upgraded with libfaad2 2.7, so you don’t have to do anything by yourself other than $ ./bootstrap && ./configure && make and # make install! Ah, actually I’m not so sure whether xmms itself builds fine on CentOS 6.8 from source these days… Maybe I’ll check that out another day. ;)

Jul 272016

x264 logoSince I’ve been doing a bit of Anime batch video transcoding with x264 and x265 in the last few months, I thought I’d document this for myself here. My goal was to loop over an arbitrary amount of episodes and just batch-transcode them all at once. And that on three different operating systems: Windows (XP x64), Linux (CentOS 6.8 x86_64) and FreeBSD 10.3 UNIX, x86_64. Since I’ve started to split the work across multiple machines, I lost track of what was where and which machine finished what, and when.

So I thought, why not let the loop send me a small notification email upon completion? And that’s what I did. On Linux and UNIX this relies on the bash shell and the mailx command. Please note that I’m talking about [Heirloom mailx], not some other mail program by the same name! I’m mentioning this, because there is a different default mailx on FreeBSD, that won’t work for this. That’s why I put alias mailx='/usr/local/bin/mailx' in my ~/.bash_profile on that OS after installing the right program to make it the default for my user.

On Windows, my loops depend on my own [colorecho] command (you can replace that with cmds’ ECHO if you want) as well as the command line mailer [blat]. Note that, if you need to use SSL/TLS encryption when mailing, blat can’t do that. A suitable replacement could be [mailsend]. Please note, that mailsend does not work on Windows XP however.

In the x265 case, avconv (from the [libav] package) is required on all platforms. You can get my build for Windows [here]. If you don’t like it, the wide-spread [ffmpeg] can be a suitable drop-in replacement.

Now, when setting up blat on Windows, make sure to run blat -help first, and learn the details about blat -install. You need to run that with certain parameters to set it up for your SMTP mail server. For whatever reason, blat reads some of that data from the registry (ew…), and blat -install will set that up for you.

Typically, when I transcode, I do so on the elementary streams rather than .mkv files directly. So I’d loop through some source files and extract the needed streams. Let’s say we have “A series – episode 01.mkv” and some more, all the way up to “A series – episode 13.mkv”, then, assuming track #0 is the video stream…

On Windows:

FOR %I IN (01,02,03,04,05,06,07,08,09,10,11,12,13) DO mkvextract tracks "A series - episode %I.mkv" ^

On Linux/UNIX:

for i in {01..13}; do mkvextract tracks "A series - episode $i.mkv" 0:$i/video.h264; done

mkvextract will create the non-existing subfolder for us, and a x264 transcoding loop would then look like this on Windows:

expand/collapse source code
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps ^
 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f ^
 -2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats %I\v.stats -t 2 --no-fast-pskip ^
 --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o %I\pass1.264 & colorecho "Pass 1 ^
 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & ^
 "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps 24000/1001 --preset veryslow --tune animation ^
 --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats ^
 %I\v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o ^
 %I\pass2.264 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10) & echo !SERIES! ^
 transcoding complete | blat - -t "" -c "" -s "x264 ^
 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET SERIES="

Note that I always write all the iteration out in full here. That’s because cmd can’t do loops with leading zeroes in the iterator. The reason for this is that those source files usually have them in their lower episode numbers. If it wasn’t 01,02, … ,12,13, but 1,2, … ,12,13 instead, you could do FOR /L %I IN (1,1,13) DO. But this isn’t possible in my case. Even if elements need alphanumeric names like here,  FOR %I IN (01,02,03,special1,special2,ova1,ova2) DO, you still won’t need that syntax on Linux/UNIX because the bash can have iterator groups like for i in {{01..13},special1,special2,ova1,ova2}; do. Makes me despise the cmd once more. ;)


Ah, according to [this], you can actually do something like cmd /V /C "FOR /L %I IN (1,1,13) DO (SET "fI=00%I" & echo "!fI!:~-2")", holy shit. It actually works and gives you leading zeroes. :~-2 for 2 digits, :~-3 for three. Expand fI for more in this example. I mean, what is this even? Some number formatting magic? I probably don’t even wanna know… Couldn’t find any way of having several groups for the iterator however. Meh. Still don’t like it.

So, well, it’s like this on Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do nice -n19 x264 --fps \
24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f \
-2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats $i/v.stats -t 2 --no-fast-pskip \
--cqm flat --non-deterministic --demuxer lavf $i/video.h264 -o $i/pass1.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && nice -n19 \
x264 --fps 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid \
normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats $i/v.stats -t 2 --no-fast-pskip --cqm flat \
--non-deterministic --demuxer lavf $i/video.h264 -o $i/pass2.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x264 notification from $MACHINE" -r \
"" -c "" -S smtp-auth="login" -S \
smtp="" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \

The variable $MACHINE or %MACHINE%/!MACHINE! specifies the machines’ host name. This will be noted in the email, so I know which machine just completed something. $EPNUM – or %EPNUM%/!EPNUM! on Windows – is used for periodic updates on the shell. The output would be like “Pass 1 done for Episode 07/13 of AnimeX” in green on Windows and bold red on Linux/UNIX (just change the color to your liking).

Finally, $SERIES aka %SERIES%/!SERIES! would be the series’ name. So say, the UNIX machine named “BEAST” above is done with this loop. The email would come with the subject line “x264 notification from BEAST” and would read “AnimeX transcoding complete” in plain text. That’s all.

Please note, that cmd batch on Windows is extremely creepy. Every whitespace (especially the leading ones when doing multi-line like this for display) needs to be exactly where it is. The same goes for double quotes where you might think they aren’t needed. They are! Also, this needs delayed variable expansion once again, which is why we see variables like !EPNUM! instead of %EPNUM% and why it’s called in a subshell by running cmd /V /C.

On Linux/UNIX we don’t need to rely on some specific API like cmds’ SetConsoleTextAttribute() to print colors, as most terminals understand ANSI color codes.

And this is what it looks like for x265:


expand/collapse source code
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO avconv -r 24000/1001 -i %I\video.h264 -f yuv4mpegpipe ^
 -pix_fmt yuv420p -r 24000/1001 - 2>NUL | "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 ^
 --fps 24000/1001 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate ^
 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 ^
 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
 --pass 1 --slow-firstpass --stats %I\v.stats --sar 1 --range full -o %I\pass1.h265 & colorecho ^
 "Pass 1 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & avconv -r 24000/1001 -i ^
 %I\video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>;NUL | ^
 "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 --fps 24000/1001 -p veryslow --pmode ^
 --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 ^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2 --stats %I\v.stats --sar ^
 1 --range full -o %I\pass2.h265 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" ^
 10) & echo !SERIES! transcoding complete | blat - -t "" -c ^
 "" -s "x265 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET ^


expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do avconv -r 24000/1001 -i \
$i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice -19 x265 - --y4m \
-D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 \
--rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq \
5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 \
--slow-firstpass --stats $i/v.stats --sar 1 --range full -o $i/pass1.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && avconv -r \
24000/1001 -i $i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice \
-19 x265 - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid \
--bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd \
1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 \
--pass 2 --stats $i/v.stats --sar 1 --range full -o $i/pass2.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x265 notification from $MACHINE" -r \
"" -c "" -S smtp-auth="login" -S \
smtp="" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \

And that’s it. The loops for audio transcoding are simpler, as that part is so fast, it doesn’t need email notifications. Runs for minutes rather than days. When all is done, I’d usually fire up the MKVToolnix GUI, and prepare a mux for the first episode. There is a nice “copy command line to clipboard” function there when you click on “Muxing” after everything is set up. With that I can build another loop that muxes everything to final .mkv files. On Windows that part is more complicated if you want Unicode support, so I needed to create input files by using a generator I wrote in Perl for that, but that’s for another day… :)

Oh, and if you wanna ssh into your Linux or UNIX boxes from afar to check on your transcoders, consider launching them on a GNU screen] session. It’s immensely useful! Too bad it won’t work on the Windows cmd. :(

Jul 152016

H.265/HEVC logoSince I’ve started to release x265 binaries for Windows XP / XP x64 (and above), I’ve started to take interest in how it’s performance has developed over time. When looking back on x264, we can see that the encoders’ performance has always gradually improved with more and faster assembly code, optimizations for new CPUs’ instruction set extensions (think AMD FMA4, SSE4a or Intel AVX/AVX2) and so on. Over many years x264 has become a ton faster, as you can also see when comparing an old x264 v0.107.1745 (white results) with newer ones (red results) for the same CPUs in the [x264 benchmark].

So, x265 is much newer, but still, it’s been around for a few years now – since 2013 to be more precise – so I’d like to take a look at its performance trends using the options I typically apply for transcoding animated content. You can find the encoding settings and other information about the test below.

Note that I don’t compile every revision of x265, so I can compare only a few, namely the ones I chose to [release] periodically, excluding the new 2.0+2, which I haven’t released yet (waiting a bit more for a release). Here we go:

x265 performance trend across versions (2017-01-26)

x265 performance trend across versions, state 2017-01-26 (click to enlarge)

previous charts
x265 performance trends across versions, 2016-07-15

x265 performance trends across versions, state 2016-07-15

Note that this runtime graph is inverted, in the sense that “higher = better”. Version 1.7 was the first which could actually encode using the options of my choice, older versions don’t understand all of them and are not comparable in my case because of that. We start off with pretty bad performance and then we get an ample boost with the earlier 1.9 versions.

From 1.9+15 to 1.9+141 we can see a gradual performance increase, as expected from continued development and optimization. Naturally, 8-bits per color channel is fastest, as it makes use of a lot of 8-bit arithmetic internally. For higher bit depths (10 and 12 bits per channel, or 30 and 36 bits per pixel respectively) the internal arithmetic is boosted to a 16 bit precision, resulting in better outputs for finer gradients. This costs performance of course.

As expected, 10 and 12 bit runtimes are relatively close in terms of speed with 8 bit being quite far ahead.

Now the most surprising thing is the nose dive we take from 1.9+141 to 1.9+200. I have no idea what happened here?! Did they fix a major bug? Or did some performance-critical options change in the --preset veryslow that I’m using? I have no idea. But to put is in numbers easier to understand: For 8 bit, my test encode went from a runtime of 30:39.312 (1.9+141) to 41:20.296 (1.9+200)! Ah, the time format being MM:SS.sss, M = minutes, S = seconds and s = milliseconds. That’s almost -35%! It’s less pronounced for the higher bit depths with -25% for 10 bits and -24% for 12 bits, but that’s still significant. Maybe I shouldn’t have deleted the output data, might have been useful to look at the H.265 stream headers.

Well, from there on out we continue to see a gradual performance increase again, in a steady and stable fashion. But what happened between 1.9+141 and 1.9+200? I don’t know. Something major must have changed, I just don’t know what it was exactly…

Maybe somebody can enlighten me a bit[1], so here are the options used, right from the benchmarking script:

expand/collapse source code
  1. @ECHO OFF
  3. :: ###################################################################################
  4. :: #                                                                                 #
  5. :: #  Please note, that command lines passed to timethis.exe may not be longer than  #
  6. :: #  1024 characters including expanded variables, so we need to keep it short      #
  7. :: #  enough. Some of the options passed to x265 may already be superflous due to    #
  8. :: #  being included in the veryslow preset, but I did not really double-check that. #
  9. :: #  Instead, I kept it just short enough by using shorter file names instead.      #
  10. :: #                                                                                 #
  11. :: #  Where to get the files:                                                        #
  12. :: #    * x265.exe     :               #
  13. :: #    * avconv.exe   :               #
  14. :: #    * colorecho.exe:                              #
  15. :: #    * timethis.exe :               #
  16. :: #                                                                                 #
  17. :: #  GPL v3 license for this scripts code, colorecho.exe, parts of avconv.exe and   #
  18. :: #  the CygWin companion libraries for avconv.exe:                                 #
  19. :: #    *                               #
  20. :: #                                                                                 #
  21. :: #  GPL v2 license for x265.exe:                                                   #
  22. :: #    *                     #
  23. :: #                                                                                 #
  24. :: #  LGPL v3 license for other parts of avconv.exe:                                 #
  25. :: #    *                                 #
  26. :: #                                                                                 #
  27. :: #  NT Resource Kit license for timethis.exe:                                      #
  28. :: #    *    #
  29. :: #                                                                                 #
  30. :: #  The x265 encoder  :                                            #
  31. :: #  The libav         :                                          #
  32. :: #  The Cygwin project:                                     #
  33. :: #                                                                                 #
  34. :: ###################################################################################
  36. :: First round, 8 bits per channel color depth (MAIN profile), 8 bit arithmetic:
  37. FOR %%I IN (1.7-8b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  38.  ""Testing v%%I, 8 bits..."" 10 & .\timethis.exe ""echo x265 v%%I-8b & .\avconv.exe -r ^
  39.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  40.  .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  41.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  42.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  43.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  44.  --slow-firstpass --stats %%I-8b.stats --sar 1 --range full -o %%I-8b-p1.h265 2>NUL & ^
  45.  .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  46.  24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow ^
  47.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  48.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  49.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
  50.  --pass 2 --stats %%I-8b.stats --sar 1 --range full -o %%I-8b-p2.h265 2>NUL"" 1> ^
  51.  .\results-%%I-8b.txt 2>.\timethis-errorlog-%%I-8b.txt
  53. :: Second round, 10 bits per channel color depth (MAIN10 profile), 16 bit arithmetic:
  54. FOR %%J IN (1.7-10b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  55.  ""Testing v%%J, 10 bits..."" 10 & .\timethis.exe ""echo x265 v%%J-10b & .\avconv.exe -r ^
  56.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  57.  .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  58.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  59.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  60.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  61.  --slow-firstpass --stats %%J-10b.stats --sar 1 --range full -o %%J-10b-p1.h265 2>NUL ^
  62.  & .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  63.  24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow ^
  64.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  65.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  66.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass ^
  67.  2 --stats %%J-10b.stats --sar 1 --range full -o %%J-10b-p2.h265 2>NUL"" 1> ^
  68.  .\results-%%J-10b.txt 2>.\timethis-errorlog-%%J-10b.txt
  70. :: Second round, 12 bits per channel color depth (MAIN12 profile), 16 bit arithmetic:
  71. FOR %%K IN (1.7-12b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230) DO .\colorecho.exe ^
  72.  ""Testing v%%K, 12 bits"" 10 & .\timethis.exe ""echo x265 v%%K-12b & .\avconv.exe -r ^
  73.  24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL | ^
  74.  .\x%%K.exe - --y4m -D 12 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes ^
  75.  16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 ^
  76.  --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
  77.  --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 ^
  78.  --slow-firstpass --stats %%K-12b.stats --sar 1 --range full -o %%K-12b-p1.h265 2>NUL ^
  79.  & .\avconv.exe -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r ^
  80.  24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 -p veryslow ^
  81.  --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2000 --rect --amp --aq-mode 3 ^
  82.  --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 ^
  83.  --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass ^
  84.  2 --stats %%K-12b.stats --sar 1 --range full -o %%K-12b-p2.h265 2>NUL"" 1> ^
  85.  .\results-%%K-12b.txt 2>.\timethis-errorlog-%%K-12b.txt
  87. .\colorecho.exe "Benchmarks completed, cleaning up..." 10
  89. :: Removing output and statistics files:
  90. del /Q *p*.h265 *stats*
  92. .\colorecho.exe "All done, results are to be found in the results-*.txt files!" 10

So if I missed some critical changes that happened in between 1.9+141 and 1.9+200, please let me know! Oh, and here are the exact system specifications, in case it matters (it probably doesn’t):

  • Intel Xeon X5690 3.33GHz (SSE4.2 is max, no AVX), running at an all-core turbo of 3.6GHz
  • 24GB DDR-III/1066 10-10-10 2T
  • X58 chipset
  • Windows XP Professional x64 Edition SP2 with all Server 2003 R2 x64 updates

I guess I’ll keep doing the performance evaluations from here on out just to see how the encoder evolves over time, performance-wise… And maybe I’ll redo 1.9+141 and one of the newer versions and parse the stream headers to see if the effective encoding options differ anywhere after all. If yes, I’ll update this post!

Update 2017-01-27:

Three new x265 versions have been added to the performance trend analysis, 2.0+54, 2.1+60 and 2.2+22. Much to my dismay, there haven’t been any big developments when it comes to x265s’ performance on x86 machines over the last 6-8 months. Since 1.9+230 it’s pretty much linear, with only minimal variance.

So there have been changes and new options and all that, but it seems that the basic speed of the encoder in the --preset veryslow, implying --no-rskip has stayed the same. Maybe a Google Summer of Code performance challenge would do something for x265? As far as I can remember this worked miracles for x264 in the past as well, if my memory serves me right. Let’s see if we’ll get any significant improvements in the future…

[1] Thanks to “Particular” this [has been clarified] in the comments.

Jun 032016

H.265/HEVC logoAnd here’s another x265 build for Windows XP and Windows XP x64, following [1.9+141]. As usual, these work on modern versions of Windows just as well. Again, built with Microsoft Visual Studio 2010 SP1 and tested for correct encodes for 8-bit, 10-bit and 12-bit color depth. The 8-bit test has been done using the x86_32 version, the 10- & 12-bit tests has been done with the x86_64 version. I’m not running complicated test suits on this, just a simple encode with manual output checking.

Here is the software for 32-bit and 64-bit systems:

As usual, the builds depend on the Microsoft Visual C++ 2010 runtime which you can download from Microsoft [for 32-bit systems] and [for 64-bit systems] if you don’t have it already.

This time around, it’s a pure binary release, giving you the x265.exe and libx265.dll. I think I’m gonna keep it that way. It’s meant for users, not developers anyway.

I’m thinking I might create a project page for this, so that all releases get consolidated on a single spot, that’d probably better than creating a new post for each and every build I’m pushing out. If I’m gonna do that, links to it will be added to each post regarding information about how to build x265 for WinXP+, and also to all binary release posts.

Such a page could also give you an avconv release on the spot, so you can work with all kinds of video input to your liking, given that x265 can only accept raw YUV video by itself. Just need to build a 32-bit version of libav as well then.

Oh well, have fun! :)

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.