Mar 142018
H.265/HEVC logo

1.) Introduction

This post is a followup to the original you can find [here]. My reason for writing a new one instead of just editing the existing one is that the new results have been measured with a slightly different parametrization and under slightly different circumstances, so direct comparability has been thrown out the window. Anyway, like in the former article, multiple x265 video encoder versions starting with 1.7+512 are going to be compared to each other to see how the performance of x265 has evolved over time.

Originally, I wanted to test this only on a machine very similar to my original testbed, maxing out at 6 cores and SSE4.2 instruction set extensions. Since it is said that x265 features significant optimizations for newer AVX and AVX2 instructions however, I decided to include a more modern machine as well, to determine how the scaling differs on both an older and a newer machine.

2.) Benchmarks

First of all, the video used for this is not very “high definition”. Well, maybe by definition as it is 720p, but still. This also puts limits on scalability across cores/threads, which shall provide some additional insights later. Let’s start.

2a.) The older machine on MS Windows

The older one goes first: Here we have an Intel Core i7 980X running CentOS 6.9 Linux. Since I already had all those [Win64 builds] lying around, I ran the test from within a VirtualBox running Windows XP Professional x64 Edition SP2. The specs:

  • CPU: Intel Core i7 980X 3.33GHz (6 cores, 12 threads), last instruction set extension: SSE4.2
  • OS: Windows XP Professional x64 Edition SP2 on VirtualBox 5.1.2 on CentOS Linux 6.9

Here we go:

x265 performance trend on the older machine

x265 performance trend on the older machine (click to enlarge)

While the developments here aren’t exactly identical to what has been posted in the former article, the trend is clearly the same. The new part starts with version 2.3+2, after which we can see a long-needed performance improvement, maybe due to improvements to the assembly code. Also, from here on out we can observe a slight upwards trend for all color channel depths. The basic drop from 8-bit to 10-bit and then 12-bit color stays rather similar though, courtesy of the more expensive 16-bit arithmetic used for the higher color depths (The 8-bit version of x265 also uses 8-bit arithmetic where applicable, reducing effective precision).

There is nothing groundbreaking to be seen like with the introduction of --no-rskip somewhere in between 1.9+141 and 1.9+200, but whatever happened between 2.3+2 and 2.4+2 isn’t too shabby either.

Now let’s see how the modern box with AVX, AVX2, BMI2 and FMA3 instructions fares here!

2b.) The newer machine on CentOS Linux

Alright, specs first:

  • CPUs: 2 × Intel Xeon E5-2620 v4 (2 × 8 cores, 16 Threads for a 16C/32T total), newest instruction set extensions: AVX2, BMI2, FMA3
  • OS: CentOS 7.3.1611 with Linux kernel 3.10.0-514.16.1

This is a modern 2017 Broadwell-EP server featuring a much higher IPC when compared to the older Westmere/Gulftown architecture from 2010. My estimate would be around +40% of raw, additional instructions per clock here. On top of that, we’ll get a peek at its AVX/AVX2 etc. scaling:

x265 performance trend on the modern machine

x265 performance trend on the modern machine

Now, before I comment on the trend to be seen in that chart, let me say one thing: Surprisingly, unlike what x264 has shown over the years, x265 didn’t seem to want to scale more across cores/threads with progressing development. So the core count was not really having much influence at all; Pass 1 of the encode would use 5-6 cores, where pass 2 would eat up 6-7, all the way from the oldest to the newest versions.

What we do see here aside from the --no-rskip drop and the performance increase from 2.3+2 to 2.4+2 are three things: First, we can see a new upwards trend in performance starting with 2.5+9. The development has been more linear for the older CPU, which is why I’m assuming that we get to see some work invested in AVX(2) code here, probably also in BMI/FMA code.

The second thing is the more significant decline starting after 2.0+11. I have no explanation for this, as new features added around here shouldn’t have more negative impacts on newer CPUs as when to compared to older ones. If anything, it should be the other way around, so that part going on up to 2.1+60 is a bit confusing.

And the third thing that is very clear here: The 12-bit version is much closer to 10-bit performance. Only thing I can think of here is that the AVX/AVX2 code paths have generally always been faster on the 12-bit version when compared to the older ≤SSE4.2 assembly code.

Seeing this, I’d like to look at one more thing: The dropoff we can see from 8-bit -> 10-bit -> 12-bit compared between the two architectures. Let’s do that:

3.) 8-/10-/12-bit scaling across platforms

Let’s compare this side-by-side across all versions tested, in a very simple chart:

Well, the modern Xeons do lose less performance even from 8- to 10-bit, but as seen above, the drop to 12-bit is surprisingly insignificant on modern CPUs. This is a bit unfortunate actually, given that all the rage is about 10-bit with the Anime communities and also the official Blu-Ray UHD/4K standards, but oh well. :roll:

4.) Summing it up

Anyway, this hasn’t been too interesting actually, aside from the 12-bit surprise that I guess nobody cares about anyway. Well, we’re seeing some more significant performance increases for recent versions of x265 on modern architectures, but those are only compensating for the loss of performance we’ve seen before, that didn’t occur on older machines.

Maybe the AVX+AVX2 code was good from the very start instead of spreading its wings only now? Anyway, we’re seeing some slow but steady increase in performance for now even on older machines. If there’s a reason to buy new processors for x265 it’s probably not the AVX/AVX2/FMA3/BMI2/AVX512/whatever, but rather the generally higher IPC of newer chips…

5.) How the benchmark was done

5a.) On Windows

Given a folder with x265 versions named x1.7+512.exe, x1.6+2.exe etc., as well as an ffmpeg.exe (like from [here]), the subfolders log\, output\, stats\ and results\, you can adapt and use the following script after putting it right next to the x265 binaries, I call it test-performance-trends.bat:

expand/collapse source code
  1. @ECHO OFF
  3. FOR %%I IN (1.7-8b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  4.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%I, 8 ^
  5.  bits... & .\timethis.exe "echo x265 version %%I 8 bit results: & .\ffmpeg.exe ^
  6.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  7.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p ^
  8.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%I-8b.stats ^
  9.  -o .\output\%%I-8b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  10.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  11.  -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow ^
  12.  --bitrate 2000 --pass 2 --stats .\stats\%%I-8b.stats -o .\output\%%I-8b-p2.h265 ^
  13.  2>NUL" 1> .\results\results-%%I-8b.txt 2>.\log\timethis-errorlog-%%I-8b.txt
  16. FOR %%J IN (1.7-10b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  17.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%J, 10 ^
  18.  bits... & .\timethis.exe "echo x265 version %%J 10 bit results: & .\ffmpeg.exe ^
  19.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  20.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p ^
  21.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%J-10b.stats ^
  22.  -o .\output\%%J-10b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  23.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  24.  -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow ^
  25.  --bitrate 2000 --pass 2 --stats .\stats\%%J-10b.stats -o ^
  26.  .\output\%%J-10b-p2.h265 2>NUL" 1> .\results\results-%%J-10b.txt ^
  27.  2>.\log\timethis-errorlog-%%J-10b.txt
  29. FOR %%K IN (1.7-12b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  30.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%K, 12 ^
  31.  bits... & .\timethis.exe "echo x265 version %%K 12 bit results: & .\ffmpeg.exe ^
  32.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  33.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  34.  -p veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats ^
  35.  .\stats\%%K-12b.stats -o .\output\%%K-12b-p1.h265 2>NUL & .\ffmpeg.exe -r ^
  36.  24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict ^
  37.  -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  38.  -p veryslow --bitrate 2000 --pass 2 --stats .\stats\%%K-12b.stats -o ^
  39.  .\output\%%K-12b-p2.h265 2>NUL" 1> .\results\results-%%K-12b.txt ^
  40.  2>.\log\timethis-errorlog-%%K-12b.txt
  42. ECHO All done, results are to be found in the results\results-*.txt files!

5b.) On Linux

Given a system-wide installation of ffmpeg and a folder with statically linked x265 binaries named x1.7+512, x1.6+2 etc., as well as the subfolder results/, the following script – – should do the job after being adapted for your list of x265 binaries and put right next to them:

expand/collapse source code
  1. #!/usr/bin/env sh
  3. for i in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  4. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$i, \
  5. 8 bits...\n" && time ( printf "x265 version $i 8 bit results:\n" \
  6. 1>./results/results-$i-8b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  7. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  8. ./x$i - --y4m -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  9. --slow-firstpass --stats ./stats/$i-8b.stats -o ./output/$i-8b-p1.h265 \
  10. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  11. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$i - --y4m \
  12. -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  13. ./stats/$i-8b.stats -o ./output/$i-8b-p2.h265 2>/dev/null ) \
  14. 2>>./results/results-$i-8b.txt; done
  16. for j in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  17. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$j, \
  18. 10 bits...\n\n" && time ( printf "x265 version $j 10 bit results:\n" \
  19. 1>./results/results-$j-10b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  20. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  21. ./x$j - --y4m -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  22. --slow-firstpass --stats ./stats/$j-10b.stats -o ./output/$j-10b-p1.h265 \
  23. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  24. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$j - --y4m \
  25. -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  26. ./stats/$j-10b.stats -o ./output/$j-10b-p2.h265 2>/dev/null ) \
  27. 2>>./results/results-$j-10b.txt; done
  29. for k in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  30. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$k, \
  31. 12 bits...\n\n" && time ( printf "x265 version $k 12 bit results:\n" \
  32. 1>./results/results-$k-12b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  33. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  34. ./x$k - --y4m -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  35. --slow-firstpass --stats ./stats/$k-12b.stats -o ./output/$k-12b-p1.h265 \
  36. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  37. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$k - --y4m \
  38. -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  39. ./stats/$k-12b.stats -o ./output/$k-12b-p2.h265 2>/dev/null ) \
  40. 2>>./results/results-$k-12b.txt; done
  42. printf "\nBenchmarks completed, results in `pwd`/results/results-*.txt!\n"

That’s it!

Jan 112018
AMI logo

1.) Introduction

This is something I’ve been wanting do do years ago, but the recent outcry regarding the Meltdown and Spectre hardware security holes have reminded me of this. Since fixing at least parts of the exploits also requires CPU microcode support (from here on: “µcode”), and not all operating systems and mainboards/machines might get them, this might be useful for the current situation as well. It can also be helpful if you wish to patch Intel Xeon support into a slightly older platform, where the manufacturer hasn’t done so already. This article does however not apply to modern UEFI-based systems.

So, instead of getting the required CPU µcodes from a donor BIOS, why not obtain the latest version from Intels’ [µcode package for Linux]? Oh, and it’s not really Linux-specific anyway.

I will show you how to extract and prepare the Intel µcodes and how to patch them so you can embed them into pretty much any somewhat modern AMI BIOS. AMI BIOSes are pretty widespread, but tools similar to the AMI-specific ones shown here might of course also exist for other BIOS brands.

The steps of this guide will be shown for both Linux/UNIX as well as Microsoft Windows.

All the tools used in this post can be found at the end of the article. You can also get all the currently released µcodes in one package there, so you don’t need to download all the archives from Intel (not all archives contain all µcodes…).

2.) Fetch and extract/convert Intels’ µcode package

You can get the µcodes directly from Intel, again, here’s the [link]. To unpack the .tgz archive on MS Windows, you may want to use [7-zip] (It’s a 2-step process with 7-zip, from .tgz to .tar and from .tar to the final files). For older releases, all you’ll get by doing this is a microcode.dat file, sometimes the file name will also contain a date. That’s all the µcodes assembled in one file in a text format. This is useless for patching the data into BIOS images, so we’ll need to extract the individual CPU µcodes and convert them into the proper binary format.

Newer releases might contain pre-built binary versions as well, but we’ll just continue to work with microcode.dat.

2a.) On Linux or UNIX

We’ll use iucode_tool for that, as at least most Linux systems should have a package ready for that program. If you can by no means get or compile iucode_tool, you might compile microdecode from source on Linux or UNIX instead, see 2b.) and 8.)! I have tested this on CentOS 6.9 Linux, and microdecode compiles just fine. But let’s continue with iucode_tool:

To extract the µcodes, switch to the directory containing microcode.dat (after unpacking the .tgz file with tar -xzf) and run the following commands:

$ mkdir ./µcodes/
$ iucode_tool -t d -L --write-named-to="./µcodes/" "microcode.dat"; done

With that, you’ll get a lot of *.fw binary files in ./µcodes/. Those are in the regular Intel µcode format and come in different sizes depending on the CPU. The file names look similar to e.g. this: s000206C2_m00000003_r00000013.fw. The first “s” part contains the CPUID, the second “m” part shows the platform ID, and the “r” part denotes the µcode revision, all of it in hexadecimal notation.

2b.) On Windows

On Windows, we’ll use the microdecode command line tool. It’s a bit easier to use than iucode_tool, as it’s only purpose is really just µcode extraction and nothing else. Switch to the directory containing the unpacked microcode.dat, make sure microdecode.exe is in your search path (or in the same directory), and run the following command:

microdecode.exe microcodes.dat

This will result in a lot of *.bin files that are identical to the .fw files the *nix tool extracts. The file names will be similar to this: cpu000206c2_plat00000003_ver00000013_date20100907.bin. The first “cpu” part contains the CPUID like above, the second “plat” part contains the platform ID, and the “ver” part shows the µcode revision. Additionally, we also get the release date in the final “date” part.

3.) Identifying your CPU

You’ll likely know your CPU model if you’re reading this. But what you’ll need to know above that is your exact CPUID, stepping and the revision of CPU µcode you’re currently running. You can look up some of the information on cpu-worlds’ [CPUID database], but ultimately, you’ll need to fetch it from your local machine anyway.

Important background information: The CPUID is a 32-bit value structured as follows:

CPUID structure

The binary structure of Intels’ 32-bit CPUID, 4 bits equal one hexadecimal character

The binary matrix is usually expressed as 4-bit hexadecimal characters. e.g. the binary CPUID 0000:0000:0000:0010:0000:0110:1100:0010bin would result in 000206C2hex.

3a.) On Linux/UNIX

On Linux, the required information can be grabbed from procfs. Like this, for example:

$ cat /proc/cpuinfo | grep -e "model" -e "stepping" -e "microcode" -e "cpu family" | sort | uniq

cpu family	: 6
microcode	: 15
model		: 44
model name	: Intel(R) Core(TM) i7 CPU       X 980  @ 3.33GHz
stepping	: 2

You have to be careful interpreting this information however, as it’s mostly decoded in decimal form. Modern kernels might show some of the data in hexadecimal, so be careful here!

But first, convert the decimals from /proc/cpuinfo to hexadecimal, where necessary, in this case we’ll do it for the µcode revision and the model number:

$ printf '%X\n' 15
$ printf'%X\n' 44

So, the model number is 2Chex. Here, the right part or Chex is the actual model number (CPUID bits 4..7) and 2hex is the extended family number (CPUID bits 16..19).

The µcode version here would be Fhex. Note that µcode versions can be wider than just 1 hexadecimal character as well. The stepping 2 is a number lower than 15, so we don’t need to convert it to hexadecimal.

To assemble our final string which is 8 hexadecimal characters wide, we walk through the bit mask on the image shown above step by step! All “reserved” parts are just filled with blanks:

  1. 0 (Bits 31..28, reserved)
  2. 0 (Bits 27..24, extended CPU family, zero for family 6)
  3. 0 (Bits 23..20, extended CPU family, zero for family 6)
  4. 2 (Bits 19..16, extended CPU model number)
  5. 0 (Bits 15..12, two reserved and two platform type bits, typically zero)
  6. 6 (Bits 11..8, CPU family code)
  7. C (Bits 7..4, CPU model number)
  8. 2 (Bits 3..0, CPU stepping ID)

As a result of this, our CPUID string looks as follows:

  • 000206C2hex, currently running µcode revision Fhex.

Please note that information down, it’s needed in the next step.

Of course, if you’re running some other UNIX, you’d be needing different tools. On FreeBSD you may be able to work with the “cpuflags” tool, like this:

$ cpuflags x 2>& | grep "Origin"

For other UNICES, you’re on your own though.

3b.) On Windows

On Windows I’d recommend Ray Hinchcliffes’ [System Information Viewer], or “SIV” in short. It’s a pretty powerful freeware system information tool that will read out the stuff we need for the next steps (You can also use something like CPU-Z of course). See the following screenshot to learn what you need to look for:

System Information Viewer (SIV)

System Information Viewer (click to enlarge)

From left to right, we need the following data as pointed out with those arrows: CPU family 6hex, CPU model 2Chex (you can ignore the decimal value “44”), stepping 2hex and µcode revision 13hex. Again, 2Chex is a combination of model number Chex and extended model number 2hex!

Just like on Linux, the final identification string needs to be assembled first. Luckily, SIV already does the decimal -> hexadecimal conversion for us. To assemble the entire CPUID, put everything together like this: 

  1. 0 (Bits 31..28, reserved)
  2. 0 (Bits 27..24, extended CPU family, zero for family 6)
  3. 0 (Bits 23..20, extended CPU family, zero for family 6)
  4. 2 (Bits 19..16, extended CPU model number)
  5. 0 (Bits 15..12, two reserved and two platform type bits, usually zero)
  6. 6 (Bits 11..8, CPU family code)
  7. C (Bits 7..4, CPU model number)
  8. 2 (Bits 3..0, CPU stepping ID)

As a result of this, our CPUID string looks as follows:

  • 000206C2hex, currently running µcode revision 13hex.

The revision is the µcode version you’re currently running. Newer version will just have higher numbers, so that’s how you can compare them with what you extracted from Intels’ package.

Note that information down, as you’ll need it in the next step.

4.) Locating the correct µcodes

4a.) On Linux/Unix

Let’s continue to assume your CPUID string is 000206C2hex. Switch to the directory where you extracted the Intel µcode package for Linux and look for the correct µcode binaries (in my case I have them grouped in folders by date as well):

$ find . -iname "*000206C2*"

Two versions are available: The newer 13hex and the older Fhex (in decimal that would amount to 19 vs. 15, higher is always newer).

4b.) On Windows

Like on Linux, let’s assume your CPUID string is 000206C2hex. Switch to the directory where your extracted Intel µcodes are, and look for the corresponding files:

DIR /B /S "*000206C2*"

There you go, the newer 13hex and the older Fhex µcodes.

5.) What’s wrong with AMI MMTool: A detailed explanation

To embed the µcodes into an actual AMI BIOS image, AMIs’ MMTool “CPU Patch” function is required. It’s a Windows program, but given its simple nature, it will run fine with Wine on Linux and FreeBSD UNIX as well:


 But if you attempt to embed Intels’ original binaries, it’ll fail right away:

MMTool failing to import Intel µcodes directly

MMTool failing to import Intel µcodes directly

If you extract existing µcodes from a given BIOS image and compare them with the same µcode revision in Intels’ format, you’ll notice a size difference:

$ ls -al ./06C2-Rev13.bin 20100914/µcodes/s000206C2_m00000003_r00000013.fw
-rw-rw-r-- 1 thrawn users 8192 Jan 10 14:04 ./06C2-Rev13.bin
-rw-r--r-- 1 thrawn users 7168 Jan 10 14:26 20100914/out/s000206C2_m00000003_r00000013.fw

Intels’ µcode file in this case is 7168 bytes or 7kiB in size, but the stuff we extracted from that AMI BIOS is 8192 bytes or 8kiB in size. So what’s the deal here? At first I had no idea at all and thought that they’d just be encoded differently in the AMI BIOS or something. But it turns out that that is not the case. If you open both files in a hex editor of your choice, you’ll notice something at the end of the file that came from the AMI BIOS:


The final 1kiB is just binary zeroes! So how do we fix that? In essence, the entire procedure is like this:

  1. Extract µcodes from the target AMI BIOS image using MMTool.
  2. Look at how large that file is (in bytes).
  3. Look at how large the desired Intel µcode file “for Linux” is for your exact CPU.
  4. Calculate the difference in bytes.
  5. Fill the end of the Intel file with as many binary zeroes as there are missing bytes.
  6. Save that file and try to load it into your BIOS image using MMTools’ “CPU Patch” function.

Alright, let’s do it!

6.) Patching those µcodes for AMI MMTool and patching the BIOS itself

First, extract some random µcode from your BIOS using AMI MMTools’ “CPU Patch” function. Look at the file and note down its’ size in bytes. Also, you may want to look at the extracted µcode to see whether it has a lot of zeroes at the end. In my case, as said, the µcodes in that AMI BIOS are 8192 bytes, or 8kiB large, while Intel’s µcode is 7kiB in size. 8192 – 7168 = 1024, so again, the difference is 1024 bytes or 1kiB.

Please triple-check this stuff, as a wrong padding will break the µcode! Of course MMTool should warn you if something’s wrong, but it doesn’t hurt to make sure it’s safe from the start.

6a.) On Linux/UNIX

Let’s make this easy, generate a 1kiB zero padding file for future use on similar µcode files:

$ dd bs=1 count=1024 if=/dev/zero of=./1kiB-zeropadding.bin
1024+0 records in
1024+0 records out
1024 bytes (1.0 kB) copied, 0.00383825 s, 267 kB/s

And now, append the zeroes to the end of the file, generating a new file in the process, so we don’t touch the originals:

$ cat ./20100914/µcodes/s000206C2_m00000003_r00000013.fw ./1kiB-zeropadding.bin > ./0206C2-rev13.bin

That file 0206C2-rev13.bin is now the final µcode file for patching into the target BIOS. As an example, I’ll patch those µcodes for Xeon X5600 processors into an ancient BIOS version 0402 of an ASUS P6T Deluxe mainboard:


Save that file, and you’re done modifying your BIOS with µcodes from Intels’ Linux package!

6b.) On Windows

On MS Windows there is no dd command by default, but we have a another preinstalled utility for creating our 1kiB zero padding file: fsutil! Like this:

fsutil File CreateNew .\1kiB-zeropadding.bin 1024

We don’t have a cat command on Windows either, but we can use a binary copy for file concatenation, no need to install any additional tools. We’ll attach the zero padding to the end of the Intel µcode and write the result to a new file, so we’re not altering the originals:

COPY /B .\20100914\cpu000206c2_plat00000003_ver00000013_date20100907.bin + .\1kiB-zeropadding.bin .\0206C2-rev13.bin
        1 file(s) copied.

That file 0206C2-rev13.bin is now the final µcode file for patching into the target BIOS. As an example, I’ll patch those µcodes for Xeon X5600 processors into an ancient BIOS version 0402 of an ASUS P6T Deluxe mainboard:


Save that file, and you’re done modifying your BIOS with µcodes from Intels’ Linux package!

7.) Flashing the modified BIOS

This step depends on your system. You may be able to flash from within the BIOS itself (e.g. from floppy disk or USB pen drive), or from within MS Windows or by using some bootable medium.

In any case, your BIOS image is ready now, and can be flashed at your convenience. After the update, re-check your µcode revision as shown in 3.). Your CPU should now be running the updated µcode!

8.) Downloads

Some of the programs used in this article can be downloaded here:

  • [AMI MMTool v3.22]
  • [microdecode], compiled by myself, contains 32-bit and 64-bit versions of the tool as well as the source code and license file. Compiles on Linux/UNIX as well.
  • [Intel µcodes], everything from 2009-03-30 up to 2018-01-08 as available on the Intel web site at the moment.

9.) Hopefully you haven’t bricked your mainboard

…all of this without any kind of guarantees of course! :roll:

May 312017
HakuNeko logo

1.) What for?

Usually, porting my favorite manga ripper [HakuNeko][1] would involve slightly more exotic target platforms like [FreeBSD UNIX]. This changed with version 1.4.2 however, as this version – the most current at the time of writing – would no longer compile on Windows machines due to some issues with its build toolchain. And that’s the most common platform for the tool!

This is what the lead developer had to say about the issue while even suggesting the use of [FMD] instead of HakuNeko on Windows:

“The latest release does not compile under windows due to some header include contradictions of sockets […]”
    -in a [comment][1] to HakuNeko ticket #142 by [Ronny Wegener], HakuNeko project leader

[1] Edit: Links have been fixed, as the HakuNeko project has now been moved to HakuNeko Legacy due to the development of its replacement, [HakuNeko S].

Normally I wouldn’t mind that much and keep using 1.4.1 for now, but unfortunately this is not an option. Quite a few Manga websites have changed by now, breaking compatibility with the previous version. As this is breaking most of HakuNekos’ functionality for some important sites, it became quite unusable on Windows, leaving Linux as the only officially supported platform.

As using virtual machines or remote VNC/X11 servers for HakuNeko proved to be too tedious for me, I thought I’d try to build this by myself. As the MSYS2/MinGW Windows toolchain seemed to be broken for 1.4.2, I tried – for the very first time – to cross-compile on Linux, choosing CentOS 7.3 x86_64 and MinGW32 4.9.3 for the task. This was quite the challenge, given that HakuNeko comes completely unprepared for cross-compiling.

2.) First, the files

Took me many hours / days to get it done – 100% statically linked too – and it’s finished now! What I won’t provide is an installer (don’t care about that), but here are my v1.4.2 builds for Windows:

As of today, those versions have been tested successfully on the following operating systems:

  • Windows XP Professional SP3 / POSReady2009
  • Windows XP Professional x64 Edition SP2 w. Server 2003 updates
  • Windows Server 2003 R2 x64 SP2
  • Windows Vista Enterprise x64 SP2
  • Windows 7 Professional x64 SP1
  • Windows 10 Professional x64 build #1607

Please be aware that not all of the functionality has been tested by me, just a few downloads that wouldn’t have worked with 1.4.1, a few more random downloads here and there, plus chapter-wise .cbz packaging. It’s looking pretty good I think. Here are some sample screen shots as “proof” of it working (click to enlarge):

HakuNeko 1.4.2 downloading "Kobayashi-san Chi no Maid Dragon" on XP x64

HakuNeko 1.4.2 downloading “Kobayashi-san Chi no Maid Dragon” on XP x64 (Note: I own that Manga in paper form)


3.) What has been done to make this work?

3a.) Initial work:

First of all, cross-compiling is a bottomless, hellish pit, a horrible place that you never want to enter unless a.) The build toolchain of the software you wanna compile is very well prepared for it or b.) you really have/want to get your hands on that build or c.) you hate yourself so much you have to hurt yourself or d.) you actually enjoy c.).

The reasons for choosing cross-compiling were that Ronny Wegener had said, that the MSYS2/MinGW32 build would fail on Windows, plus it would require GCC version 5.3 to link with the bundled, pre-built static libraries (OpenSSL, cURL, wxWidgets).

So I thought it would be more likely to work if I were to run my own MinGW setup on Linux, not relying on the bundled stuff but linking against the libraries that come with MinGW32 4.9.3 on my platform of choice – CentOS 7.3 Linux.

One exception was the GUI library wxWidgets 3.0.2 that I had to cross-compile and statically link by myself as well, but luckily, that’s easy despite its size. wxWidgets is one piece of software that does come well-prepared for cross-compiling! In my case, that made it as simple as this (parallel compile with 6 CPUs):

$ ./configure --prefix=/usr/local/i686-w64-mingw32 --host=i686-w64-mingw32 --build=x86_64-linux \
 --enable-unicode --with-expat --with-regex --with-opengl --with-libpng --with-libjpeg --with-libtiff \
 --with-zlib --with-msw --enable-ole --enable-uxtheme --disable-shared
$ make -j6
# make install

3b.) HakuNeko build toolchain / Makefile modifications for cross-compiling:

HakuNeko is much harder, and I don’t even remember half of what I did, but most of it was manually editing the ./Makefile after $ ./configure --config-mingw32 would have produced something rather broken.

Let’s get to it, file paths are relative to the source root. First, edit the following parts of the ./Makefile (you need to look for them in different places of the file). First, the PREFIX, should be in the bottom half of the file:

PREFIX = /usr/local/i686-w64-mingw32/

CC and the CFLAGS:

CC = i686-w64-mingw32-g++
CFLAGS = -c -Wall -O2 -std=c++11 \
 -I/usr/local/i686-w64-mingw32/lib/wx/include/i686-w64-mingw32-msw-unicode-static-3.0 \
 -I/usr/local/i686-w64-mingw32/include/wx-3.0 -D_FILE_OFFSET_BITS=64 -D__WXMSW__ -mthreads \
 -DCURL_STATICLIB -I/usr/i686-w64-mingw32/sys-root/mingw/include

Add -DPORTABLE, if you want to build the portable version of HakuNeko.

Then, the Windows resource compiler, controlled by RC and RCFLAGS:

RC = /usr/bin/i686-w64-mingw32-windres
RCFLAGS = -J rc -O coff -F pe-i386 -I/usr/i686-w64-mingw32/sys-root/mingw/include \

And finally, the static linking part, which is the hardest stuff to get done right, LD, LDFLAGS and LDLIBS:

LD = i686-w64-mingw32-g++
LDFLAGS = -s -static -static-libgcc -static-libstdc++ -mwindows -DCURL_STATICLIB
LDLIBS = -L/usr/local/i686-w64-mingw32/lib   -Wl,--subsystem,windows -mwindows \
 -lwx_mswu_xrc-3.0-i686-w64-mingw32 -lwx_mswu_webview-3.0-i686-w64-mingw32 \
 -lwx_mswu_qa-3.0-i686-w64-mingw32 -lwx_baseu_net-3.0-i686-w64-mingw32 \
 -lwx_mswu_html-3.0-i686-w64-mingw32 -lwx_mswu_adv-3.0-i686-w64-mingw32 \
 -lwx_mswu_core-3.0-i686-w64-mingw32 -lwx_baseu_xml-3.0-i686-w64-mingw32 \
 -lwx_baseu-3.0-i686-w64-mingw32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lcurl -lidn -liconv \
 -lssh2 -lssl -lcrypto -lpng -ljpeg -ltiff -lexpat -lwxregexu-3.0-i686-w64-mingw32 -lz -lrpcrt4 \
 -lwldap32 -loleaut32 -lole32 -luuid -lws2_32 -lwinspool -lwinmm -lshell32 -lcomctl32 -lcomdlg32 \
 -ladvapi32 -lwsock32 -lgdi32

Took a while to find the libraries (and static library order!) necessary to satisfy all the dependencies properly.

If you need it, here is the modified Makefile I’ve used to cross-compile:

  • [HakuNeko Makefile] for cross-compiling HakuNeko 1.4.2 for Windows on CentOS 7.3 x86_64 Linux (needs statically linked & installed wxWidgets first).

3c.) Source code modifications:

However, something will still not be quite right, because some of the crypto libraries will provide the MD5 functions MD5_Init(), MD5_Update() as well as MD5_Final(), and those are already defined by HakuNeko itself. This will break the static linking, as redundant definitions won’t work. We’ll rely on the libraries (libcrypto, libssl), and comment the built-in stuff out in src/v7/v7.c:

void MD5_Init(MD5_CTX *c);
void MD5_Update(MD5_CTX *c, const unsigned char *data, size_t len);
void MD5_Final(unsigned char *md, MD5_CTX *c);


/* void MD5_Init(MD5_CTX *c);
 * void MD5_Update(MD5_CTX *c, const unsigned char *data, size_t len);
 * void MD5_Final(unsigned char *md, MD5_CTX *c);

On top of that, the configure system may have generated src/main.cpp as well as src/main.h. Those are bogus files, turning the entire tool into nothing but one large “Hello World” program. Plus, that’s hard to debug, as the binary won’t even output “Hello World” on a Windows terminal when it’s built as a GUI tool. I only found the issue when testing it with Wine on Linux. ;)

Please delete src/main.cpp and src/main.h before continuing.

Now, if you’re really lucky, you should be able to run something like $ make -j6 and watch everything work out nicely. Or watch it crash and burn, which is the much, much more likely scenario, given I’ve likely only given you half of what I did to the build tools.

Well, in any case, no need to run $ make install of course, just grab the binary build/msw/bin/hakuneko.exe and copy it off to some Windows machine, then try to run it. If you’ve built the portable version, you may wish to rename the file to hakuneko-portable.exe, just like the official developers do.

4.) The future

Let’s just hope that the developers of HakuNeko can get this fixed for versions >=1.4.3, because I really, really don’t want to keep doing this. It’s extremely painful, as cross-compiling is exactly the kind of living hell I heard a lot of people saying it is! I think it’s a miracle I managed to compile and run it at all, and it was so frustrating and tedious for somebody like me (who isn’t a developer).

The statement that this took “hours / days” wasn’t an exaggeration. I think it was something like 10-12 man hours of pure frustration to get it done. I mean, it does feel pretty nice when it finally works, but I wouldn’t bet on myself being able to do this again for future versions…

So please, make it work on Windows again, if possible, and keep HakuNeko cross-platform! It’s still my favorite tool for the task! Thanks! :)

Mar 012017
Notepadqq @ CentOS 6 Linux logo

It’s rather rare for me to look for a replacement of some good Windows software for Linux/UNIX instead of the other way around, but the source code editor [Notepad++] is one example of such software. The program gedit on the Gnome 2 desktop environment of my old CentOS 6 enterprise Linux isn’t bad, but it isn’t exactly good either. The thing I was missing most was a search & replace engine capable of regular expressions.

Of course, vi can do it, but at times, vi can be a bit hard to use, so I kinda looked for a Notepad++ replacement. What I found was [Notepadqq], which is basically a clone using the Qt5 UI. However, this editor is clearly made for more modern systems, but I still looked for a way to get it to compile and run on my CentOS 6.8 x86_64 Linux system. And I found one. Here are the most important prerequisites:

  • A new enough GCC (I used 6.2.0), because the v4.4.7 platform compiler won’t work with the modern C++ stuff
  • Qt5 libraries from the [EPEL] repository
  • git

First, you’ll want a new compiler. That part is surprisingly easy, but a bit time consuming. First, download a fresh GCC tarball from a server on the [mirrors list], those are in the releases/ subdirectory, so a file like gcc-6.3.0.tar.bz2 (My version is still 6.2.0). It seems Notepadqq only needs some GCC 5, but since our platform compiler won’t cut it anyway, why not just use the latest?

Now, once more, this will take time, could well be hours, so you might wanna do the compilation step over night, the last step needs root privileges:

$ tar -xzvf ./gcc-6.3.0.tar.bz2
$ cd ./gcc-6.3.0/
$ ./configure --program-suffix="-6.3.0"
$ make
# make install

And when you do this, please never forget to add a --program-suffix for the configuration step!  You might seriously fuck things up if you miss that! So double-check it!

When that’s finally done, let’s handle Qt5 next. I’ll be using a binary distribution to make things easy, and yeah, I didn’t just install the necessary packages, I got the whole Qt5 blob instead, too lazy to do the cherry picking. Oh, and if you don’t have it, add git as well:

# yum install
# yum install qt5* git

I assume # yum install qt5-qtwebkit qt5-qtwebkit-devel qt5-qtsvg qt5-qtsvg-devel qt5-qttools qt5-qttools-devel should also be enough according to the requirements, but I didn’t try that. Now, enter a free directory or one you generally use for source code and fetch the latest Notepadqq version (this will create a subfolder we’ll cd to):

$ git clone
$ cd ./notepadqq

After that, we need to make sure that we’ll be using the correct compiler and that we’re linking against the correct libraries that came with it (like*). To do that, set the following environment variables, assuming you’re using the bash as your shell (use lib/ instead of lib64/ folders if you’re on 32-bit x86):

$ export CC="gcc-6.3.0"
$ export CXX="g++-6.3.0"
$ export CPP="cpp-6.3.0"
$ export CFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export CXXFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export LDFLAGS="-L/usr/local/lib64/"

The C-related settings are probably not necessary as Qt5 stuff should be pure C++, but you’ll never know, so let’s play it safe.

With that we’re including and linking against the correct libraries and we’ll be using our modern compiler as well. Time to actually compile Notepadqq. To do that, we’ll still need to tell it where to find the Qt5 versions of the qmake and lrelease binaries, but luckily, we can solve that with some simple configuration options. So, let’s do this, the last step requires root privileges again, from within the notepadqq/ directory that git clone created for us:

$ ./configure --qmake /usr/bin/qmake-qt5 --lrelease /usr/bin/lrelease-qt5
$ make
# make install

Now, there are some weird linking issues that I never got fixed on CentOS (some developer please tell me how, I have the same crap when building x265!). Because of that we still can’t launch Notepadqq as-is, we need to give it an LD_LIBRARY_PATH to find the proper libraries at runtime. Let’s just create an executable launcher script /usr/local/sbin/ for that. Edit it and enter the following code:

LD_LIBRARY_PATH="/usr/local/lib64" /usr/local/bin/notepadqq "$@"

Use this as your launcher script for Notepadqq and you’re good to go with your Notepad++ replacement on good old CentOS 6.x:

Running Notepadqq on CentOS 6 Linux

Running the latest Notepadqq on CentOS 6 Linux with Qt5 version 5.6.1, state 2017-03-01

Now, let’s see whether it’s even that good actually… :roll:

Nov 192016
FreeBSD GMABoost logo

Recently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # 200 or # 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Aug 032016
xmms logo

Yeah, I’m still using the good old xmms v1 on Linux and UNIX. Now, the boxes I used to play music on were all 32-bit x86 so far. Just some old hardware. Recently I tried to play some of my [AAC-LC] files on my 64-bit Linux workstation, and to my surprise, it failed to play the file, giving me the error Pulse coding not allowed in short blocks when started from the shell (just so I can read the error output).

I thought this had to have something to do with my file and not the player, so I tried .aac files with ADTS headers instead of audio packed into an mp4/m4a container. The result was the same. Also, the encoding (SBR/VBR or even HE-AAC, which sucks for music anyway) didn’t make a difference. Then I found [this post] on the Gentoo forums, showing that the problem was really the architecture. 32-bit builds wouldn’t fail, but the source code of the [libfaad2] decoding library of xmms’ mp4 plugin wasn’t ready for 64-bit.

xmms’ xmms_mp4Plugin-0.4 comes with a pretty old libfaad2, 2.0 or something. I will show you how to upgrade that to version 2.7, which is fixed for x86_64 (Readily fixed source code is also provided at the bottom). We’ll also fix up other parts of that plugin so it can compile using more modern versions of GCC and clang, and my test platform for this would be a rather conservative CentOS 6.8 Linux. First, get the source code:

I’m assuming you already have xmms installed, otherwise obtain version 1.2.11 from [here]!

Step 1, libfaad2:

Unpack both archives from above, then enter the mp4 plugins’ source directory. You’ll find a libfaad2/ directory in there. Delete or move it and all of its contents. From the faad2 source tree, copy the directory libfaad/ from there to libfaad2/ in the plugins’ directory, replacing the deleted one. Now that’s the source, but we also need the updated headers so that the xmms plugin can link against the newer libfaad2. To do that, copy the two files include/faad.h and include/neaacdec.h from the faad2 2.7 source tree to the directory include/ in the plugin source tree. Overwrite faad.h if prompted.

Step 2, libmp4v2:

Also, the bundled libmp4v2 of the mp4 plugin is very old and broken on modern systems due to code issues. Let’s fix them, so back to the mp4 plugin source tree. We need to fix some invalid pure specifiers in the following files: libmp4v2/mp4property.h, libmp4v2/mp4property.cpp, libmp4v2/rtphint.h and libmp4v2/rtphint.cpp.

To do so, open them in a text editor and search and replace all occurences of = NULL with = 0. This will fix all assignments and comparisons that would otherwise break. When using vi or vim, you can do :%s/=\sNULL/= 0/g for this.

On top of that, we need to fix a invalid const char* to char* conversion in libmp4v2/rtphint.cpp as well. Open it, and hop to line number 325, you’ll see this:

  1. char* pSlash = strchr(pRtpMap, '/');

Replace it with this:

  1. const char* pSlash = strchr(pRtpMap, '/');

And we’re done!

Now, in the mp4 plugins’ source root directory, run $ ./bootstrap && ./configure. Hopefully, no errors will occur. If all is ok, run $ make. Given you have xmms 1.2.11 installed, it should compile and link fine now. Last step: # make install.

This has been tested with my current GCC 4.4.7 platform compiler as well as GCC 4.9.3 on CentOS 6.8 Linux. Also, it has been tested with clang 3.4.1 on FreeBSD 10.3 UNIX, also x86_64. Please note that FreeBSD 10.3 needs extensive modifications to its build tools as well, so I can’t provide fixed source for this. However, packages on FreeBSD have already been fixed in that regard, so you can just # pkg install xmms-faad2 and it’s done anyway. This is likely the case for several modern Linux distros as well.

Let’s play:

xmms playing AAC-LC Freebsd 10.3 UNIX on x86_64

Yeah, it works. In this case on FreeBSD.

And the shell would say:

2-MPEG-4 AAC Low Complexity profile
MP4 - 2 channels @ 44100 Hz

Perfect! And here is the [fixed source code] upgraded with libfaad2 2.7, so you don’t have to do anything by yourself other than $ ./bootstrap && ./configure && make and # make install! Ah, actually I’m not so sure whether xmms itself builds fine on CentOS 6.8 from source these days… Maybe I’ll check that out another day. ;)

Jan 272016

HakuNeko logoJust yesterday I’ve showed you how to modify and compile the [HakuNeko] Manga Ripper so it can work on FreeBSD 10 UNIX, [see here] for some background info. I also mentioned that I couldn’t get it to build on CentOS 6 Linux, something I chose to investigate today. After flying blind for a while trying to fix include paths and other things, I finally got to the core of the problem, which lies in HakuNekos’ src/CurlRequest.cpp, where the code calls CURLOPT_ACCEPT_ENCODING from cURLs typecheck-gcc.h. This is critical stuff, as [cURL] is the software library needed for actually connecting to websites and downloading the image files of Manga/Comics.

It turns out that CURLOPT_ACCEPT_ENCODING wasn’t always called that. With cURL version 7.21.6, it was renamed to that from the former CURLOPT_ENCODING as you can read [here]. And well, CentOS 6 ships with cURL 7.19.7…

When running $ ./configure && make from the unpacked HakuNeko source tree without fixing anything, you’ll run into this problem:

g++ -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread -o obj/CurlRequest.cpp.o src/CurlRequest.cpp
src/CurlRequest.cpp: In member function ‘void CurlRequest::SetCompression(wxString)’:
src/CurlRequest.cpp:122: error: ‘CURLOPT_ACCEPT_ENCODING’ was not declared in this scope
make: *** [obj/CurlRequest.cpp.o] Error 1

So you’ll have to fix the call in src/CurlRequest.cpp! Look for this part:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Change CURLOPT_ACCEPT_ENCODING to CURLOPT_ENCODING. The rest can stay the same, as the name is all that has really changed here. It’s functionally identical as far as I can tell. So it should look like this:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Save the file, go back to the main source tree and you can do:

  • $ ./configure && make
  • # make install

And done! Works like a charm:

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux!

And now, for your convenience I fixed up the Makefile and rpm/SPECS/specfile.spec a little bit to build proper rpm packages as well. I can provide them for CentOS 6.x Linux in both 32-bit as well as 64-bit x86 flavors:

You need to unzip these first, because I was too lazy to allow the rpm file type in my blogging software.

The naked rpms have also been submitted to the HakuNeko developers as a comment to their [More Linux Packages] support ticket which you’re supposed to use for that purpose, so you can get them from there as well. Not sure if the developers will add the files to the projects’ official downloads.

This build of HakuNeko has been linked against the wxWidgets 2.8.12 GUI libraries, which come from the official CentOS 6.7 package repositories. So you’ll need to install wxGTK to be able to use the white kitty:

  • # yum install wxGTK

After that you can install the .rpm package of your choice. For a 64-bit system for instance, enter the folder where the hakuneko_1.3.12_el6_x86_64.rpm file is, run # yum localinstall ./hakuneko_1.3.12_el6_x86_64.rpm and confirm the installation.

Now it’s time to have fun using HakoNeko on your Enterprise Linux system! Totally what RedHat intended you to use it for! ;) :roll:

Jan 262016

HakuNeko logoSince I’ve started using FreeBSD as a Linux and Windows replacement, I’ve naturally always been looking at porting my “known good” software over to the UNIX OS, or at replacing it by something that gets the job done without getting on my nerves too much at the same time. For most parts other than TrueCrypt, that was quite achievable, even though I had to endure varying degrees of pain getting there. Now, my favorite Manga / Comic ripper on Windows, [HakuNeko] was the next piece of software on the list. It’s basically just a more advanced Manga website parser and downloader based on stuff like [cURL], [OpenSSL] or the [wxWidgets] GUI libraries.

I didn’t even know this until recently (shame on me for never looking closely), but HakuNeko is actually free software licensed under the MIT license. Unfortunately, the source code and build system are quite Linux- and Windows-centric, and there exist neither packages nor ports of it for FreeBSD UNIX. Actually, the code doesn’t even build on my CentOS 6.7 Linux right now (I have yet to figure out the exact problem), but I managed to fix it up so it can compile and work on FreeBSD! And here’s how, step by step:

1.) Prerequisites

Please note that from here on, terminal commands are shown in this form: $ command or # command. Commands starting with a $ are to be executed as a regular user, and those starting with # have to be executed as the superuser root.

Ok, this has been done on FreeBSD 10.2 x86_32 using HakuNeko 1.3.12, both are current at the time of writing. I guess it might work on older and future releases of FreeBSD with different releases of HakuNeko as well, but hey, who knows?! That having been said, you’ll need the following software on top of FreeBSD for the build system to work (I may have missed something here, if so, just install the missing stuff like shown below):

  • cURL
  • GNU sed
  • GNU find
  • bash
  • OpenSSL
  • wxWidgets 2.8.x

Stuff that’s not on your machine can be fetched and installed as root from the official package repository, Like e.g.: # pkg install gsed findutils bash wx28-gtk2 wx28-gtk2-common wx28-gtk2-contrib wx28-gtk2-contrib-common

Of course you’ll need the HakuNeko source code as well. You can get it from official sources (see the link in first paragraph) or download it directly from here in the version that I’ve used successfully. If you take my version, you need 7zip for FreeBSD as well: # pkg install p7zip.

Unpack it:

  • $ 7z x hakuneko_1.3.12_src.7z (My version)
  • $ tar -xzvf hakuneko_1.3.12_src.tar.gz (Official version)

The insides of my archive are just vanilla as well however, so you’ll still need to do all the modifications by yourself.

2.) Replace the shebang lines in all scripts which require it

Enter the unpacked source directory of HakuNeko and open the following scripts in your favorite text editor, then replace the leading shebang lines #!/bin/bash with #!/usr/local/bin/bash:

  • ./configure
  • ./
  • ./
  • ./res/parsers/

It’s always the first line in each of those scripts, see for example:

  1. #!/bin/bash
  3. # import setings from config-default
  4. . ./
  6. # overwrite settings from config-default
  8. CC="clang++"
  9. LD="clang++"

This would have to turn into the following (I also fixed that comment typo while I was at it):

  1. #!/usr/local/bin/bash
  3. # import settings from config-default
  4. . ./
  6. # overwrite settings from config-default
  8. CC="clang++"
  9. LD="clang++"

3.) Replace all sed invocations with gsed invocations in all scripts which call sed

This is needed because FreeBSDs sed and Linux’ GNU sed aren’t exactly that compatible in how they’re being called, different options and all.

In the text editor vi, the expression :%s/sed /gsed /g can do this globally over an entire file (mind the whitespaces, don’t omit them!). Or just use a convenient graphical text editor like gedit or leafpad for searching and replacing all occasions. The following files need sed replaced with gsed:

  • ./configure
  • ./res/parsers/

4.) Replace all find invocations with gfind invocations in ./configure

Same situation as above with GNU find, like :%s/find /gfind /g or so, but only in one file:

  • ./configure

5.) Fix the make check

This is rather cosmetic in nature as $ ./configure won’t die if this test fails, but you may still wish to fix this. Just replace the string make --version with gmake --version (there is only one occurrence) in:

  • ./configure

6.) Fix the DIST variables’ content

I don’t think that this is really necessary either, but while we’re at it… Change the DIST=linux default to DIST=FreeBSD in:

  • ./configure

Again, only one occurrence.

7.) Run ./configure to create the Makefile

Enough with that, let’s run the first part of the build tools:

  • $ ./configure --config-clang

Notice the --config-clang option? We could use GCC as well, but since clang is FreeBSDs new and default platform compiler, you should stick with that whenever feasible. It works for HakuNeko, so we’re gonna use the default compiler, which means you don’t need to install the entire GCC for just this.

There will be error messages looking quite intimidating, like the basic linker test failing, but you can safely ignore those. Has something to do with different function name prefixes in FreeBSDs libc (or whatever, I don’t really get it), but it doesn’t matter.

However, there is one detail that the script will get wrong, and that’s a part of our include path. So let’s handle that:

8.) Fix the includes in the CFLAGS in the Makefile

Find the line containing the string CFLAGS = -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread or similar in the newly created ./Makefile. After the option -O2 add the following: -I/usr/local/include. So it looks like this: CFLAGS = -c -Wall -O2 -I/usr/local/include -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread. That’s it for the Makefile.

9.) Fix the Linux-specific conditionals across the C++ source code

And now the real work starts, because we need to fix up portions of the C++ code itself as well. While the code would build and run fine on FreeBSD, those relevant parts are hidden behind some C++ preprocessor macros/conditionals looking for Linux instead. Thus, important parts of the code can’t even compile on FreeBSD, because the code only knows Linux and Windows. Fixing that isn’t extremely hard though, just a bit of copy, paste and/or modify. First of all, the following files need to be fixed:

  • ./src/MangaConnector.cpp
  • ./src/Logger.cpp
  • ./src/MangaDownloaderMain.cpp
  • ./src/MangaDownloaderConfiguration.cpp

Now, what you should look for are all conditional blocks which look like #ifdef __LINUX__. Each will end with an #endif line. Naturally, there are also #ifdef __WINDOWS__ blocks, but those don’t concern us, as we’re going to use the “Linux-specific” code, if you can call it that. Let me give you an example right out of MangaConnector.cpp, starting at line #20:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Now given that the Linux code builds just fine on FreeBSD, the most elegant and easier version would be to just alter all those #ifdef conditionals to inclusive #if defined ORs, so that they trigger for both Linux and FreeBSD. If you do this, the block from above would need to change to this:

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Should you ever want to create different code paths for Linux and FreeBSD, you can also just duplicate it. That way you could later make changes for just Linux or just FreeBSD separately:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif
  4. #ifdef __FreeBSD__
  5. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  6. #endif

Whichever way you choose, you’ll need to find and update every single one of those conditional blocks. There are three in Logger.cpp, three in MangaConnector.cpp, two in MangaDownloaderConfiguration.cpp and again three in MangaDownloaderMain.cpp. Some are more than 10 lines long, so make sure to not make any mistakes if duplicating them.

Note that you can maybe extend compatibility even further with additional directives like __OpenBSD__ or __NetBSD__ for additional BSDs or __unix__ for a wide range of UNIX systems like AIX or HP-UX. None of which has been tested by me of course.

When all of that is done, it’s compile and install time:

10.) Compile and install

You can compile as a regular user, but the installation needs root by default. I’ll assume you’ll want to install HakuNeko system-wide, so, we’ll leave the installation target directories on their defaults below /usr/local/. While sitting in the unpacked source directory, run:

  • $ gmake
  • # gmake install

If nothing starts to crash and burn, this should compile and install the code. clang will show some warnings during compilation, but you can safely ignore that.

11.) Start up the white kitty

The installation procedure will also conveniently update your window manager as well, if you’re using panels/menus. Here it’s Xfce4:

HakuNeko is showing up as an "Internet" tool

HakuNeko (“White Cat”) is showing up as an “Internet” tool. Makes sense.

With the modifications done properly it should fire up just fine after initializing its Manga connectors:

HakuNeko with the awesomeness that is "Gakkou Gurashi!" being selected from the HTTP source "MangaReader"

HakuNeko with the awesomeness that is “Gakkou Gurashi!” being selected from the HTTP source [MangaReader].

Recently the developers have also added [DynastyScans] as a source, which provides access to partially “rather juicy” Yuri Dōjinshi (self-published amateur and sometimes semi-professional works) of well-known Manga/Anime, if you’re into that. Yuri, that is (“girls love”). Mind you, not all, but a lot of the stuff on DynastyScans can be considered NSFW and likely 18+, just as a word of warning:

HakuNeko fetching a Yuru Yuri Dōjinshi from DynastyScans, bypassing their download limits by not fetching packaged ZIPs - it works perfectly!

HakuNeko fetching a Yuru Yuri Dōjinshi called “Secret Flowers” from DynastyScans, bypassing their download limits by not fetching packaged ZIPs – it works perfectly!

Together with a good comic book reader that can read both plain JPEG-filled folders and stuff like packaged .cbz files, HakuNeko makes FreeBSD a capable comic book / Manga reading system. My personal choice for a reader to accompany HakuNeko would be [QComicBook], which you can easily get on FreeBSD. There are others you can fetch from the official package repository as well though.

Final result:

HakuNeko and QComicBook make a good team on FreeBSD UNIX

HakuNeko and QComicBook make a good team on FreeBSD UNIX – I like the reader even more than ComicRack on Windows.

And, at the very end, one more thing, even though you’re likely going to be aware of this already: Just like Anime fansubs, fan-translated Manga or even Dōjinshi are sitting in a legal grey zone, as long as the book in question hasn’t been licensed in your country. It’s being tolerated, but if it does get licensed, ownership of a fan-translated version will likely become illegal, which means you should actually buy the stuff at that point in time.

Just wanted to have that said as well.

Should you have trouble building HakuNeko on FreeBSD 10 UNIX (maybe because I missed something), please let me know in the comments!

Sep 022015

colorecho logoRecently, I am doing a lot of audio/video transcoding, and for that I’m using tools like mkvextract, mkvmerge, x264, avconv and fghaacenc on the command line. Mostly this is to get rid of old files that are either very inefficiently encoded – like DivX3 or MPEG-2 – or are just too large/high-bitrate for what the content’s worth. Since I do have large batches of relatively similar files, I wrote myself a few nice loops that would walk through an entire stack of videos and process all of it automatically. Because I want to see at first glance how far through a given video stack a script is, I wanted to output some colored notifications on the shell after each major processing step, telling me the current video number. That way, I could easily see how far the job had progressed. We’re talking processes here that take multiple days, so it makes sense.

Turns out that this was harder than expected. In the old DOS days, you would just load ANSI.SYS for that, and get ANSI color codes to work with. The same codes still work in modern UNIX terminals. But not within the Windows cmd for whatever reason. Now there is an interface for that kind of stuff, but seemingly no tools to use it. Like e.g. the echo command simply can’t make use of it.

Now there is project porting ANSI codes back to the cmd, and it’s called [ANSICON]. Basically just a DLL hooked into the cmd. Turns out I can’t use that, because I’m already using [clink] to make my cmd more UNIX-like and actually usable, which also is additional code hooked into the cmd, and the two confict with each other. More information about that [here].

So how the hell can I do this then? Turns out there is a batch script that can do it, but it’s a very outlandish hack that doesn’t seem to want to play nice with my shells if I do @echo off instead of echo off as well. I guess it’s clinks work breaking stuff here, but I’m not sure. Still, here is the code, as found by [Umlüx]:

expand/collapse source code
  2. SETLOCAL EnableDelayedExpansion
  4. FOR /F "tokens=1,2 delims=#" %%A IN ('"PROMPT #$H#$E# & ECHO ON & FOR %%b IN (1) DO REM"') DO (
  5.   SET "DEL=%%A"
  6. )
  8. ECHO say the name of the colors, dont read
  10. CALL :ColorText 0a "blue"
  11. CALL :ColorText 0C "green"
  12. CALL :ColorText 0b "red"
  13. ECHO.
  14. CALL :ColorText 19 "yellow"
  15. CALL :ColorText 2F "black"
  16. CALL :ColorText 4e "white"
  18. GOTO :eof
  20. :ColorText
  22. ECHO OFF
  24.  "%~2"
  25. findstr /v /a:%1 /R "^$" "%~2" NUL
  26. DEL "%~2" > NUL 2>&1
  28. GOTO :eof

So since those two solutions didn’t work for me, what else could I try?

Then I had that crazy idea: Since the interface is there, why not write a little C program (actually ended up being C++ instead) that uses it? Since I was on Linux at the time, I tried to write it there and attempt my first cross-compile for Windows using a pre-built [MinGW-w64] compiler, that luckily just played along nicely on my CentOS 6.6 Linux. You can get pre-built binaries for 32-Bit Windows targets [here], and for 64-bit Windows targets [here]. Thing is, I know zero C++, so that took some time and it’s quite crappy, but here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include "windows.h"
  4. // Set NT commandline color:
  5. void SetColor(int value) {
  6.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  7. }
  9. int main(int argc, char *argv[]) {
  10.   SetColor(atoi(argv[2]));
  11.   std::cout << "\r\n" << argv[1] << "\r\n";
  12.   SetColor(7);
  13.   return 0;
  14. }

Update 2016-06-22: Since I wanted to work with wide character strings in recent days (so, non-ANSI Unicode characters), my colorecho command failed, because it couldn’t handle them at all. There was no wide character support. I decided to update the program to also work with that using Windows-typical UCS-2le/UTF-16, here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include <io.h>
  3. #include <fcntl.h>
  4. #include "windows.h"
  6. // Set NT commandline color:
  7. void SetColor(int value) {
  8.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  9. }
  11. // Unicode main
  12. int wmain(int argc, wchar_t *argv[]) {
  13.   SetColor(_wtoi(argv[2]));
  14.   _setmode(_fileno(stdout), _O_U16TEXT);      // Set stdout to UCS-2le/UTF-16
  15.   std::wcout << "\r\n" << argv[1] << "\r\n";  // Write unicode
  16.   SetColor(7);                                // Set color back to default
  17.   return 0;
  18. }

I also updated the download section and the screenshots further below. End of update.

This builds nicely using MinGW on Linux for 64-bit Windows targets. Like so: While sitting in the MinGW directory (I put my code there as echo.cpp as well, for simplicities’ sake), run: ./bin/x86_64-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe. If you want to target 32-bit Windows instead, you’d need to use the proper MinGW for that (it comes in a separate archive): ./bin/i686-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe

When building like that, you’d also need to deploy and libgcc_s_seh-1.dll along with the EXE files though. The DLLs can be obtained from your local MinGW installation directory, subfolder ./x86_64-w64-mingw32/lib/ for the 64-bit or ./i686-w64-mingw32/lib/ for the 32-bit compiler. If you don’t want to do that and rather rely on Microsofts own C++ redistributables, you can also compile it with Microsoft VisualStudio Express, using its pre-prepared command line. You can find that here, if you have Microsofts VisualStudio installed – version 2010 in my case:

VC2010 command lines

The command lines of Visual Studio (click to enlarge)

Here, the “Visual Studio Command Prompt” is for a 32-bit build target, and “Visual Studio x64 Win64 Command Prompt” is for building 64-bit command line programs. Choose the appropriate one, then change into a directory where you have echo.cpp and run the following command: cl /EHsc /W4 /nologo /Fecolorecho.exe echo.cpp, giving you the executable colorecho.exe.

Alternatively, you can just download my pre-compiled versions here:

Update 2016-06-22: And here are the Unicode versions:

End of update.

Note that this tool does exactly what I need it to do, but it’ll likely not do exactly what you’d need it to do. Like e.g. the line breaks it adds before and after its output. That’s actually a job for the shells echo command to do, not some command line tool. But I just don’t care. So that’s why it’s basically a piece of crap for general use. The syntax is as follows, as shown for the 64-bit VC10 build:

colorecho-x86_64-vc10.exe "I am yellow!" 14

When run, it looks like this:

Colorecho running

colorecho invoked on the Windows cmd

So the first parameter is the string to be echoed, the second one is the color number. That number is 2-digit and can affect both the foreground and the background. Here the first 16 of them, which are foreground only:

  • 0: Black
  • 1: Dark blue
  • 2: Dark green
  • 3: Dark cyan
  • 4: Dark red
  • 5: Dark purple
  • 6: Dark yellow
  • 7: Light grey
  • 8: Dark grey
  • 9: Light blue
  • 10: Light green
  • 11: Light cyan
  • 12: Light red
  • 13: Light purple
  • 14: Light yellow
  • 15: White

If you go higher than that, you’ll also start changing the background colors and you’ll get different combinations of foreground and background colorization.

The background colors actually follow the same order, black, dark blue, dark green, etc. Numbers from 0..15 are on black background, numbers from 16..31 are on a dark blue background and so on. This makes up pretty much the same list:

  • 0..15: 16 foreground colors as listed above on the default black background, and
  • 16..31: on a dark blue background
  • 32..47: on a dark green background
  • 48..63: on a dark cyan background
  • 64..79: on a dark red background
  • 80..95: ona dark purple background
  • 96..111: on a dark yellow background
  • 112..127: on a light grey background
  • 128..143: on a dark grey background
  • 144..159: on a light blue background
  • 160..175: on a light green background
  • 176..191: on a light cyan background
  • 192..207: on a light red background
  • 208..223: on a light purple background
  • 224..239: on a light yellow background
  • 240..255: on a white background

Going over 255 will simply result in an overflow causing the system to start from the beginning, so 256 is equal to 0, 257 to 1, 260 to 4 and so on.

Update 2016-06-22: With the Unicode version, you can now even do stupid stuff like this:

colorecho unicode with DejaVu Sans Mono font on Windows cmd

It could be useful for some people? Maybe?

Note that this has been rendered using the DejaVu Sans Mono font, you can get it from [here]. Also, you need to tell the Windows cmd which fonts to allow (only monospaced TTFs work), you can read how to do that [here], it’s a relatively simple registry hack. I have yet to find a font that would work and also render CJK characters though, but with the fonts I myself have available at the moment, asian stuff won’t work, you’ll just get some boxes instead:

colorecho unicode CJK fail

Only characters which have proper glyphs for all desired codepoints in the font in use can be rendered, at least on Windows XP.

Note that this is not a limitation of colorecho, it does output the characters correctly, “ですね” in this case. It’s just that DejaVu Sans Mono has no font glyphs to actually draw them on screen. If you ever come across a neat cmd-terminal-compatible font that can render most symbols and CJK, please tell me in the comments, I’d be quite happy about that! End of update.

If somebody really wants to use this (which I doubt, but hey), but wishes it to do things in a more sane way, just request it in the comments. Or, if you can, just take the code, change and recompile it by yourself. You can get Microsofts VisualStudio Express – in the meantime called [VisualStudio Community] – free of charge anyway, and MinGW is free software to begin with.