Mar 152018
Windows 2000 + Let's Encrypt logo

[1] 1.) Introduction

There has been an issue with my [stoneage server]German flag (which is also hosting this weblog) that has been bugging me for quite a while now: I’ve been scared of the time when other server operators and software developers would start to seriously disable ancient SSL/TLS ciphers such as SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA and SSL_RSA_WITH_3DES_EDE_CBC_SHA, or in other words: SHA1, RC4 and 3DES. The services are actually made of both non-free as well as free services that are using several different cryptographic libraries such as different OpenSSL versions but also WinSSL CryptAPI / schannel. Naturally it’s the latter which is presenting the biggest issue: The CryptAPI of Windows 2000 Server is ancient. And while it can do at least TLS v1.0, the ciphers have been becoming the true burden. Unfortunately (?!), some services on do require some form of widely accepted encryption.

Some of the servers do have interchangeable OpenSSL libraries – so you can just swap those .dll files for an upgrade – while others do not. Some from the latter category can be backported to / recompiled for Windows 2000 to upgrade OpenSSL, like I’ve been [able to do] for the UnrealIRCd IRC server. Some however can not.

And then there are services which are using Windows SSL or in other words the CryptAPI / schannel. Those are the most inflexible of the pack.

Another problem is that self-signed certificates such as the ones I’ve been using have also become increasingly problematic. Most client software like web browsers in particular, but also email clients, IRC clients and others keep pestering their users quite a bit when being presented with such an “untrustworthy” SSL certificate. They really want one signed by a fully trusted certificate authority (CA) or an intermediate CA. The best solution for that problem right now is to get a Let’s Encrypt[2] certificate, as it’s free and well-trusted. Only reason I haven’t done that so far is that it needs automatisms (=ACME client software) in place to remain manageable.

I wanted to solve both issues in one fell swoop.

2.) The trigger

eMail logoThen again, you know how people are – they tend to act only when something bad has already happened. In the tradition of that (bad) behavior I’ve chosen to act only where the problem at hand was truly easy to solve. After I’ve managed to compile a modern enough OpenSSL library, I’ve just swapped it on the spot where possible, but I haven’t touched the more problematic services, where such a thing is not doable.

However, there has been a serious issue recently when mail server operators started to beef up their cryptography across the board. It seems there are some updates being rolled out about now which block the ancient ciphers and/or protocols I mentioned in the introduction.

The truly serious part is that seemingly, all mail servers who encounter another that does feature STARTTLS on SMTP port 25, but has no secure ciphers to offer just drop the connection on the spot without generating any error. The mail will not be delivered, but no delivery error will be returned to the sender either! It’s truly a case where the eMails just “disappear” without anyone ever noticing a problem at first. To make matters worse, there was no way to disable just STARTTLS for SMTP with my server software. It’s either no SSL/TLS at all or fully enabled STARTTLS on all plain eMail ports.

See this log to get an idea (remote host names have been replaced with “” and IP addresses as well as security-relevant data have been masked here):

expand/collapse log file

The most interesting part is clearly at the bottom, where it says “SSL negotiation successful (TLS 1.0, 2048 bit key exchange, 168 bit 3DES encryption)”. 3DES is the best Windows 2000 can do by itself, and it looks as if the remote server would accept the implementation just fine. But what happens next is that the remote machine just drops the connection, and that’s it!

The remote side who sent the eMail (that was myself from my account at work actually) never got any delivery failure notification, and after asking other people about it, it was just the same. As said, the mails just vanish!

I also talked to the operator of the mail gateway at work, and he said he didn’t see anything in the logs either, other than that the connection was just being dropped. Pretty stupid to not generate a proper error here, if you ask me.

Well, that was enough of a reason to act! eMails just disappearing is completely unacceptable after all!

3.) What can you do?

I thought about this issue in the past already, and one of my ideas was to use something like [BlackWingCats’] kernel API extensions for Windows 2000 (“KernelEx”). The issue with that is though, that it’s modifying large parts of the operating system, and it’s very incompatible to my German version, wrecking half of the system upon installation during my tests in a VM. So that idea went down the drain pretty fast.

Very recently however, I got the idea that maybe I could use [stunnel] to solve the cipher problem. It’s basically a port mapper wrapping plain text protocols up in SSL/TLS. It’s a program coming from the Linux/UNIX world, but it also works on Windows. On top of that, it miraculously runs on Windows 2000 as well, even in its newest version bundling modern OpenSSL 1.0.2.

Additionally, I thought I’d once again try to get some Let’s Encrypt ACME client to work on Windows 2000 for certificate issuing and renewal, even though I failed to get any of them working before.

3a.) Let’s Encrypt configuration via ZeroSSL

ZeroSSL bannerSince we need an SSL certificate for stunnel anyway, let’s cover this part first. For ease of use, I chose ZeroSSLs’ [Crypt::LE] Perl tool. I didn’t like their [web-based configuration] too much, as they’ll be issuing your private key for you on their servers.

Generally, when dealing with cryptography and trust concepts, I wouldn’t recommend anyone other than yourself generate and use your private key, so that’s one reason why I chose their Perl tool instead. Plus, you can’t automate the certificate renewal process with some web tool, but you can with Perl.

If you don’t have Strawberry Perl installed on your machine, you can still rely on their [Windows binaries] as well. Those are basically just and the necessary Perl parts wrapped in an le32.exe file. Crypt::LE also supports ActiveState Perl, but the free Strawberry Perl should be preferable, if you’re choosing that path.

Unfortunately, if you choose to use the .exe version, it won’t run on Windows 2000 out of the box, as it calls the WinSock 2.0 function freeaddrinfo(), which is only available on Windows XP and newer. For an easy fix, you can use a modified WS2_32.dll WinSock library found in the dllfiles\ subfolder of this [winsock2_getaddrinfo.rar] archive by [Martin Brenner]German flag, if you choose to trust the DLL.

If you do, just put it next to le32.exe, and make sure you launch the program only while sitting within the installation folder of le32.exe itself. Do not overwrite your system-wide %WINDIR%\system32\WS2_32.dll, you don’t need to do that, and you shouldn’t either!

As said, if you don’t trust the hack for the binary, you need to install Strawberry Perl and follow the instructions for the compilation & installation of Crypt::LE from their website.

3a1.) Ok, I have the tools, now what?

I won’t cover the process of using Crypt::LE, as there is an excellent manual [right here at ZeroSSL]! There is one thing that needs to be said though: You need an openssl.exe for the initial certificate request and key generation part. There is one bundled with stunnel in the subdirectory bin\ of its installation folder, you can just use that.

But before you start with it on a cmd terminal, you’ll need to tell OpenSSL where to look for its configuration file. Say you’ve installed stunnel in %PROGRAMFILES%\stunnel\, then run the following command before starting to work with OpenSSL:

SET "OPENSSL_CONF=%PROGRAMFILES%\stunnel\config\openssl.cnf"

This sets %OPENSSL_CONF% to the file name of the OpenSSL configuration, and OpenSSL will automatically parse that environment variable. Adjust paths as necessary, then follow ZeroSSLs’ manual to get your first Let’s Encrypt certificate(s)!

3b.) stunnel

stunnel on Windows 2000

stunnel on Windows 2000

If you’ve been following this article, you’ll already have stunnel installed by now. On Linux & UNIX, stunnel is just a command line tool and/or xinetd service, but on Windows, you also get a bit of a GUI and a tray icon with it. You’ll still have to configure it by editing its config\stunnel.conf configuration file with your favorite text editor however.

Say you wanted to protect a web server on port 80 by adding HTTPS on port 443 for it. The corresponding configuration entry in that configuration file would look like that (here: with separate key file not stored directly within the certificate):

accept  = 443
connect = 80
cert    = mydomain.pem
key     = mydomain.key

stunnel will listen on port 443 with implicit TLS for you, and then redirect the traffic to port 80 on the same machine. To a web browser connecting to, it’ll look like just another TLS-enabled web server.

The same goes for other implicit SSL/TLS services such as SMTPS, IMAPS, POP3S, etc.

The exception of course are explicit SSL/TLS implementations using the STARTTLS command. One classic example for that would be SMTP on port 587, which is typically STARTTLS-enabled. To make that work, stunnel has to emulate parts of the specific protocol to secure, so support for this is rather limited. Currently, stunnel supports the following network protocols for explicit STARTTLS:

  • CIFS (SMB, Samba, older implementation)
  • CONNECT (Client only)
  • IMAP (as per RFC 2595)
  • NNTP (as per RFC 4642)
  • POP3 (as per RFC 2449)
  • SMTP (as per RFC 2487)
  • SOCKS (versions 4, 4a and 5)

To use it, you’d need to configure the service as follows, this example is for SMTP with STARTTLS on port 587, mapping it to your local, unencrypted SMTP server:

accept   = 587
connect  = 25
cert     = mydomain.pem
key      = mydomain.key
protocol = smtp

Now if present, switch off your existing SSL/TLS services first (like make your web server stop listening to port 443), and then fire up the stunnel program and everything should work. You can also install it as a system service on Windows by the way, it’s very easy: stunnel -install.

Together with a properly requested and signed Let’s Encrypt certificate this will give any ancient server things like TLS v1.2 with AES256-GCM-SHA384 and any modern client will trust your certificate implicitly, as most vendors have by now added the ISRG CA certificates to their CACert bundles. Even Microsoft trusts them by now.

4.) Really putting it all together

Let's Encrypt logoEven if you’ve managed to do all of this manually, it won’t solve the problem forever. Let’s Encrypt certificates have a lifetime of just 90 days, so you will have to renew them pretty often. That can be done with just a batch script launching Crypt::LE as well as doing the rollout of the certificates and relaunch of the servers, so that they can re-read the certificates and key files.

Here’s an example script for stunnel and some other hypothetical servers that accept certificate files in different configurations. It assumes that you’re using Crypt::LE together with Strawberry Perl. As you can see, aside from Perl and Crypt::LE you won’t need anything else, as the rest can be done with regular Windows cmd builtins and the NetShell:

expand/collapse certificate-renewal.bat
  1. :: Renew our certificate if it expires within the next 30 days, put HTTP
  2. :: challenge files in C:\MyHTTProot\.well-known\acme-challenge\ and return
  3. :: code 42 if there was a renewal
  4. --------------------------------------------------------------------------
  5. SET "PERLBIN=C:\StrawberryPerl\perl\bin\perl.exe"
  6. SET "LE=C:\StrawberryPerl\perl\bin\"
  7. "%PERLBIN%" "%LE%" -renew 30 -generate-missing -unlink -live -legacy ^
  8.  -key "C:\MyCerts\my-account-key.key" -csr "C:\MyCerts\my-cert-request.csr" ^
  9.  -csr-key "C:\MyCerts\mydomain-key.key" -crt "C:\MyCerts\mydomain-cert.cert" ^
  10.  -domains "," -issue-code 42 ^
  11.  -path "C:\MyHTTProot\.well-known\acme-challenge\"
  13. :: Check whether there was a renewal, and roll out certs + restart servers
  14. :: if so, otherwise just terminate
  15. :: -----------------------------------------------------------------------
  16. IF %ERRORLEVEL% EQU 42 (
  17.   :: stunnel for servers with old cryptographic implementations, this requires
  18.   :: a domain+intermediate certificate bundle with a separate private key file
  19.   TYPE "C:\MyCerts\mydomain.cert" > "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  20.   ECHO. >> "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  21.   TYPE "C:\MyCerts\" >> "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  22.   :: They key needs copying only once
  23.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server\stunnel\mydomain.key"
  25.   :: A server that needs domain certificate, intermediate CA cert and key all
  26.   :: separately
  27.   COPY /V /Y "C:\MyCerts\mydomain.cert" "C:\Server1\sslconf\"
  28.   :: They key and intermediate certificate need copying only once
  29.   :: COPY /V /Y "C:\MyCerts\" "C:\Server1\sslconf\"
  30.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server1\sslconf\"
  32.   :: A server that needs domain and intermediate certificates in one bundle, but
  33.   :: the private key as a separate file, like stunnel
  34.   TYPE "C:\MyCerts\mydomain.cert" > "C:\Server2\sslconf\mydomain.pem"
  35.   ECHO. >> "C:\Server2\sslconf\mydomain.pem"
  36.   TYPE "C:\MyCerts\" >> "C:\Server2\sslconf\mydomain.pem"
  37.   :: They key needs copying only once
  38.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server2\sslconf\"
  40.   :: Yet another server, that needs both certificates and your private key all
  41.   :: in one bundled file
  42.   TYPE "C:\MyCerts\mydomain.key" > "C:\Server3\sslconf\mydomain.pem"
  43.   ECHO. >> "C:\Server3\sslconf\mydomain.pem"
  44.   TYPE "C:\MyCerts\mydomain.cert" >> "C:\Server3\sslconf\mydomain.pem"
  45.   ECHO. >> "C:\Server3\sslconf\mydomain.pem"
  46.   TYPE "C:\MyCerts\" >> "C:\Server3\sslconf\mydomain.pem"
  48.   :: Restart all services to reload the certificate and key
  49.   :: ------------------------------------------------------
  50.   :: Restart stunnel
  51.   net stop "stunnel"
  52.   net start "stunnel"
  54.   :: Restart Server 1
  55.   net stop "Server 1 system service name"
  56.   net start "Server 1 system service name"
  58.   :: Restart Server 2
  59.   net stop "Server 2 system service name"
  60.   net start "Server 2 system service name"
  62.   :: Restart Server 3
  63.   net stop "Server 3 system service name"
  64.   net start "Server 3 system service name"
  65. )
  67. :: All done!

Now you can automate the process by creating a job in Windows task scheduler to launch that certificate-renewal.bat like every 20 days or so. The script will of course vary quite a bit depending on your environment and your services, so take it only as a rough guide.

And that’s how you get Let’s Encrypt certificates and modern cryptography up and running on an 18 years old Windows operating system, for what it’s worth. :roll:

5.) Bonus information

You can actually use stunnel in client mode as well. Say you’re using Windows 2000 Pro as your client operating system (*cough*) and your software is so old and insecure, that some remote server at say won’t talk to you anymore. Just run stunnel on your client on demand, with a per-host configuration such as this:

client = yes
accept = localhost:9999
connect =
CAfile = ca-certs.pem

Open up your web browser, surf to https://localhost:9999, and stunnel’ll redirect you to that server, which will now see a secure clientside SSL implementation. Only thing is that you might need to create an exception in your browser, because the host names don’t match between what you entered in the address field and what’s in the remote servers’ certificate, but no way around that.

And now, time for some cold beer! Beer Smilie

[1] The “Let’s Encrypt Radiant Lock” design mark is a trademark of the Internet Security Research group and is licensed under the CC-BY-NC 4.0. All rights reserved.

[2] The “Let’s Encrypt®” word mark is a trademark of the Internet Security Research group. All rights reserved.

Mar 142018
H.265/HEVC logo

1.) Introduction

This post is a followup to the original you can find [here]. My reason for writing a new one instead of just editing the existing one is that the new results have been measured with a slightly different parametrization and under slightly different circumstances, so direct comparability has been thrown out the window. Anyway, like in the former article, multiple x265 video encoder versions starting with 1.7+512 are going to be compared to each other to see how the performance of x265 has evolved over time.

Originally, I wanted to test this only on a machine very similar to my original testbed, maxing out at 6 cores and SSE4.2 instruction set extensions. Since it is said that x265 features significant optimizations for newer AVX and AVX2 instructions however, I decided to include a more modern machine as well, to determine how the scaling differs on both an older and a newer machine.

2.) Benchmarks

First of all, the video used for this is not very “high definition”. Well, maybe by definition as it is 720p, but still. This also puts limits on scalability across cores/threads, which shall provide some additional insights later. Let’s start.

2a.) The older machine on MS Windows

The older one goes first: Here we have an Intel Core i7 980X running CentOS 6.9 Linux. Since I already had all those [Win64 builds] lying around, I ran the test from within a VirtualBox running Windows XP Professional x64 Edition SP2. The specs:

  • CPU: Intel Core i7 980X 3.33GHz (6 cores, 12 threads), last instruction set extension: SSE4.2
  • OS: Windows XP Professional x64 Edition SP2 on VirtualBox 5.1.2 on CentOS Linux 6.9

Here we go:

x265 performance trend on the older machine

x265 performance trend on the older machine (click to enlarge)

While the developments here aren’t exactly identical to what has been posted in the former article, the trend is clearly the same. The new part starts with version 2.3+2, after which we can see a long-needed performance improvement, maybe due to improvements to the assembly code. Also, from here on out we can observe a slight upwards trend for all color channel depths. The basic drop from 8-bit to 10-bit and then 12-bit color stays rather similar though, courtesy of the more expensive 16-bit arithmetic used for the higher color depths (The 8-bit version of x265 also uses 8-bit arithmetic where applicable, reducing effective precision).

There is nothing groundbreaking to be seen like with the introduction of --no-rskip somewhere in between 1.9+141 and 1.9+200, but whatever happened between 2.3+2 and 2.4+2 isn’t too shabby either.

Now let’s see how the modern box with AVX, AVX2, BMI2 and FMA3 instructions fares here!

2b.) The newer machine on CentOS Linux

Alright, specs first:

  • CPUs: 2 × Intel Xeon E5-2620 v4 (2 × 8 cores, 16 Threads for a 16C/32T total), newest instruction set extensions: AVX2, BMI2, FMA3
  • OS: CentOS 7.3.1611 with Linux kernel 3.10.0-514.16.1

This is a modern 2017 Broadwell-EP server featuring a much higher IPC when compared to the older Westmere/Gulftown architecture from 2010. My estimate would be around +40% of raw, additional instructions per clock here. On top of that, we’ll get a peek at its AVX/AVX2 etc. scaling:

x265 performance trend on the modern machine

x265 performance trend on the modern machine

Now, before I comment on the trend to be seen in that chart, let me say one thing: Surprisingly, unlike what x264 has shown over the years, x265 didn’t seem to want to scale more across cores/threads with progressing development. So the core count was not really having much influence at all; Pass 1 of the encode would use 5-6 cores, where pass 2 would eat up 6-7, all the way from the oldest to the newest versions.

What we do see here aside from the --no-rskip drop and the performance increase from 2.3+2 to 2.4+2 are three things: First, we can see a new upwards trend in performance starting with 2.5+9. The development has been more linear for the older CPU, which is why I’m assuming that we get to see some work invested in AVX(2) code here, probably also in BMI/FMA code.

The second thing is the more significant decline starting after 2.0+11. I have no explanation for this, as new features added around here shouldn’t have more negative impacts on newer CPUs as when to compared to older ones. If anything, it should be the other way around, so that part going on up to 2.1+60 is a bit confusing.

And the third thing that is very clear here: The 12-bit version is much closer to 10-bit performance. Only thing I can think of here is that the AVX/AVX2 code paths have generally always been faster on the 12-bit version when compared to the older ≤SSE4.2 assembly code.

Seeing this, I’d like to look at one more thing: The dropoff we can see from 8-bit -> 10-bit -> 12-bit compared between the two architectures. Let’s do that:

3.) 8-/10-/12-bit scaling across platforms

Let’s compare this side-by-side across all versions tested, in a very simple chart:

Well, the modern Xeons do lose less performance even from 8- to 10-bit, but as seen above, the drop to 12-bit is surprisingly insignificant on modern CPUs. This is a bit unfortunate actually, given that all the rage is about 10-bit with the Anime communities and also the official Blu-Ray UHD/4K standards, but oh well. :roll:

4.) Summing it up

Anyway, this hasn’t been too interesting actually, aside from the 12-bit surprise that I guess nobody cares about anyway. Well, we’re seeing some more significant performance increases for recent versions of x265 on modern architectures, but those are only compensating for the loss of performance we’ve seen before, that didn’t occur on older machines.

Maybe the AVX+AVX2 code was good from the very start instead of spreading its wings only now? Anyway, we’re seeing some slow but steady increase in performance for now even on older machines. If there’s a reason to buy new processors for x265 it’s probably not the AVX/AVX2/FMA3/BMI2/AVX512/whatever, but rather the generally higher IPC of newer chips…

5.) How the benchmark was done

5a.) On Windows

Given a folder with x265 versions named x1.7+512.exe, x1.6+2.exe etc., as well as an ffmpeg.exe (like from [here]), the subfolders log\, output\, stats\ and results\, you can adapt and use the following script after putting it right next to the x265 binaries, I call it test-performance-trends.bat:

expand/collapse source code
  1. @ECHO OFF
  3. FOR %%I IN (1.7-8b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  4.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%I, 8 ^
  5.  bits... & .\timethis.exe "echo x265 version %%I 8 bit results: & .\ffmpeg.exe ^
  6.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  7.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p ^
  8.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%I-8b.stats ^
  9.  -o .\output\%%I-8b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  10.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  11.  -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow ^
  12.  --bitrate 2000 --pass 2 --stats .\stats\%%I-8b.stats -o .\output\%%I-8b-p2.h265 ^
  13.  2>NUL" 1> .\results\results-%%I-8b.txt 2>.\log\timethis-errorlog-%%I-8b.txt
  16. FOR %%J IN (1.7-10b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  17.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%J, 10 ^
  18.  bits... & .\timethis.exe "echo x265 version %%J 10 bit results: & .\ffmpeg.exe ^
  19.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  20.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p ^
  21.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%J-10b.stats ^
  22.  -o .\output\%%J-10b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  23.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  24.  -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow ^
  25.  --bitrate 2000 --pass 2 --stats .\stats\%%J-10b.stats -o ^
  26.  .\output\%%J-10b-p2.h265 2>NUL" 1> .\results\results-%%J-10b.txt ^
  27.  2>.\log\timethis-errorlog-%%J-10b.txt
  29. FOR %%K IN (1.7-12b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  30.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%K, 12 ^
  31.  bits... & .\timethis.exe "echo x265 version %%K 12 bit results: & .\ffmpeg.exe ^
  32.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  33.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  34.  -p veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats ^
  35.  .\stats\%%K-12b.stats -o .\output\%%K-12b-p1.h265 2>NUL & .\ffmpeg.exe -r ^
  36.  24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict ^
  37.  -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  38.  -p veryslow --bitrate 2000 --pass 2 --stats .\stats\%%K-12b.stats -o ^
  39.  .\output\%%K-12b-p2.h265 2>NUL" 1> .\results\results-%%K-12b.txt ^
  40.  2>.\log\timethis-errorlog-%%K-12b.txt
  42. ECHO All done, results are to be found in the results\results-*.txt files!

5b.) On Linux

Given a system-wide installation of ffmpeg and a folder with statically linked x265 binaries named x1.7+512, x1.6+2 etc., as well as the subfolder results/, the following script – – should do the job after being adapted for your list of x265 binaries and put right next to them:

expand/collapse source code
  1. #!/usr/bin/env sh
  3. for i in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  4. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$i, \
  5. 8 bits...\n" && time ( printf "x265 version $i 8 bit results:\n" \
  6. 1>./results/results-$i-8b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  7. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  8. ./x$i - --y4m -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  9. --slow-firstpass --stats ./stats/$i-8b.stats -o ./output/$i-8b-p1.h265 \
  10. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  11. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$i - --y4m \
  12. -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  13. ./stats/$i-8b.stats -o ./output/$i-8b-p2.h265 2>/dev/null ) \
  14. 2>>./results/results-$i-8b.txt; done
  16. for j in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  17. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$j, \
  18. 10 bits...\n\n" && time ( printf "x265 version $j 10 bit results:\n" \
  19. 1>./results/results-$j-10b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  20. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  21. ./x$j - --y4m -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  22. --slow-firstpass --stats ./stats/$j-10b.stats -o ./output/$j-10b-p1.h265 \
  23. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  24. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$j - --y4m \
  25. -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  26. ./stats/$j-10b.stats -o ./output/$j-10b-p2.h265 2>/dev/null ) \
  27. 2>>./results/results-$j-10b.txt; done
  29. for k in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  30. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$k, \
  31. 12 bits...\n\n" && time ( printf "x265 version $k 12 bit results:\n" \
  32. 1>./results/results-$k-12b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  33. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  34. ./x$k - --y4m -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  35. --slow-firstpass --stats ./stats/$k-12b.stats -o ./output/$k-12b-p1.h265 \
  36. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  37. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$k - --y4m \
  38. -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  39. ./stats/$k-12b.stats -o ./output/$k-12b-p2.h265 2>/dev/null ) \
  40. 2>>./results/results-$k-12b.txt; done
  42. printf "\nBenchmarks completed, results in `pwd`/results/results-*.txt!\n"

That’s it!

Sep 222017
Changeicon logo

[1] 1. Introduction

Note: A more step-by-step guide is to be found farther below, see point 5.!

Recently, [Umlüx] has reminded me of his idea to be able to (visually) tag folders on Windows for specific purposes. For convenience, it’s supposed to work by right-clicking a folder in Windows Explorer, opening a submenu from the context menu, and then by picking the proper tag. After that, the folder icon should change, indicating that something’s special about this one. Windows can actually do the “tagging” part by itself using desktop.ini files, but manually writing them is a pain of course, hence the right-click idea.

The thing is, there already are tools to address folder tagging on Windows, but they’re often not feature complete, have limited XP compatibility or they lack very important features like timestamp preservation. Others have that last part, but only in paid versions of their software. So, time to do it by ourselves!

I picked his Powershell code up, and while it would run on my ancient XP x64 machines, implementing the required menu structure proved to be impossible, especially the cascading part. Ah, let me just show you the final product first, so you know what it was that I wanted (in my case, it’s meant for tagging media folders):

Changeicon context menu

“Changeicon” context menu (click to enlarge)

2. Creating the menu structure (XP + Vista compatible)

Starting with Windows 7, Microsoft introduced a new way to create submenus in the context menu, including icons, all in the Windows Registry. Older versions of Windows like Vista or XP however can’t do that, and I wanted a solution that works on all of them, from XP to Win10. So how can programs like 7-zip create such Explorer submenus with icons on legacy systems? They do so by injecting a COM .dll into the graphical shell, extending its capabilities. Typically, those are written in C++, and that’s not something I want to or can even do.

Luckily, we don’t have to develop that by ourselves, as somebody has already done it: Meet the [KuShellExtension]!

KuShellExtension – or KuShellExt in short – is a set of two library files, KuShellExtension.dll as well as KuShellExtension64.dll, the latter being meant for 64-bit Vista machines. It’s compiled for NT 5.2 though, so it’ll also work on XP x64 and Server 2003 x64. On top of that, it still works even on Windows 10! With the libraries comes an XML configuration file and some simple installer/uninstaller shell scripts.

In the example we’ve seen in the screenshot above, the respective configuration in config.xml would look like this:

expand/collapse source code
  1. <?xml version="1.0" encoding="utf-8"?>
  2. <config version="1">
  4.   <!-- Gobal variables -->
  5.   <var name="LEGACY_STYLE">false</var>
  6.   <var name="HIDE_MISSING">false</var>
  7.   <var name="ICON_DIR">${var:APPDATA}\changeicon\icons\</var>
  8.   <var name="INSTALL_DIR">${var:APPDATA}\changeicon\bin\</var>
  10.   <!-- Menu -->
  12.   <menu name="Tag Folder" class="folder" icon="${var:ICON_DIR}MainIcon.ico">
  13.     <!-- Submenus -->
  14.     <menuitem name="Tag as &quot;Currently watching&quot;" class="folder" icon="${var:ICON_DIR}CurrentlyWatching.ico" action="execute" console="false" multiple="N" workdir="C:\">
  15.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;CurrentlyWatching&quot;
  16.     </menuitem>
  17.     <menuitem name="Tag as &quot;Ongoing release&quot;" class="folder" icon="${var:ICON_DIR}WorkInProgress.ico" action="execute" console="false" multiple="N" workdir="C:\">
  18.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;WorkInProgress&quot;
  19.     </menuitem>
  20.     <menuitem name="Tag as &quot;Freshly completed&quot;" class="folder" icon="${var:ICON_DIR}NewlyCompleted.ico" action="execute" console="false" multiple="N" workdir="C:\">
  21.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;NewlyCompleted&quot;
  22.     </menuitem>
  23.     <menuitem name="----"></menuitem> <!-- Separator -->
  24.     <menuitem name="Tag as &quot;Favorite&quot;" class="folder" icon="${var:ICON_DIR}Favorite.ico" action="execute" console="false" multiple="N" workdir="C:\">
  25.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Favorite&quot;
  26.     </menuitem>
  27.     <menuitem name="Tag as &quot;Top series&quot;" class="folder" icon="${var:ICON_DIR}Star.ico" action="execute" console="false" multiple="N" workdir="C:\">
  28.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Star&quot;
  29.     </menuitem>
  30.     <menuitem name="Tag as &quot;Keep an Eye on for later (High priority)&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOnHighPrio.ico" action="execute" console="false" multiple="N" workdir="C:\">
  31.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOnHighPrio&quot;
  32.     </menuitem>
  33.     <menuitem name="Tag as &quot;Keep an Eye on for later&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOn.ico" action="execute" console="false" multiple="N" workdir="C:\">
  34.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOn&quot;
  35.     </menuitem>
  36.     <menuitem name="Tag as &quot;Keep an Eye on for later (Low priority)&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOnLowPrio.ico" action="execute" console="false" multiple="N" workdir="C:\">
  37.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOnLowPrio&quot;
  38.     </menuitem>
  39.     <menuitem name="Tag as &quot;Not interested&quot;" class="folder" icon="${var:ICON_DIR}NotInterested.ico" action="execute" console="false" multiple="N" workdir="C:\">
  40.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;NotInterested&quot;
  41.     </menuitem>
  42.     <menuitem name="----"></menuitem> <!-- Separator -->
  43.     <menuitem name="Tag as &quot;Fluff&quot;" class="folder" icon="${var:ICON_DIR}SweetFluff.ico" action="execute" console="false" multiple="N" workdir="C:\">
  44.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;SweetFluff&quot;
  45.     </menuitem>
  46.     <menuitem name="Tag as &quot;Sweet, sweet Yuri!&quot;" class="folder" icon="${var:ICON_DIR}Yuri.ico" action="execute" console="false" multiple="N" workdir="C:\">
  47.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Yuri&quot;
  48.     </menuitem>
  49.     <menuitem name="----"></menuitem> <!-- Separator -->
  50.     <menuitem name="Tag as &quot;A/V main folder&quot;" class="folder" icon="${var:ICON_DIR}AVMain.ico" action="execute" console="false" multiple="N" workdir="C:\">
  51.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;AVMain&quot;
  52.     </menuitem>
  53.     <menuitem name="----"></menuitem>
  54.     <menuitem name="Remove tag" class="folder" icon="X:\icons\Trash.ico" action="execute" console="false" multiple="N" workdir="C:\">
  55.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;del&quot;
  56.     </menuitem>
  57.   </menu>
  58. </config>

As you can see, it defines the main menu with its icon and then several submenu entries with their own icons. Also, it’s not calling my modified version of Umlüx’ Powershell script, but a Visual Basic Script instead. The reason for this shall be explained in point 4., for now we only care about the menu structure.

Looking at the variables defined on top of the XML data, ICON_DIR and INSTALL_DIR, they reference an icon folder and a program folder, both in %APPDATA%\changeicon\, so I put that into the current users’ profile folder. The icons to be used have to be put into the ICON_DIR, the scripts and an unlocker program are to be put into INSTALL_DIR. That doesn’t affect KuShellExt itself though, you can install that anywhere.

Note that while KuShellExt is loaded, you can edit its configuration on the fly. The library will detect changes made to it, and reload the updated configuration automatically, so you don’t have to unload and load the .dll when making changes to the menus or icon names.

Now, we still need the scripts and [Unlocker 1.9.2] (this is just the .exe, without the adware that usually comes with version 1.9.2). As to why Unlocker is required, well, let’s talk about that and also give you the first script:

3. The timestamp issue and the Powershell script that does the tagging

Note: Windows XP or Vista users may have to install the Windows Management Framework Core package (including Powershell 2.0) first.

This whole solution will alter folder appearances by placing hidden desktop.ini files into them. Writing such a file will alter the folders’ modification time though, or in short it’s “mtime”. And that’s bad if you have software that relies on that piece of meta data. One typical example would be backup software, that decides whether files have to be backupped based on the mtime.

In my own case, the backup problem applies to my rsync backup system, but there’s even more; I’ve written myself a Perl script that walks through my entire video folder, generating an HTML report out of the data, sorted by “latest first”. That way, I can see what movies or series have been added or modified recently. That script also depends on the mtime, so having it change when tagging folders is not acceptable!

Time to look at Umlüx’ script, or rather my modified version of it, changeicon.ps1:

changeicon.ps1, expand/collapse source code
  1. # Change Icon v0.815 Karl Veratschnig 2017
  2. # Modified by Michael Lackner
  3. #
  4. # Licensed under the GNU General Public license version 3.0 with express
  5. # permission by Karl Veratschnig.
  6. #
  7. # usage: 
  8. # .\changeicon.ps1 *dir_path* *icon*
  9. # i.e. ./changeicon.ps1 c:\testfolder Favorite
  10. #
  11. # run from anywhere:
  12. # Powershell.exe set-executionpolicy remotesigned -File "path to ps1"
  13. #
  14. # use "del" icon to revert folder to normal
  16. ###
  17. # Update-ExplorerIcon refreshes icons in Windows Explorer by rebuilding its icon cache
  18. # Written by Idera,
  19. #
  20. ###
  21. function Update-ExplorerIcon {
  22.   [CmdletBinding()]
  23.   param()
  25.   $code = @'
  26.   private static readonly IntPtr HWND_BROADCAST = new IntPtr(0xffff); 
  27.   private const int WM_SETTINGCHANGE = 0x1a; 
  28.   private const int SMTO_ABORTIFHUNG = 0x0002; 
  30.   [System.Runtime.InteropServices.DllImport("user32.dll", SetLastError=true, CharSet=CharSet.Auto)]
  31.   static extern bool SendNotifyMessage(IntPtr hWnd, uint Msg, UIntPtr wParam,IntPtr lParam);
  33.   [System.Runtime.InteropServices.DllImport("user32.dll", SetLastError = true)] 
  34.   private static extern IntPtr SendMessageTimeout ( IntPtr hWnd, int Msg, IntPtr wParam, string lParam, uint fuFlags, uint uTimeout, IntPtr lpdwResult ); 
  36.   [System.Runtime.InteropServices.DllImport("Shell32.dll")] 
  37.   private static extern int SHChangeNotify(int eventId, int flags, IntPtr item1, IntPtr item2);
  39.   public static void Refresh() {
  40.     SHChangeNotify(0x8000000, 0x1000, IntPtr.Zero, IntPtr.Zero);
  41.     SendMessageTimeout(HWND_BROADCAST, WM_SETTINGCHANGE, IntPtr.Zero, null, SMTO_ABORTIFHUNG, 100, IntPtr.Zero); 
  42.   }
  43. '@
  44.   Add-Type -MemberDefinition $code -Namespace MyWinAPI -Name Explorer 
  45.   [MyWinAPI.Explorer]::Refresh()
  46. }
  48. ###
  49. # User-configurable block
  50. ###
  51. $iconpath = "$env:APPDATA\icons\"           # Where icon files reside
  52. $installpath = "$env:APPDATA\bin\"          # Where scripts & binaries reside
  53. $unlocker = "unlocker.exe"                  # Name of the handle unlocker to use
  54. $ulparams = "/S"                            # Unlock parameter to supply to the unlocker
  56. ###
  57. # Non-user-configurable block / main program
  58. ###
  59. $folder = Get-Item $args[0]                 # Get folder from arguments
  60. & $installpath$unlocker "$folder" $ulparams # Remove open handles from folder
  61.                                             # WARNING: If files inside *are* open in
  62.                                             # other programs, their behavior might
  63.                                             # become unstable. Data loss is possible!
  64. $icon = $args[1]                            # Get icon name from arguments
  65. $olddate = $folder.LastWriteTime            # Get modification time stamp (mtime)
  66. $folder.attributes="Normal"                 # Reset folder attributes
  67. if (Test-Path "$folder\desktop.ini") {      # Check for existing desktop.ini
  68.   & $installpath$unlocker "$folder\desktop.ini" $ulparams # Unlock it, just to make sure
  69.   Remove-Item "$folder\desktop.ini" -Force    # Delete it if present
  70. }
  71. if($icon -ne "del") {                       # If op is to tag, not to delete...
  72.   $stream = [System.IO.StreamWriter] "$folder\desktop.ini" #... build new desktop.ini
  73.   $stream.WriteLine("[.ShellClassInfo]")            # Category
  74.   $stream.WriteLine("IconFile=$iconpath\$icon.ico") # Icon file
  75.   $stream.WriteLine("IconIndex=0")                  # Icon #0 in file
  76.   $stream.WriteLine("IconResource=$iconpath\$icon.ico,0") # Icon #0 in file (for modern OS)
  77.   $stream.WriteLine("Infotip=$icon")                # Info tip set to icon name
  78.   $stream.WriteLine("ConfirmFileOp=0")              # Disable special folder handling
  79.   $stream.WriteLine("TimeStamp=$olddate")           # Remark time stamp in the file for
  80.                                                     # safety/recovery purposes
  81.   $stream.close()                                   # Close file
  82.   $folder.attributes="ReadOnly"                     # Set folder to RO to enable special
  83.                                                     # desktop.ini handling
  84.   (Get-Item "$folder\desktop.ini").attributes = "Hidden,System" # Set desktop.ini
  85.                                                     # attributes to Hidden+System, also
  86.                                                     # to enable special folder handling
  87. } else {                                    # If op is to delete, not to tag...
  88.   if (Test-Path "$folder\desktop.ini") {      # Check whether desktop.ini really exists
  89.     & $installpath$unlocker "$folder\desktop.ini" $ulparams # Unlock it, just to make sure
  90.     Remove-Item "$folder\desktop.ini" -Force    # Delete it if present
  91.   }
  92. }
  94. Update-ExplorerIcon # Rebuild Windows Explorers' icon cache
  96. Start-Sleep -s 2    # This is to work around a race condition against writes/flushes on
  97.                     # some systems, so we give the system a little bit of time for its
  98.                     # "beauty sleep" before resetting the mtime.
  100. $folder.LastWriteTime = Get-Date $olddate   # Reset original mtime stamp


As you can see, the mtime is first being read from the folder by using $folder.LastWriteTime, and then reset to that value after writing the desktop.ini to the target directory. Also, it calls unlocker.exe /S on that folder to unlock it first, no matter whether there are open files within that folder or anything. This is done to avoid the dangerous situation, where the code would write the desktop.ini, but would then fail to update the folders’ timestamp due to open handles on the directory. Often this would also be caused by Windows Explorer itself, especially if you have open subfolders when tagging.

Forcefully unlocking the folder first deals with that problem. Please keep in mind that doing this may make programs relying on their open handles to behave in an undefined way however. Loss of data in that folder could be possible, e.g. if a text editor loses its handle for writing to a file in that folder.

In case something still goes wrong, the mtime is also being remarked in the desktop.ini file, in the value TimeStamp.

Here’s a sample .ini, as generated by changeicon, the format of the timestamp depends on your locale:

  1. [.ShellClassInfo]
  2. IconFile=C:\Documents and Settings\USERNAME\Application Data\changeicon\icons\WorkInProgress.ico
  3. IconIndex=0
  4. IconResource=C:\Documents and Settings\USERNAME\Application Data\changeicon\icons\WorkInProgress.ico,0
  5. Infotip=WorkInProgress
  6. ConfirmFileOp=0
  7. TimeStamp=09/22/2017 08:59:08

Now, there is one cosmetic issue left here…

4. Hiding that ugly terminal window

You can try to hide the window that pops up when calling a Powershell script, but it never really works reliably. So, are we going to tolerate that thing popping up every time we tag a folder? As if! That’s where the VBScript code comes in. This is an old trick I’ve used to run cmd batch scripts in a hidden way before, and this also works for calling Powershell scripts. I call this changeicon.vbs:

changeicon.vbs, expand/collapse source code
  1. ' Declare variables and create shell object:
  2. Dim AppData, WinDir, PowerShell, arg0, arg1
  3. Set WshShell = Wscript.CreateObject("WScript.Shell")
  5. '''
  6. ' User-configurable part (file/folder locations)
  7. '''
  8. AppData = wshShell.ExpandEnvironmentStrings("%APPDATA%")
  9. WinDir = wshShell.ExpandEnvironmentStrings("%WINDIR%")
  10. PowerShell = WinDir & "\system32\WindowsPowerShell\v1.0\powershell.exe"
  12. '''
  13. ' Non-user-configurable part
  14. '''
  15. ' Pull command line arguments into variables:
  16. arg0 = WScript.Arguments.Item(0)
  17. arg1 = WScript.Arguments.Item(1)
  18. ' Execute program:
  19. WshShell.Run """" & PowerShell & """ """ & "-File" & """ """ & AppData & "\changeicon\bin\changeicon.ps1" & """ """ & arg0 & """ """ & arg1 & """", 0
  20. Set WshShell = Nothing


5. Actually running that Frankenstein solution

In essence, you need to do the following:

  1. Pick a folder for changeicon.ps1, changeicon.vbs and unlocker.exe, and put those three files into it.
  2. Pick a folder for your icons, and place all your desired icons into it, no subfolders!
  3. Edit changeicon.ps1 and changeicon.vbs and change the install/icon paths.
  4. Install KuShellExtension and run its DLL hook script install.cmd.
  5. Edit KuShellExts’ settings.xml to reflect your menu structure and the corresponding menu icons and commands to execute. Better don’t delete all the comments in that file, the documentation can be pretty helpful at times.

Unfortunately I can’t share the icons I’ve created because they’re based on Microsofts’ icons, but you can easily find icons online or make your own with Microangelo or IcoFX. Both are commercial software for Windows, but you could also use the Gimp for that.

6. Enjoy tagging folders

Folders tagged with changeicon

Folders tagged with changeicon

With that, it’s much, much easier to keep track of things and to not forget what kind of stuff (*cough* tons of Anime *cough*) I still need to watch or keep an eye on for later.

But it’s not limited to that; You could use tagged folders for pretty much anything, like designating them to specific purposes or use them for document or work classification, whatever.

It’s interesting that even Windows 10 still can’t do that via the GUI by default by now…

Anyway, thanks fly out to [Umlüx] for writing the most important part at the core of this mess, the Powershell script, and also to [Idera] for the icon refreshing code I grabbed from their site! Also, if you want Umlüx’ modern solution for Windows 7+, which is based on pure Powershell code and Registry entries, you’d need to contact him directly. You may wish to do so, if you don’t need XP or Vista, because then you wouldn’t need to rely on the KuShellExtension anymore.

Happy tagging!


[1] Logo based on the Windows 10 Custom Folder Icons Pack made by Terraromaster

Jul 072017
Nekopara Vol.3 logo

1. Introduction

Of course I would never play something like Nekopara *cough*, so this is just a post describing a technical solution to a compatibility problem! Ok?! Good.

Yeah, it’s another one of those “something broke on XP / XP x64, so let’s fix it” articles. I’ve already been pla…  eh.. investigating Nekopara Volumes 0, 1 and 2, and while the developer claims it needs Windows Vista or higher, those titles worked just fine on XP and XP x64. The final Volume 3 however broke.

I wondered why, given they’re all pretty similar, so I started unpacking the .exe files, looking for information. What I found in the meta data was that Vol.0-2 have been using the TVP(Kirikiri) or maybe the forked [Kirikiri Z] game scripting engine, whereas Vol.3 swapped that for the [Ares CatSystem2] engine, for whatever reason. My assumption would be, that the CatSystem2 thingy was actually built for Vista+ for real, thus breaking XP compatibility. Plus, some other minor components are broken as well (some installers, patches, etc., just like the older volumes).

Now, I’ve already been talking to a guy called UncleVasya / Oleg Ovcharenko, who built a [stub DLL solution] for games based on the Clausewitz Engine (Europa Universalis 4, Hearts of Iron 4, Crusader Kings 2 and finally Stellaris), making it work on XP. It’s pretty similar to the XCOM hacks[1][2]. So I asked him about this one as well, and with quite some work and some additional (important) hints from him regarding the Steam version, I managed to make it run!

So, first things first: Thanks Oleg, you’re doing great work! :)

I will now show you how to make this visual novel / game work on XP x64 and XP, both for the slightly trickier Steam version (whether you choose to play the censored or the uncensored version doesn’t matter, the corresponding patch will be discussed as well), as well as the normal version.

Note: All screenshots in this post are 8-bit (256 color) PNG files. They may look a bit bad at times, but better than JPEG in the case of those specific images. Reason for not using truecolor PNG: 8-bit saves a ton of bandwidth.

2. How to make the non-Steam version work on XP / XP x64

Software required:

  1. [Nekopara Vol.3]
  2. [7-zip] archiver
  3. NTCore [CFF Explorer] (optional; only needed for patches)
  4. Olegs’ [patcher]

2a. The main game

First, buy the game and download it. Do not pirate it! You suck if you do (I actually fooled around with a pirated version as well, but only after buying the game). When running the installer, you’ll notice that it already breaks early on after invoking the launcher:

Nekopara Vol.3s' installer already breaks

Nekopara Vol.3s’ installer already fails to execute on XP

As you can see, it calls InitializeCriticalSectionEx(), which is a newer, Vista+ version of InitializeCriticalSection(), see the MSDN[1][2] for details. Since the new version works differently, you can’t just hex edit your way out of this one.

First, unpack Olegs’ patcher to some subdirectory of your choice. Then, unpack the Nekopara Vol.3 installer (the .exe file) into a subfolder using 7-zip, and look for a file called INSTALL.exe. Copy that file into the directory where Olegs patcher resides, so where files like xp_EU4_1.21.cmd and xp_Stellaris_1.6.cmd can be found.

Since the scripts from Oleg aren’t made for hacking our files, we’ll write a new one for this, let’s call it xp_installer.cmd. Edit that with a text editor, and add the following lines:

rundll32.exe zernel32.dll,PatchFile INSTALL.exe

Make sure xp_installer.cmd, zernel32.dll and the INSTALL.exe from Nekopara are in the same directory, then execute xp_installer.cmd. either by just double clicking it, or by opening a cmd terminal and by running it from there. Like this (you don’t need to run the extra commands, they’re just there to show you more information):

Olegs' patch doing its magic on INSTALL.exe

Olegs’ patch doing its magic on INSTALL.exe!

After that, rename your original INSTALL.exe in the directory where you unpacked the Nekopara Vol.3 installer, creating a backup file. Copy the following files from the patcher directory back to the installer directory: INSTALL.exe, zernel32.dll, z3d9.dll, zs2_32.dll and normaliz.dll. The “z” files are now implementing the missing functions, while redirecting all the others to the real Windows libraries like kernel32.dll, d3d9.dll, ws2_32.dll etc.

You don’t need to repack anything, just run INSTALL.exe directly, and you’ll no longer be greeted with an error message, but with this:

The installer works now

The installer works now, great

Install the game to a directory of your choice. Now, if you click the NEKOPARAvol3.exe in the directory where the game was installed, the same launcher comes up again, but now it allows you to configure and play the actual game instead of installing it…

Nekopara Vol.3s' launcher after installation

Nekopara Vol.3s’ launcher after installation

…or does it? Well, the “System settings” part’ll work, yes, but when clicking that alluring “Start” button, you’ll run into yet another wall:

Nekopara still won't execute due to GetTickCount64()

What now? GetTickCount64(), that’s what.

Guess which function call doesn’t exist on XP? See the MSDN[1][2] again. GetTickCount64() really is an improvement over GetTickCount(), but still, XP simply doesn’t have this either. As you can see from the title bar, the offending binary is cs2.exe, which is the actual game. We can get rid of the issue by using Olegs’ patcher again, so it’s the same process as with INSTALL.exe, just use this script instead, call it xp_cs2.cmd or something:

rundll32.exe zernel32.dll,PatchFile cs2.exe

Again, in case something goes wrong, rename your original cs2.exe before copying back the patched version with its .dll files. After copying back, you can run the game either by invoking cs2.exe directly, or by launching it from the NEKOPARAvol3.exe launcher:

Running the non-Steam version of Nekopara Vol.3 on XP x64

Running the non-Steam version of Nekopara Vol.3 on XP x64 (click to enlarge)

2b. Making patches work as well

Patches are essentially also just self-extracting archives that execute a launcher after unpacking. We’ll discuss the patch 11 in this case. Running it will produce a different kind of error (people who know the content restoration patches for the Steam version may have seen this error as well):

Nekopara Vol.3 patch failure

Nekopara Vol.3 patch failure, due to it not being “a valid Win32 application”.

This error means that the header of the binary is asking for a more modern platform. This may make sense, if the program really calls modern functions, but you know, there are modern applications that don’t ask for it and then fail with calls to things like GetTickCount64(), and there are programs which ask for a modern platform without ever having an actual need for it. The patchers are in the latter category of programs.

Unpack the patcher nekopara3_v11_update.exe using 7-zip, and look for a file called updater.exe. Create a backup copy of it, then open this file in NTCores’ CFF Explorer, and click on the “Optional Header” part. You’ll see something like this, I’ve marked the relevant lines with some red blocks for you:

updater.exe in CFF Explorer

updater.exe in CFF Explorer (click to enlarge)

The marked fields show values like 0006 and 0000, as you can see. The significant number is the last or rightmost, so 6 and 0. This corresponds to the platform target Windows NT 6.0, or in other words: Windows Vista. Just rewrite that to show the following numbers, then save the file:

Patch the header to NT 5.1

Patch the header to NT 5.1 (click to enlarge)

NT 5.1 (0005, 0001) equals Windows XP. Note that the kernel versions 5.0 mean Windows 2000, 5.2 means Server 2003 or XP x64 (slightly more modern). Again, no need to repack anything, just save the file after the modifications have been made and execute updater.exe afterwards, you should be getting this:

Nekopara Vol.3 non-Steam patcher working on XP

And here we have a working patcher (click to enlarge)

Yay! And now, for the Steam version of Nekopara Vol.3…

3. How to make the Steam version work on XP / XP x64

Software required:

  1. [Nekopara Vol.3] on Steam (a censored version)
  2. [Content restoration patch] (optional; only required if you have to do perverted things to the cat girls)
  3. at0ms’ [Steamless]
  4. A Windows Vista or newer machine (needed to run Steamless, can be a virtual machine)
  5. [7-zip] archiver
  6. NTCore [CFF Explorer] (optional; only needed for the content restoration patch)
  7. Olegs’ [patcher]

3a. The main game

First, buy the game on Steam and download it. If you really need the uncensored version (you probably do, heh?), buy the content restoration patch at Denpasoft and download that as well. Of course, running the game as-is won’t work, otherwise we wouldn’t need this article in the first place:

The Steam version of Nekopara Vol.3 breaks on XP as well of course

The Steam version of Nekopara Vol.3 breaks on XP as well of course, due to GetTickCount64() call, a newer and better version of GetTickCount(), see MSDN[1][2].

Now, what I didn’t get at first was that patching the Steam versions’ NEKOPARAvol3.exe can never work out of the box. The reason is, that the offending function calls aren’t plainly there for us to see – the actual game binary cs2.exe is encrypted and packed into a SteamStub binary as its payload data. This is a part of the Steamworks DRM system wrapping our program up.

To be able to patch it, we (unfortunately) need to crack its cryptographic DRM protection system first. Now, let me say this again: I do not condone piracy. Don’t fucking crack and distribute this game. You’re an ass if you do. Removing the DRM part is only being done so we can fix the game on XP, keep that in mind!

Well, let’s start; First, boot up a Vista or newer Windows, and install Steamless on it. I actually tried to compile Steamless for XP, but this is .Net 4.5.2 stuff. To make it work on .Net 4.0 would require modifications of its build files / source code, which is a bit over my head right now. So we’re stuck with needing a modern Windows OS to do this. Copy the problematic NEKOPARAvol3.exe from your Steam game installation directory over to that machine, or just install Steam and the game on the modern Windows OS as well (which is what I actually did).

Launch Steamless, open that .exe and decrypt / unpack it, Steamless will leave your binary alone, and create a new, fixed one, so you don’t need to create a manual backup copy:

Steamless cracking NEKOPARAvol3.exe

Steamless cracking NEKOPARAvol3.exe (click to enlarge)

Copy the fixed file back to XP, and rename it back to NEKOPARAvol3.exe. Create a backup of the original .exe in your Steam game installation directory, while you’re at it.

Unpack Olegs’ patcher in a directory of your choice, and move the NEKOPARAvol3.exe there as well, that’s where files akin to xp_EU4_1.21.cmd and xp_Stellaris_1.6.cmd can be found. Since those patcher scripts aren’t targeted at Nekopara Vol.3, we’ll write our own, call it xp_neko_3.cmd or something, open it in a text editor and enter the following lines:

rundll32.exe zernel32.dll,PatchFile NEKOPARAvol3.exe

Make sure that NEKOPARAvol3.exe, zernel32.dll and xp_neko_3.cmd are together in the same folder, then execute xp_neko_3.cmd either by double-clicking it, or by opening a cmd terminal and executing it from there. Like this:

Olegs' patcher handling the decrypted NEKOPARAvol3.exe

Olegs’ patcher handling the now-decrypted Steam version of NEKOPARAvol3.exe

Copy the fully fixed .exe back into the Steam game installation directory, together with the patchers’ stub libraries zernel32.dll, z3d9.dll, zs2_32.dll and normaliz.dll, which will handle the functions usually missing on XP.

Now, run the game either by executing NEKOPARAvol3.exe, or by launching it from within Steam, and you should be greeted with something like this:

Nekopara Vol.3 running on XP x64 in its Steam version

Nekopara Vol.3 running on XP x64 in its Steam version (click to enlarge)

Great (or something)!

Please be aware that if the binary is ever overwritten by Steam because of some update or whatever, you have to re-do the procedure, meaning the Steamless unpacking plus applying Olegs’ patch. If the game terminates without any error when launched from within Steam, try to run NEKOPARAvol3.exe directly instead, and you’ll see the error messages – Steam tends to suppress them.

3b. The content restoration patch (this also applies to the patches for Nekopara Vol.1 and Vol.2)

So you want to lewd the cat girls? Perverted! Plus, Windows XP / XP x64 won’t let you, because the patch is asking for a newer platform (despite not actually requiring it though):

Nekopara Vol.3 content restoration patch failure on XP

Nekopara Vol.3 content restoration patch failure on XP, due to the patch not being “a valid Win32 application”.

But if you absolutely have to, here’s how. Unpack the nekopara_vol3_Steam_R18DLC.exe you bought and downloaded from Denpasoft using 7-zip. Look for the file SteamPatch.exe, and open it in CFF Explorer:

The Nekopara Vol.3 Content restoration patchs' SteamPatch.exe in CFF Explorer

The Nekopara Vol.3 Content restoration patchs’ SteamPatch.exe in CFF Explorer

Now, this is similar to the procedure described for updater.exe for a non-Steam versions’ patch. The significant (rightmost) numbers in the fields where it days 0006 and 0000 represent Windows NT 6.0, or in other words Windows Vista. Since the patcher doesn’t really need any Vista-specific functions, we’ll just fix the header that is currently asking for a NT 6.0 platform as follows:

The Nekopara Vol.3 Content restoration patchs' SteamPatch.exe in CFF Explorer, fixed for XP

Change the fields to 5.1 (0005 and 0001 respectively) to have it check for XP+ instead, and it’s fixed!

Save the file after modifying it. Just like for the non-Steam version patches, there is no need to repack anything. Just run updater.exe directly, and you’ll now get this:

The Nekopara Vol.3 content restoration patch for the Steam version working on XP x64

The Nekopara Vol.3 content restoration patch for the Steam version working on XP x64. Because you’re in it for the Hentai.

There you go, pervert! You now have the fully restored version of Nekopara Vol.3 on Steam, running on XP or XP x64.

And last but not least: Thanks again, Oleg! I made you touch some weird shit, but you still fixed it and gave me the right ideas about the Steam version as well, yay! ;)

4. Bonus feature: How to make Mechwarrior Online work on XP / XP x64 after their launcher upgrade

While entirely unrelated to the weird Japanese shit above, I’ll just mention this here as well, because it doesn’t deserve its own post, given the simplicity of the “solution”; Piranha Games decided to give Mechwarrior Online (MWO) a new game launcher called “MWO Portal”, that is now built with .Net 4.5.2, just like Steamless, breaking it on XP. Mind you, the game itself would still work just fine, even the 64-bit version on XP x64.

The new MWOPortal launcher

Windows XP / XP x64 users will likely never see this launcher work on their OS (Unless ExtendedXP really takes off, it’s pretty good already, but yeah).

Since hacking .Net 4.5 stuff to run on .Net 4 / .Net 4 CP is not something I can do yet, MWO would be gone from all XP machines. There is an easy fix for this though:

Get the game on Steam! The Steam version doesn’t include the launcher, as Steam itself is handling both the execution and the updates of MWO. Without the launcher, MWO still works just fine! :)

May 312017
HakuNeko logo

1.) What for?

Usually, porting my favorite manga ripper [HakuNeko][1] would involve slightly more exotic target platforms like [FreeBSD UNIX]. This changed with version 1.4.2 however, as this version – the most current at the time of writing – would no longer compile on Windows machines due to some issues with its build toolchain. And that’s the most common platform for the tool!

This is what the lead developer had to say about the issue while even suggesting the use of [FMD] instead of HakuNeko on Windows:

“The latest release does not compile under windows due to some header include contradictions of sockets […]”
    -in a [comment][1] to HakuNeko ticket #142 by [Ronny Wegener], HakuNeko project leader

[1] Edit: Links have been fixed, as the HakuNeko project has now been moved to HakuNeko Legacy due to the development of its replacement, [HakuNeko S].

Normally I wouldn’t mind that much and keep using 1.4.1 for now, but unfortunately this is not an option. Quite a few Manga websites have changed by now, breaking compatibility with the previous version. As this is breaking most of HakuNekos’ functionality for some important sites, it became quite unusable on Windows, leaving Linux as the only officially supported platform.

As using virtual machines or remote VNC/X11 servers for HakuNeko proved to be too tedious for me, I thought I’d try to build this by myself. As the MSYS2/MinGW Windows toolchain seemed to be broken for 1.4.2, I tried – for the very first time – to cross-compile on Linux, choosing CentOS 7.3 x86_64 and MinGW32 4.9.3 for the task. This was quite the challenge, given that HakuNeko comes completely unprepared for cross-compiling.

2.) First, the files

Took me many hours / days to get it done – 100% statically linked too – and it’s finished now! What I won’t provide is an installer (don’t care about that), but here are my v1.4.2 builds for Windows:

As of today, those versions have been tested successfully on the following operating systems:

  • Windows XP Professional SP3 / POSReady2009
  • Windows XP Professional x64 Edition SP2 w. Server 2003 updates
  • Windows Server 2003 R2 x64 SP2
  • Windows Vista Enterprise x64 SP2
  • Windows 7 Professional x64 SP1
  • Windows 10 Professional x64 build #1607

Please be aware that not all of the functionality has been tested by me, just a few downloads that wouldn’t have worked with 1.4.1, a few more random downloads here and there, plus chapter-wise .cbz packaging. It’s looking pretty good I think. Here are some sample screen shots as “proof” of it working (click to enlarge):

HakuNeko 1.4.2 downloading "Kobayashi-san Chi no Maid Dragon" on XP x64

HakuNeko 1.4.2 downloading “Kobayashi-san Chi no Maid Dragon” on XP x64 (Note: I own that Manga in paper form)


3.) What has been done to make this work?

3a.) Initial work:

First of all, cross-compiling is a bottomless, hellish pit, a horrible place that you never want to enter unless a.) The build toolchain of the software you wanna compile is very well prepared for it or b.) you really have/want to get your hands on that build or c.) you hate yourself so much you have to hurt yourself or d.) you actually enjoy c.).

The reasons for choosing cross-compiling were that Ronny Wegener had said, that the MSYS2/MinGW32 build would fail on Windows, plus it would require GCC version 5.3 to link with the bundled, pre-built static libraries (OpenSSL, cURL, wxWidgets).

So I thought it would be more likely to work if I were to run my own MinGW setup on Linux, not relying on the bundled stuff but linking against the libraries that come with MinGW32 4.9.3 on my platform of choice – CentOS 7.3 Linux.

One exception was the GUI library wxWidgets 3.0.2 that I had to cross-compile and statically link by myself as well, but luckily, that’s easy despite its size. wxWidgets is one piece of software that does come well-prepared for cross-compiling! In my case, that made it as simple as this (parallel compile with 6 CPUs):

$ ./configure --prefix=/usr/local/i686-w64-mingw32 --host=i686-w64-mingw32 --build=x86_64-linux \
 --enable-unicode --with-expat --with-regex --with-opengl --with-libpng --with-libjpeg --with-libtiff \
 --with-zlib --with-msw --enable-ole --enable-uxtheme --disable-shared
$ make -j6
# make install

3b.) HakuNeko build toolchain / Makefile modifications for cross-compiling:

HakuNeko is much harder, and I don’t even remember half of what I did, but most of it was manually editing the ./Makefile after $ ./configure --config-mingw32 would have produced something rather broken.

Let’s get to it, file paths are relative to the source root. First, edit the following parts of the ./Makefile (you need to look for them in different places of the file). First, the PREFIX, should be in the bottom half of the file:

PREFIX = /usr/local/i686-w64-mingw32/

CC and the CFLAGS:

CC = i686-w64-mingw32-g++
CFLAGS = -c -Wall -O2 -std=c++11 \
 -I/usr/local/i686-w64-mingw32/lib/wx/include/i686-w64-mingw32-msw-unicode-static-3.0 \
 -I/usr/local/i686-w64-mingw32/include/wx-3.0 -D_FILE_OFFSET_BITS=64 -D__WXMSW__ -mthreads \
 -DCURL_STATICLIB -I/usr/i686-w64-mingw32/sys-root/mingw/include

Add -DPORTABLE, if you want to build the portable version of HakuNeko.

Then, the Windows resource compiler, controlled by RC and RCFLAGS:

RC = /usr/bin/i686-w64-mingw32-windres
RCFLAGS = -J rc -O coff -F pe-i386 -I/usr/i686-w64-mingw32/sys-root/mingw/include \

And finally, the static linking part, which is the hardest stuff to get done right, LD, LDFLAGS and LDLIBS:

LD = i686-w64-mingw32-g++
LDFLAGS = -s -static -static-libgcc -static-libstdc++ -mwindows -DCURL_STATICLIB
LDLIBS = -L/usr/local/i686-w64-mingw32/lib   -Wl,--subsystem,windows -mwindows \
 -lwx_mswu_xrc-3.0-i686-w64-mingw32 -lwx_mswu_webview-3.0-i686-w64-mingw32 \
 -lwx_mswu_qa-3.0-i686-w64-mingw32 -lwx_baseu_net-3.0-i686-w64-mingw32 \
 -lwx_mswu_html-3.0-i686-w64-mingw32 -lwx_mswu_adv-3.0-i686-w64-mingw32 \
 -lwx_mswu_core-3.0-i686-w64-mingw32 -lwx_baseu_xml-3.0-i686-w64-mingw32 \
 -lwx_baseu-3.0-i686-w64-mingw32 -L/usr/i686-w64-mingw32/sys-root/mingw/lib -lcurl -lidn -liconv \
 -lssh2 -lssl -lcrypto -lpng -ljpeg -ltiff -lexpat -lwxregexu-3.0-i686-w64-mingw32 -lz -lrpcrt4 \
 -lwldap32 -loleaut32 -lole32 -luuid -lws2_32 -lwinspool -lwinmm -lshell32 -lcomctl32 -lcomdlg32 \
 -ladvapi32 -lwsock32 -lgdi32

Took a while to find the libraries (and static library order!) necessary to satisfy all the dependencies properly.

If you need it, here is the modified Makefile I’ve used to cross-compile:

  • [HakuNeko Makefile] for cross-compiling HakuNeko 1.4.2 for Windows on CentOS 7.3 x86_64 Linux (needs statically linked & installed wxWidgets first).

3c.) Source code modifications:

However, something will still not be quite right, because some of the crypto libraries will provide the MD5 functions MD5_Init(), MD5_Update() as well as MD5_Final(), and those are already defined by HakuNeko itself. This will break the static linking, as redundant definitions won’t work. We’ll rely on the libraries (libcrypto, libssl), and comment the built-in stuff out in src/v7/v7.c:

void MD5_Init(MD5_CTX *c);
void MD5_Update(MD5_CTX *c, const unsigned char *data, size_t len);
void MD5_Final(unsigned char *md, MD5_CTX *c);


/* void MD5_Init(MD5_CTX *c);
 * void MD5_Update(MD5_CTX *c, const unsigned char *data, size_t len);
 * void MD5_Final(unsigned char *md, MD5_CTX *c);

On top of that, the configure system may have generated src/main.cpp as well as src/main.h. Those are bogus files, turning the entire tool into nothing but one large “Hello World” program. Plus, that’s hard to debug, as the binary won’t even output “Hello World” on a Windows terminal when it’s built as a GUI tool. I only found the issue when testing it with Wine on Linux. ;)

Please delete src/main.cpp and src/main.h before continuing.

Now, if you’re really lucky, you should be able to run something like $ make -j6 and watch everything work out nicely. Or watch it crash and burn, which is the much, much more likely scenario, given I’ve likely only given you half of what I did to the build tools.

Well, in any case, no need to run $ make install of course, just grab the binary build/msw/bin/hakuneko.exe and copy it off to some Windows machine, then try to run it. If you’ve built the portable version, you may wish to rename the file to hakuneko-portable.exe, just like the official developers do.

4.) The future

Let’s just hope that the developers of HakuNeko can get this fixed for versions >=1.4.3, because I really, really don’t want to keep doing this. It’s extremely painful, as cross-compiling is exactly the kind of living hell I heard a lot of people saying it is! I think it’s a miracle I managed to compile and run it at all, and it was so frustrating and tedious for somebody like me (who isn’t a developer).

The statement that this took “hours / days” wasn’t an exaggeration. I think it was something like 10-12 man hours of pure frustration to get it done. I mean, it does feel pretty nice when it finally works, but I wouldn’t bet on myself being able to do this again for future versions…

So please, make it work on Windows again, if possible, and keep HakuNeko cross-platform! It’s still my favorite tool for the task! Thanks! :)

Mar 012017
Notepadqq @ CentOS 6 Linux logo

It’s rather rare for me to look for a replacement of some good Windows software for Linux/UNIX instead of the other way around, but the source code editor [Notepad++] is one example of such software. The program gedit on the Gnome 2 desktop environment of my old CentOS 6 enterprise Linux isn’t bad, but it isn’t exactly good either. The thing I was missing most was a search & replace engine capable of regular expressions.

Of course, vi can do it, but at times, vi can be a bit hard to use, so I kinda looked for a Notepad++ replacement. What I found was [Notepadqq], which is basically a clone using the Qt5 UI. However, this editor is clearly made for more modern systems, but I still looked for a way to get it to compile and run on my CentOS 6.8 x86_64 Linux system. And I found one. Here are the most important prerequisites:

  • A new enough GCC (I used 6.2.0), because the v4.4.7 platform compiler won’t work with the modern C++ stuff
  • Qt5 libraries from the [EPEL] repository
  • git

First, you’ll want a new compiler. That part is surprisingly easy, but a bit time consuming. First, download a fresh GCC tarball from a server on the [mirrors list], those are in the releases/ subdirectory, so a file like gcc-6.3.0.tar.bz2 (My version is still 6.2.0). It seems Notepadqq only needs some GCC 5, but since our platform compiler won’t cut it anyway, why not just use the latest?

Now, once more, this will take time, could well be hours, so you might wanna do the compilation step over night, the last step needs root privileges:

$ tar -xzvf ./gcc-6.3.0.tar.bz2
$ cd ./gcc-6.3.0/
$ ./configure --program-suffix="-6.3.0"
$ make
# make install

And when you do this, please never forget to add a --program-suffix for the configuration step!  You might seriously fuck things up if you miss that! So double-check it!

When that’s finally done, let’s handle Qt5 next. I’ll be using a binary distribution to make things easy, and yeah, I didn’t just install the necessary packages, I got the whole Qt5 blob instead, too lazy to do the cherry picking. Oh, and if you don’t have it, add git as well:

# yum install
# yum install qt5* git

I assume # yum install qt5-qtwebkit qt5-qtwebkit-devel qt5-qtsvg qt5-qtsvg-devel qt5-qttools qt5-qttools-devel should also be enough according to the requirements, but I didn’t try that. Now, enter a free directory or one you generally use for source code and fetch the latest Notepadqq version (this will create a subfolder we’ll cd to):

$ git clone
$ cd ./notepadqq

After that, we need to make sure that we’ll be using the correct compiler and that we’re linking against the correct libraries that came with it (like*). To do that, set the following environment variables, assuming you’re using the bash as your shell (use lib/ instead of lib64/ folders if you’re on 32-bit x86):

$ export CC="gcc-6.3.0"
$ export CXX="g++-6.3.0"
$ export CPP="cpp-6.3.0"
$ export CFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export CXXFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export LDFLAGS="-L/usr/local/lib64/"

The C-related settings are probably not necessary as Qt5 stuff should be pure C++, but you’ll never know, so let’s play it safe.

With that we’re including and linking against the correct libraries and we’ll be using our modern compiler as well. Time to actually compile Notepadqq. To do that, we’ll still need to tell it where to find the Qt5 versions of the qmake and lrelease binaries, but luckily, we can solve that with some simple configuration options. So, let’s do this, the last step requires root privileges again, from within the notepadqq/ directory that git clone created for us:

$ ./configure --qmake /usr/bin/qmake-qt5 --lrelease /usr/bin/lrelease-qt5
$ make
# make install

Now, there are some weird linking issues that I never got fixed on CentOS (some developer please tell me how, I have the same crap when building x265!). Because of that we still can’t launch Notepadqq as-is, we need to give it an LD_LIBRARY_PATH to find the proper libraries at runtime. Let’s just create an executable launcher script /usr/local/sbin/ for that. Edit it and enter the following code:

LD_LIBRARY_PATH="/usr/local/lib64" /usr/local/bin/notepadqq "$@"

Use this as your launcher script for Notepadqq and you’re good to go with your Notepad++ replacement on good old CentOS 6.x:

Running Notepadqq on CentOS 6 Linux

Running the latest Notepadqq on CentOS 6 Linux with Qt5 version 5.6.1, state 2017-03-01

Now, let’s see whether it’s even that good actually… :roll:

Nov 242016
Broken Windows logo

[1] I know what I should do if a system service on Microsoft Windows starts crashing of course; Fixing it is the way to go! But sometimes you simply can’t, because the component causing a certain instability can’t be swapped out or updated. Now Windows services do have a mechanism for monitoring and restarting a service upon failure, but it seems that only works if the system gets an actual error code back from the service upon termination. But it doesn’t seem to work (at least for me) if the service just dies abnormally. Windows recognizes the service has stopped somehow of course, but the restart procedure just doesn’t kick in.

So I thought I’d do it myself, programmatically. And it’s actually pretty easy. I solved this with VBScript, Windows Batch and Mark Russinovichs’ pslist plus grep. So the prerequisites are:

  • Microsoft Windows (well, huh..)
  • MS Windows Script(ing) Host / VBScript, Windows should come with this preinstalled since Windows 2000.
  • [pslist]
  • [grep][src] (grep is optional, I used GNU grep 2.5.4 in this case, licensed under the [GPLv3+])

Make sure the pstools and grep are within your %PATH%, so Windows can find those .exe files. If you don’t want to use grep, you can also use Microsofts’ own find command, if your version of Windows has it.

I divided this into two small scripts. Since the main part is Batch, it might be problematic if you run it at very short intervals, checking for the services’ status, because you get a command window popping up on the desktop. Since most users wouldn’t want that, another script acts as a launcher, hiding the cmd.exe window so it’s run fully in the background without disturbing any potential users or administrators. The launcher looks like this, in my case it’s meant to watch over an Apache web server:

  1. Set WshShell = CreateObject("WScript.Shell")
  2. WshShell.Run chr(34) & "C:\Server\Scripts\monitor-httpd.bat" & Chr(34), 0
  3. Set WshShell = Nothing

And that script C:\Server\Scripts\monitor-httpd.bat we’re launching looks like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| grep httpd`) DO SET HTTPDSTATUS=%%I

A version relying on Microsoft find instead of GNU grep could look like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| find /I "httpd"`) DO SET HTTPDSTATUS=%%I

To get a services’ exact name, just launch services.msc from Start \ Run or run the command net start on a cmd terminal.

As you can see, this greps “httpd” from the process list and pushes its output into %%I and finally into %HTTPDSTATUS%. We have to use a FOR /F for that, as Windows has no way of pushing command outputs from subshells into shell variables like UNIX has (like e.g. var=`command` or var=$(command)). Then we check for the status of that variable. If it’s not defined, then the process http.exe was nowhere to be found! In that case we restart the associated system service (needs proper permissions!). If the variable is defined, we do nothing but unsetting it, since we can assume the service is operating normally. Or at the very least it’s running. ;)

You can automate that by using the Windows task scheduler:

Scheduling an Apache web server "watchdog"

Scheduling an Apache web server “watchdog” (German Windows)

Create a Schedule to your liking and you’re done! If you can afford the affected service to be down for 5 minutes and no longer, just run it every 4 minutes or so.

The solution shown above can easily be adapted to monitor and restart any Windows service you have, as long as the service isn’t fundamentally broken so that it wouldn’t even start up anymore. Also, you can do a lot more, like sending notification eMails with a command line mailer like [blat] when crashes do occur. Of course, this is only useful for services that crash rarely. If it dies every few minutes, you should reaaally fix it instead of just pushing the restart button all the time… ;)

And that’s that!

[1] © Mar.0007. Original Version for

Nov 192016
FreeBSD GMABoost logo

Recently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # 200 or # 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Jul 272016
x264 logo

Since I’ve been doing a bit of Anime batch video transcoding with x264 and x265 in the last few months, I thought I’d document this for myself here. My goal was to loop over an arbitrary amount of episodes and just batch-transcode them all at once. And that on three different operating systems: Windows (XP x64), Linux (CentOS 6.8 x86_64) and FreeBSD 10.3 UNIX, x86_64. Since I’ve started to split the work across multiple machines, I lost track of what was where and which machine finished what, and when.

So I thought, why not let the loop send me a small notification email upon completion? And that’s what I did. On Linux and UNIX this relies on the bash shell and the mailx command. Please note that I’m talking about [Heirloom mailx], not some other mail program by the same name! I’m mentioning this, because there is a different default mailx on FreeBSD, that won’t work for this. That’s why I put alias mailx='/usr/local/bin/mailx' in my ~/.bash_profile on that OS after installing the right program to make it the default for my user.

On Windows, my loops depend on my own [colorecho] command (you can replace that with cmds’ ECHO if you want) as well as the command line mailer [blat]. Note that, if you need to use SSL/TLS encryption when mailing, blat can’t do that. A suitable replacement could be [mailsend]. Please note, that mailsend does not work on Windows XP however.

In the x265 case, avconv (from the [libav] package) is required on all platforms. You can get my build for Windows [here]. If you don’t like it, the wide-spread [ffmpeg] can be a suitable drop-in replacement.

Now, when setting up blat on Windows, make sure to run blat -help first, and learn the details about blat -install. You need to run that with certain parameters to set it up for your SMTP mail server. For whatever reason, blat reads some of that data from the registry (ew…), and blat -install will set that up for you.

Typically, when I transcode, I do so on the elementary streams rather than .mkv files directly. So I’d loop through some source files and extract the needed streams. Let’s say we have “A series – episode 01.mkv” and some more, all the way up to “A series – episode 13.mkv”, then, assuming track #0 is the video stream…

On Windows:

FOR %I IN (01,02,03,04,05,06,07,08,09,10,11,12,13) DO mkvextract tracks "A series - episode %I.mkv" ^

On Linux/UNIX:

for i in {01..13}; do mkvextract tracks "A series - episode $i.mkv" 0:$i/video.h264; done

mkvextract will create the non-existing subfolder for us, and a x264 transcoding loop would then look like this on Windows:

expand/collapse source code
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps ^
 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f ^
 -2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats %I\v.stats -t 2 --no-fast-pskip ^
 --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o %I\pass1.264 & colorecho "Pass 1 ^
 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & ^
 "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps 24000/1001 --preset veryslow --tune animation ^
 --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats ^
 %I\v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o ^
 %I\pass2.264 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10) & echo !SERIES! ^
 transcoding complete | blat - -t "" -c "" -s "x264 ^
 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET SERIES="

Note that I always write all the iteration out in full here. That’s because cmd can’t do loops with leading zeroes in the iterator. The reason for this is that those source files usually have them in their lower episode numbers. If it wasn’t 01,02, … ,12,13, but 1,2, … ,12,13 instead, you could do FOR /L %I IN (1,1,13) DO. But this isn’t possible in my case. Even if elements need alphanumeric names like here,  FOR %I IN (01,02,03,special1,special2,ova1,ova2) DO, you still won’t need that syntax on Linux/UNIX because the bash can have iterator groups like for i in {{01..13},special1,special2,ova1,ova2}; do. Makes me despise the cmd once more. ;)


Ah, according to [this], you can actually do something like cmd /V /C "FOR /L %I IN (1,1,13) DO (SET "fI=00%I" & echo "!fI!:~-2")", holy shit. It actually works and gives you leading zeroes. :~-2 for 2 digits, :~-3 for three. Expand fI for more in this example. I mean, what is this even? Some number formatting magic? I probably don’t even wanna know… Couldn’t find any way of having several groups for the iterator however. Meh. Still don’t like it.

So, well, it’s like this on Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do nice -n19 x264 --fps \
24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f \
-2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats $i/v.stats -t 2 --no-fast-pskip \
--cqm flat --non-deterministic --demuxer lavf $i/video.h264 -o $i/pass1.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && nice -n19 \
x264 --fps 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid \
normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats $i/v.stats -t 2 --no-fast-pskip --cqm flat \
--non-deterministic --demuxer lavf $i/video.h264 -o $i/pass2.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x264 notification from $MACHINE" -r \
"" -c "" -S smtp-auth="login" -S \
smtp="" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \

The variable $MACHINE or %MACHINE%/!MACHINE! specifies the machines’ host name. This will be noted in the email, so I know which machine just completed something. $EPNUM – or %EPNUM%/!EPNUM! on Windows – is used for periodic updates on the shell. The output would be like “Pass 1 done for Episode 07/13 of AnimeX” in green on Windows and bold red on Linux/UNIX (just change the color to your liking).

Finally, $SERIES aka %SERIES%/!SERIES! would be the series’ name. So say, the UNIX machine named “BEAST” above is done with this loop. The email would come with the subject line “x264 notification from BEAST” and would read “AnimeX transcoding complete” in plain text. That’s all.

Please note, that cmd batch on Windows is extremely creepy. Every whitespace (especially the leading ones when doing multi-line like this for display) needs to be exactly where it is. The same goes for double quotes where you might think they aren’t needed. They are! Also, this needs delayed variable expansion once again, which is why we see variables like !EPNUM! instead of %EPNUM% and why it’s called in a subshell by running cmd /V /C.

On Linux/UNIX we don’t need to rely on some specific API like cmds’ SetConsoleTextAttribute() to print colors, as most terminals understand ANSI color codes.

And this is what it looks like for x265:


expand/collapse source code
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO avconv -r 24000/1001 -i %I\video.h264 -f yuv4mpegpipe ^
 -pix_fmt yuv420p -r 24000/1001 - 2>NUL | "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 ^
 --fps 24000/1001 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate ^
 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 ^
 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
 --pass 1 --slow-firstpass --stats %I\v.stats --sar 1 --range full -o %I\pass1.h265 & colorecho ^
 "Pass 1 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & avconv -r 24000/1001 -i ^
 %I\video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>;NUL | ^
 "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 --fps 24000/1001 -p veryslow --pmode ^
 --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 ^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2 --stats %I\v.stats --sar ^
 1 --range full -o %I\pass2.h265 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" ^
 10) & echo !SERIES! transcoding complete | blat - -t "" -c ^
 "" -s "x265 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET ^


expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do avconv -r 24000/1001 -i \
$i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice -19 x265 - --y4m \
-D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 \
--rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq \
5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 \
--slow-firstpass --stats $i/v.stats --sar 1 --range full -o $i/pass1.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && avconv -r \
24000/1001 -i $i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice \
-19 x265 - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid \
--bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd \
1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 \
--pass 2 --stats $i/v.stats --sar 1 --range full -o $i/pass2.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x265 notification from $MACHINE" -r \
"" -c "" -S smtp-auth="login" -S \
smtp="" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \

And that’s it. The loops for audio transcoding are simpler, as that part is so fast, it doesn’t need email notifications. Runs for minutes rather than days. When all is done, I’d usually fire up the MKVToolnix GUI, and prepare a mux for the first episode. There is a nice “copy command line to clipboard” function there when you click on “Muxing” after everything is set up. With that I can build another loop that muxes everything to final .mkv files. On Windows that part is more complicated if you want Unicode support, so I needed to create input files by using a generator I wrote in Perl for that, but that’s for another day… :)

Oh, and if you wanna ssh into your Linux or UNIX boxes from afar to check on your transcoders, consider launching them on a GNU screen] session. It’s immensely useful! Too bad it won’t work on the Windows cmd. :(

Jan 292016

WP comment nesting logoIt’s not like a lot of people are actively commenting on this weblog, but there are at least two posts which do have quite a few replies (well, for my standards), the one about using [48GB of RAM on an X58 chipset mainboard], and the other about [SSD TRIM, exFAT, EXT3/4, ASPI and UDF 2.5 on Windows XP / XP x64]. To make it possible for users to interact and discuss in a better fashion, I had enabled nested comments. However, the comments took too much horizontal space away, limiting the usable depth of nesting levels. The last (10th) comment would be like ~2cm narrow, and extremely hard to read.

Finally, I thought I should really fix that. So I found that the corresponding CSS code for nested comments was in my themes’ subfolder, in a file called wp-content/themes/<theme name>/style.css. By inspecting my website with the [Vivaldi] web browser (based on Chromium), i found that the CSS class ul.children was likely to blame, as the comments are actually a combination of ordered and unordered HTML lists, see here:

  1. ul.children { padding-left: 20px; }

On top of that, <ul> gets a wide margin-left set by default as well, worsening the situation. This resulted in child comments indenting by something like 40 pixels. For nothing. So I fixed that:

  1. ul.children {
  2.         padding-left: 0px;
  3.         margin-left: 5px;
  4. }

That way it looks much, much more compact and much nicer. This gave me enough space to make the nesting levels even deeper without sacrificing readability (like it was before at 10 levels), so I decided to go for 16 levels. Better than nothing.

The WordPress guys however – in their infinite wisdom – limited the depth to 10 levels, so I was already at the maximum?! Back to inspecting with Vivaldi – the comment settings page this time. That way I found out that the limit is set in wp-admin/options-discussion.php. This is the corresponding code:

  1. <?php
  2. /**
  3.  * Filter the maximum depth of threaded/nested comments.
  4.  *
  5.  * @since 2.7.0.
  6.  *
  7.  * @param int $max_depth The maximum depth of threaded comments. Default 10.
  8.  */
  9. $maxdeep = (int) apply_filters( 'thread_comments_depth_max', 10 );
  11. $thread_comments_depth = '</label><label for="thread_comments_depth"><select name="thread_comments_depth" id="thread_comments_depth">';
  12. for ( $i = 2; $i <= $maxdeep; $i++ ) {
  13.         $thread_comments_depth .= "<option value='" . esc_attr($i) . "'";
  14.         if ( get_option('thread_comments_depth') == $i ) $thread_comments_depth .= " selected='selected'";
  15.         $thread_comments_depth .= ">$i</option>";
  16. }
  17. $thread_comments_depth .= '</select>';

Yeah, ok. So I just changed the $maxdeep assignment to this:

  1. $maxdeep = (int) apply_filters( 'thread_comments_depth_max', 16 );

And with that we’re done, as everything seems to be working properly:

Nesting more than 10 WordPress comments

Nesting more than 10 WordPress comments without relying on some additional CPU-hungry plugin.

Of course, I now need to keep track of my changes, because updating either WordPress itself or my theme will revert my changes back to the previous behavior. But since I haven’t modified my WordPress installation that much so far, I can still live with patching everything back in manually after an upgrade.

And another tiny improvement made…

Update: I just noticed, that sometimes, very long “words” (like HTTP or FTP links without any spaces in them) would overflow the barriers of the div layers they were sitting in in the comments, so while I was dealing with restoring my modifications after a theme update, I fixed the word wrapping as well. The related CSS code sits in wp-content/themes/<theme name>/style.css again. So, I look for the following part…

  1. .comment-body p { line-height: 1.5em; }

…and change it to this:

  1. .comment-body p {
  2.         line-height: 1.5em;
  3.         word-break: break-word;
  4. }

Now even very long words like links will be wrapped properly! With word-break: break-word; only words without spaces will be broken where necessary, so normal sentences with whitespaces for delimiters will still be broken at the spaces, like it should be!