Mar 292018
TLS v1.3 logo

1.) Introduction

Now that the Internet Engineering Task Force has approved the new TLS v1.3 cryptographic protocol, we’ll surely see implementation based on the final specification rather soon, given how far work has progressed in OpenSSL and probably other libraries. In the second to last article I have [shown] how to upgrade servers running on Windows 2000 with modern cryptography and Let’s Encrypt certificates. That software was based on OpenSSL 1.0.2 and stunnel 5.44, which both fully support Windows 2000. However, when looking at the [current state] of the stunnel project in its latest 5.45 beta 6 version, it seems there isn’t even a 32-bit Windows version of it anymore.

Since compiling OpenSSL with some old Microsoft compiler is especially tricky, I tried to cross-compile the latest beta version with mingw 4.9.2 on CentOS 6.9 Linux instead, to get my 32-bit version. The same goes for stunnel. When tried that, I ran into several parts of code that simply can’t work on Windows 2000 any longer as-is.

For my ancient server, TLS v1.3 would be helpful though, as it would lower the CPU load due to its more efficient way of doing TLS handshakes. So there is a pretty good reason for trying this for real.

2.) Required software

Naturally, you need a Linux machine with the 32-bit x86 version of mingw installed for this. It might very well be easy to do on BSDs and other UNIX systems as well. You’ll also need the usual essentials for building programs under Linux like GNU autoconf, make, etc. Then, fetch the latest versions of OpenSSL and stunnel. At the time of writing, that’d be OpenSSL 1.1.1-pre3 and stunnel 5.45b6.

Unpack those, and get your favorite text editor ready, as we’ll need to modify parts of the source code before attempting a build.

3.) OpenSSL with TLS v1.3

First we need OpenSSL, as stunnel will be linking against it. Enter its source directory, then open the header file crypto/bio/bio_lcl.h. After the initial include lines, you’ll need to switch off the AI_PASSIVE macro. This results in OpenSSL using it’s own implementation for name resolutions. If you don’t do this, it would call the Winsock API functions getnameinfo(), getaddrinfo(), freeaddrinfo(), as well as use the struct addrinfo, none of which exist on Windows 2000, but only on Windows XP and newer. So add the following as described, just after the first #include block:

/* mod: Disable AI_PASSIVE so we don't call freeaddrinfo()
* and getaddrinfo() etc. on Windows 2000 */
# undef AI_PASSIVE

Save the file, then compile (here, for 16 CPU threads in parallel with make -j16) and install with the following options set:

$ ./Configure enable-tls1_3 no-asm no-async no-dso no-engine --cross-compile-prefix=i686-w64-mingw32- --prefix=/opt/openssl-mingw mingw shared
$ make -j16
# make install

Just a little explanation for the ./Configure options, why they need to be there, and what the implications are:

  1. enable-tls1_3:

    Like the name says, this enables TLS v1.3. Currently, as it’s still only a draft and not the final version, TLS v1.3 won’t be enabled by default, hence this option.

  2. no-asm:

    This is optional depending on your hardware. If you remove it, OpenSSL will be built including SSE2 code, so you’d need to run Windows on a machine with a new enough processor, meaning either Intels’ Pentium 4 or AMDs’ Athlon 64 CPU. With that option gone, it won’t work on older chips. As my Pentium Pro CPUs don’t have any SSE, I have to fall back to pure C code without assembly optimizations by setting this parameter.

  3. no-async:

    This disables asynchronous sockets in WinSock. This is required as those need light-weight co-operatively multitasking threads called “Fibres”, which don’t exist on Windows 2000. This disables the otherwise failing calls ConvertFiberToThread() and ConvertThreadToFiber().

  4. no-dso:

    Disables the shared object abstraction layer, and with it the call to GetModuleHandeEx(), which is used to communicate with kernel drivers for cryptographic hardware acceleration engines. This also means that we have to switch off support for things like VIA Padlock unfortunately. So no more hardware acceleration, as no-dso implies no-engine.

  5. no-engine:

    Required by no-dso. This disables all cryptographic hardware acceleration engines. As a side-effect, this also disables the native Windows CryptAPI/schannel support of stunnel, so you can no longer store certificates in Windows’ own certificate store either. For us, the CryptAPI is useless anyway, as it’s far too old on Windows 2000, no matter how you look at it (SSL v3, TLS v1.0, SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA).

  6. --cross-compile-prefix:

    Here: A mingw running on a 64-bit host operating system targeting a 32-bit Windows system. You can determine this string by the names of your mingw programs. Usually that string triplet should be correct as-is for what we’re doing.

  7. --prefix:

    Don’t change it! This is the installation directory for our cross-compiled Windows version of OpenSSL, and this location is exactly where the stunnel build system will look for it!

Allright, that settles it with OpenSSL. Next: stunnel!

4.) stunnel 5.45

Enter stunnels’ source directory, and open str/str.c in a text editor. Then, look for the following code block:

expand/collapse source code
/* reportedly, malloc does not always return 16-byte aligned addresses
* for 64-bit targets as specified by
* */
#ifdef USE_WIN32
#define system_malloc(n) _aligned_malloc((n),16)
#define system_realloc(p,n) _aligned_realloc((p),(n),16)
#define system_free(p) _aligned_free(p)
#define system_malloc(n) malloc(n)
#define system_realloc(p,n) realloc((p),(n))
#define system_free(p) free(p)


This shows that a fix of some memory allocation functions has been implemented to correct the old functions’ behavior for 64-bit code. However, this also replaces the functions for 32-bit versions of Windows with aligning ones. Problem: Those don’t exist on Windows 2000 either. We’re talking about _aligned_malloc(), _aligned_realloc() and _aligned_free() replacing the good old malloc(), realloc() and free() functions.

Mind you, the x86 architecture isn’t as strict as not to allow for unaligned memory access, but the CPU fixing things transparently in the background does reduce performance when misaligning your accesses to RAM (You can compare this to filesystem misalignments on solid state drives). So forcing the use of memory alignment is a good thing. But it’s not something Windows 2000 needs, as for 32-bit, the old functions should align properly anyway. Replace the above code with the following:

expand/collapse source code
/* mod: Removed aligning memory functions required for x86_64
* to restore Windows 2000 compatibility. Will be using the regular calls
* instead, as they should work fine for x86_32 */
#define system_malloc(n) malloc(n)
#define system_realloc(p,n) realloc((p),(n))
#define system_free(p) free(p)


Now run the following commands for a cross-compiled build of stunnel linked against the modern OpenSSL you’ve built before. In my case, I’m doing a parallel build across 16 CPU threads again. What ./configure does might look wrong to you in terms of cross-compiling, but just ignore that. It’s going to work just fine (Hopefully at least, heh)!

$ ./configure
$ cd src/
$ make -j16 mingw
$ cd ../doc/
$ make
$ cd ..

And we’re set! All you have to do now is copy off the required files to replace your original stunnel installation. The following files are required; On the left side: The location of the file on a typical Windows installation of stunnel. On the right side: Where to find the corresponding new file on your Linux machine. You might need to adjust the paths a little depending on where you have extracted the source code and where exactly mingw is installed. Also, your current installation might not yet have libssp-0.dll, but you’ll need that one too:

  • %PROGRAMFILES%\stunnel\bin\libssp-0.dll -> /usr/i686-w64-mingw32/sys-root/mingw/bin/libssp-0.dll
  • %PROGRAMFILES%\stunnel\bin\openssl.exe -> /opt/openssl-mingw/bin/openssl.exe
  • %PROGRAMFILES%\stunnel\bin\libssl-1_1.dll -> /opt/openssl-mingw/bin/libssl-1_1.dll
  • %PROGRAMFILES%\stunnel\bin\libcrypto-1_1.dll -> /opt/openssl-mingw/bin/libcrypto-1_1.dll
  • %PROGRAMFILES%\stunnel\config\openssl.cnf -> /opt/openssl-mingw/ssl/openssl.cnf
  • %PROGRAMFILES%\stunnel\bin\stunnel.exe -> ~/yourbuilddir/stunnel/bin/mingw/stunnel.exe
  • %PROGRAMFILES%\stunnel\bin\tstunnel.exe -> ~/yourbuilddir/stunnel/bin/mingw/tstunnel.exe
  • %PROGRAMFILES%\stunnel\doc\stunnel.html -> ~/yourbuilddir/stunnel/doc/stunnel.html

5.) Using it

stunnel with the modern OpenSSL can be used as-is. Just copy the files over your existing installation of stunnel 5.44 or older, then restart the program or service.

Other applications which do not work in tandem with stunnel might need to be recompiled though. There is some software where OpenSSL DLLs can just be swapped out all the way from version 0.9.6 to 1.0.2, but for those, it failed with 1.1.1 for me. The DLLs just wouldn’t load even if a full [dependency walk] checks out ok. Could also be because they’ve been built with mingw instead of Microsoft Visual C though, who knows. For closed source software or open source software that is too hard to recompile, using stunnel is usually a good alternative, as has been shown [here].

6.) The truly final TLS v1.3

As soon as OpenSSL 1.1.1 gets released with official TLS v1.3 support, I will update this article in case the process shown here will require any changes to work with the final versions.

Anyway, I’m looking forward to how much speed can be gained using TLS v1.3 over TLS v1.2 on CPUs from 1997! ;)

7.) Hey, I just want the binary programs to use on Windows 2000!

Oh my… Well, here you go:

As for the source code, you can find the original sources of OpenSSL [here] and the stunnel sources [here]. The patches shown by me here shall be released under the [OpenSSL license] for the code added to OpenSSL and under the [GPL v3] for the code added to stunnel.

Mar 222018
Responsive Design with Encryption logo

1.) Encryption on this weblog

[1][2] In the last article you can see how I recently managed to implement Let’s Encrypt SSL certificates on Windows 2000 Server, including regular renewals via an ACME client that works on said platform. Since I believe in user choice though, I decided not to force people to use encryption on my web sites, as it also lowers performance even further on my already extremely slow server. Instead, I aimed to make this blog software and all of its plugins fully protocol-independent, which proved to be quite a piece of work requiring modification of the main and theme source code, massive changes in the database (lots of static HTTP:// links there) and also changes to one of the CSS stylesheets and the code generating it. But with that done, the visiting user may choose whether to use encryption or whether not to. For this, the weblogs subdomain has been added to the Let’s Encrypt multi-domain SSL certificate.

So from now on, you can visit this place either in an unencrypted fashion via [], or in an encrypted one via []!

Encryption on the desktop

Encryption on the desktop, finally working after several days of work invested

2.) Mobile-friendly design

Now, before that I had worked on another update solely for this weblog, which is to finally make it mobile device friendly. Reading the desktop version of this site is quite painful on smartphones and even tablets after all. Don’t get me wrong, my main target audience will still be people sitting in front of PCs and workstations, but still, I wanted to make this work. Sometimes even I dig up some of my own pages on a smartphone after all.

Instead of using a fully responsive design – which I just can’t bring myself to like at all – I installed a plugin that allows me to have a secondary mobile theme solely for smartphones and tablets. This is a better solution for now I think, as it makes the site very low-traffic and pretty fast on mobile devices without the need to throw away the current desktop theme. This is what it looks like on a Blackberry KEYone Android 7.1.1 smartphone at a 1080×1620 resolution:


And thank god, the mobile design plugin didn’t break the encryption like several other plugins did! It just works with both protocols now, so the mobile site can be reached via HTTP:// and HTTPS:// as well:

Encryption on mobile devices

Encryption also works on mobile devices (Click to enlarge)

There are still a few things that are a bit broken for the mobile version though. I hadn’t thought of setting a “featured image” for each article, instead inserting a small logo on the top left of each article manually. I have updated most recent articles to use a featured image, but the majority of posts still hasn’t been updated, so older posts don’t have those distorted little logos in their mobile versions yet.

The bigger problem is the source code though. The mobile version renders the HTML lists used for line numbers very wrongly, so source code with line numbers looks like total crap. Probably not something I can easily fix, but maybe I’ll look into it in the future. It’s likely going to require a change somewhere in the CSS stylesheets (not that I understand CSS all that well).

If there are any issues with either the encrypted version of the site (like if you encounter pages with a broken padlock in the browsers’ address bar due to mixed content) or with the mobile theme, please report them in the comments! I will fix them – if I can.

[1] The lock icon is © by Svengraph and is licensed under the CC-BY 3.0 unported license

[2] The responsive design logo is © by Encinitas Web Design and is licensed under a not otherwise specified CC license

Mar 152018
Windows 2000 + Let's Encrypt logo

[1] 1.) Introduction

There has been an issue with my [stoneage server]German flag (which is also hosting this weblog) that has been bugging me for quite a while now: I’ve been scared of the time when other server operators and software developers would start to seriously disable ancient SSL/TLS ciphers such as SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA and SSL_RSA_WITH_3DES_EDE_CBC_SHA, or in other words: SHA1, RC4 and 3DES. The services are actually made of both non-free as well as free services that are using several different cryptographic libraries such as different OpenSSL versions but also WinSSL CryptAPI / schannel. Naturally it’s the latter which is presenting the biggest issue: The CryptAPI of Windows 2000 Server is ancient. And while it can do at least TLS v1.0, the ciphers have been becoming the true burden. Unfortunately (?!), some services on do require some form of widely accepted encryption.

Some of the servers do have interchangeable OpenSSL libraries – so you can just swap those .dll files for an upgrade – while others do not. Some from the latter category can be backported to / recompiled for Windows 2000 to upgrade OpenSSL, like I’ve been [able to do] for the UnrealIRCd IRC server. Some however can not.

And then there are services which are using Windows SSL or in other words the CryptAPI / schannel. Those are the most inflexible of the pack.

Another problem is that self-signed certificates such as the ones I’ve been using have also become increasingly problematic. Most client software like web browsers in particular, but also email clients, IRC clients and others keep pestering their users quite a bit when being presented with such an “untrustworthy” SSL certificate. They really want one signed by a fully trusted certificate authority (CA) or an intermediate CA. The best solution for that problem right now is to get a Let’s Encrypt[2] certificate, as it’s free and well-trusted. Only reason I haven’t done that so far is that it needs automatisms (=ACME client software) in place to remain manageable.

I wanted to solve both issues in one fell swoop.

2.) The trigger

eMail logoThen again, you know how people are – they tend to act only when something bad has already happened. In the tradition of that (bad) behavior I’ve chosen to act only where the problem at hand was truly easy to solve. After I’ve managed to compile a modern enough OpenSSL library, I’ve just swapped it on the spot where possible, but I haven’t touched the more problematic services, where such a thing is not doable.

However, there has been a serious issue recently when mail server operators started to beef up their cryptography across the board. It seems there are some updates being rolled out about now which block the ancient ciphers and/or protocols I mentioned in the introduction.

The truly serious part is that seemingly, all mail servers who encounter another that does feature STARTTLS on SMTP port 25, but has no secure ciphers to offer just drop the connection on the spot without generating any error. The mail will not be delivered, but no delivery error will be returned to the sender either! It’s truly a case where the eMails just “disappear” without anyone ever noticing a problem at first. To make matters worse, there was no way to disable just STARTTLS for SMTP with my server software. It’s either no SSL/TLS at all or fully enabled STARTTLS on all plain eMail ports.

See this log to get an idea (remote host names have been replaced with “” and IP addresses as well as security-relevant data have been masked here):

expand/collapse log file

The most interesting part is clearly at the bottom, where it says “SSL negotiation successful (TLS 1.0, 2048 bit key exchange, 168 bit 3DES encryption)”. 3DES is the best Windows 2000 can do by itself, and it looks as if the remote server would accept the implementation just fine. But what happens next is that the remote machine just drops the connection, and that’s it!

The remote side who sent the eMail (that was myself from my account at work actually) never got any delivery failure notification, and after asking other people about it, it was just the same. As said, the mails just vanish!

I also talked to the operator of the mail gateway at work, and he said he didn’t see anything in the logs either, other than that the connection was just being dropped. Pretty stupid to not generate a proper error here, if you ask me.

Well, that was enough of a reason to act! eMails just disappearing is completely unacceptable after all!

3.) What can you do?

I thought about this issue in the past already, and one of my ideas was to use something like [BlackWingCats’] kernel API extensions for Windows 2000 (“KernelEx”). The issue with that is though, that it’s modifying large parts of the operating system, and it’s very incompatible to my German version, wrecking half of the system upon installation during my tests in a VM. So that idea went down the drain pretty fast.

Very recently however, I got the idea that maybe I could use [stunnel] to solve the cipher problem. It’s basically a port mapper wrapping plain text protocols up in SSL/TLS. It’s a program coming from the Linux/UNIX world, but it also works on Windows. On top of that, it miraculously runs on Windows 2000 as well, even in its newest version bundling modern OpenSSL 1.0.2.

Additionally, I thought I’d once again try to get some Let’s Encrypt ACME client to work on Windows 2000 for certificate issuing and renewal, even though I failed to get any of them working before.

3a.) Let’s Encrypt configuration via ZeroSSL

ZeroSSL bannerSince we need an SSL certificate for stunnel anyway, let’s cover this part first. For ease of use, I chose ZeroSSLs’ [Crypt::LE] Perl tool. I didn’t like their [web-based configuration] too much, as they’ll be issuing your private key for you on their servers.

Generally, when dealing with cryptography and trust concepts, I wouldn’t recommend anyone other than yourself generate and use your private key, so that’s one reason why I chose their Perl tool instead. Plus, you can’t automate the certificate renewal process with some web tool, but you can with Perl.

If you don’t have Strawberry Perl installed on your machine, you can still rely on their [Windows binaries] as well. Those are basically just and the necessary Perl parts wrapped in an le32.exe file. Crypt::LE also supports ActiveState Perl, but the free Strawberry Perl should be preferable, if you’re choosing that path.

Unfortunately, if you choose to use the .exe version, it won’t run on Windows 2000 out of the box, as it calls the WinSock 2.0 function freeaddrinfo(), which is only available on Windows XP and newer. For an easy fix, you can use a modified WS2_32.dll WinSock library found in the dllfiles\ subfolder of this [winsock2_getaddrinfo.rar] archive by [Martin Brenner]German flag, if you choose to trust the DLL.

If you do, just put it next to le32.exe, and make sure you launch the program only while sitting within the installation folder of le32.exe itself. Do not overwrite your system-wide %WINDIR%\system32\WS2_32.dll, you don’t need to do that, and you shouldn’t either!

As said, if you don’t trust the hack for the binary, you need to install Strawberry Perl and follow the instructions for the compilation & installation of Crypt::LE from their website.

3a1.) Ok, I have the tools, now what?

I won’t cover the process of using Crypt::LE, as there is an excellent manual [right here at ZeroSSL]! There is one thing that needs to be said though: You need an openssl.exe for the initial certificate request and key generation part. There is one bundled with stunnel in the subdirectory bin\ of its installation folder, you can just use that.

But before you start with it on a cmd terminal, you’ll need to tell OpenSSL where to look for its configuration file. Say you’ve installed stunnel in %PROGRAMFILES%\stunnel\, then run the following command before starting to work with OpenSSL:

SET "OPENSSL_CONF=%PROGRAMFILES%\stunnel\config\openssl.cnf"

This sets %OPENSSL_CONF% to the file name of the OpenSSL configuration, and OpenSSL will automatically parse that environment variable. Adjust paths as necessary, then follow ZeroSSLs’ manual to get your first Let’s Encrypt certificate(s)!

3b.) stunnel

stunnel on Windows 2000

stunnel on Windows 2000

If you’ve been following this article, you’ll already have stunnel installed by now. On Linux & UNIX, stunnel is just a command line tool and/or xinetd service, but on Windows, you also get a bit of a GUI and a tray icon with it. You’ll still have to configure it by editing its config\stunnel.conf configuration file with your favorite text editor however.

Say you wanted to protect a web server on port 80 by adding HTTPS on port 443 for it. The corresponding configuration entry in that configuration file would look like that (here: with separate key file not stored directly within the certificate):

accept  = 443
connect = 80
cert    = mydomain.pem
key     = mydomain.key

stunnel will listen on port 443 with implicit TLS for you, and then redirect the traffic to port 80 on the same machine. To a web browser connecting to, it’ll look like just another TLS-enabled web server.

The same goes for other implicit SSL/TLS services such as SMTPS, IMAPS, POP3S, etc.

The exception of course are explicit SSL/TLS implementations using the STARTTLS command. One classic example for that would be SMTP on port 587, which is typically STARTTLS-enabled. To make that work, stunnel has to emulate parts of the specific protocol to secure, so support for this is rather limited. Currently, stunnel supports the following network protocols for explicit STARTTLS:

  • CIFS (SMB, Samba, older implementation)
  • CONNECT (Client only)
  • IMAP (as per RFC 2595)
  • NNTP (as per RFC 4642)
  • POP3 (as per RFC 2449)
  • SMTP (as per RFC 2487)
  • SOCKS (versions 4, 4a and 5)

To use it, you’d need to configure the service as follows, this example is for SMTP with STARTTLS on port 587, mapping it to your local, unencrypted SMTP server:

accept   = 587
connect  = 25
cert     = mydomain.pem
key      = mydomain.key
protocol = smtp

Now if present, switch off your existing SSL/TLS services first (like make your web server stop listening to port 443), and then fire up the stunnel program and everything should work. You can also install it as a system service on Windows by the way, it’s very easy: stunnel -install.

Together with a properly requested and signed Let’s Encrypt certificate this will give any ancient server things like TLS v1.2 with AES256-GCM-SHA384 and any modern client will trust your certificate implicitly, as most vendors have by now added the ISRG CA certificates to their CACert bundles. Even Microsoft trusts them by now.

4.) Really putting it all together

Let's Encrypt logoEven if you’ve managed to do all of this manually, it won’t solve the problem forever. Let’s Encrypt certificates have a lifetime of just 90 days, so you will have to renew them pretty often. That can be done with just a batch script launching Crypt::LE as well as doing the rollout of the certificates and relaunch of the servers, so that they can re-read the certificates and key files.

Here’s an example script for stunnel and some other hypothetical servers that accept certificate files in different configurations. It assumes that you’re using Crypt::LE together with Strawberry Perl. As you can see, aside from Perl and Crypt::LE you won’t need anything else, as the rest can be done with regular Windows cmd builtins and the NetShell:

expand/collapse certificate-renewal.bat
  1. :: Renew our certificate if it expires within the next 30 days, put HTTP
  2. :: challenge files in C:\MyHTTProot\.well-known\acme-challenge\ and return
  3. :: code 42 if there was a renewal
  4. --------------------------------------------------------------------------
  5. SET "PERLBIN=C:\StrawberryPerl\perl\bin\perl.exe"
  6. SET "LE=C:\StrawberryPerl\perl\bin\"
  7. "%PERLBIN%" "%LE%" -renew 30 -generate-missing -unlink -live -legacy ^
  8.  -key "C:\MyCerts\my-account-key.key" -csr "C:\MyCerts\my-cert-request.csr" ^
  9.  -csr-key "C:\MyCerts\mydomain-key.key" -crt "C:\MyCerts\mydomain-cert.cert" ^
  10.  -domains "," -issue-code 42 ^
  11.  -path "C:\MyHTTProot\.well-known\acme-challenge\"
  13. :: Check whether there was a renewal, and roll out certs + restart servers
  14. :: if so, otherwise just terminate
  15. :: -----------------------------------------------------------------------
  16. IF %ERRORLEVEL% EQU 42 (
  17.   :: stunnel for servers with old cryptographic implementations, this requires
  18.   :: a domain+intermediate certificate bundle with a separate private key file
  19.   TYPE "C:\MyCerts\mydomain.cert" > "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  20.   ECHO. >> "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  21.   TYPE "C:\MyCerts\" >> "%PROGRAMFILES%\stunnel\config\mydomain.pem"
  22.   :: They key needs copying only once
  23.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server\stunnel\mydomain.key"
  25.   :: A server that needs domain certificate, intermediate CA cert and key all
  26.   :: separately
  27.   COPY /V /Y "C:\MyCerts\mydomain.cert" "C:\Server1\sslconf\"
  28.   :: They key and intermediate certificate need copying only once
  29.   :: COPY /V /Y "C:\MyCerts\" "C:\Server1\sslconf\"
  30.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server1\sslconf\"
  32.   :: A server that needs domain and intermediate certificates in one bundle, but
  33.   :: the private key as a separate file, like stunnel
  34.   TYPE "C:\MyCerts\mydomain.cert" > "C:\Server2\sslconf\mydomain.pem"
  35.   ECHO. >> "C:\Server2\sslconf\mydomain.pem"
  36.   TYPE "C:\MyCerts\" >> "C:\Server2\sslconf\mydomain.pem"
  37.   :: They key needs copying only once
  38.   :: COPY /V /Y "C:\MyCerts\mydomain.key" "C:\Server2\sslconf\"
  40.   :: Yet another server, that needs both certificates and your private key all
  41.   :: in one bundled file
  42.   TYPE "C:\MyCerts\mydomain.key" > "C:\Server3\sslconf\mydomain.pem"
  43.   ECHO. >> "C:\Server3\sslconf\mydomain.pem"
  44.   TYPE "C:\MyCerts\mydomain.cert" >> "C:\Server3\sslconf\mydomain.pem"
  45.   ECHO. >> "C:\Server3\sslconf\mydomain.pem"
  46.   TYPE "C:\MyCerts\" >> "C:\Server3\sslconf\mydomain.pem"
  48.   :: Restart all services to reload the certificate and key
  49.   :: ------------------------------------------------------
  50.   :: Restart stunnel
  51.   net stop "stunnel"
  52.   net start "stunnel"
  54.   :: Restart Server 1
  55.   net stop "Server 1 system service name"
  56.   net start "Server 1 system service name"
  58.   :: Restart Server 2
  59.   net stop "Server 2 system service name"
  60.   net start "Server 2 system service name"
  62.   :: Restart Server 3
  63.   net stop "Server 3 system service name"
  64.   net start "Server 3 system service name"
  65. )
  67. :: All done!

Now you can automate the process by creating a job in Windows task scheduler to launch that certificate-renewal.bat like every 20 days or so. The script will of course vary quite a bit depending on your environment and your services, so take it only as a rough guide.

And that’s how you get Let’s Encrypt certificates and modern cryptography up and running on an 18 years old Windows operating system, for what it’s worth. :roll:

5.) Bonus information

You can actually use stunnel in client mode as well. Say you’re using Windows 2000 Pro as your client operating system (*cough*) and your software is so old and insecure, that some remote server at say won’t talk to you anymore. Just run stunnel on your client on demand, with a per-host configuration such as this:

client = yes
accept = localhost:9999
connect =
CAfile = ca-certs.pem

Open up your web browser, surf to https://localhost:9999, and stunnel’ll redirect you to that server, which will now see a secure clientside SSL implementation. Only thing is that you might need to create an exception in your browser, because the host names don’t match between what you entered in the address field and what’s in the remote servers’ certificate, but no way around that.

And now, time for some cold beer! Beer Smilie

[1] The “Let’s Encrypt Radiant Lock” design mark is a trademark of the Internet Security Research group and is licensed under the CC-BY-NC 4.0. All rights reserved.

[2] The “Let’s Encrypt®” word mark is a trademark of the Internet Security Research group. All rights reserved.

Mar 142018
H.265/HEVC logo

1.) Introduction

This post is a followup to the original you can find [here]. My reason for writing a new one instead of just editing the existing one is that the new results have been measured with a slightly different parametrization and under slightly different circumstances, so direct comparability has been thrown out the window. Anyway, like in the former article, multiple x265 video encoder versions starting with 1.7+512 are going to be compared to each other to see how the performance of x265 has evolved over time.

Originally, I wanted to test this only on a machine very similar to my original testbed, maxing out at 6 cores and SSE4.2 instruction set extensions. Since it is said that x265 features significant optimizations for newer AVX and AVX2 instructions however, I decided to include a more modern machine as well, to determine how the scaling differs on both an older and a newer machine.

2.) Benchmarks

First of all, the video used for this is not very “high definition”. Well, maybe by definition as it is 720p, but still. This also puts limits on scalability across cores/threads, which shall provide some additional insights later. Let’s start.

2a.) The older machine on MS Windows

The older one goes first: Here we have an Intel Core i7 980X running CentOS 6.9 Linux. Since I already had all those [Win64 builds] lying around, I ran the test from within a VirtualBox running Windows XP Professional x64 Edition SP2. The specs:

  • CPU: Intel Core i7 980X 3.33GHz (6 cores, 12 threads), last instruction set extension: SSE4.2
  • OS: Windows XP Professional x64 Edition SP2 on VirtualBox 5.1.2 on CentOS Linux 6.9

Here we go:

x265 performance trend on the older machine

x265 performance trend on the older machine (click to enlarge)

While the developments here aren’t exactly identical to what has been posted in the former article, the trend is clearly the same. The new part starts with version 2.3+2, after which we can see a long-needed performance improvement, maybe due to improvements to the assembly code. Also, from here on out we can observe a slight upwards trend for all color channel depths. The basic drop from 8-bit to 10-bit and then 12-bit color stays rather similar though, courtesy of the more expensive 16-bit arithmetic used for the higher color depths (The 8-bit version of x265 also uses 8-bit arithmetic where applicable, reducing effective precision).

There is nothing groundbreaking to be seen like with the introduction of --no-rskip somewhere in between 1.9+141 and 1.9+200, but whatever happened between 2.3+2 and 2.4+2 isn’t too shabby either.

Now let’s see how the modern box with AVX, AVX2, BMI2 and FMA3 instructions fares here!

2b.) The newer machine on CentOS Linux

Alright, specs first:

  • CPUs: 2 × Intel Xeon E5-2620 v4 (2 × 8 cores, 16 Threads for a 16C/32T total), newest instruction set extensions: AVX2, BMI2, FMA3
  • OS: CentOS 7.3.1611 with Linux kernel 3.10.0-514.16.1

This is a modern 2017 Broadwell-EP server featuring a much higher IPC when compared to the older Westmere/Gulftown architecture from 2010. My estimate would be around +40% of raw, additional instructions per clock here. On top of that, we’ll get a peek at its AVX/AVX2 etc. scaling:

x265 performance trend on the modern machine

x265 performance trend on the modern machine

Now, before I comment on the trend to be seen in that chart, let me say one thing: Surprisingly, unlike what x264 has shown over the years, x265 didn’t seem to want to scale more across cores/threads with progressing development. So the core count was not really having much influence at all; Pass 1 of the encode would use 5-6 cores, where pass 2 would eat up 6-7, all the way from the oldest to the newest versions.

What we do see here aside from the --no-rskip drop and the performance increase from 2.3+2 to 2.4+2 are three things: First, we can see a new upwards trend in performance starting with 2.5+9. The development has been more linear for the older CPU, which is why I’m assuming that we get to see some work invested in AVX(2) code here, probably also in BMI/FMA code.

The second thing is the more significant decline starting after 2.0+11. I have no explanation for this, as new features added around here shouldn’t have more negative impacts on newer CPUs as when to compared to older ones. If anything, it should be the other way around, so that part going on up to 2.1+60 is a bit confusing.

And the third thing that is very clear here: The 12-bit version is much closer to 10-bit performance. Only thing I can think of here is that the AVX/AVX2 code paths have generally always been faster on the 12-bit version when compared to the older ≤SSE4.2 assembly code.

Seeing this, I’d like to look at one more thing: The dropoff we can see from 8-bit -> 10-bit -> 12-bit compared between the two architectures. Let’s do that:

3.) 8-/10-/12-bit scaling across platforms

Let’s compare this side-by-side across all versions tested, in a very simple chart:

Well, the modern Xeons do lose less performance even from 8- to 10-bit, but as seen above, the drop to 12-bit is surprisingly insignificant on modern CPUs. This is a bit unfortunate actually, given that all the rage is about 10-bit with the Anime communities and also the official Blu-Ray UHD/4K standards, but oh well. :roll:

4.) Summing it up

Anyway, this hasn’t been too interesting actually, aside from the 12-bit surprise that I guess nobody cares about anyway. Well, we’re seeing some more significant performance increases for recent versions of x265 on modern architectures, but those are only compensating for the loss of performance we’ve seen before, that didn’t occur on older machines.

Maybe the AVX+AVX2 code was good from the very start instead of spreading its wings only now? Anyway, we’re seeing some slow but steady increase in performance for now even on older machines. If there’s a reason to buy new processors for x265 it’s probably not the AVX/AVX2/FMA3/BMI2/AVX512/whatever, but rather the generally higher IPC of newer chips…

5.) How the benchmark was done

5a.) On Windows

Given a folder with x265 versions named x1.7+512.exe, x1.6+2.exe etc., as well as an ffmpeg.exe (like from [here]), the subfolders log\, output\, stats\ and results\, you can adapt and use the following script after putting it right next to the x265 binaries, I call it test-performance-trends.bat:

expand/collapse source code
  1. @ECHO OFF
  3. FOR %%I IN (1.7-8b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  4.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%I, 8 ^
  5.  bits... & .\timethis.exe "echo x265 version %%I 8 bit results: & .\ffmpeg.exe ^
  6.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  7.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p ^
  8.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%I-8b.stats ^
  9.  -o .\output\%%I-8b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  10.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  11.  -r 24000/1001 - 2>NUL | .\x%%I.exe - --y4m -D 8 --fps 24000/1001 -p veryslow ^
  12.  --bitrate 2000 --pass 2 --stats .\stats\%%I-8b.stats -o .\output\%%I-8b-p2.h265 ^
  13.  2>NUL" 1> .\results\results-%%I-8b.txt 2>.\log\timethis-errorlog-%%I-8b.txt
  16. FOR %%J IN (1.7-10b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  17.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%J, 10 ^
  18.  bits... & .\timethis.exe "echo x265 version %%J 10 bit results: & .\ffmpeg.exe ^
  19.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  20.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p ^
  21.  veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats .\stats\%%J-10b.stats ^
  22.  -o .\output\%%J-10b-p1.h265 2>NUL & .\ffmpeg.exe -r 24000/1001 -i ^
  23.  .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 ^
  24.  -r 24000/1001 - 2>NUL | .\x%%J.exe - --y4m -D 10 --fps 24000/1001 -p veryslow ^
  25.  --bitrate 2000 --pass 2 --stats .\stats\%%J-10b.stats -o ^
  26.  .\output\%%J-10b-p2.h265 2>NUL" 1> .\results\results-%%J-10b.txt ^
  27.  2>.\log\timethis-errorlog-%%J-10b.txt
  29. FOR %%K IN (1.7-12b 1.9+15 1.9+108 1.9+141 1.9+200 1.9+210 1.9+230 2.0+11 2.0+54 ^
  30.  2.1+2 2.1+60 2.2+22 2.3+2 2.4+2 2.5+9 2.6+2 2.7) DO ECHO Testing x265-%%K, 12 ^
  31.  bits... & .\timethis.exe "echo x265 version %%K 12 bit results: & .\ffmpeg.exe ^
  32.  -r 24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le ^
  33.  -strict -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  34.  -p veryslow --bitrate 2000 --pass 1 --slow-firstpass --stats ^
  35.  .\stats\%%K-12b.stats -o .\output\%%K-12b-p1.h265 2>NUL & .\ffmpeg.exe -r ^
  36.  24000/1001 -i .\video-teekyuu.h264 -f yuv4mpegpipe -pix_fmt yuv420p10le -strict ^
  37.  -1 -r 24000/1001 - 2>NUL | .\x%%K.exe - --y4m -D 12 --fps 24000/1001 ^
  38.  -p veryslow --bitrate 2000 --pass 2 --stats .\stats\%%K-12b.stats -o ^
  39.  .\output\%%K-12b-p2.h265 2>NUL" 1> .\results\results-%%K-12b.txt ^
  40.  2>.\log\timethis-errorlog-%%K-12b.txt
  42. ECHO All done, results are to be found in the results\results-*.txt files!

5b.) On Linux

Given a system-wide installation of ffmpeg and a folder with statically linked x265 binaries named x1.7+512, x1.6+2 etc., as well as the subfolder results/, the following script – – should do the job after being adapted for your list of x265 binaries and put right next to them:

expand/collapse source code
  1. #!/usr/bin/env sh
  3. for i in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  4. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$i, \
  5. 8 bits...\n" && time ( printf "x265 version $i 8 bit results:\n" \
  6. 1>./results/results-$i-8b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  7. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  8. ./x$i - --y4m -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  9. --slow-firstpass --stats ./stats/$i-8b.stats -o ./output/$i-8b-p1.h265 \
  10. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  11. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$i - --y4m \
  12. -D 8 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  13. ./stats/$i-8b.stats -o ./output/$i-8b-p2.h265 2>/dev/null ) \
  14. 2>>./results/results-$i-8b.txt; done
  16. for j in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  17. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$j, \
  18. 10 bits...\n\n" && time ( printf "x265 version $j 10 bit results:\n" \
  19. 1>./results/results-$j-10b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  20. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  21. ./x$j - --y4m -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  22. --slow-firstpass --stats ./stats/$j-10b.stats -o ./output/$j-10b-p1.h265 \
  23. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  24. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$j - --y4m \
  25. -D 10 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  26. ./stats/$j-10b.stats -o ./output/$j-10b-p2.h265 2>/dev/null ) \
  27. 2>>./results/results-$j-10b.txt; done
  29. for k in {1.7+512,1.9+15,1.9+108,1.9+141,1.9+200,1.9+210,1.9+230,2.0+11,2.0+54,\
  30. 2.1+2,2.1+60,2.2+22,2.3+2,2.4+2,2.5+9,2.6+2,2.7}; do printf "\nTesting x265-$k, \
  31. 12 bits...\n\n" && time ( printf "x265 version $k 12 bit results:\n" \
  32. 1>./results/results-$k-12b.txt && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 \
  33. -f yuv4mpegpipe -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | \
  34. ./x$k - --y4m -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 1 \
  35. --slow-firstpass --stats ./stats/$k-12b.stats -o ./output/$k-12b-p1.h265 \
  36. 2>/dev/null && ffmpeg -r 24000/1001 -i ./video-teekyuu.h264 -f yuv4mpegpipe \
  37. -pix_fmt yuv420p10le -strict -1 -r 24000/1001 - 2>/dev/null | ./x$k - --y4m \
  38. -D 12 --fps 24000/1001 -p veryslow --bitrate 2000 --pass 2 --stats \
  39. ./stats/$k-12b.stats -o ./output/$k-12b-p2.h265 2>/dev/null ) \
  40. 2>>./results/results-$k-12b.txt; done
  42. printf "\nBenchmarks completed, results in `pwd`/results/results-*.txt!\n"

That’s it!

Jan 162018
The Red Solstice logo

1.) Introduction

Yes, it’s another one of “those” posts. Reason for this is, that Humble Bundle is currently [giving away] the crowdfunded title “The Red Solstice” for free, so I picked up a copy to see whether I could get it to work on Windows XP x64 (and also 32-bit XP). Why not, right? Ah, you need Steam for this by the way.

This game is the first one I’ve seen on the Desktop which is actually using OpenGL ES instead of OpenGL or just Direct3D. Typically, that API is more widespread on mobile devices running Android or something, as it’s slimmer and brings a reduced feature set of the “real” OpenGL to the table. The bundled version is OpenGL ES 2.0, and instead of some modern GPGPU language like HLSL it uses nVidias’ old Cg library, which is luckily compatible with XP. The game developer [Ironward] seems to claim that this game, which is based on something called the “TRSEngine” needs Windows 7 or newer to run however.

2.) Required tools

To be able to reproduce the hack described in this post you will need the following tools:

3.) Fixing the platform target

The game has been compiled for a NT 6.0 or Windows Vista platform target, so launching it as-is will result in the dreaded “not a Win32 application” error:

The Red Solstice is "not a valid Win32 application" on XP

Hey, it’s not a Win32 application! Pfh, as if…

The fact that the program was built with an NT 6.0 platform target doesn’t necessarily mean that it actually needs any modern functions of Vista/7/8/10, so the first step is usually to fix the binary headers and change that platform level to something lower, usually 5.1 (XP) for 32-bit code or 5.2 (XP x64 / Server 2003) for 64-bit code. The Red Solstice is a 32-bit game, so it’s going to be 5.1. Sometimes, this can be enough.

As an example, launch CFF Explorer and open the Game.exe from the games’ installation folder, then click on Nt Headers -> Optional Header. I’ve marked the important parts in red for you:

The Red Solstice stock Game.exe

The Red Solstice stock Game.exe (click to enlarge)

To fix this, change the values from 0006 and 0000 to 0005 and 0001, this corresponds to changing the required platform from NT 6.0 to 5.1.

Game.exe with modified binary header

Game.exe with modified binary header (click to enlarge)

After the change, save and overwrite the existing file, but create a backup first, if you wish to be able to revert from the change quickly.

Fixing that one file won’t be enough though. In total, you’ll need to apply the hack seen above to the following files:

  • Game.exe
  • icudt51.dll (Unicode library)
  • icuin51.dll (Unicode library)
  • icuuc51.dll (Unicode library)
  • libEGL.dll (Platform-specific glue layer for OpenGL ES on Windows)
  • libGLESv2.dll (OpenGL ES 2.0)
  • libogg.dll (OGG vorbis audio decoder)
  • libtheoraplayer.dll (Theora audio decoder)
  • OpenAL32.dll (OpenAL audio API)
  • Qt5Core.dll (Qt5 2D GUI library)
  • Qt5Gui.dll (Qt5 2D GUI library)
  • Qt5Widgets.dll (Qt5 2D GUI library)
  • platforms\qwindows.dll (Platform-specific glue layer for Qt5 on Windows)

It’ll take about 1-2 minutes to fix all of them I guess. However, if you attempt to run the game after that, it’ll still fall on its face with a different error:

The XP kernel API doesn't provide GetTickCount64()

Ah, GetTickCount64() is actually an [old acquaintance] of mine by now…

Inspecting the Game.exe with Steve Millers’ Dependency Walker shows, that the problem is actually limited to GetTickCount64(), so it’s the one and only missing kernel API function call we’ll have to deal with:

Tracking the dependencies of Game.exe brings up a missing GetTickCount64() call in kernel32.dll

Tracking the dependencies of Game.exe brings up a missing GetTickCount64() call to kernel32.dll, but no other problems. (Click to enlarge)

Alright, let’s get to it!

4.) Fixing the GetTickCount64() call

This is a bit trickier, as we need to present a GetTickCount64() function to the game, that will do something useful. For this we’ll use Oleg Ovcharenkos’ Stellaris hack, which allows us to redirect that missing call to the legacy GetTickCount() that Windows XP does have.

Unpack the software into some directory of your choice and copy the Game.exe of “The Red Solstice” into that directory as well. Olegs’ hack isn’t prepared for this game yet, so copy one of the existing scripts such as xp_Stellaris_1.6.cmd to a new script file, e.g. xp_TheRedSolstice_1.6.cmd. Open the file in your favorite text editor, and make sure it contains exactly and only the following lines:

rundll32 zernel32.dll,PatchFile Game.exe

Save the file, and run it by double-clicking on it or by launching it directly from a command line terminal. It’ll patch the game so it calls the stub DLL zernel32.dll and some other z* libraries, which fixes some missing calls and redirect the rest to the platforms kernel API as-is. It’ll also create a backup copy of Game.exe for you. Now you have to copy the following files from the folder containing Olegs’ hack back to the actual game folder:

  • Game.exe
  • zernel32.dll
  • zs2_32.dll

That’s it!

5.) Finally: Running the game on XP

With the platform target hack done with CFF Explorer and the GetTickCount64() => GetTickCount() redirection done by Olegs’ Stellaris hack we can finally launch the game:

The Red Solstice fired up on Windows XP x64

The Red Solstice fired up on Windows XP x64 (click to enlarge)

I have to say that I haven’t tested this extensively so far, but a few brief gameplay tests have been conducted:

Gameplay test

Gameplay test (click to enlarge)

There you go, another modern game hacked to run on ancient Windows XP & XP x64! Beer Smilie

Jan 112018
AMI logo

1.) Introduction

This is something I’ve been wanting do do years ago, but the recent outcry regarding the Meltdown and Spectre hardware security holes have reminded me of this. Since fixing at least parts of the exploits also requires CPU microcode support (from here on: “µcode”), and not all operating systems and mainboards/machines might get them, this might be useful for the current situation as well. It can also be helpful if you wish to patch Intel Xeon support into a slightly older platform, where the manufacturer hasn’t done so already. This article does however not apply to modern UEFI-based systems.

So, instead of getting the required CPU µcodes from a donor BIOS, why not obtain the latest version from Intels’ [µcode package for Linux]? Oh, and it’s not really Linux-specific anyway.

I will show you how to extract and prepare the Intel µcodes and how to patch them so you can embed them into pretty much any somewhat modern AMI BIOS. AMI BIOSes are pretty widespread, but tools similar to the AMI-specific ones shown here might of course also exist for other BIOS brands.

The steps of this guide will be shown for both Linux/UNIX as well as Microsoft Windows.

All the tools used in this post can be found at the end of the article. You can also get all the currently released µcodes in one package there, so you don’t need to download all the archives from Intel (not all archives contain all µcodes…).

2.) Fetch and extract/convert Intels’ µcode package

You can get the µcodes directly from Intel, again, here’s the [link]. To unpack the .tgz archive on MS Windows, you may want to use [7-zip] (It’s a 2-step process with 7-zip, from .tgz to .tar and from .tar to the final files). For older releases, all you’ll get by doing this is a microcode.dat file, sometimes the file name will also contain a date. That’s all the µcodes assembled in one file in a text format. This is useless for patching the data into BIOS images, so we’ll need to extract the individual CPU µcodes and convert them into the proper binary format.

Newer releases might contain pre-built binary versions as well, but we’ll just continue to work with microcode.dat.

2a.) On Linux or UNIX

We’ll use iucode_tool for that, as at least most Linux systems should have a package ready for that program. If you can by no means get or compile iucode_tool, you might compile microdecode from source on Linux or UNIX instead, see 2b.) and 8.)! I have tested this on CentOS 6.9 Linux, and microdecode compiles just fine. But let’s continue with iucode_tool:

To extract the µcodes, switch to the directory containing microcode.dat (after unpacking the .tgz file with tar -xzf) and run the following commands:

$ mkdir ./µcodes/
$ iucode_tool -t d -L --write-named-to="./µcodes/" "microcode.dat"; done

With that, you’ll get a lot of *.fw binary files in ./µcodes/. Those are in the regular Intel µcode format and come in different sizes depending on the CPU. The file names look similar to e.g. this: s000206C2_m00000003_r00000013.fw. The first “s” part contains the CPUID, the second “m” part shows the platform ID, and the “r” part denotes the µcode revision, all of it in hexadecimal notation.

2b.) On Windows

On Windows, we’ll use the microdecode command line tool. It’s a bit easier to use than iucode_tool, as it’s only purpose is really just µcode extraction and nothing else. Switch to the directory containing the unpacked microcode.dat, make sure microdecode.exe is in your search path (or in the same directory), and run the following command:

microdecode.exe microcodes.dat

This will result in a lot of *.bin files that are identical to the .fw files the *nix tool extracts. The file names will be similar to this: cpu000206c2_plat00000003_ver00000013_date20100907.bin. The first “cpu” part contains the CPUID like above, the second “plat” part contains the platform ID, and the “ver” part shows the µcode revision. Additionally, we also get the release date in the final “date” part.

3.) Identifying your CPU

You’ll likely know your CPU model if you’re reading this. But what you’ll need to know above that is your exact CPUID, stepping and the revision of CPU µcode you’re currently running. You can look up some of the information on cpu-worlds’ [CPUID database], but ultimately, you’ll need to fetch it from your local machine anyway.

Important background information: The CPUID is a 32-bit value structured as follows:

CPUID structure

The binary structure of Intels’ 32-bit CPUID, 4 bits equal one hexadecimal character

The binary matrix is usually expressed as 4-bit hexadecimal characters. e.g. the binary CPUID 0000:0000:0000:0010:0000:0110:1100:0010bin would result in 000206C2hex.

3a.) On Linux/UNIX

On Linux, the required information can be grabbed from procfs. Like this, for example:

$ cat /proc/cpuinfo | grep -e "model" -e "stepping" -e "microcode" -e "cpu family" | sort | uniq

cpu family	: 6
microcode	: 15
model		: 44
model name	: Intel(R) Core(TM) i7 CPU       X 980  @ 3.33GHz
stepping	: 2

You have to be careful interpreting this information however, as it’s mostly decoded in decimal form. Modern kernels might show some of the data in hexadecimal, so be careful here!

But first, convert the decimals from /proc/cpuinfo to hexadecimal, where necessary, in this case we’ll do it for the µcode revision and the model number:

$ printf '%X\n' 15
$ printf'%X\n' 44

So, the model number is 2Chex. Here, the right part or Chex is the actual model number (CPUID bits 4..7) and 2hex is the extended family number (CPUID bits 16..19).

The µcode version here would be Fhex. Note that µcode versions can be wider than just 1 hexadecimal character as well. The stepping 2 is a number lower than 15, so we don’t need to convert it to hexadecimal.

To assemble our final string which is 8 hexadecimal characters wide, we walk through the bit mask on the image shown above step by step! All “reserved” parts are just filled with blanks:

  1. 0 (Bits 31..28, reserved)
  2. 0 (Bits 27..24, extended CPU family, zero for family 6)
  3. 0 (Bits 23..20, extended CPU family, zero for family 6)
  4. 2 (Bits 19..16, extended CPU model number)
  5. 0 (Bits 15..12, two reserved and two platform type bits, typically zero)
  6. 6 (Bits 11..8, CPU family code)
  7. C (Bits 7..4, CPU model number)
  8. 2 (Bits 3..0, CPU stepping ID)

As a result of this, our CPUID string looks as follows:

  • 000206C2hex, currently running µcode revision Fhex.

Please note that information down, it’s needed in the next step.

Of course, if you’re running some other UNIX, you’d be needing different tools. On FreeBSD you may be able to work with the “cpuflags” tool, like this:

$ cpuflags x 2>& | grep "Origin"

For other UNICES, you’re on your own though.

3b.) On Windows

On Windows I’d recommend Ray Hinchcliffes’ [System Information Viewer], or “SIV” in short. It’s a pretty powerful freeware system information tool that will read out the stuff we need for the next steps (You can also use something like CPU-Z of course). See the following screenshot to learn what you need to look for:

System Information Viewer (SIV)

System Information Viewer (click to enlarge)

From left to right, we need the following data as pointed out with those arrows: CPU family 6hex, CPU model 2Chex (you can ignore the decimal value “44”), stepping 2hex and µcode revision 13hex. Again, 2Chex is a combination of model number Chex and extended model number 2hex!

Just like on Linux, the final identification string needs to be assembled first. Luckily, SIV already does the decimal -> hexadecimal conversion for us. To assemble the entire CPUID, put everything together like this: 

  1. 0 (Bits 31..28, reserved)
  2. 0 (Bits 27..24, extended CPU family, zero for family 6)
  3. 0 (Bits 23..20, extended CPU family, zero for family 6)
  4. 2 (Bits 19..16, extended CPU model number)
  5. 0 (Bits 15..12, two reserved and two platform type bits, usually zero)
  6. 6 (Bits 11..8, CPU family code)
  7. C (Bits 7..4, CPU model number)
  8. 2 (Bits 3..0, CPU stepping ID)

As a result of this, our CPUID string looks as follows:

  • 000206C2hex, currently running µcode revision 13hex.

The revision is the µcode version you’re currently running. Newer version will just have higher numbers, so that’s how you can compare them with what you extracted from Intels’ package.

Note that information down, as you’ll need it in the next step.

4.) Locating the correct µcodes

4a.) On Linux/Unix

Let’s continue to assume your CPUID string is 000206C2hex. Switch to the directory where you extracted the Intel µcode package for Linux and look for the correct µcode binaries (in my case I have them grouped in folders by date as well):

$ find . -iname "*000206C2*"

Two versions are available: The newer 13hex and the older Fhex (in decimal that would amount to 19 vs. 15, higher is always newer).

4b.) On Windows

Like on Linux, let’s assume your CPUID string is 000206C2hex. Switch to the directory where your extracted Intel µcodes are, and look for the corresponding files:

DIR /B /S "*000206C2*"

There you go, the newer 13hex and the older Fhex µcodes.

5.) What’s wrong with AMI MMTool: A detailed explanation

To embed the µcodes into an actual AMI BIOS image, AMIs’ MMTool “CPU Patch” function is required. It’s a Windows program, but given its simple nature, it will run fine with Wine on Linux and FreeBSD UNIX as well:


 But if you attempt to embed Intels’ original binaries, it’ll fail right away:

MMTool failing to import Intel µcodes directly

MMTool failing to import Intel µcodes directly

If you extract existing µcodes from a given BIOS image and compare them with the same µcode revision in Intels’ format, you’ll notice a size difference:

$ ls -al ./06C2-Rev13.bin 20100914/µcodes/s000206C2_m00000003_r00000013.fw
-rw-rw-r-- 1 thrawn users 8192 Jan 10 14:04 ./06C2-Rev13.bin
-rw-r--r-- 1 thrawn users 7168 Jan 10 14:26 20100914/out/s000206C2_m00000003_r00000013.fw

Intels’ µcode file in this case is 7168 bytes or 7kiB in size, but the stuff we extracted from that AMI BIOS is 8192 bytes or 8kiB in size. So what’s the deal here? At first I had no idea at all and thought that they’d just be encoded differently in the AMI BIOS or something. But it turns out that that is not the case. If you open both files in a hex editor of your choice, you’ll notice something at the end of the file that came from the AMI BIOS:


The final 1kiB is just binary zeroes! So how do we fix that? In essence, the entire procedure is like this:

  1. Extract µcodes from the target AMI BIOS image using MMTool.
  2. Look at how large that file is (in bytes).
  3. Look at how large the desired Intel µcode file “for Linux” is for your exact CPU.
  4. Calculate the difference in bytes.
  5. Fill the end of the Intel file with as many binary zeroes as there are missing bytes.
  6. Save that file and try to load it into your BIOS image using MMTools’ “CPU Patch” function.

Alright, let’s do it!

6.) Patching those µcodes for AMI MMTool and patching the BIOS itself

First, extract some random µcode from your BIOS using AMI MMTools’ “CPU Patch” function. Look at the file and note down its’ size in bytes. Also, you may want to look at the extracted µcode to see whether it has a lot of zeroes at the end. In my case, as said, the µcodes in that AMI BIOS are 8192 bytes, or 8kiB large, while Intel’s µcode is 7kiB in size. 8192 – 7168 = 1024, so again, the difference is 1024 bytes or 1kiB.

Please triple-check this stuff, as a wrong padding will break the µcode! Of course MMTool should warn you if something’s wrong, but it doesn’t hurt to make sure it’s safe from the start.

6a.) On Linux/UNIX

Let’s make this easy, generate a 1kiB zero padding file for future use on similar µcode files:

$ dd bs=1 count=1024 if=/dev/zero of=./1kiB-zeropadding.bin
1024+0 records in
1024+0 records out
1024 bytes (1.0 kB) copied, 0.00383825 s, 267 kB/s

And now, append the zeroes to the end of the file, generating a new file in the process, so we don’t touch the originals:

$ cat ./20100914/µcodes/s000206C2_m00000003_r00000013.fw ./1kiB-zeropadding.bin > ./0206C2-rev13.bin

That file 0206C2-rev13.bin is now the final µcode file for patching into the target BIOS. As an example, I’ll patch those µcodes for Xeon X5600 processors into an ancient BIOS version 0402 of an ASUS P6T Deluxe mainboard:


Save that file, and you’re done modifying your BIOS with µcodes from Intels’ Linux package!

6b.) On Windows

On MS Windows there is no dd command by default, but we have a another preinstalled utility for creating our 1kiB zero padding file: fsutil! Like this:

fsutil File CreateNew .\1kiB-zeropadding.bin 1024

We don’t have a cat command on Windows either, but we can use a binary copy for file concatenation, no need to install any additional tools. We’ll attach the zero padding to the end of the Intel µcode and write the result to a new file, so we’re not altering the originals:

COPY /B .\20100914\cpu000206c2_plat00000003_ver00000013_date20100907.bin + .\1kiB-zeropadding.bin .\0206C2-rev13.bin
        1 file(s) copied.

That file 0206C2-rev13.bin is now the final µcode file for patching into the target BIOS. As an example, I’ll patch those µcodes for Xeon X5600 processors into an ancient BIOS version 0402 of an ASUS P6T Deluxe mainboard:


Save that file, and you’re done modifying your BIOS with µcodes from Intels’ Linux package!

7.) Flashing the modified BIOS

This step depends on your system. You may be able to flash from within the BIOS itself (e.g. from floppy disk or USB pen drive), or from within MS Windows or by using some bootable medium.

In any case, your BIOS image is ready now, and can be flashed at your convenience. After the update, re-check your µcode revision as shown in 3.). Your CPU should now be running the updated µcode!

8.) Downloads

Some of the programs used in this article can be downloaded here:

  • [AMI MMTool v3.22]
  • [microdecode], compiled by myself, contains 32-bit and 64-bit versions of the tool as well as the source code and license file. Compiles on Linux/UNIX as well.
  • [Intel µcodes], everything from 2009-03-30 up to 2018-01-08 as available on the Intel web site at the moment.

9.) Hopefully you haven’t bricked your mainboard

…all of this without any kind of guarantees of course! :roll:

Dec 212017
Oxenfree logo

Thanks to GOG giving Oxenfree away for free recently, I had the opportunity to test this game for its compatibility with Windows XP and Windows XP x64 Edition. So I can’t vouch for the Steam version, but I do assume that it’s the same as for the GOG version, as the Steam API itself is still compatible with XP and XP x64. Oxenfree itself is using the Unity engine, which is typically highly compatible when it comes to different operating system platforms. On Windows XP and Vista, Oxenfree still breaks though, so let’s take a closer look!

For this test, the 64-bit version of the game was used on XP x64, while the regular Windows XP machine got the 32-bit version of course.

1.) Installation

If you get/got the game from GOG, please do not download the Galaxy version, but the regular one, as the GOG Galaxy client really won’t run on XP.

Now, the first thing you might encounter are some installer errors, if you’re using the GOG installer. These might ring a bell for some users of Wine on Linux, as they can happen there as well. I’ve seen those on one of the two XP x64 machines that I’ve been using for testing, so they don’t always pop up, but they may appear for you:


Error 2/3 actually just appears twice, but at a slightly different address. Luckily, all of those can be safely ignored. I have no idea what the GOG installer is doing that some Windows boxes just don’t like, but it won’t mess up the installation!

Now let’s try to run it:

2.) Windows XP 32-bit

Hint: Note that the trick shown here may also help for XP x64!

Running the game as-is on XP 32-bit resulted in the following error on my system:

Oxenfree Direct3D error

Oxenfrees’ Unity Engine 5 failing to render to Direct3D 9

The reason for this seems to be the bundled d3d9.dll, which is actually compiled for NT 6.0 (Vista), and won’t load on XP. Even when hacking that librarys’ header to support NT 5.2 (XP x64) or NT 5.1 (XP) with CFF Explorer, the same error would pop up (why? No idea). Instead of hacking it, just rename the file to make Oxenfree use the systems’ own Direct3D 9.0c library in %WINDIR%\system32\. Launching the game after that resulted in this:

Oxenfree running on Windows XP 32-bit

Oxenfree running on a German Windows XP 32-bit (click to enlarge)

There we go! Just in case this doesn’t help, and the game isn’t even launching at all, try to rename the dxgi.dll as well, an explanation follows below.

3.) Windows XP Professional x64 Edition

On XP x64, the game didn’t even show any errors for me at all, it would just instantly terminate, even if you patch all the libraries to NT 5.2 headers with CFF Explorer (this isn’t required actually, as you’ll see).

The primary problem on XP x64 is the DirectX Graphics Infrastructure library, so dxgi.dll. I don’t even understand why it was bundled with the game, as Oxenfree uses Direct3D 9, but DXGI is meant to work as a kind of glue layer between Direct3D 10, 11 and 12. More information [here] and [here]. DXGI can do some interesting things like making simple and faster full screen / windowed switches by calling SetFullscreenState() or like resizing windows with ResizeTarget(). The idea is probably that everything works the same way, no matter whether you’re using D3D 10, 11 or 12.

However, DXGI has nothing to do with Direct3D 9?! So just rename the dxgi.dll to something else and give it a shot, it should work just like that:

A 64-bit Oxenfree running on Windows XP x64

A 64-bit Oxenfree running on Windows XP x64 (click to enlarge)

Clearly, Oxenfree isn’t using DXGI for switching to its windowed mode or for window resizing. Everything works just as intended without that DLL.

In case you now get the Direct3D error shown for the XP 32-bit solution above, just rename the bundled d3d9.dll as well. I’ve seen the error appear on one of two XP x64 machines tested, and renaming the DLL worked just like it did for 32-bit XP too.

4.) Windows Vista

I have no real idea about Vista. There was one Steam user who reported the game breaking in a similar way to what I’ve seen on XP x64 however. You can read the thread [on the Steam forums], I’ve replied there yesterday as well.

I assume that the fixes described above with help with that as well though. I just can’t test it, as I don’t have a physical Windows Vista machine at the moment.

5.) What else?

I’m not sure what exactly the bundled xinput9_1_0.dll is for. Probably some DirectInput component of DirectX handling game controllers or something? No idea. Anyway, this can be safely renamed just like the others. You should have that library installed system-wide anyway, in %WINDIR%\system32\. There are more libraries that you’ll likely never need for the regular GOG version however. If you want, you can experiment with renaming or removing the following as well (at your own risk of course):

  • All files in Oxenfree_Data\Plugins\, but leave the Steam libraries alone if you have the Steam version of the game. Also, the 64-bit version puts an sqlite3.dll there. I’m not sure if the removal of that database DLL is safe, it’s untested.
  • GogGalaxyHooks.dll, as we don’t have the Galaxy client anyway.
  • steam_api.dll or steam_api64.dll for the 64-bit version. Again, leave those alone if you’re using the Steam version.

Never touch Oxenfree_Data\Mono\mono.dll though. That’s a critical part of the Unity3D engine itself, as it’s mostly written in .Net/Mono.

And with that, just play the game! :)

Oct 162017
Fraps logo

Does anyone still remember [Fraps]? It seems development has come to a halt in 2013, but it was once the standard tool for measuring frame rates in 3D games and programs under Windows. Also, of course, for recording in-game video footage. All that up to Direct3D 11 and any version of OpenGL. And while the latest (and probably last) version 3.5.99 still officially supports Windows XP, it was [recently brought to my attention]German flag that it throws the “not a valid Win32 application” error.

Since I’ve been using Fraps myself, I thought I’d take a quick look at this. After confirming the error, I looked at its binary files fraps32.dll, fraps64.dat, fraps64.dll, frapslcd.dll and fraps.exe with NTCores’ [CFF Explorer]. It was as expected, its platform target was set to Windows NT 6.0 (Windows Vista), which was why it refused to start on older operating systems.

So I patched the headers of the 32-bit files to a NT 5.1 target (Windows XP) and of the 64-bit files to a NT 5.2 target (XP x64, Server 2003):

Patching Fraps with CFF Explorer

Patching Fraps to a NT 5.1 platform target with CFF Explorer

And that’s all, because Fraps 3.5.99 is still fully API compatible to Windows XP. This basically proves, that choosing that NT 6.0 platform target was either a wrong decision on the developers’ side, or just a slip-up that went unnoticed because nobody’s testing this on XP any longer:

Running the binary-patched Fraps 3.5.99 on XP x64

Running the binary-patched Fraps 3.5.99 on XP x64 (Click to enlarge)

So much for that.

If you want to run Fraps 3.5.99 on XP, please install the [official version] first, and then unpack the following archive into your Fraps installation folder, overwriting all of the files there:

  • [Fraps 3.5.99], patched for Windows XP / XP x64 compatibility.

And no, I’m not going to build a complete installer with the patched files inside, because I’m far too lazy to do that! :roll:

I tested this briefly with 3DMark 2001 SE b330 on XP and XP x64 (I didn’t have anything better / more modern on the machine at hand right now), and it did indeed work, showing its fps counter just fine!

Fraps 3.5.99 measuring the frame rate in 3DMark 2001 SE under XP

Fraps 3.5.99 measuring the frame rate in 3DMark 2001 SE under XP. And no, I didn’t pick a good scene to snap, I know… (click to enlarge)

So that’s it, cheers! :)

Oct 092017
Visual Basic 6.0 logo

A long, long time ago, there was a pretty useful little IRC bot for Windows called the [AnGeL] bot. It seems like nobody remembers it anymore, but it was born from the Anime / Sailor Moon scene back in the late 90s to early 2000s and developed by a German software engineer going by the name of [Benedikt Hübschen]. The bot was pretty widespread for a while, at least in the German speaking parts of the Internet, and it was extensible by writing VBScript code using Microsofts’ Windows Script Host.

So, essentially, it was what you’d have used if you couldn’t run UNIX or Linux with an eggdrop bot. And I sure couldn’t, because back then I barely even knew about the existence of such systems.

Recently, I ran into a small little problem though; I wanted the bot to create and maintain an SSL/TLS-only channel. So, an IRC channel that would let users join and chat with each other only if they’re connecting to the IRC server via an encrypted connection. This is usually done by setting the +z flag on the channel, which might be followed by the IRC server itself also setting the encryption indicator mode +Z automatically.

However, I found that the bot wouldn’t enforce +z at all. It wouldn’t even set the mode when asked to do so explicitly. It was possible to add it to the list of enforced modes, but it just wouldn’t work, with the same being true for +P (protect channel modes even when nobody is in the channel).

Luckily, Mr. Hübschen made the source code available under the GPL license (that’s what he told me personally) [here]! And yes, that is VisualBasic 6 code. And yes, VB6 is a part of the infamous Visual Studio 6, you might know the abbreviation “VC6” from C/C++ programs compiled with it. So I though I’d inspect the source code and attempt to fix this issue.

I fired up my Windows 2000 Pro SP4 virtual machine for that, installed Visual Studio 6 (thank you, MSDNAA/Dreamspark/Imagine) and its service pack 6, and then loaded the project file:

The AnGeL IRC bot source code loaded in VB6

That development environment is ancient, and it sure looks the part… what a mess.

I identified the part of the code that would need changing, it’s in the public function ChangeMode() in SourceCode/Modules/Server/Server_Functions.bas. I simply copied some code and adapted it for my purpose, adding just +z and +P support for now:

Server_Functions.bas, expand/collapse source code
  1. Public Function ChangeMode(Should As String, Current As String) ' : AddStack "Routines_ChangeMode(" & Should & ", " & Current & ")"
  2. Dim u As Long, CurMode As Long, Changes As String, InsertWhat As String
  3. Dim CurPos As Long, LimitPos As Long, KeyPos As Long
  4.   ' Added by GrandAdmiralThrawn (
  5.   ' modes +z (SSL/TLS enforce) and +P (permanent channel with
  6.   ' modes preservation even when empty):
  7.   CurMode = GetModeChar(Current, "z")
  8.   Select Case GetModeChar(Should, "z")
  9.     Case -1: If CurMode = 1 Then Changes = Changes & "-z"
  10.     Case 1: If CurMode = 0 Then Changes = Changes & "+z"
  11.   End Select
  12.   CurMode = GetModeChar(Current, "P")
  13.   Select Case GetModeChar(Should, "P")
  14.     Case -1: If CurMode = 1 Then Changes = Changes & "-P"
  15.     Case 1: If CurMode = 0 Then Changes = Changes & "+P"
  16.   End Select
  17.   ' End of modification by GAT.
  18.   CurMode = GetModeChar(Current, "p")
  19.   Select Case GetModeChar(Should, "p")
  20.     Case -1: If CurMode = 1 Then Changes = Changes & "-p"
  21.     Case 1: If CurMode = 0 Then Changes = Changes & "+p"
  22.   End Select
  23.   CurMode = GetModeChar(Current, "s")
  24.   Select Case GetModeChar(Should, "s")
  25.     Case -1: If CurMode = 1 Then Changes = Changes & "-s"
  26.     Case 1: If CurMode = 0 Then Changes = Changes & "+s"
  27.   End Select
  28.   CurMode = GetModeChar(Current, "m")
  29.   Select Case GetModeChar(Should, "m")
  30.     Case -1: If CurMode = 1 Then Changes = Changes & "-m"
  31.     Case 1: If CurMode = 0 Then Changes = Changes & "+m"
  32.   End Select
  33.   CurMode = GetModeChar(Current, "t")
  34.   Select Case GetModeChar(Should, "t")
  35.     Case -1: If CurMode = 1 Then Changes = Changes & "-t"
  36.     Case 1: If CurMode = 0 Then Changes = Changes & "+t"
  37.   End Select
  38.   CurMode = GetModeChar(Current, "i")
  39.   Select Case GetModeChar(Should, "i")
  40.     Case -1: If CurMode = 1 Then Changes = Changes & "-i"
  41.     Case 1: If CurMode = 0 Then Changes = Changes & "+i"
  42.   End Select
  43.   CurMode = GetModeChar(Current, "n")
  44.   Select Case GetModeChar(Should, "n")
  45.     Case -1: If CurMode = 1 Then Changes = Changes & "-n"
  46.     Case 1: If CurMode = 0 Then Changes = Changes & "+n"
  47.   End Select
  48.   If InStr(ServerChannelModes, "c") Then
  49.     CurMode = GetModeChar(Current, "c")
  50.     Select Case GetModeChar(Should, "c")
  51.       Case -1: If CurMode = 1 Then Changes = Changes & "-c"
  52.       Case 1: If CurMode = 0 Then Changes = Changes & "+c"
  53.     End Select
  54.   End If
  55.   If InStr(ServerChannelModes, "C") Then
  56.     CurMode = GetModeChar(Current, "C")
  57.     Select Case GetModeChar(Should, "C")
  58.       Case -1: If CurMode = 1 Then Changes = Changes & "-C"
  59.       Case 1: If CurMode = 0 Then Changes = Changes & "+C"
  60.     End Select
  61.   End If
  63.   For u = 1 To Len(Should)
  64.     Select Case Mid(Should, u, 1)
  65.       Case "l": If GetModeChar(Should, "l") = 1 Then CurPos = CurPos + 1: LimitPos = CurPos + 1
  66.       Case "k": CurPos = CurPos + 1: KeyPos = CurPos + 1
  67.       Case " ": Exit For
  68.     End Select
  69.   Next u
  71.   CurMode = GetModeChar(Current, "l")
  72.   Select Case GetModeChar(Should, "l")
  73.     Case -1: If CurMode = 1 Then Changes = Changes & "-l"
  74.     Case 1
  75.       If CurMode = 0 Then Changes = Changes & "+l": InsertWhat = " " & Param(Should, LimitPos)
  76.       If CurMode = 1 Then If Param(Current, 2) <> Param(Should, LimitPos) Then Changes = Changes & "+l": InsertWhat = " " & Param(Should, LimitPos)
  77.   End Select
  78.   CurMode = GetModeChar(Current, "k")
  79.   Select Case GetModeChar(Should, "k")
  80.     Case -1: If CurMode = 1 Then Changes = Changes & "-k" & InsertWhat & " " & Param(Current, ParamCount(Current)): InsertWhat = ""
  81.     Case 1: If CurMode = 0 Then Changes = Changes & "+k" & InsertWhat & " " & Param(Should, KeyPos): InsertWhat = ""
  82.   End Select
  84.   Changes = CleanModes(Changes)
  85.   ChangeMode = Changes & InsertWhat
  86. End Function

Honestly, I didn’t think it would actually compile at all. But just hit File \ Make AnGeL.exe, and it all builds just fine! And it’s fast as well. At least the building process is.

I chose to have VB6 “Optimize for fast code” and to have it “favor Pentium Pro(tm)”, whatever that means. But I assume it’s faster on P6 / i686 architectures now (Pentium Pro, Pentium II/III and other more modern chips). Probably also requires such a processor now, breaking compatibility with 586 and earlier chips, but I’m not sure whether that’s true.

I gave it the version number 1.6.3, with the latest I could ever find on the web before having been 1.6.2. You can download this version together with the source code here:

If you just want to run it instead of an existing one, all you need to do is to copy the AnGeL.exe over the one you have now, and that’s it. To edit the code, you need VisualBasic 6, just load the project file ANGEL.VBP and you can start to modify and recompile it.

Hah, changing the AnGeL bot and building its source code after so many years… felt a little bit like touching the holy grail or something. ;)

My thanks fly out for Benedikt Hübschen for developing the AnGeL bot, and for open-sourcing it! Also, I would like to thank all the contributors to the project as well! I’ll continue to use the bot, probably for a long time to come. :)

Sep 222017
Changeicon logo

[1] 1. Introduction

Note: A more step-by-step guide is to be found farther below, see point 5.!

Recently, [Umlüx] has reminded me of his idea to be able to (visually) tag folders on Windows for specific purposes. For convenience, it’s supposed to work by right-clicking a folder in Windows Explorer, opening a submenu from the context menu, and then by picking the proper tag. After that, the folder icon should change, indicating that something’s special about this one. Windows can actually do the “tagging” part by itself using desktop.ini files, but manually writing them is a pain of course, hence the right-click idea.

The thing is, there already are tools to address folder tagging on Windows, but they’re often not feature complete, have limited XP compatibility or they lack very important features like timestamp preservation. Others have that last part, but only in paid versions of their software. So, time to do it by ourselves!

I picked his Powershell code up, and while it would run on my ancient XP x64 machines, implementing the required menu structure proved to be impossible, especially the cascading part. Ah, let me just show you the final product first, so you know what it was that I wanted (in my case, it’s meant for tagging media folders):

Changeicon context menu

“Changeicon” context menu (click to enlarge)

2. Creating the menu structure (XP + Vista compatible)

Starting with Windows 7, Microsoft introduced a new way to create submenus in the context menu, including icons, all in the Windows Registry. Older versions of Windows like Vista or XP however can’t do that, and I wanted a solution that works on all of them, from XP to Win10. So how can programs like 7-zip create such Explorer submenus with icons on legacy systems? They do so by injecting a COM .dll into the graphical shell, extending its capabilities. Typically, those are written in C++, and that’s not something I want to or can even do.

Luckily, we don’t have to develop that by ourselves, as somebody has already done it: Meet the [KuShellExtension]!

KuShellExtension – or KuShellExt in short – is a set of two library files, KuShellExtension.dll as well as KuShellExtension64.dll, the latter being meant for 64-bit Vista machines. It’s compiled for NT 5.2 though, so it’ll also work on XP x64 and Server 2003 x64. On top of that, it still works even on Windows 10! With the libraries comes an XML configuration file and some simple installer/uninstaller shell scripts.

In the example we’ve seen in the screenshot above, the respective configuration in config.xml would look like this:

expand/collapse source code
  1. <?xml version="1.0" encoding="utf-8"?>
  2. <config version="1">
  4.   <!-- Gobal variables -->
  5.   <var name="LEGACY_STYLE">false</var>
  6.   <var name="HIDE_MISSING">false</var>
  7.   <var name="ICON_DIR">${var:APPDATA}\changeicon\icons\</var>
  8.   <var name="INSTALL_DIR">${var:APPDATA}\changeicon\bin\</var>
  10.   <!-- Menu -->
  12.   <menu name="Tag Folder" class="folder" icon="${var:ICON_DIR}MainIcon.ico">
  13.     <!-- Submenus -->
  14.     <menuitem name="Tag as &quot;Currently watching&quot;" class="folder" icon="${var:ICON_DIR}CurrentlyWatching.ico" action="execute" console="false" multiple="N" workdir="C:\">
  15.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;CurrentlyWatching&quot;
  16.     </menuitem>
  17.     <menuitem name="Tag as &quot;Ongoing release&quot;" class="folder" icon="${var:ICON_DIR}WorkInProgress.ico" action="execute" console="false" multiple="N" workdir="C:\">
  18.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;WorkInProgress&quot;
  19.     </menuitem>
  20.     <menuitem name="Tag as &quot;Freshly completed&quot;" class="folder" icon="${var:ICON_DIR}NewlyCompleted.ico" action="execute" console="false" multiple="N" workdir="C:\">
  21.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;NewlyCompleted&quot;
  22.     </menuitem>
  23.     <menuitem name="----"></menuitem> <!-- Separator -->
  24.     <menuitem name="Tag as &quot;Favorite&quot;" class="folder" icon="${var:ICON_DIR}Favorite.ico" action="execute" console="false" multiple="N" workdir="C:\">
  25.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Favorite&quot;
  26.     </menuitem>
  27.     <menuitem name="Tag as &quot;Top series&quot;" class="folder" icon="${var:ICON_DIR}Star.ico" action="execute" console="false" multiple="N" workdir="C:\">
  28.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Star&quot;
  29.     </menuitem>
  30.     <menuitem name="Tag as &quot;Keep an Eye on for later (High priority)&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOnHighPrio.ico" action="execute" console="false" multiple="N" workdir="C:\">
  31.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOnHighPrio&quot;
  32.     </menuitem>
  33.     <menuitem name="Tag as &quot;Keep an Eye on for later&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOn.ico" action="execute" console="false" multiple="N" workdir="C:\">
  34.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOn&quot;
  35.     </menuitem>
  36.     <menuitem name="Tag as &quot;Keep an Eye on for later (Low priority)&quot;" class="folder" icon="${var:ICON_DIR}KeepAnEyeOnLowPrio.ico" action="execute" console="false" multiple="N" workdir="C:\">
  37.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;KeepAnEyeOnLowPrio&quot;
  38.     </menuitem>
  39.     <menuitem name="Tag as &quot;Not interested&quot;" class="folder" icon="${var:ICON_DIR}NotInterested.ico" action="execute" console="false" multiple="N" workdir="C:\">
  40.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;NotInterested&quot;
  41.     </menuitem>
  42.     <menuitem name="----"></menuitem> <!-- Separator -->
  43.     <menuitem name="Tag as &quot;Fluff&quot;" class="folder" icon="${var:ICON_DIR}SweetFluff.ico" action="execute" console="false" multiple="N" workdir="C:\">
  44.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;SweetFluff&quot;
  45.     </menuitem>
  46.     <menuitem name="Tag as &quot;Sweet, sweet Yuri!&quot;" class="folder" icon="${var:ICON_DIR}Yuri.ico" action="execute" console="false" multiple="N" workdir="C:\">
  47.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;Yuri&quot;
  48.     </menuitem>
  49.     <menuitem name="----"></menuitem> <!-- Separator -->
  50.     <menuitem name="Tag as &quot;A/V main folder&quot;" class="folder" icon="${var:ICON_DIR}AVMain.ico" action="execute" console="false" multiple="N" workdir="C:\">
  51.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;AVMain&quot;
  52.     </menuitem>
  53.     <menuitem name="----"></menuitem>
  54.     <menuitem name="Remove tag" class="folder" icon="X:\icons\Trash.ico" action="execute" console="false" multiple="N" workdir="C:\">
  55.       wscript &quot;//B&quot; &quot;//Nologo&quot; &quot;${var:INSTALL_DIR}changeicon.vbs&quot; &quot;%1&quot; &quot;del&quot;
  56.     </menuitem>
  57.   </menu>
  58. </config>

As you can see, it defines the main menu with its icon and then several submenu entries with their own icons. Also, it’s not calling my modified version of Umlüx’ Powershell script, but a Visual Basic Script instead. The reason for this shall be explained in point 4., for now we only care about the menu structure.

Looking at the variables defined on top of the XML data, ICON_DIR and INSTALL_DIR, they reference an icon folder and a program folder, both in %APPDATA%\changeicon\, so I put that into the current users’ profile folder. The icons to be used have to be put into the ICON_DIR, the scripts and an unlocker program are to be put into INSTALL_DIR. That doesn’t affect KuShellExt itself though, you can install that anywhere.

Note that while KuShellExt is loaded, you can edit its configuration on the fly. The library will detect changes made to it, and reload the updated configuration automatically, so you don’t have to unload and load the .dll when making changes to the menus or icon names.

Now, we still need the scripts and [Unlocker 1.9.2] (this is just the .exe, without the adware that usually comes with version 1.9.2). As to why Unlocker is required, well, let’s talk about that and also give you the first script:

3. The timestamp issue and the Powershell script that does the tagging

Note: Windows XP or Vista users may have to install the Windows Management Framework Core package (including Powershell 2.0) first.

This whole solution will alter folder appearances by placing hidden desktop.ini files into them. Writing such a file will alter the folders’ modification time though, or in short it’s “mtime”. And that’s bad if you have software that relies on that piece of meta data. One typical example would be backup software, that decides whether files have to be backupped based on the mtime.

In my own case, the backup problem applies to my rsync backup system, but there’s even more; I’ve written myself a Perl script that walks through my entire video folder, generating an HTML report out of the data, sorted by “latest first”. That way, I can see what movies or series have been added or modified recently. That script also depends on the mtime, so having it change when tagging folders is not acceptable!

Time to look at Umlüx’ script, or rather my modified version of it, changeicon.ps1:

changeicon.ps1, expand/collapse source code
  1. # Change Icon v0.815 Karl Veratschnig 2017
  2. # Modified by Michael Lackner
  3. #
  4. # Licensed under the GNU General Public license version 3.0 with express
  5. # permission by Karl Veratschnig.
  6. #
  7. # usage: 
  8. # .\changeicon.ps1 *dir_path* *icon*
  9. # i.e. ./changeicon.ps1 c:\testfolder Favorite
  10. #
  11. # run from anywhere:
  12. # Powershell.exe set-executionpolicy remotesigned -File "path to ps1"
  13. #
  14. # use "del" icon to revert folder to normal
  16. ###
  17. # Update-ExplorerIcon refreshes icons in Windows Explorer by rebuilding its icon cache
  18. # Written by Idera,
  19. #
  20. ###
  21. function Update-ExplorerIcon {
  22.   [CmdletBinding()]
  23.   param()
  25.   $code = @'
  26.   private static readonly IntPtr HWND_BROADCAST = new IntPtr(0xffff); 
  27.   private const int WM_SETTINGCHANGE = 0x1a; 
  28.   private const int SMTO_ABORTIFHUNG = 0x0002; 
  30.   [System.Runtime.InteropServices.DllImport("user32.dll", SetLastError=true, CharSet=CharSet.Auto)]
  31.   static extern bool SendNotifyMessage(IntPtr hWnd, uint Msg, UIntPtr wParam,IntPtr lParam);
  33.   [System.Runtime.InteropServices.DllImport("user32.dll", SetLastError = true)] 
  34.   private static extern IntPtr SendMessageTimeout ( IntPtr hWnd, int Msg, IntPtr wParam, string lParam, uint fuFlags, uint uTimeout, IntPtr lpdwResult ); 
  36.   [System.Runtime.InteropServices.DllImport("Shell32.dll")] 
  37.   private static extern int SHChangeNotify(int eventId, int flags, IntPtr item1, IntPtr item2);
  39.   public static void Refresh() {
  40.     SHChangeNotify(0x8000000, 0x1000, IntPtr.Zero, IntPtr.Zero);
  41.     SendMessageTimeout(HWND_BROADCAST, WM_SETTINGCHANGE, IntPtr.Zero, null, SMTO_ABORTIFHUNG, 100, IntPtr.Zero); 
  42.   }
  43. '@
  44.   Add-Type -MemberDefinition $code -Namespace MyWinAPI -Name Explorer 
  45.   [MyWinAPI.Explorer]::Refresh()
  46. }
  48. ###
  49. # User-configurable block
  50. ###
  51. $iconpath = "$env:APPDATA\icons\"           # Where icon files reside
  52. $installpath = "$env:APPDATA\bin\"          # Where scripts & binaries reside
  53. $unlocker = "unlocker.exe"                  # Name of the handle unlocker to use
  54. $ulparams = "/S"                            # Unlock parameter to supply to the unlocker
  56. ###
  57. # Non-user-configurable block / main program
  58. ###
  59. $folder = Get-Item $args[0]                 # Get folder from arguments
  60. & $installpath$unlocker "$folder" $ulparams # Remove open handles from folder
  61.                                             # WARNING: If files inside *are* open in
  62.                                             # other programs, their behavior might
  63.                                             # become unstable. Data loss is possible!
  64. $icon = $args[1]                            # Get icon name from arguments
  65. $olddate = $folder.LastWriteTime            # Get modification time stamp (mtime)
  66. $folder.attributes="Normal"                 # Reset folder attributes
  67. if (Test-Path "$folder\desktop.ini") {      # Check for existing desktop.ini
  68.   & $installpath$unlocker "$folder\desktop.ini" $ulparams # Unlock it, just to make sure
  69.   Remove-Item "$folder\desktop.ini" -Force    # Delete it if present
  70. }
  71. if($icon -ne "del") {                       # If op is to tag, not to delete...
  72.   $stream = [System.IO.StreamWriter] "$folder\desktop.ini" #... build new desktop.ini
  73.   $stream.WriteLine("[.ShellClassInfo]")            # Category
  74.   $stream.WriteLine("IconFile=$iconpath\$icon.ico") # Icon file
  75.   $stream.WriteLine("IconIndex=0")                  # Icon #0 in file
  76.   $stream.WriteLine("IconResource=$iconpath\$icon.ico,0") # Icon #0 in file (for modern OS)
  77.   $stream.WriteLine("Infotip=$icon")                # Info tip set to icon name
  78.   $stream.WriteLine("ConfirmFileOp=0")              # Disable special folder handling
  79.   $stream.WriteLine("TimeStamp=$olddate")           # Remark time stamp in the file for
  80.                                                     # safety/recovery purposes
  81.   $stream.close()                                   # Close file
  82.   $folder.attributes="ReadOnly"                     # Set folder to RO to enable special
  83.                                                     # desktop.ini handling
  84.   (Get-Item "$folder\desktop.ini").attributes = "Hidden,System" # Set desktop.ini
  85.                                                     # attributes to Hidden+System, also
  86.                                                     # to enable special folder handling
  87. } else {                                    # If op is to delete, not to tag...
  88.   if (Test-Path "$folder\desktop.ini") {      # Check whether desktop.ini really exists
  89.     & $installpath$unlocker "$folder\desktop.ini" $ulparams # Unlock it, just to make sure
  90.     Remove-Item "$folder\desktop.ini" -Force    # Delete it if present
  91.   }
  92. }
  94. Update-ExplorerIcon # Rebuild Windows Explorers' icon cache
  96. Start-Sleep -s 2    # This is to work around a race condition against writes/flushes on
  97.                     # some systems, so we give the system a little bit of time for its
  98.                     # "beauty sleep" before resetting the mtime.
  100. $folder.LastWriteTime = Get-Date $olddate   # Reset original mtime stamp


As you can see, the mtime is first being read from the folder by using $folder.LastWriteTime, and then reset to that value after writing the desktop.ini to the target directory. Also, it calls unlocker.exe /S on that folder to unlock it first, no matter whether there are open files within that folder or anything. This is done to avoid the dangerous situation, where the code would write the desktop.ini, but would then fail to update the folders’ timestamp due to open handles on the directory. Often this would also be caused by Windows Explorer itself, especially if you have open subfolders when tagging.

Forcefully unlocking the folder first deals with that problem. Please keep in mind that doing this may make programs relying on their open handles to behave in an undefined way however. Loss of data in that folder could be possible, e.g. if a text editor loses its handle for writing to a file in that folder.

In case something still goes wrong, the mtime is also being remarked in the desktop.ini file, in the value TimeStamp.

Here’s a sample .ini, as generated by changeicon, the format of the timestamp depends on your locale:

  1. [.ShellClassInfo]
  2. IconFile=C:\Documents and Settings\USERNAME\Application Data\changeicon\icons\WorkInProgress.ico
  3. IconIndex=0
  4. IconResource=C:\Documents and Settings\USERNAME\Application Data\changeicon\icons\WorkInProgress.ico,0
  5. Infotip=WorkInProgress
  6. ConfirmFileOp=0
  7. TimeStamp=09/22/2017 08:59:08

Now, there is one cosmetic issue left here…

4. Hiding that ugly terminal window

You can try to hide the window that pops up when calling a Powershell script, but it never really works reliably. So, are we going to tolerate that thing popping up every time we tag a folder? As if! That’s where the VBScript code comes in. This is an old trick I’ve used to run cmd batch scripts in a hidden way before, and this also works for calling Powershell scripts. I call this changeicon.vbs:

changeicon.vbs, expand/collapse source code
  1. ' Declare variables and create shell object:
  2. Dim AppData, WinDir, PowerShell, arg0, arg1
  3. Set WshShell = Wscript.CreateObject("WScript.Shell")
  5. '''
  6. ' User-configurable part (file/folder locations)
  7. '''
  8. AppData = wshShell.ExpandEnvironmentStrings("%APPDATA%")
  9. WinDir = wshShell.ExpandEnvironmentStrings("%WINDIR%")
  10. PowerShell = WinDir & "\system32\WindowsPowerShell\v1.0\powershell.exe"
  12. '''
  13. ' Non-user-configurable part
  14. '''
  15. ' Pull command line arguments into variables:
  16. arg0 = WScript.Arguments.Item(0)
  17. arg1 = WScript.Arguments.Item(1)
  18. ' Execute program:
  19. WshShell.Run """" & PowerShell & """ """ & "-File" & """ """ & AppData & "\changeicon\bin\changeicon.ps1" & """ """ & arg0 & """ """ & arg1 & """", 0
  20. Set WshShell = Nothing


5. Actually running that Frankenstein solution

In essence, you need to do the following:

  1. Pick a folder for changeicon.ps1, changeicon.vbs and unlocker.exe, and put those three files into it.
  2. Pick a folder for your icons, and place all your desired icons into it, no subfolders!
  3. Edit changeicon.ps1 and changeicon.vbs and change the install/icon paths.
  4. Install KuShellExtension and run its DLL hook script install.cmd.
  5. Edit KuShellExts’ settings.xml to reflect your menu structure and the corresponding menu icons and commands to execute. Better don’t delete all the comments in that file, the documentation can be pretty helpful at times.

Unfortunately I can’t share the icons I’ve created because they’re based on Microsofts’ icons, but you can easily find icons online or make your own with Microangelo or IcoFX. Both are commercial software for Windows, but you could also use the Gimp for that.

6. Enjoy tagging folders

Folders tagged with changeicon

Folders tagged with changeicon

With that, it’s much, much easier to keep track of things and to not forget what kind of stuff (*cough* tons of Anime *cough*) I still need to watch or keep an eye on for later.

But it’s not limited to that; You could use tagged folders for pretty much anything, like designating them to specific purposes or use them for document or work classification, whatever.

It’s interesting that even Windows 10 still can’t do that via the GUI by default by now…

Anyway, thanks fly out to [Umlüx] for writing the most important part at the core of this mess, the Powershell script, and also to [Idera] for the icon refreshing code I grabbed from their site! Also, if you want Umlüx’ modern solution for Windows 7+, which is based on pure Powershell code and Registry entries, you’d need to contact him directly. You may wish to do so, if you don’t need XP or Vista, because then you wouldn’t need to rely on the KuShellExtension anymore.

Happy tagging!


[1] Logo based on the Windows 10 Custom Folder Icons Pack made by Terraromaster