The GAT at XIN.at – GrandAdmiralThrawns technology (or whatever this is right now) weblog at XIN.at. Incompatibility is weakness!

XIN.at: Maintenance(s) and one complete infrastructural change

Feb 072023

There are two things to be announced around here, with one hopefully being rather minor and one being major. So let’s do the probably minor one first: On Wednesday, 2023-02-15 at 01:00 – 06:00 UTC+1, there will be a planned maintenance in the automatic exchange. This is probably linked to the other upcoming thing, but yeah. According to my internet service provider Magenta this might actually result in a full-blown 5 hour downtime, just so you’re warned. Alright, now to the big change!

Magenta, likely together with many other ISPs and network carriers has now announced that the older, purely symmetric [G.SHDSL] technology will be completely dropped, client and DSLAM hardware both. Granted, the technology is pretty old by now, having been standardized in 2001 and introduced around here somewhere in 2003, if I remember correctly. It does have some advantages even today though, such as low latency (or “pings”) as well as full duplex, just like the CAPI ISDN I had before. So full bandwidth up and down at the same time, all the time.

Paradyne/Zhone SNE2040G G.SHDSL network extender

The Zhone SNE2040G G.SHDSL network extender that connects XIN.at to the world at the time of writing

In the XIN.at case, G.SHDSL is implemented as bridged ethernet from the local extender box above to the DSLAM over multiple copper wire pairs, four in parallel in its final version. As a leased line it also comes with contractually guaranteed bandwidths for both up- and downstream. It had quite the history around here, first being offered by the purely Austrian company “iNode”, then “UPC” and now [Magenta], a daughter company of the German [Telekom Deutschland], which acquired UPC’s Austrian divisions to gain access to their infrastructure for feeding their T.Mobile LTE and 5G towers into the internet. They keep doing the regular DSL stuff as well though.

So, what will they replace the good old tech with? Since FTTH (“Fiber To The Home”) is way too expensive, it’ll be virtually unbundled VDSL2. Technically, the main difference should be higher bandwidth through wider frequency spectrum coverage at a significantly lower range. Or in other words: Higher bandwidth as long as you’re close enough to the fiber. I guess this is “FTTN” then, “Fiber To The Node”? Given that the automatic exchange should now be hooked up to Fiber, which started arriving in my town somewhat belatedly last year.

The question is whether I should even keep the symmetric stuff – which VDSL2 does support to some extent – or switch to a much cheaper asymmetric product, when both will be using VDSL2 anyway. I’ll have to decide upon that soon, and it’ll be mostly depending on costs, given that symmetric lines tend to be much more expensive per given bandwidth.

Naturally, I’ll update you on the progress. Maybe I can make operating the ancient XIN.at monster both faster and cheaper, that’d be the optimum, especially now, where money’s running dry left and right…

Update 2022-02-16: In the end I decided to keep a symmetric link with VDSL2, at least for the time being (see below for details). Reasons: I get +50% bandwidth at 12/12 mbit/s – still not much, but okay – at the same price with a 100% bandwidth guarantee. Asymmetric links are much cheaper, but they guarantee nothing in terms of speed, which is risky. Also, the new product would cost significantly more upon forming a new contract. So getting an asymmetric link and finding out it sucks and then going back to the symmetric VDSL2 line would seriously drain my wallet. And 12/12 would be the slowest available product, so ugh… We’ll see whether I’ll reconsider some time in the future, depending on how critical the financial situation will get.

The migration will happen on 2023-03-07 at 09:00 – 11:00 am UTC+1, so expect this server to go offline for some time during that period.

That’s it for the update.

Update 2023-0308: The migration has been completed yesterday morning at around 10:00 am UTC+1. Surprisingly, what I got was not VDSL2, but instead [G.fast] with more modern vectoring. The fiber solution here being FTTC, as the optical cables terminate in a box at the end of the road, making the last 100 – 2o0 or so metres copper. Instead of being an OSI level 2 network extender, the new device is is an OSI level 3 router by Cisco, which runs [IOS]. The model name being C1113-8P:

: Cisco C1113-8P G.fast router

: Rear of the box

Replacing the old SNE2040, this thing is significantly larger, works with only one instead of four copper pairs on the DSL side and features no fan, being passively cooled. Another difference is that it doesn’t have Rx/Tx LEDs but instead some really bright green activity LEDs and a brightly illuminated blue Cisco logo. Looks strangely purple in the photos, but whatever. The serial terminal LED is also crazy bright. Who even needs this? When switching off the ceiling lamp, the entire room is illuminated in green and blue now.

Fun facts: Apple actually licensed the “IOS” name from Cisco just so they could call their mobile operating system [iOS], and the same is actually true for Apple’s [iPhone] name, which Cisco’s Linksys division had used to release [VoIP phones under] before Apple had released its first smartphone. After that release, they were sued by Cisco for trademark infringement. Guess those two have a history.

In any case, XIN.at now got that +50% bandwidth increase at no extra cost. At least something, even though the uplink is still slow with its 12/12 Mbit/s. I’ll still have to think about that asymmetrical connection, as the maximum net bandwidth using G.fast in my flat would be 80/15 Mbit/s according to the technician who installed the Cisco box in my home. Another idea would be to get a second, cheap DSL line and segment my networks in such a way that e.g. the gaming machine would be routed out via that secondary connection with higher downstream speed. Makes more sense than a mobile connection, as the fastest possible link via that would go via LTE at a maximum of 18/3 Mbit/s, also according to a measurement by that same technician. We’ll see.

How to get FreeCiV 2.6.7 running on Windows XP / XP x64

Gaming, Windows 1 Response »

Nov 282022

0. Introduction

Yeah, it’s another one of “those” posts. I would have never even found [FreeCiV] by myself (I only played Civilization V), but IRC user Srandista let me know that the free and open-source re-imagination of the “Civilization” games has issues on Windows XP and that they can be fixed by dropping in a specific DLL. The solution was initially [posted on the FreeCiV forums] by user [log65536]. Mind you that the newer 3.x releases won’t work and would be quite hard to fix, so this is specifically for the older version 2.6.7, latest in the 2.x series of releases. The reason for this is mostly that development of the game was shifted from a MSYS/mingw to a newer, MSYS2/mingw build toolchain, which introduces many new Windows API dependencies that cannot be easily fixed for XP. At least it’s far too much for me to binary-hack, and also too hard to recompile & link from source in an older MSYS framework. So 2.6.7 it is. You can get all versions for Windows including 2.6.7 from [here].

1. The issue

The game links against dbghelp.dll at runtime. Problem is that Microsoft bundles a crippled version of the library with each version of Windows. This bundled library has less features than the ones shipped with the Debugging Tools for Windows in the Windows SDK, or with Microsoft Visual Studio. With a stock dbghelp.dll you’ll run into the following error when trying to run FreeCiV 2.6.7 on any Windows XP or XP x64 system:

EnumDirTreeW() missing in the stock dbghelp.dll, version 5.2.3790.1830

EnumDirTreeW() is just one of the functions missing. %WINDIR%\system32\dbghelp.dll (32-bit) and/or %WINDIR%\SysWOW64\dbghelp.dll (32-bit on XP x64) cannot easily be replaced, as the libraries are protected by WFP, the Windows File Protection. But we can still put a sufficiently capable library into the game’s folder, where it’ll be looked for first.

2. How to fix it, the quick way

Files before more details, so download the following archive and unpack it with [7-Zip]:

[dbghelp library] (version 6.12.2.633, x86_32)

Take the resulting dbghelp.dll and drop it in the main game folder after installation, so in the folder where the game’s main EXE files are. Do not attempt to replace the system libraries with it. Even if you bypass WFP, this could affect your system’s stability. Then, launch the game and it’ll just work:

FreeCiV 2.6.7 with SDL GUI running on XP x64 (click to enlarge)

3. Background information

3a. Origins of dbghelp.dll

dbghelp.dll is a debugging library for things like stack traces, symbol table lookups etc. User log65536 originally suggested just taking it from the good old MPC-HC media player, which bundles version 6.12.2.633 of the DLL in its final [1.7.13 release] in the subfolder CrashReporter\. While this works, I wanted to know what that library actually is and where to originally obtain it from.

It was originally shipped in constantly upgraded versions with the Microsoft “Debugging Tools for Windows”, which are also included in the Windows SDK. The corresponding SDK v7.1 can still be [officially downloaded] in its final version from Microsoft at the time of writing. The DLL was also bundled with different versions of Microsoft’s development environment. The exact version bundled with MPC-HC 1.7.13 is actually from the very same SDK, as included in Microsoft Visual Studio 2010 and possibly other, newer versions. So if you already have that, you don’t need to fetch the SDK from Microsoft. You’ll already have a version of the library that’ll likely get the job done.

When looking at the 5.2.3790.1830 version on the left and 6.12.2.633 version on the right side by side in [CFF Explorer], it can be seen that the symbol table of the newer version from the Windows SDK is much larger than the one of the DLL bundled with Windows XP x64 in this case, so it implements and exports far more functions:

Comparing the symbol tables of two different dbghelp.dll versions (click to enlarge)

For a quick comparison, just look at the main scrollbars to the right of each window. The list is just longer for the newer DLL, the file also being larger. A closer look shows that the newer DLL exports EnumDirTreeW() as well, whereas the older one doesn’t. That’s the reason for the missing function from the error message in the beginning. If you want to learn more about that function, see [the documentation].

3b. Getting dbghelp.dll out of the Microsoft Windows SDK v7.1

For me, the ISO version of the Windows SDK v7.1 failed to install on XP x64, as the installer wanted to install a 64-bit version of the tools that wasn’t even on the DVD image. Instead, I chose to unpack the required DLL from its MSI installers, because the main MSI would not allow an installation without running the broken setup.exe. To do that, launch the Windows command prompt by running cmd.exe. Windows ships with a command line tool msiexec.exe that can then do the job like this:

msiexec.exe /a "<msifile>" /qb TARGETDIR="<targetfolder>"

So, e.g.:

msiexec.exe /a "D:\Setup\WinSDKDebugToolsRedist\WinSDKDebugToolsRedist_x86.msi" /qb TARGETDIR="X:\debugtools-extracted\"

Then, in X:\debugtools-extracted\Program Files\Microsoft SDKs\Windows\v7.1\Redist\Debugging Tools for Windows\ you’ll find dbg_x86.msi, dbg_amd64.msi and dbg_ia64.msi. Just double-click the x86 version and install it like you normally would. This one actually allows it. That’ll give you e.g. X:\Program Files (x86)\Debugging Tools for Windows (x86)\dbghelp.dll amongst many other programs and libraries. Just copy that file to the FreeCiV 2.6.7 game installation folder.

Naturally, it’s much easier to obtain it from MPC-HC 1.7.13 in its 32-bit release, but the Debugging Tools would be the official source.

3c. Licensing

Note that there is one other, non-technical difference between the DLLs shipped with Windows and the ones shipped with the Debugging Tools for Windows. The former cannot be shared via the Internet or in any other way, as it’s a part of the operating system. The [Microsoft Windows license] simply does not permit this.

The version from the Debugging Tools however is a redistributable released under a [different license] bundled with a corresponding [redist.txt] file showing that the dbghelp.dll included is indeed legally redistributable. That’s why I can serve it here.

Thanks fly out to Srandista for letting me know about this hack as well as FreeCiV in the first place, and also to log65536 for discovering that there are more full-featured versions of dbghelp.dll available!

FreeBSD with lots of SIGSEGV & SIGBUS crashes: A diagnosis

Hardware No Responses »

Nov 222022

0. Introduction

Over the course of several months (actually, years even), my Threadripper 3970X machine which I have already shown off here[1][2][3][4] started producing catastrophic program crashes. I didn’t pay too much attention to it until recently, as the frequency of crashes started increasing into what was more like a storm rather than an occasional occurrence. Clearly, something had to be done about it. At first, I had suspected the (expensive!) processor, because I already know somebody who had his own Threadripper 3970X break down during 3D rendering work. In my case, I could observe multiple processes receive signals 10 (Meaning SIGBUS on FreeBSD, a bus error) and 11 (SIGSEGV, a segmentation fault). The former means that the machine was told to access memory at a memory address that isn’t even an address, so doesn’t describe anything on the address bus. It’s like I tell you to drive a banana. It’s a banana, not a car or bike, so something’s clearly very wrong with that statement. Unless a bad programming error causes it, this typically means broken hardware. Like, CPU, RAM, mainboard, etc.

When a segmentation fault happens, the memory address itself was okay, but the memory addressed did not belong to the process attempting the access. This is illegal at least for security reasons. This is less indicative of a hardware fault, but since it’s mixed together with bus errors…

1. Super short version

You get lots of programs crashing on you with SIGBUS and SIGSEGV errors in /var/log/messages when they clearly shouldn’t? Try checking your memory first!

2. The failing processes

Here’s a list of total failures, grouped by signal over the course of about two years:

Program	num(SIGSEGV)	num(SIGBUS)
x265	89	7
php-fpm	4	55
bash	2	24
ffmpeg	4	1
tmux	0	2
bsdisks	1	1
smartctl	0	1
nmbd	1	0
mkvtoolnix-gui	1	0
zabbix_agentd	1	0
VirtualBoxVM	1	0

And here’s a list showing how the frequency rose dramatically in recent months, it follows the log rotation, one line per log file, which is why the time periods are all over the place:

Time frame	num(SIGSEGV)	num(SIGBUS)
2021, Jan 1st – Jul 15th (≈6½ months)	4	0
2021, Jul 15th – Dec 31st (≈6½ months)	20	1
2022, Jan 1st – Feb 8th (≈1¼ months)	7	0
2022, Feb 9th – May 25th (≈3½ months)	23	0
2022, May 26th – Sep 21st (≈4 months)	10	0
2022, Sep 22nd – Nov 22nd (≈4 months)	44	91

I made another clumsy attempt to visualize this in Zabbix by monitoring x265 processes and detecting stuck / broken ones. They only stay in this state for a limited time before they’re killed by the kernel, so this isn’t really all that representative. But it still shows how the problem got much more intense towards the end:

Attempt to visualize x265 crashes

php-fpm is doing quite a bit of stuff all the time due to the Zabbix web monitoring system. x265 is the real heavy lifter though, doing SDTV, 1080p and 4K/UHD encoding work. And lots of crashing. ffmpeg interestingly shows only few crashes, despite it also doing quite a bit of work. Errors like the above show up in /var/log/messages as shown below, this is just a few extracted samples:

Jun  1 14:57:52 BEAST kernel: pid 42555 (getpgid), jid 0, uid 1001: exited on signal 11 (core dumped)
Sep 23 19:17:27 BEAST kernel: pid 25018 (php-fpm), jid 0, uid 80: exited on signal 10
Sep 29 17:06:35 BEAST kernel: pid 24064 (x265), jid 0, uid 1001: exited on signal 11 (core dumped)
Oct  3 08:00:21 BEAST kernel: pid 90061 (bash), jid 0, uid 1001: exited on signal 10 (core dumped)
Oct 26 08:14:06 BEAST kernel: pid 36978 (bash), jid 0, uid 1001: exited on signal 11 (core dumped)
Oct 10 01:18:39 BEAST kernel: pid 77471 (grep), jid 0, uid 1001: exited on signal 11 (core dumped)
Oct 14 07:08:57 BEAST kernel: pid 37931 (ffmpeg), jid 0, uid 1001: exited on signal 11 (core dumped)
Oct 15 07:31:27 BEAST nmbd[1702]: [2022/10/15 07:31:27.286361,  0] ../../lib/util/fault.c:80(fault_report)
Oct 15 07:31:27 BEAST nmbd[1702]:   INTERNAL ERROR: Signal 11 in pid 1702 (4.12.15)
Oct 15 07:31:28 BEAST kernel: pid 1702 (nmbd), jid 0, uid 0: exited on signal 11 (core dumped)
Oct 17 18:39:36 BEAST kernel: pid 8340 (x265), jid 0, uid 1001: exited on signal 10 (core dumped)

Another result of this was the corruption of the MySQL database holding Zabbix’ data. One of the tables had its index corrupted, and setting the innodb_force_recovery option in my.cnf to 1 could not recover it. So it had to be recreated the mysqldump / DROP TABLE tablename; / restore way.

3. The reason

So, it was not the CPU. I would’ve hated having to send it back to AMD for replacement. So, when you start seeing a mix of SIGSEGV and SIGBUS errors in your logs and programs start crashing seemingly randomly, you may want to try testing your main memory first. I didn’t suspect it, because it’s Kingston, and Kingston RAM hasn’t ever failed me, not in over 20 years. Guess there’s a first time for everything. I asked a colleague and software engineer – who does some Linux Kernel hacking every now and then – about those SIGSEGV/SIGBUS errors. He recommended testing the RAM first, because it’s easy and means testing the cheapest component before any other. On top of that, some components are harder to diagnose than others, like e.g. mainboards.

On FreeBSD, you can try [memtester] first, which is available as a package for most architectures. Run it as superuser root, so it can lock the memory to be tested. Try to lock as much as you can afford to to improve detection probability. Naturally, this has its limits, as it cannot test any memory currently in use by the operating system, services and programs. In my case, it showed errors in its initial “stuck address” test already, which indicates something is really horribly wrong with some parts of the memory subsystem. This also means that further results are unreliable, but I still let it run for a while, and errors kept piling up pretty quickly.

I had no prior experience with the tool however, so I additionally booted [memtest86] (yeah I know, that’s not the free software version…) from a USB pen drive to see whether it would confirm memtester’s results. And yes, it did so pretty quickly as well.

All that was left to do was to narrow it down. Took a while, because I had 8 sticks in there. I tested them in groups of 4, then 2, then single ones. This also showed that it had to be the memory, not the memory controllers or the mainboard. After a while, I managed to identify a single, broken DIMM. So, one of those, the photo is from an older post around here:

256GiB Kingston HyperX DDR4/3200 16-20-20

256GiB Kingston HyperX DDR4/3200 (click to enlarge)

Thankfully, it’s already replaced, as Kingston’s RMA process is simple to use and works fast, as long as lifetime warranty still applies. You can access it [here].

Backporting ffmpeg 5.1.2 to Microsoft Windows XP / XP x64

Powerful Tools, Programming, Uncategorized, Windows No Responses »

Nov 032022

0. Introduction

This post is more documentation for myself rather than anything else. As I compiled ffmpeg 2.2.16 for my (mostly) successful backport of Audacium 1.0.0, which can be seen [in the comments here], I thought it’d be fun to once again attempt compilation of an XP-patched version of the latest ffmpeg release, which at the time of writing is [5.1.2 “Riemann”]. The idea was to build both a 32-bit version for regular Windows XP as well as a more powerful 64-bit version for Windows XP Pro x64 Edition and Server 2003 x64. As before, these are not as complete as the releases you might be used to; Most external libraries are missing here (Theora, Vorbis, WebP, lamemp3, FdK-AAC and many, many more), but the latest [libzimg release version 3.0.4] is included, as I need that one myself for high quality spatial and colorspace scaling, so for HDR → SDR tonemapping.

1. The downloads

For those who just want the release files, here they are, license is the [GNU General Public license version 3] or later, any modified source code is included:

[ffmpeg 5.1.2 for Windows XP x86_32 (32-bit)] (including GnuTLS and libzimg)
[ffmpeg 5.1.2 for Windows XP x86_64 (64-bit)] (including GnuTLS and libzimg)

Here’s a quick screenshot showing how ffmpeg was configured for this backport:

ffmpeg 5.1.2 x86_64 running on XP x64

As you can see, support for cryptographic network connections is built-in using GnuTLS. I can’t vouch for much though, as it’s a relatively old version of the library, so likely no modern ciphers and surely no TLS v1.3. GDI grabbing for desktop screen recording had to be disabled, because I couldn’t get the stuff to compile. Modern GPU hardware acceleration has been disabled as well, because Windows XP doesn’t have Direct3D 11, DXVA or Vulkan anyway.

Hence I do not recommend using this backport on modern systems like Windows 7, 8.1, 10 or 11. Makes no sense, you’d be better off with other releases. For XP, it might be good enough, depending on what you need. I’m open to requests for additional, built-in libraries, but can’t promise anything. And please forget about AV1, no dice there, at least not with my skills. I tried libaom, libdav1d and librav1e, and couldn’t even get the build systems working.

2. How the backport was done

First of all, this is no MSYS/mingw build, but a dynamically linked CygWin one. For a platform, CygWin 1.7.32 was used. You can still get those older CygWin versions from the [CygWin time machine] project. Additionally, I compiled myself newer build tools and a C/C++ compiler to handle modern instruction set extensions and C/C++ standards. The following software was used for both the 32-bit and the 64-bit builds:

CygWin 1.7.32 (0.274/5/3)
GNU autoconf 2.69
GNU make 4.0
GNU binutils 2.33.1
nasm 2.14.02
GCC 9.2.0

2a. libzimg 3.0.4

For libzimg, I had problems with certain functions being called from the std namespace, where the compiler wouldn’t find them. So whenever it died, complaining about something like std::somefunction not being defined, I would just change the calls to somefunction in the corresponding source files, removing the std namespace identifiers. Just do so until the whole code compiles, then install it as usually done with autoconf+make based build systems.

2b. ffmpeg 5.1.2 “Riemann”

Before starting with ffmpeg, make sure that its build toolchain will be able to locate zimg’s pkg-config metadata file /usr/local/lib/pkgconfig/zimg.pc. Otherwise, it won’t be able to find libzimg at all and fail during the initial configuration step. Do so by exporting both the default and this additional path: $ export PKG_CONFIG_PATH="/usr/lib/pkgconfig:/usr/local/lib/pkgconfig". It may generally be a good idea to just add this to ~/.bash_profile, as it’s very useful to make build systems look for .pc files in both locations, since most autoconf+make build systems will by default install to /usr/local/.

Additionally to that, export some C/C++ environment variables for picking the desired, modern compiler and for including all required headers:

$ export CC="gcc-9.2.0"
$ export CXX="c++-9.2.0"
$ export CFLAGS="-I/usr/include -I/usr/local/include"
$ export CXXFLAGS="-I/usr/include -I/usr/local/include"

Then, start modifying build system and code! First, the configure script needs to be edited. Go to around line 2320, and look for the following block:

SYSTEM_LIBRARIES="
    bcrypt
    vaapi_drm
    vaapi_x11
    vdpau_x11
"

What we need to get rid of is the bcrypt (bcrypt.dll) library, as this modern, cryptographic library does not exist on any Windows XP version. Replace it with the following:

SYSTEM_LIBRARIES="
    vaapi_drm
    vaapi_x11
    vdpau_x11
"

Further down, more bcrypt stuff needs to be ripped out, jump to about line 3800 and look for this:

avutil_suggest="clock_gettime ffnvcodec libm libdrm libmfx opencl user32 vaapi vulkan videotoolbox
corefoundation corevideo coremedia bcrypt stdatomic"

Replace it with the following:

avutil_suggest="clock_gettime ffnvcodec libm libdrm libmfx opencl user32 vaapi vulkan videotoolbox
corefoundation corevideo coremedia stdatomic"

We’re not done removing it yet, jump to line 6330 or so and look for the following code:

check_lib bcrypt   "windows.h bcrypt.h"   BCryptGenRandom     -lbcrypt &&
    check_cpp_condition bcrypt bcrypt.h "defined BCRYPT_RNG_ALGORITHM"

Just comment it out:

#check_lib bcrypt   "windows.h bcrypt.h"   BCryptGenRandom     -lbcrypt &&
#    check_cpp_condition bcrypt bcrypt.h "defined BCRYPT_RNG_ALGORITHM"

That’s it for the configuration script. Unfortunately, even with that, the code will still try to include the Windows bcrypt header file for some random number generation code. Open the C file libavutil/random_seed.c and hop to about line 30. Look for the following preprocessor conditional block:

#if HAVE_BCRYPT
#include 
#include 
#endif

Actually, we should just make sure we don’t have HAVE_BCRYPT set, but the configuration step somehow doesn’t take care of that. While not a clean solution, let’s just comment it out to be done with Microsoft’s bcrypt library:

/*
 * #if HAVE_BCRYPT
 * #include 
 * #include 
 * #endif
 */

And that’s it, as long as your build tools are new enough. As for configuring the code, it depends on what external libraries you have and wish to include and what your licensing restrictions might be like, but as shown in the screenshot in the beginning, I configured ffmpeg as follows:

$ ./configure --cc="gcc-9.2.0.exe" --cxx="c++-9.2.0.exe" --enable-shared --enable-pic --enable-gpl \
--enable-version3 --enable-bsf=hevc_metadata --enable-libzimg --disable-w32threads \
--disable-schannel --enable-gnutls --disable-vulkan --disable-dxva2 --disable-d3d11va \
--disable-indev=gdigrab --disable-htmlpages --disable-manpages --disable-podpages --disable-debug

After that, just compile it by running $ make, as with libzimg. As ffmpeg is rather large, you may wish to speed up the process by utilizing multiple CPUs. To see how many CPUs are usable by CygWin, just run $ nproc. Let’s assume that number would be 16, then just run $ make -j 16, and it will compile much faster.

3. Deployment

To know which .dll files I need to include, I’d usually just copy the resulting ffmpeg.exe and ffprobe.exe files to some temporary, empty path like X:\release\ffmpeg512\, outside of the CygWin environment. Then, just launch a CMD terminal and clear the local search paths to ensure you’re not pulling in any libraries from anywhere at load time by accident:

SET PATH=

Alright, now switch to the folder with the two .exe files, e.g. CD /D X:\release\ffmpeg512\ and run .\ffmpeg.exe -version. It will surely complain about some missing DLL. Locate that DLL inside of your CygWin bin\ folder or the usr\local\bin\ one. Then, rinse and repeat for as many times as necessary – about 30 times in my case – and copy over each missing DLL to that X:\release\ffmpeg512\ folder. Once done, you can zip it up and redeploy it on any Windows XP machine, as all required libraries are included without any ballast.

Naturally, 64-bit builds will only run on Windows XP Professional x64 Edition and Windows Server 2003 x64 or newer, whereas the 32-bit one would work on any XP machine or newer.

So much for libzimg 3.0.4 and ffmpeg 5.1.2 on some very old Microsoft operating systems.

FreeBSD: Tuning a GEOM RAID-0 with fast SSDs, with a focus on BIO_DELETE under very high CPU load

Storage, Unix No Responses »

Sep 152022

[1] For the past two years I’ve been running all my H.265/HEVC + AAC-LC media encodes plus some others (some H.264/AVC video) on a Threadripper 3970X machine, which i described in some other posts: [Part 1], [Part 2], [Part 3] & [Part 4]. This was to consolidate the encoding that was previously done on multiple, older boxes. For an operating system it uses the UNIX-like [FreeBSD]. In January last year I had this powerful machine [upgraded] with four 1 TB SSDs as the 2 TB system SSD became too small to handle certain jobs without running out of disk space. I avoided the use of the powerful ZFS file system due to a lack of experience with it, plus my assumption that it would consume more RAM and CPU resources than the older and simpler UFS. Since UFS has no built-in RAID functionality, I chose FreeBSD’s [GEOM RAID] to create a simple, striped RAID-0, as no data safety is required for this volume.

: Ryzen Threadripper 3970X being put into its socket

: Silverstone Primera case

: “Hyper Quad” riser with M.2 SSDs

GEOM RAID was chosen instead of [GEOM stripe], because the latter showed worse performance in my tests and does not support [GEOM bio controlling functionality], which means no support for BIO_DELETE, and hence no support for ATA TRIM, SCSI UNMAP or NVMe Deallocate for SSDs. The machine has five SSDs in total: A single Corsair MP600 with 2 TB and four Corsair MP600 with 1 TB each. They’re connected via a bifurcated PCI Express 4.0 x16 slot, so 4 PCIe 4.0 lanes per SSD.

The array was created with a 16 kiB stripe block size, so a full 4-disk strip would be 64 kiB large. The UFS file system on it was created with a similar 16 kiB block size, so a full RAID strip would be 4 file system blocks wide and each file system block would be the same size as a RAID stripe block.

I don’t remember it clearly, but I believe I determined that size by inspecting the MKV formats’ interleaving and the sizes of individual stream block sizes with a focus on the largest part: The video stream(s). So I thought I should pick a stripe block size closest to something that would match the I/O done by the [MKVToolNix] toolset for (de)multiplexing .mkv containers, e.g. mostly the use of mkvextract and mkvmerge.

The manufacturer specifications for the single 2 TB SSD and the 1 TB SSDs are actually the same, they’re as follows:

read IOPS:	680000/s
write IOPS:	600000/s
read throughput:	4950 MiB/s
SLC-cached write throughput:	4250 MiB/s

I found the resulting RAID array to perform sub-par under very high CPU loads, but somewhat okay under very low CPU loads, with the exception of one thing: BIO_DELETE requests. Those always performed badly. I’ll go into more detail about that below. Here’s what I mean by “very high CPU loads” by the way:

BEAST Threadripper 3970X CPU load over 6 months

“BEAST” CPU load over a 6 month period (green line = physical core count, blue line = SMT/thread count)

As you can see, load is almost always high. And surprisingly, that had a big impact on I/O performance, as RAID and file system processes seemingly had to compete with user land processes – mostly video encoders – for CPU.

My first step to boost I/O performance under high CPU load was to give the GEOM framework and its RAID class real-time priority. While that may seem dangerous, it’s been working well for me for more than a year now:

#!/usr/bin/env sh
 
# Main GEOM framework
printf 'Uplifting the GEOM framework kernel process to real-time...\n'
ps ax | grep '\[geom\]' | grep -v 'grep' | sed 's/^ *//g' | cut -d' ' -f1 | while read -r pid; do
  printf "  PID: ${pid}\n"
  rtprio 20 -"${pid}"
done
printf 'Done.\n\n'
 
# GEOM RAID
printf 'Uplifting the GEOM RAID kernel process for arrays using\n'
printf 'DDF style meta data to real-time...\n'
ps ax | grep '\[g_raid DDF\]' | grep -v 'grep' | sed 's/^ *//g' | cut -d' ' -f1 | while read -r pid; do
  printf "  PID: ${pid}\n"
  rtprio 20 -"${pid}"
done
printf 'Done.\n\n'

This makes I/O quite a bit faster, but BIO_DELETE requests would still perform badly, see here, briefly after the deletion of some pretty large files, ≈110 GiB in total:

BIO_DELETE on GEOM RAID-0 with 16 kiB stripe and 16 kiB UFS block sizes

The interesting columns are d/s (deletes per second), kB/d (kiB per delete) and kB/s d (kiB/s for deletes). So let’s focus on that part for now.

Note that I’m using [gstat-rs] instead of the base [gstat] for those GEOM statistics. Just because it can sort its output better. I modified the Rust source code minimally to change the header color myself (really didn’t like the red+blue), but did not change anything else, the change starts at around line 620 in gstat/src/main.rs:

    // Modified by M. Lackner to turn the blue header background into black
    // let normal_style = Style::default().bg(Color::Blue);
    let normal_style = Style::default().bg(Color::Black);
 
    terminal.clear()?;
    loop {
        terminal.draw(|f| {
            let header_cells = columns.cols.iter()
                .enumerate()
                .filter(|(_i, col)| col.enabled)
                .map(|(i, col)| {
                    // Modified by M. Lackner to turn the red text
                    // color into a shade of orange
                    // let style = Style::default().fg(Color::Red);
                    let style = Style::default().fg(Color::Rgb(255, 215, 135));

So what do we see on the above screenshot? /dev/nvd1 to /dev/nvd4 are the components or members of the RAID-0. /dev/raid/r0 is the array and /dev/raid/r0p1 is the UFS-formatted file system.

Now the interesting part: The RAID itself is performing only 62 BIO_DELETEs per second (d/s), but each delete command being sent to the RAID is relatively large at 5758 kiB, or ≈5,62 MiB. That size also fluctuates a bit, so it’s dynamic. But when you look at the individual disks, the size of each delete command is only 16 kiB! And that is completely static, it stays like that for the entire sequence of BIO_DELETE / NVMe Deallocate commands. This seems to be making the whole process pretty slow, as we’re seeing a throughput of roughly 88000 kiB/s per SSD or 356300 kiB/s for the whole RAID. Just shy of 350 MiB/s.

It seems that when running BIO_DELETE over a GEOM RAID, the delete commands sent to the array get split up into small chunks equal to the size of the stripe blocks before sending them to the component SSDs.

Let’s do something similar on the single system SSD while the operation is still on its way on the RAID:

BIO_DELETE on the single SSD’s UFS (/dev/nvd0p2), which has been formatted with a 32 kiB block size

As you can see, the commands are larger for the system SSD when compared to the 16 kiB deletes being issued to the RAID component disks. Here, the delete size also fluctuates a bit, but not too much. In the screenshot above you can see a delete size of 1242 kiB (≈1,21 MiB) with 3570 deletes being done per second. This results in a whopping 4430512 kiB/s of throughput! That’s a nice and healthy ≈4,23 GiB/s! Wow. Like, that’s as fast as it gets, because it’s very close to what the manufacturer says the SSD can do in terms of SLC-cached write speed.

It appears that the breaking up of large into many small delete blocks for the RAID is really costly. Even in terms of CPU consumption (probably?). It’s also bound to be a waste of I/O operations.

Mind you, the SSDs are all of the same make and almost the same model, with just the size being a bit different. So the problem’s got to be the software.

After the completion of all compute jobs that needed that RAID, I decided to backup all data and destroy/reconstruct it as superuser root:

gpart destroy -F /dev/raid/r0
graid delete raid/r0

After that, I rebuilt the array, partitioned it with a new GUID partition table and a single partition in an aligned fashion, then formatted it with UFS. All at higher block sizes, 256 kiB for the RAID stripe blocks, and 64 kiB for UFS, which is the maximum the file system supports:

graid label -s 262144 DDF BEASTRAID RAID0 /dev/nvd1 /dev/nvd2 /dev/nvd3 /dev/nvd4
gpart create -s GPT /dev/raid/r0
gpart add -t freebsd-ufs -a 1024 -b 512 -l BEASTRAID /dev/raid/r0
newfs -E -L BEASTRAID -b 65536 -d 1048576 -f 65536 -g 1073741824 -h 32 -t -U /dev/raid/r0p1

Here, the -E option will BIO_DELETE the whole file system on creation. So it’s going to give us a first glance at what speeds we’re going to get for those operations:

BIO_DELETE during [newfs] on a GEOM RAID-0 with 256 kiB stripe and 64 kiB UFS block sizes

Again, we can see a static delete size, but now it’s 16 times larger at 256 kiB. We’re getting about 5243 deletes per second and a throughput of about 1342144 kiB/s (≈1,28 GiB/s) per component SSD. This is still not remotely near the maximum the SSD could do, but much better already. Let’s replay the actual file deletion on a fully formatted UFS, just like we did before:

BIO_DELETE on a GEOM RAID-0 with 256 kiB stripe and 64 kiB UFS block sizes

Alright, performance is similar to what newfs showed us. We can now see that the RAID device is getting very few deletes per second, but they’re really large per delete: It’s 33 deletes/s with each one being 17486 kiB or ≈17 MiB in size. Those are once again being broken up into much smaller deletes sent to the SSDs, but they’re at least 256 kiB in size instead of 16 kiB. Overall speed on the array is now 5762128 kiB/s or a nice ≈5,5 GiB/s. Yup, that I can most definitely live with! Only a bit faster than a single disk, but much better than before.

Actually, it scales nearly linearly with stripe block size! From ≈350 MiB/s to ≈5,5 GiB/s. That’s almost exactly 16 times as fast! And we upped the stripe block size by factor 16 as well…

I’m thinking this might scale even further, maybe all the way up near to the theoretical maximum SSD performance. If true, that would result in about ≈16 GiB/s over the entire array.

Maybe I’ll try reconstructing that RAID-0 again at an even larger stripe block size in the future, when it’s idle once more. I’m thinking 1 MiB stripe blocks here. I’ll update this article once I do that.

All those tests were done at a load value of around 110, so the 32-core CPU with 64 SMTs was really more than fully loaded on all cores and all hardware threads.

Okay, to be fair, I’ve only looked at GEOM BIO_DELETE today and nothing else really, but the improvement in performance isn’t bad in any case.

Update 2022-09-21: Alright, the idle state came faster than expected, and with it an opportunity to try and push this even further. I deleted and reconstructed the array once more, but this time with a very large stripe block size of 1 MiB. So four times the size since the first optimization, and 64 times the original size I had chosen in the beginning. And here are the results:

BIO_DELETE on a GEOM RAID-0 with 1 MiB stripe and 64 kiB UFS block sizes during newfs

During the initial file system creation including block deallocation like before, the individual SSDs now show a performance of about 5,25 GiB/s, which is nothing short of impressive, as this is even beyond manufacturer specification! We can see that each individual delete command is sent at a block size of exactly 1 MiB.

Something funny happened for the actual file deletion test though, see below.

BIO_DELETE on a GEOM RAID-0 with 1 MiB stripe and 64 kiB UFS block sizes

gstat-rs is actually not capable of showing the full number for throughput on the array anymore, as it’s too big to display in its column. The program simply truncates the least significant number.

Also, the I/O fluctuates a lot more now, and the delete block size interestingly is not always exactly 1 MiB, but sometimes just a few kiB short of that.

To better display this, I decided to show you a regular gstat screenshot instead. It breaks the alignment to be able to display the full number, and sorting is bad, but still:

BIO_DELETE on a GEOM RAID-0 with 1 MiB stripe and 64 kiB UFS block sizes, shown by regular gstat

Looking at the above screenshots, we can see a Deallocate bandwidth of 12,5 – 15 GiB/s! Now that’s fast! Scaling is no longer completely linear though, so this might be getting close to the end of scaling by enlarging stripe block sizes. These values translate to about 3,12 – 3,87 GiB/s per component SSD, which is not too far away from the specifications.

Load values were between 110 – 130 during these latest tests, so comparable to before.

Now, when compared to the state in the very beginning, this means an improvement by a factor of roughly 36× – 44× by making the stripe block size 64× as large. Crazy results, indeed!

I had not originally assumed that making the stripe blocks significantly larger would really help that much, but it appears I was mistaken! I must say, I’m quite happy with this, as a delete command now completes in just a few seconds instead of minutes for sizes around 100 GiB, where other I/O would be impacted in the meantime.

Nice!

[1] Original drawing is © by ASK (Pixiv profile). 【PFFK】琉璃, Pixiv Fantasia: Fallen Kings series. Altered and used with express permission.

Backporting MP4Box from GPAC 2.0 to Windows XP & XP x64

Media Encoding, Windows 4 Responses »

Sep 072022

0. Introduction

While this whole thing is actually pretty unnecessary, I still got that itch to compile myself a new version of GPAC’s MP4Box tool on Windows XP & XP x64. The trigger was just that I the same thing on my RedHat Enterprise Linux 8 workstation now, so I wanted all machines to have the same version of it. If you don’t know MP4Box, it’s a command line tool for UNIX-like and Windows systems that can multiplex to and work with .mp4 media container files. You’ll likely know those from YouTube and other streaming sites, cameras and/or smartphones. It’s the quasi-standard for media streaming files on the Internet these days, I suppose. What few videos are being hosted on this site also use it instead of the .mkv container, which is more widespread for offline, file-based playback.

To do this, I picked the current release version 2.0.0 of the [GPAC multimedia library], which is being developed for academic and professional use by the french engineering school [Télécom Paris] and can be obtained here: [GPAC 2.0.0].

1. The files

Before I go into the details, let’s provide the files including the finished programs, dependencies and the C99 source code first:

[GPAC 2.0.0 source code] (unmodified version)
[GPAC 2.0.0 source code] (modified to support Windows XP & XP x64)
[MP4Box from GPAC 2.0.0] (32 & 64-bit versions with XP & XP x64 support)
[Microsoft VisualC++2017 redistributables] (32 & 64-bit versions)

The GPAC program is licensed under the [GNU Lesser General Public License version 2.1] or later versions. A part of the code I added to the program (details below) comes from the [PlibC project] and is licensed under the [ISC license].

Since Windows XP support was dropped from the project at least since version [0.8.0-rev1-gc1990d5c], this once again had to be a backport instead of just a recompilation.

Please note that this is not a full GPAC build, but only a limited MP4Box-only one! So encoder/decoder support, the GPAC media player and other things are not included.

The 32-bit version has been tested on three 32-bit Windows XP machines, one of them a physical box, the other two being VMs. For some reason, it crashed on one of the VMs, but then again, it’s quite the modified OS in there, so it may be just that. The 64-bit version has been tested on two XP x64 machines, one physical, one virtual. In all cases, the test case was a simple H.264/AVC elementary video stream and AAC-LC audio stream multiplex to .mp4, something like this: mp4box.exe -add "video.h264" -add "audio.aac" "output.mp4"

2. The source code modification

First, we’ll modify some thread-related code in src\include\gpac\thread.h around line 71, look for the following code:

#include <winbase.h>

After that, add a new static inline function to wrap around the function InterlockedCompareExchange64() for 32-bit builds. We need this later to replace calls to InterlockedAdd64(), which does not exist on XP at all. There is also a problem with InterlockedExchangeAdd64(), which does exist only on XP x64 and newer, but not on 32-bit platforms at all. So we need to wrap it around InterlockedCompareExchange64() instead, but for 32-bit XP this also doesn’t exist (ugh…), so it needs to be reimplemented. I call this new function InterlockedCompareExchange64xp():

/* Addendum by M. Lackner: Wrap InterlockedExchangeAdd64() around
   InterlockedCompareExchange64() and name it specifically as an XP 
   function */
#ifndef GPAC_64_BITS
#pragma intrinsic(_InterlockedCompareExchange64)
#define InterlockedCompareExchange64xp _InterlockedCompareExchange64 
static inline LONGLONG InterlockedExchangeAdd64xp(_Inout_ LONGLONG volatile *Addend, _In_ LONGLONG Value)
{
  LONGLONG Old;
  do {
    Old = *Addend;
  } while (InterlockedCompareExchange64(Addend, Old + Value, Old) != Old);
  return Old;
}
#endif
/* End of addendum */

In the same header file, look for the following code block around line 90 or so:

/*! atomic integer addition */
#define safe_int_add(__v, inc_val) InterlockedAdd((int *) (__v), inc_val)
/*! atomic integer subtraction */
#define safe_int_sub(__v, dec_val) InterlockedAdd((int *) (__v), -dec_val)
/*! atomic large integer addition */
#define safe_int64_add(__v, inc_val) InterlockedAdd64((LONG64 *) (__v), inc_val)
/*! atomic large integer subtraction */
#define safe_int64_sub(__v, dec_val) InterlockedAdd64((LONG64 *) (__v), -dec_val)

We replace this with corresponding InterlockedExchangeAdd() calls, which are present on all XP platforms, with InterlockedExchangeAdd64() for XP x64 and our new InterlockedExchangeAdd64xp() wrapper for 32-bit XP:

/* Modification by M. Lackner for Windows XP & XP x64 compatibility */
//#define safe_int_add(__v, inc_val) InterlockedAdd((int *) (__v), inc_val)
/*! atomic integer subtraction */
//#define safe_int_sub(__v, dec_val) InterlockedAdd((int *) (__v), -dec_val)
/*! atomic large integer addition */
//#define safe_int64_add(__v, inc_val) InterlockedAdd64((LONG64 *) (__v), inc_val)
/*! atomic large integer subtraction */
//#define safe_int64_sub(__v, dec_val) InterlockedAdd64((LONG64 *) (__v), -dec_val)
 
/* InterlockedAdd and InterlockedAdd64 do not exist on Win XP, so we use
   the older InterlockedExchangeAdd and InterlockedExchangeAdd64 functions
   as well as a reimplementation of the latter for 32-bit XP.
   See:
   https://docs.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-interlockedexchangeadd
   https://docs.microsoft.com/en-us/windows/win32/api/winnt/nf-winnt-interlockedexchangeadd64
 */
#define safe_int_add(__v, inc_val) InterlockedExchangeAdd((int *) (__v), inc_val)
#define safe_int_sub(__v, dec_val) InterlockedExchangeAdd((int *) (__v), -dec_val)
#ifdef GPAC_64_BITS
#define safe_int64_add(__v, inc_val) InterlockedExchangeAdd64((LONGLONG *) (__v), inc_val)
#define safe_int64_sub(__v, dec_val) InterlockedExchangeAdd64((LONGLONG *) (__v), -dec_val)
#else
#define safe_int64_add(__v, inc_val) InterlockedExchangeAdd64xp((LONGLONG *) (__v), inc_val)
#define safe_int64_sub(__v, dec_val) InterlockedExchangeAdd64xp((LONGLONG *) (__v), -dec_val)
#endif

For the next step, open src\utils\os_net.c and jump to around line 135. Look for the following line:

#endif /*WIN32||_WIN32_WCE*/

After that, we have to add a very slightly modified version of PlibC’s inet_ntop() function for IPv6 support, as this function is also missing on all XP systems. This approach was mentioned by [Yuriy Petrovskiy] on [Stackoverflow]. This is the part licensed under the ISC license. I’m really not sure if that stuff is needed by anyone (or if network support is even needed at all…), but it’s probably more clean that way. Windows XP’s do support IPv6 after all. To avoid a redefinition, I call this new function inet_ntop_xp():

expand/collapse source code

/* Added by M. Lackner to ensure Windows XP & XP x64 compatibility */
/* Adding PlibC's inet_ntop() function in a very slightly modified
   and renamed variant for IPv6 address conversion support on Win XP's,
   which is called by gf_sk_get_remote_address() further below.
   The renamed function is called inet_ntop_xp().
   Original source code taken from here:
   https://sourceforge.net/projects/plibc/ 
 
   This inet_ntop() implementation is (C) 1996-2001 Internet Software
   Consortium and is licensed under the ISC license.
   License text: https://opensource.org/licenses/ISC */
#define NS_INT16SZ   2
#define NS_IN6ADDRSZ  16
 
/*
 * WARNING: Don't even consider trying to compile this on a system where
 * sizeof(int) < 4.  sizeof(int) > 4 is fine; all the world's not a VAX.
 */
 
static const char *inet_ntop4(const unsigned char *src, char *dst, size_t size);
 
#ifdef AF_INET6
static const char *inet_ntop6(const unsigned char *src, char *dst, size_t size);
#endif
 
/* char *
 * isc_net_ntop(af, src, dst, size)
 *  convert a network format address to presentation format.
 * return:
 *  pointer to presentation format address (`dst'), or NULL (see errno).
 * author:
 *  Paul Vixie, 1996.
 */
const char * inet_ntop_xp(int af, const void *src, char *dst, size_t size)
{
  switch (af) {
    case AF_INET:
      return (inet_ntop4(src, dst, size));
#ifdef AF_INET6
    case AF_INET6:
      return (inet_ntop6(src, dst, size));
#endif
    default:
      errno = EAFNOSUPPORT;
      return (NULL);
  }
  /* NOTREACHED */
}
 
/* const char *
 * inet_ntop4(src, dst, size)
 *  format an IPv4 address
 * return:
 *  `dst' (as a const)
 * notes:
 *  (1) uses no statics
 *  (2) takes a unsigned char* not an in_addr as input
 * author:
 *  Paul Vixie, 1996.
 */
static const char * inet_ntop4(const unsigned char *src, char *dst, size_t size)
{
  static const char *fmt = "%u.%u.%u.%u";
  char tmp[sizeof "255.255.255.255"];
  size_t len;
 
  len = snprintf(tmp, sizeof tmp, fmt, src[0], src[1], src[2], src[3]);
  if (len >= size) {
    errno = ENOSPC;
    return (NULL);
  }
  memcpy(dst, tmp, len + 1);
 
  return (dst);
}
 
/* const char *
 * isc_inet_ntop6(src, dst, size)
 *  convert IPv6 binary address into presentation (printable) format
 * author:
 *  Paul Vixie, 1996.
 */
#ifdef AF_INET6
static const char * inet_ntop6(const unsigned char *src, char *dst, size_t size)
{
  /*
   * Note that int32_t and int16_t need only be "at least" large enough
   * to contain a value of the specified size.  On some systems, like
   * Crays, there is no such thing as an integer variable with 16 bits.
   * Keep this in mind if you think this function should have been coded
   * to use pointer overlays.  All the world's not a VAX.
   */
  char tmp[sizeof "ffff:ffff:ffff:ffff:ffff:ffff:255.255.255.255"], *tp;
  struct { int base, len; } best, cur;
  unsigned int words[NS_IN6ADDRSZ / NS_INT16SZ];
  int i, inc;
 
  /*
   * Preprocess:
   *  Copy the input (bytewise) array into a wordwise array.
   *  Find the longest run of 0x00's in src[] for :: shorthanding.
   */
  memset(words, '\0', sizeof words);
  for (i = 0; i < NS_IN6ADDRSZ; i++)
    words[i / 2] |= (src[i] << ((1 - (i % 2)) << 3));
  best.base = -1;
  best.len = 0;
  cur.base = -1;
  cur.len = 0;
  for (i = 0; i < (NS_IN6ADDRSZ / NS_INT16SZ); i++) {
    if (words[i] == 0) {
      if (cur.base == -1)
        cur.base = i, cur.len = 1;
      else
        cur.len++;
    } else {
      if (cur.base != -1) {
      if (best.base == -1 || cur.len > best.len)
        best = cur;
        cur.base = -1;
      }
    }
  }
  if (cur.base != -1) {
    if (best.base == -1 || cur.len > best.len)
      best = cur;
  }
  if (best.base != -1 && best.len < 2)
    best.base = -1;
 
  /*
   * Format the result.
   */
  tp = tmp;
  for (i = 0; i < (NS_IN6ADDRSZ / NS_INT16SZ); i++) {
    /* Are we inside the best run of 0x00's? */
    if (best.base != -1 && i >= best.base && i < (best.base + best.len)) {
      if (i == best.base)
        *tp++ = ':';
      continue;
    }
    /* Are we following an initial run of 0x00s or any real hex? */
    if (i != 0)
      *tp++ = ':';
    /* Is this address an encapsulated IPv4? */
    if (i == 6 && best.base == 0 && (best.len == 6 || (best.len == 5 && words[5] == 0xffff))) {
      if (!inet_ntop4(src + 12, tp, sizeof tmp - (tp - tmp)))
        return (NULL);
      tp += strlen(tp);
      break;
    }
    inc = snprintf(tp, 5, "%x", words[i]);
    tp += inc;
  }
  /* Was it a trailing run of 0x00's? */
  if (best.base != -1 && (best.base + best.len) == (NS_IN6ADDRSZ / NS_INT16SZ))
    *tp++ = ':';
  *tp++ = '\0';
 
  /*
   * Check for overflow, copy, and we're done.
   */
  if ((size_t)(tp - tmp) > size) {
    errno = ENOSPC;
    return (NULL);
  }
  memcpy(dst, tmp, tp - tmp);
  return (dst);
}
#endif /* AF_INET6 */
/* End of inet_ntop() reimplementation */

In the same source file we need to change the call to that function as well. Luckily there is only a single one. Look for the following call somewhere around line 1630:

inet_ntop(AF_INET, addrptr, clienthost, NI_MAXHOST);

Simply change it to this:

/* Modification by M. Lackner */
/* Replace inet_ntop() with XP-compatible and renamed version
   inet_ntop_xp() from PlibC */
//inet_ntop(AF_INET, addrptr, clienthost, NI_MAXHOST);
inet_ntop_xp(AF_INET, addrptr, clienthost, NI_MAXHOST);
/* End of inet_ntop() call replacement */

And that’s it, now thread-safe atomic operations should work (hopefully in an actually safe way given our wrapper function, not exactly sure), and IPv6 support is solved as well.

The MP4Box part of the code can now hopefully be compiled and linked correctly for both 32-bit and 64-bit Windows XP targets. As can be seen in the files section at the top I used Microsoft VisualStudio 2017 with a v141_xp platform toolset for that, but the 2015 version with its v140_xp toolset should work just as well. If you use that, the redistributables linked to above should also work, as they’re typically compatible between the 2015 and 2017 versions.

32 & 64-bit MP4Box from GPAC 2.0.0 running on XP x64

As always with such hacks, you should know that I’m not really a hacker or programmer, so some of this stuff may be stupid or unsafe. If you can spot some obvious issues with the above modification, just let me know in the comments and I’ll try to fix it!

Repairing a DELL 3008WfP; A failure so far

Hardware 9 Responses »

Aug 172022

A while ago I decided to attempt a repair of what was basically the “reference” 30″ 2560×1600 monitor, an LG Flatron W3000H that they built to demonstrate their new, large panel. Back then I got the monitor out of an apartment liquidation and had to replace the inverter board driving the screens’ failed backlight consisting of 9 CCFL lamps. But long before that, I had bought my own 30″ screen which features the same LCD panel, but with other added features, like many more connectors, a card reader with USB hub and a more aggressive anti-glare coating: The DELL UltraSharp 3008WfP. I guess it’s most known for its wide array of supported connectors, which allow the user to hook up anything from a Commodore 64 home computer via composite or S-Video up to a modern graphics card via dual-link DVI-D, HDMI or DisplayPort (as always, Ctrl+click to enlarge images):

There’s a whole lot of connectors

For a while now, the screen had been showing erratic behavior: It’s backlight would sometimes flicker, and it would show a single or two vertical, fuzzy and tansparent yellow lines across the entire panel height at about a 2-3 pixels width for some time before they just disappeared after a few hours of operation. Additionally, it would show a single, vertical, 1 pixel wide opaque purple line about a 150 pixels high on the lower left constantly. My assumptions were that the flickering was caused by the inverter board generating the high-voltage A/C required by the CCFL lamps, and that the panel errors were caused by the LCD driver board. Naturally, it could just be one or more lamps failing and the mainboard having issues, but this seemed like the most likely candidates to me.

So, like for the LG Flatron W3000H, I got myself another LGIT-LM300WQ5 inverter board with DELL part number 6632L-0440A, plus a LM300WQ5-STA1 LCD panel driver or “T-CON” board with DELL part number 6870C-0183D:

Monitor and replacement parts

Now I already knew this device would not be an easy one to disassemble. To reach the insides I basically followed a power board repair article that can be found on [this weblog] and sources a forum thread that is still online on [overclockers.co.uk]. Just like it is said there, the hardest part is to get the front bezel off. And I did it wrong, partially breaking it in the process.

But first comes the stand. You need to remove it to be able to continue. To do so, you’ll require a Torx T15 screw driver. This is the only Torx screws I encountered during disassembly. Carefully put the monitor face down, then remove the screws and rise the lower part at an angle. It’ll easily slide out.

DELL 3008WfP stand mount using Torx T15 screws

Stand mount on the rear

Now, for the hard part: The front bezel is actually a 2-part assembly with the metallic bezel and a black, plastic insert that the metallic part clips into. What I did at first was to try and get just the metallic front part off, and that was a bad idea. If you do that, you will permanently break the plastic clips holding the metallic bezel and its insert together, which means you’ll need to glue it back on afterwards. You’ll notice when little black plastic pieces start falling out:

Parts broken off the DELL 3008WfP front bezel assembly

This is bad! When you see those, stop whatever you’re doing, because you’re doing it wrong, just like me!

On the following photo you can see how the 2-part front bezel assembly should not come apart:

DELL UltraSharp 3008WfP front bezel, now partially broken

While it ain’t easy, you should try to make sure to keep the front part and the insert together when removing them from the base frame! There are little notches in the insert you can get into with a small flat-head screwdriver to use leverage to pry it off. You may need to insert multiple of them at once to get it off on all sides without having it snap back into place by itself. The process is much easier once you’ve opened it at least once, but there will be a lot of resistance the first time around, and it will make you curse and flip tables over a lot. When the insert snaps out of the metal frame it’ll come with clear, audible clicks. This may cost you the better part of an hour to get done and needs a surprising amount of force.

Important: Do not just tear it away too far from the screen! The cable connecting the front bezel buttons allows only about a centimeter of travel, so be especially careful around the spot where those buttons are! You really don’t want to tear that cable off its soldering joints!

Also, it’s best done with the monitor lying on its back, if you’re alone. If you have two people you could have somebody hold it in place in an upright position while you pry off the bezel. To ensure the bezel stays detached for what comes next, use some old manuals or something similar as spacers. Make sure they’re relatively thick at a 100+ pages. You’ll see why soon:

Monitor with front bezel detached

With the bezel in place just like this (remember the cable connecting the front buttons!), flip the monitor upright, and then put it face-down again. Make sure the manuals stay in place to protect the panel from the plastic insert’s clamps! You don’t want to scratch the panel! If you do, you may be able to polish those scratches out with a soft piece of cloth later, but believe me, you don’t wanna have to do that. It’ll take hours to get done, I’ve already been down that road. The manuals I used to do this are all reasonably thick, and that’s good, because they need to withstand a heavy weight pressing down on them from above after flipping the monitor over. They are what makes sure that the front bezel’s plastic clamps don’t accidentally push into the otherwise almost unprotected panel:

DELL 3008WfP flipped over with the front bezel remaining detached

Ah right, the rear plastic cover is already off here, I forgot to photo-document that part; It can be a bit hard to remove the first time as well. Make sure you don’t put too much strain on the card reader, and maybe use a flat-head screwdriver or some plastic tool to put some leverage on the plastic frame as it gets caught on the display connectors. It takes a bit of force, but should come off after 5-10 minutes on your first try.

First things first: Just tear all the metallic tape off. It serves zero purpose anyway. Okay, maybe not “zero” in case of the card reader… so let’s say “near-zero”. Realistically, there’s no need to keep or replace it though.

Before continuing further, make some post-its or something, so you can document where each screw came from. There a multiple types here, so documenting the location a screw or a set of screws came from can help with making reassembly quicker and easier.

About components: The left shroud that goes almost from top to bottom covers the CCFL inverter board, so the part that drives the backlight. The small one on the right holds the USB 2.0 card reader, which internally consists of the larger card reader board PTB-1767 and the smaller USB 2.0 connector board PTB-1872. The small one on top covers the LCD driver or “T-CON” board. The main shroud covering a large part around the center of the screen holds the power board and the mainboard with all the display connectors.

For a first step, I attempted to remove the inverter board, because it looks the easiest. And again, I managed to mess up.

DELL 3008WfP inverter board shroud… uhm. Yeah. It obviously shouldn’t look like this!

One of the two screws holding the shroud in place just refused to come out, so I messed up it’s head. Nothing left to do but to just bend it off to the side. Bad. Anyway, for the inverter, you need to remove three cables on the side towards the center: The power cable in the middle, a small monopolar cable on the right and another small ribbon cable on the top left, which connects the inverter with the LCD driver. The right one can be a bit tricky, so you might want to (carefully!) use pliers here.

Next come the power cables on the other side of the board, connecting it to the nine CCFL tubes. Those should come off real easy:

CCFL inverter board removed

The inverter board itself is held into place by six small screws. After having removed those, you can just lift it off and replace it with a new one. If you want to remove more than just that part, re-connect neither the LCD driver connection cable, nor the power cable yet!

Next, remove the single screw fixing the card reader in place. The reader then slides out towards the bottom side of the screen. Also, remove the two screws holding the top LCD driver shroud in place, and tear it off. I say “tear”, because there are some weak glue pads involved here as well. This shouldn’t take too much force:

DELL 3008WfP reassembly omitting the seconary LM300WQ5-STA1 LCD driver board shroud

Card reader detached and LCD driver board top shroud removed

You should also disconnect the ribbon cable connecting the front bezel buttons from the mainboard now. It runs right over the card reader shroud and is also affixed to it. You should also pull that cable off the spot where it’s glued to the main shroud for safety! The glue is weak, so doing this won’t damage it. You can just put it back into place later, and it’ll still stick. You can leave it affixed to the card reader though, that part won’t hurt:

Card reader and front bezel button connector cable

Remove all the screws holding the main shroud in place now. There are six larger ones near the top with two on each side of the the LCD driver board assembly and one on each side on the bottom of the screen. Also, there are two smaller ones close to the LCD driver board as well. Lift it off on the top side of the screen, but be careful! Not more than a few centimeters!

Main shroud lift off a bit (here the top LCD driver board shroud is still installed – doesn’t matter)

When looking at it from this angle, the whole main shroud slides a bit to the left after you lift it off a bit. After having done that, it’ll be loose, but don’t lift it off entirely just yet! First the data and power cables between the mainboard and the LCD driver board need to be detached from the latter:

: Power cable

: iTMDS connector

The power cable on the left can just be carefully pulled out, but do not do that for the iTMDS data cable: It’s got some tiny metal locks on each side. Push both of them inwards and carefully pull it out at the same time. If you cannot manage that, do it on one side first, just pull it out for a bit at an angle, then repeat the process on the other side.

In my own case, there was a big (and bad!) surprise waiting for me here. I did not notice immediately, so I’ll talk about it towards the bottom.

Anyway, carefully turn the main shroud over vertically, so top to bottom. You’ll get to see the mainboard on the left and the smaller power board – which is also prone to failure by the way – on the right. Both run seriously hot when the screen’s powered on:

DELL 3008WfP main shroud flipped over and LCD driver board shroud almost coming off

You can now access the four screws holding the final LCD driver board shroud in place. Because for some reason DELL thought it was smart to put a shroud over a shroud over a board… However, as you can see on the picture above, that alone is not enough to remove it. You also need to unscrew and remove the top metal rail first. Take a look at the two black screws, one all the way to the left and one on the right: Those two need to go, so you can remove the rail and with it the shroud covering the LCD driver board. When done, it looks like this:

DELL 3008WfP LCD driver board (LM300WQ5-STA1) uncovered

LCD driver board finally uncovered

The driver or “T-CON” board connects to the panel with two thin ribbon cables. Do not just pull them out forcefully though! While you can do that (I sure did, heh…) it’s not how it’s supposed to be done. To avoid unnecessary damage, you should lift the small, black plastic clamps on the connectors, which are what fixes the cables into place and ensures firm contact between the traces and the connector pins:

LCD driver board with open panel connector clamp

Once the clamps are open, the cables can be pulled out and back in with almost zero force. When reconnecting the replacement board, just slide the cables back in under the small plastic noses on each side of the connector and close the clamps again:

LCD driver board with cable inserted and clamp closed

Ah right, before I forget: There is also a small ribbon cable going from this board to the inverter. You can just pull that one out very easily. Remove the board, and replace it with a new one, then reconnect all cables.

With the cables firmly back in place, the T-CON board and the panel (as well as the inverter if installed at this point) are reconnected:

LM300WQ5-STA1 LCD driver board hooked up

Driver board hooked up to the panel

From here on out, essentially just put everything back in reverse order and make sure not to forget (or damage!) any of the cables and connectors.

Personally, I would omit the outer LCD driver board shroud. It really serves zero purpose other than further impacting what little air convection happens in there. Leaving it off means there will be one left-over screw as well.

Alright, now I did aaaall of that, and what did I end up with?

A screen that was even more broken than before.

After powering it back up, it went into factory panel diagnostics mode, cycling between solid colors: Black, white, red, green, blue. Apparently, this mode is supposed to help with dead & hot pixel detection. And the panel errors from before were visible as well. As I couldn’t manage to disable the diagnostics mode with the undocumented key combinations that are supposed to deactivate it, I assumed something must’ve gone wrong with the data connection between the mainboard and the LCD driver board. Or maybe between the latter and the panel itself? The only thing that seemed to be working perfectly was the backlight. Too early to judge it yet though, as the flickering issue sometimes showed up only briefly and only after hours of operation.

Alright, so I had to do a check on all cables, requiring me to tear it apart again! And what fun that is in case of this device!

Suffice to say, all cables were firmly in place. But I noticed something unbelievable for the dual-link iTMDS cable connecting mainboard and the LCD driver. That cable is even documented by the chip maker Silicon Image, which built the two single-link SiI7172 transmitters and the dual-link receiver chip, which together make up the Silicon Image VastLane SiI7189CMHU iTMDS chipset. Let’s take a look at its block diagram:

Silicon Image 2 × SiI7172 + SiI7181 block diagram

Silicon Image VastLane SiI7189CMHU block diagram

In case of this DELL monitor, the top part consisting of the video processor and the SiI7172 chips are on the mainboard. From there, a “Dual Link iTMDS Over Flat Cable” runs to the SiI7181 on the T-CON board, which then drives the panel controller. In this case, the panel controller is a LG Philips LCD TL23210D chip.

Let’s take a closer look at that dual-link iTMDS cable on the mainboard side:

: Dual-link iTMDS cable, left SiI7172 transmitter side

: Dual-link iTMDS cable, right SiI7172 transmitter side

Holy hell… What’s happening here? Probably as a result of both age and excessive heat exposure over the course of tens of thousands of hours of operation, the plastic insulation had become hard and brittle. It’s enough to just run your finger over the cable, and the insulation just falls off like dry old paint! See all the colorful little crumbs on the metal shroud? Yes, that’s all cable insulation material! Let’s take a closer look so you can see the problem more clearly:

Dual-link iTMDS cable insulation coming off, likely due to heat and aging

Cable insulation coming off like crazy

Naturally, this means there’s a lot of short-circuiting now. And that explains why there is no longer any working data connection between the mainboard and the LCD driver. This might have even blown the LCD driver board’s fuse “F2”, so I’ll have to check whether it’s still conductive.

The disintegrating dual-link iTMDS cable supposed to connect two SiI7172 iTMDS transmitters on the mainboard with a single SiI7181 receiver on the LCD driver board

The disintegrating dual-link iTMDS cable – I couldn’t find any LG or DELL part numbers for it so far

For now, I call this a complete failure.

Next steps: Try to find a replacement dual-link iTMDS cable (that does not come with a full, expensive mainboard attached to it) or re-insulate the entire cable by myself. Manually.

This post will be edited in the future once I’ve found a solution to this mess and once I’ll have checked that fuse.

Seriously, this monitor is a b**ch to repair, and I didn’t even have to touch the power board yet…

Running a CalDAV/CardDAV server on Windows 2000 for multi-user calendars, tasks and contacts synchronization across devices

Server, Windows No Responses »

Jul 152022

0. Introduction

Recently, I found it more and more troublesome that I couldn’t synchronize my personal calendar(s) across all my devices. Originally, I’ve been running a SyncML server for that purpose, as it was conveniently integrated with my mail server suite. Many years ago I used a Nokia E72 “smart” phone *cough* to synchronize against that server. There is even an app for it on Android called [Synthesis], which I am currectly using on modern Android. However, I could not find any software nor software plugins for SyncML for any one of my Desktop operating systems (WinXP x64, RedHat Enterprise Linux 8 & FreeBSD 12/13). Seems SyncML never really made it onto the desktop. And where it did – like in the form of the Funambol plugin for Mozilla Thunderbird – it’s been long abandoned and no longer works with modern software. Heck, it doesn’t even work with TB 52.9 on XP / XP x64 anymore, it’s too old for even that!

At first I tried to bring SyncML and modern CalDAV/CardDAV solutions together by running my own bridge server using [SyncEvolution] on Linux. In essence, I would run my own DAV server and bridge it to my SyncML server in the background. All of it over HTTPS. DAV as a protocol, originally called WebDAV (Web Distributed Authoring & Versioning) is a series of extensions to HTTP/HTTPS that can be used for sharing and collaboratively working on certain data sets. Like version control for software development (cvs+https://, svn+https:// etc., as opposed to doing it over e.g. svn+ssh://). CalDAV and CardDAV are just altered versions used for calendar, tasks and contact information data.

Unfortunately, that bridge thing did not work. While SyncEvolution can do it the other way around, running as a SyncML server and linking it to a CalDAV/CardDAV backend, it can not run e.g. a CalDAV server and bridge it to a backend SyncML one, like it is in my case. Actually, the original developer told me it could, but I just didn’t find any way to do so, even with the latest version.

So, I gave up on the whole idea, when user M477 on the XIN IRC chat suggested just running a DAV server on my stoneage Windows 2000 Server machine directly. I had thought that to be an exercise in futility, but he proved me wrong! This can indeed be done, and when combining the Radicale CalDAV/CardDAV server with modern HTTPS using my own OpenSSL + stunnel backports it can satisfy the security requirements of modern client software as well!

1. The server software

As for software, some pretty old versions are required, at least partially. I’m using Python 2.7.10, the Radicale 1.1.7 DAV server and as mentioned, my own backport of relatively modern OpenSSL 1.1 and stunnel for bridging HTTPS to a localhost HTTP socket (unpack the files below with [7-Zip]):

[Python 2.7.10]
[Radicale 1.1.7]
[stunnel] (XIN build of v5.45b6 for Windows 2000 with modern OpenSSL 1.1.1)
[Microsoft instsrv.exe & srvany] (Optional: For system service creation)
[htpasswd.exe] (Optional: For using htpasswd file authentication in Radicale)

1a. Python

Setting up Python is straight-forward, so I won’t discuss this here. Just run the installer and optimally install it into a path not containing any whitespaces.

1b. Radicale

1b1. Installation and basic configuration

Unpack it and put it wherever you wish, e.g. X:\servers\radicale\. It’s all written in interpreted Python language, so to execute it interactively for tests, you’d do something akin to this:

CD /D X:\servers\radicale\
"C:\Program Files\Python27\python.exe" .\radicale.py

I do suggest creating a restricted user to run it as though, just to make sure no part of the software unnecessarily runs with higher privileges. If you wish to create such a user including its user profile folder without having to log in with it interactively, you can run the following as an administrative user on a cmd terminal after regular user creation via Administrative Tools in the system control panel:

runas.exe /profile /user:<username> cmd.exe

This’ll create the profile and give you a terminal where you can work as the target user to set things up. If you’re authenticating against a domain controller’s ActiveDirectory, do it like this:

runas.exe /profile /user:<domainname>\<username> cmd.exe

Note there is a file called config coming with Radicale. This will be expected at the UNIX-style path ~/.config/radicale/config. Python translates the ~ home directory shorthand to %USERPROFILE% on Windows transparently, so pre-create the folders there and copy the config file to that location for the user which will be expected to run the software: %USERPROFILE%\.config\radicale\config. Slash-to-backslash translation will also be done by Python internally, so no need to change the conventions in config.

Edit the config file with your favorite text editor, and take a look at the options hosts, daemon, ssl & realm in the [server] block for now. I’d suggest the following settings:

[server]
hosts = 127.0.0.1:5232
daemon = False
ssl = False
realm = CalDAV/CardDAV

Since we’re not going to rely on the old SSL implementation of Python, the server should listen only on localhost with no encrpytion instead of on the actual, public network. Also, it cannot run as a “daemon”, which is a type of background service on UNIX and Linux, specifically. As for the realm option, you can just enter any arbitrary string. Its value will be presented to you likely as a dialog window title when the server asks your calendar/tasks/contacts client for login information.

1b2. Authentication

Radicale v1 supports multiple authentication backends by default. Newer versions 2 and 3 need additional plugins to gain the same flexibility, but with this version, a lot of them are bundled, like LDAP, IMAP and the file-based htpasswd. Look for the authentication block [auth].

htpasswd-style authentication is probably the easiest. This means creating a users & passwords file using the command line program htpasswd.exe, while picking a compatible hash function that works with Python 2.7.10 (e.g. SHA1). For help on usage, just run htpasswd.exe --help on a cmd terminal with htpasswd.exe on your search path. Creating a new user+password file with SHA1 password hashes is as easy as this:

htpasswd.exe -c -s <passwdfile> <username>

In config, you’d then specify it like this:

[auth]
type = htpasswd
htpasswd_filename = ~/.config/radicale/htpasswd

In my own case, I am running an eMail server with IMAPv4 on the same host, so I chose the IMAP authentication backend instead. As it’s running locally, no SSL is required when authenticating against it:

[auth]
type = IMAP
imap_hostname = localhost
imap_port = 143
imap_ssl = False

1b3. Rights management

Next, look at the [rights] block. For simple setups, where each user only needs to access their own calendar and contacts, I’d suggest setting the rights management type as such: type = owner_only. If you require more complex setups with certain users sharing specific calendars & contacts, I suggest reading up on the from_file value and its implementation. It’s not exactly trivial, but here’s the [documentation]!

1b4. Data storage

The most well-supported way of storing the data on the server side is to do so as such text files. Continue to the [storage] block in config. Let me suggest the following directives:

[storage]
type = filesystem
filesystem_folder = ~/.config/radicale/db

Note that you need to create the folder %USERPROFILE%\.config\radicale\db\ in advance! Also make sure the user who is supposed to be running Radicale has write access to it.

1c. stunnel

Full stunnel documentation with examples towards the end can be found [here]. Documentation on my own Windows 2000 backport can be found [here]. For now, let’s just assume you won’t be changing Radicale’s TCP socket, so it’ll run on localhost:5232. Then, a proper service definition in config\stunnel.conf would be like the following, assuming your public server IP address were 233.204.248.135:

[cdav]
accept  = 233.204.248.135:5232
connect = 5232
cert    = Your-SSL-certificate.pem
key     = Your-SSL-certificate-private-key.pem

Naturally, you’d need to create SSL certificates first. For this I suggest [Let’s Encrypt] if you don’t have any server-side SSL implemented yet. Their certificates are trusted by all modern software vendors, so it’s a good pick when working with modern client software.

1d. instsrv & srvany

Those two are for installing and running interactive programs such as Python as background system services. This is optional, but I’d recommend doing so. Service installation works as follows on a cmd terminal, assuming instsrv.exe is on your search path and srvany.exe is in %WINDIR%\system32\:

instsrv.exe "<service name>" %WINDIR%\system32\srvany.exe

So, e.g.:

instsrv.exe "Radicale CalDAV and CardDAV server" %WINDIR%\system32\srvany.exe

Required system service properties are then edited in the system registry using regedit.exe. Please be careful with that! Look for a registry key named after the service name you just picked, e.g.:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Radicale CalDAV and CardDAV server\

There need to be three sub-keys: Enum, Parameters and Security. If Parameters doesn’t exist yet, create it, then enter it. Now, here you’ll create three string values called AppDirectory, Application and AppParameters. The first is the current working directory of the server. You’ll enter the path to Radicale as its value. The second is the full path to the Program being called. Enter the full path to python.exe here. And the last is a list of parameters to be passed to the invoked program. In our case this is the full path to the Radicale main program radicale.py. So, for example:

Property	Value
`AppDirectory`	`X:\servers\radicale`
`Application`	`C:\Programs\Python27\python.exe`
`AppParameters`	`X:\servers\radicale\radicale.py`

Also, don’t forget to go to Win+r, run services.msc and reconfigure that service by setting the user to run the service as, if you’re using a dedicated account for it!

Before starting the service, you should run it interactively on a cmd terminal (e.g. the one you launched in 1b1. with runas.exe). To do that, just replicate the Registry settings on a terminal:

CD /D "X:\servers\radicale\"
"C:\Programs\Python27\python.exe" "X:\servers\radicale\radicale.py"

Make sure it works and you can reach it with a browser. Following the examples above, you’d just need to connect to https://233.204.248.135:5232, and you should get a login prompt. Test authentication, and if it breaks, take a look at the output on the terminal to debug it! The result should be either a blank, white page or alternatively a “Radicale works” message. Once you’re certain it works so far, terminate the program and start the configured system service in its stead, then test again.

2. The client software

In my case, I decided to use three clients: First, [Thunderbird] (both old and new) on Microsoft Windows, Linux and FreeBSD. Second, [KOrganizer] on Linux. And finally, [DAVx⁵] on Google Android.

When configuring a client for a server-side calendar, the full URL would be in the format https://<server socket>/<user name>/<collection>. A “collection” is essentially a contacts list or a calendar+tasks list. You can pick its name freely when initially creating the collection. E.g. you can just call it “calendar” as well. The user name should match your login credentials.

For example: https://233.204.248.135:5232/johndoe/calendar. Let’s say you own mydomain.org, which points to 233.204.248.135, then of course the following would be much prettier, and won’t trigger any SSL warnings or errors, as long as your SSL certificate’s common name is mydomain.org:

https://mydomain.org:5232/johndoe/calendar

In Thunderbird, just switch to the calendars tab and press the “+” symbol on the left pane to add a new one. Select “On the network”, then enter your user name into the “User name” as well as the URL into the “Location” fields. You may also check “Offline Support” to make sure you can see the calendar and get notifications even while your Radicale server is down. Continue and you’ll be promped for a password. Authenticate, then scan for calendars, select the server’s offer and you’re done!

Once the calendar has been created on the server side, you can also import any calendar backups you may have (e.g. from Google Calendar or other software) onto the server. Thunderbird supports the very common iCalendar format (.ics) for this. Wait for the entries to be uploaded, and you should be set!

Here are some example screenshots of two versions of Thunderbird as well as KOrganizer working with a Radicale Server running on ancient Windows 2000, connected via TLSv1.2. Note that there is an additional, local-only calendar added to the mix on Linux (click to enlarge pictures):

: Thunderbird 91.7 on Linux (Color import didn’t work?)

: Thunderbird 52.9 on XP x64

: KOrganizer 5.18 on Linux

As for Android, I decided to use the DAVx⁵ app, as mentioned. Setup is pretty similar to the desktop programs, here are some sample screenshots:

: DAVx⁵ splash screen

: DAVx⁵ CalDAV profile

: Android 8 calendar

Note that Apple iOS supports this natively! CardDAV for contact synchronization should be found under Settings / Contacts / Accounts / Add Account / Other, “Add CardDAV Account” and CalDAV for calendar synchronization under Settings / Calendar / Accounts / Add Account, “Other”, “Add a calendar account”, “Add CalDAV Account”. I have not tried this though, as I do not have an iPhone or iPad.

So while I can’t really use good old SyncML in a cross-platform context, this still enables me to share a centralized set of self-hosted contacts, tasks and calendars for me or even for multiple users on an ancient server. While security may be quite debatable in this case, at least the cryptographic part is by no means ancient for both login and data transfer.

While the SyncML server will remain online, XIN eMail server users may now additionally request CalDAV/CardDAV access for synchronizing their calendars, tasks and contacts to an ancient museum server, without having to rely on “the cloud” to hold that data.

A Windows XP x64 backport of S.T.A.L.K.E.R. Anomaly 1.5.1

Gaming, Programming, Windows 5 Responses »

Jun 152022

0. Introduction

A short while ago, user Ergo asked about a possible backport for S.T.A.L.K.E.R. Anomaly to make it run on Windows XP in a discussion about a similar Ion Fury backport, [see here]. Having looked through the code and the MSVC build system of the latest version 1.5.1 of that game made me think I couldn’t probably fix all this… but i still gave it a shot. The required changes were quite extensive actually, and mostly revolved around ripping out the game’s [Discord] integration. That being the major part, some DXGI, Direct3D 10.x & 11.x error reporting code was also failing on top of some other, rather minor things related to the project configuration for the compiler and linker.

Way I understand it, the game’s Discord integration is mainly for reporting your player status (faction, level, current task, etc.) on the game’s Discord channel, so let’s just call this feature “non-essential”.

In the end it took me multiple days and many failures to get there, but against all my expectations I actually managed to compile and link the code to a Windows XP x64 platform target. That – naturally – was only the first step. Code that compiles and links is one thing, code that runs and works is an entirely different story however.

Before I start with showing you my modification documentation, let’s look at the results first though:

1. The results

There are some components that I couldn’t compile from source code in one go, as only libraries and headers are coming with the [S.T.A.L.K.E.R. Anomaly sources] for those. The Intel oneAPI (formerly “Thread Building Blocks”) library was one such thing, as were some unicode libraries, but also the launcher itself, which seems to be its own little software project carrying its own version number. The Intel TBB library and the launcher don’t appear to have any Windows Vista/7/8/10 dependencies though, they’ll work without any mod whatsoever:

S.T.A.L.K.E.R. Anomaly launcher on XP x64

Anybody can get this far though (click to enlarge)

But this is where things usually break apart with S.T.A.L.K.E.R. Anomaly 1.5.1 on XP x64. No version will actually launch (e.g. bin\AnomalyDX9.exe, bin\AnomalyDX8.exe, etc.). Actually attempting to do so will just produce a crash in the launcher. Trying to start e.g. bin\AnomalyDX9.exe manually gives us a first hint as to what’s wrong:

The most common error when running new programs on Windows XP

The most common error when running newer programs on Windows XP

Out of curiosity, I inspected the PE32+ / PE64 optional header of the program with [CFF Explorer] to check out the values of MajorSubsystemVersion and MinorSubsystemVersion as well. They were set to 0006 and 0000, so NT 6.0, Windows Vista. As a quick test I set them to 0005 and 0002, reflecting NT 5.2, so Windows XP x64 or Windows Server 2003. But of course, just patching the program’s header wasn’t enough, there are a lot more dependencies on modern API functionality to be found here, many more than any binary fix known to me can deal with. I’ll spare you the full list:

A call to SetThreadErrorMode(), which does not exist on any XP

A call to the Kernel32 API function SetThreadErrorMode(), which [requires] NT 6.1 / Windows 7 as a minimum

So I had to try and actually backport the source code. A simple recompile/relink with just a few changes in the build configuration is not enough in case of S.T.A.L.K.E.R. Anomaly 1.5.1. My backport only covers the DirectX 9 version of the game, so you’ll have to pick that one in the launcher, then configure the rest of the settings and hit that “Play S.T.A.L.K.E.R. Anomaly” button. While inside of my test VM, this is the last thing I saw before a crash due to the lack of any hardware Direct3D renderers:

The splash screen – a first sign that at least something works

The stock version crashes before even showing a splash screen, so I was looking forward to a first real test on my physical XP x64 machine, which happened the evening before yesterday. And the result was a big surprise for me as I really didn’t expect even limited success given my code mutilations:

S.T.A.L.K.E.R. Anomaly 1.5.1 on XP x64: Main menu

S.T.A.L.K.E.R. Anomaly 1.5.1 running on XP x64: Main menu (click to enlarge)

The main menu is already based on modified code, so this was pretty nice already! On top of that, the OpenAL sound subsystem and OGG media decoder were obviously also working as I could hear some background music at this point. Now, I still fully expected it to crash and burn when launching an actual game, but nope:

S.T.A.L.K.E.R. Anomaly 1.5.1 on XP x64: Ingame (click to enlarge)

This is at a low resolution of 1024×768 with 8xMSAA and 8xSSTAA enabled through the nVidia driver, running on a GeForce GTX Titan Black (think: GeForce GTX 780 Ti). Performance was okay even if not stellar. I guess much higher resolutions like 1920×1200, 2560×1600 or even my current 3840×1600 will likely require some reduction of subsample counts, especially for the transparency supersampling. I have not played around with that yet.

2. Patch download

And here’s the patch to make the game run on Windows XP Professional x64 Edition:

[S.T.A.L.K.E.R. Anomaly 1.5.1 XP x64 patch][modified source code][license]

First, unpack [S.T.A.L.K.E.R. Anomaly 1.5.1] using [7-Zip], then do the same with the patch. You’ll see that the patch files are an exact subset of the game. Just copy the AnomalyLauncher.exe file and bin\ subdirectory including all its files from the patch over the stock version of the game. The launcher and .dll files are probably not required, but I still decided to include them.

As described above, start the launcher and make sure to pick the DirectX 9 version of the game, then configure the rest and launch it!

Note: A version optimized for the use of the AVX instruction set extension is not included, as AVX cannot work on any Windows XP system anyway. The kernel thread scheduler lacks the necessary XSAVE and XRSTOR instruction support to make [context switching work for AVX code], so no AVX support on XP or XP x64.

Now if all you want to do is play the game on XP x64, this is all you’ll need. No guarantees for stability or anything though. If you want to know how the backport was made, read on!

3. Build system modifications

Note: The modified source code is meant to be compiled with MS Visual Studio 2017 with the Visual Studio 2015 platform toolset and Windows XP support for it installed to provide the correct compiler and platform SDK. The below steps should hopefully show you how to reach the XP-compatible, modified state starting from the [original version 1.5.1 source code].

But before looking at any source code modifications, let’s look at the MSVC++ build system first!

Having opened the solution src\engine.sln in MS VisualStudio 2017, we’ll need to handle the solution & projects import first. This is easy: Leave the “Windows SDK Version” field alone. The “Platform toolset” field will suggest an upgrade to v141. Click on the dropdown and select “No Upgrade”, then click “OK”:

Solution import into VisualStudio 2017

These steps should not be necessary using Visual Studio 2015 directly, but this is what I have around here, so yeah.

Once all projects have been loaded, click on “Solution ‘engine’ (29 projects)” on top, and in the engine Solution Properties on the bottom right, switch from DX10|x64 to DX9|x64. Be sure not to pick DX9-AVX|64, as the resulting code would terminate due to causing an illegal instruction exception on XP x64:

Changing the solution configuration to DX9|x64

Alright, now we need to change the toolset for all eligible projects from v140 to v140_xp. This is only possible if you have Windows XP support for Microsoft Visual Studio 2015 installed, so if there is no v140_xp to select for you, that would be the reason. You can install it using the original VS2015 installer. Mark all projects other than “LuaJIT-2”, which is not MSVC project file based, but Makefile-based:

S.T.A.L.K.E.R. Anomaly project list

Right click them, and select “Properties”. Now a dialog with the General project configuration options should show up. Change the “Platform toolset” from “Visual Studio 2015 (v140)” to “Visual Studio 2015 – Windows XP (v140_xp)”:

Retargeting Anomaly projects from v140 to v140_xp

Click “Apply” and “OK”. The project list will now reflect the change:

Projects retargeted

Having changed the toolset will likely mess up detection of the header Windows.h, as parts of the SDK cannot be found, so we now need to reconfigure some header include and library linking options. Right-click the project “3rd Party\lua_extensions” and select “Properties”. There’s no screenshot for this, but switch to “VC++ Directories” in the tree on the left and for the two drop-down fields “Include Directories” and “Library Directories”, choose “<Inherit from parent or project defaults>”. Click “Apply” and “OK”.

Another issue is that having changed the platform toolset to v140_xp will likely have added a “Minimum Required Version” field to projects, set to 5.02 for Windows NT 5.2 (XP x64, Server 2003). When present, this field needs to be passed to the linker together with a “SubSystem” specification, or the linker will die. We may now be missing that for the main xrEngine project, though this doesn’t happen always. I’m not sure what exactly this depends on.

Right-click the “xrEngine” project and select “Properties” again. Navigate to Linker \ System. Click the dropdown field next to “SubSystem”, and select “Windows (/SUBSYSTEM:WINDOWS)”. The “Minimum Required Version” field should contain 5.02. If not, enter it manually! It should look like this:

xrEngine project linker properties

Now, time for Discord to be murderized! First, we’ll violently break the Interface of xrEngine to the Discord library. Expand the “xrEngine\Interfaces” part in solution explorer to the right, mark “Discord” and delete it:

Remove xrEngine’s Discord interface

Now, on top of that, fire up a file browser like Windows Explorer and completely delete the subdirectory src\3rd party\discord. This completes all steps for build system reconfiguration. Time to mess up some source code!

4. Source code modifications

If only changing the build system would’ve been enough, but no! We need to subject the source code to quite a bit of abuse as well! Practically, pretty much all we’re gonna do here is remove stuff.

4a. DXGI, Direct3D 10.x & Direct3D 11.x error reporting code

Before looking at the larger scale modifications needed for complete removal of the Discord integration, let’s show you one other part, which is about error reporting related to the DirectX Graphics Infrastructure (DXGI), Direct3D 10.x and Direct3D 11.x. That stuff wouldn’t compile when targeting XP and Direct3D 9.0 using a v140_xp platform toolset.

The following code blocks need to be removed or commented out. In the Windows XP case, they wouldn’t serve any purpose anyway. The paths shown in the code block headers are not actual file system paths, but reflect the structure as shown in Microsoft VisualStudio’s Solution Explorer. The line numbers shown here are not always exact due to my edits, so it may happen that code is found on ±10 or so lines, but it should be close enough. Click to expand/collapse individual blocks:

src\3rd party\DXERR\Source Files\dxerr.cpp

```
include <d3d10_1.h>
```
```
include <d3d11_1.h>
```

src\3rd party\DXERR\Source Files\dxerr.cpp

// -------------------------------------------------------------

```
// dxgi.h error codes
```

// -------------------------------------------------------------

```
        CHK_ERRA(DXGI_STATUS_OCCLUDED)
```
```
        CHK_ERRA(DXGI_STATUS_CLIPPED)
```

        CHK_ERRA(DXGI_STATUS_NO_REDIRECTION)

        CHK_ERRA(DXGI_STATUS_NO_DESKTOP_ACCESS)

        CHK_ERRA(DXGI_STATUS_GRAPHICS_VIDPN_SOURCE_IN_USE)

        CHK_ERRA(DXGI_STATUS_MODE_CHANGED)

        CHK_ERRA(DXGI_STATUS_MODE_CHANGE_IN_PROGRESS)

        CHK_ERRA(DXGI_ERROR_INVALID_CALL)

```
        CHK_ERRA(DXGI_ERROR_NOT_FOUND)
```
```
        CHK_ERRA(DXGI_ERROR_MORE_DATA)
```

        CHK_ERRA(DXGI_ERROR_UNSUPPORTED)

        CHK_ERRA(DXGI_ERROR_DEVICE_REMOVED)

        CHK_ERRA(DXGI_ERROR_DEVICE_HUNG)

        CHK_ERRA(DXGI_ERROR_DEVICE_RESET)

        CHK_ERRA(DXGI_ERROR_WAS_STILL_DRAWING)

        CHK_ERRA(DXGI_ERROR_FRAME_STATISTICS_DISJOINT)

        CHK_ERRA(DXGI_ERROR_GRAPHICS_VIDPN_SOURCE_IN_USE)

        CHK_ERRA(DXGI_ERROR_DRIVER_INTERNAL_ERROR)

        CHK_ERRA(DXGI_ERROR_NONEXCLUSIVE)

        CHK_ERRA(DXGI_ERROR_NOT_CURRENTLY_AVAILABLE)

        CHK_ERRA(DXGI_ERROR_REMOTE_CLIENT_DISCONNECTED)

        CHK_ERRA(DXGI_ERROR_REMOTE_OUTOFMEMORY)

src\3rd party\DXERR\Source Files\dxerr.cpp

// -------------------------------------------------------------

```
// d3d11.h error codes
```

// -------------------------------------------------------------

        CHK_ERRA(D3D11_ERROR_TOO_MANY_UNIQUE_STATE_OBJECTS)

        CHK_ERRA(D3D11_ERROR_FILE_NOT_FOUND)

        CHK_ERRA(D3D11_ERROR_TOO_MANY_UNIQUE_VIEW_OBJECTS)

        CHK_ERRA(D3D11_ERROR_DEFERRED_CONTEXT_MAP_WITHOUT_INITIAL_DISCARD)

src\3rd party\DXERR\Source Files\dxerr.cpp

// -------------------------------------------------------------

```
// d3d10.h error codes
```

// -------------------------------------------------------------

        CHK_ERR(D3D10_ERROR_TOO_MANY_UNIQUE_STATE_OBJECTS, "There are too many unique state objects.")

        CHK_ERR(D3D10_ERROR_FILE_NOT_FOUND, "File not found")

```
 
```

// -------------------------------------------------------------

```
// dxgi.h error codes
```

// -------------------------------------------------------------

        CHK_ERR(DXGI_STATUS_OCCLUDED, "The target window or output has been occluded. The application should suspend rendering operations if possible.")

        CHK_ERR(DXGI_STATUS_CLIPPED, "Target window is clipped.")

        CHK_ERR(DXGI_STATUS_NO_REDIRECTION, "")

        CHK_ERR(DXGI_STATUS_NO_DESKTOP_ACCESS, "No access to desktop.")

        CHK_ERR(DXGI_STATUS_GRAPHICS_VIDPN_SOURCE_IN_USE, "")

        CHK_ERR(DXGI_STATUS_MODE_CHANGED, "Display mode has changed")

        CHK_ERR(DXGI_STATUS_MODE_CHANGE_IN_PROGRESS, "Display mode is changing")

        CHK_ERR(DXGI_ERROR_INVALID_CALL, "The application has made an erroneous API call that it had enough information to avoid. This error is intended to denote that the application should be altered to avoid the error. Use of the debug version of the DXGI.DLL will provide run-time debug output with further information.")

        CHK_ERR(DXGI_ERROR_NOT_FOUND, "The item requested was not found. For GetPrivateData calls, this means that the specified GUID had not been previously associated with the object.")

        CHK_ERR(DXGI_ERROR_MORE_DATA, "The specified size of the destination buffer is too small to hold the requested data.")

        CHK_ERR(DXGI_ERROR_UNSUPPORTED, "Unsupported.")

        CHK_ERR(DXGI_ERROR_DEVICE_REMOVED, "Hardware device removed.")

        CHK_ERR(DXGI_ERROR_DEVICE_HUNG, "Device hung due to badly formed commands.")

        CHK_ERR(DXGI_ERROR_DEVICE_RESET, "Device reset due to a badly formed commant.")

        CHK_ERR(DXGI_ERROR_WAS_STILL_DRAWING, "Was still drawing.")

        CHK_ERR(DXGI_ERROR_FRAME_STATISTICS_DISJOINT, "The requested functionality is not supported by the device or the driver.")

        CHK_ERR(DXGI_ERROR_GRAPHICS_VIDPN_SOURCE_IN_USE, "The requested functionality is not supported by the device or the driver.")

        CHK_ERR(DXGI_ERROR_DRIVER_INTERNAL_ERROR, "An internal driver error occurred.")

        CHK_ERR(DXGI_ERROR_NONEXCLUSIVE, "The application attempted to perform an operation on an DXGI output that is only legal after the output has been claimed for exclusive owenership.")

        CHK_ERR(DXGI_ERROR_NOT_CURRENTLY_AVAILABLE, "The requested functionality is not supported by the device or the driver.")

        CHK_ERR(DXGI_ERROR_REMOTE_CLIENT_DISCONNECTED, "Remote desktop client disconnected.")

        CHK_ERR(DXGI_ERROR_REMOTE_OUTOFMEMORY, "Remote desktop client is out of memory.")

```
 
```

// -------------------------------------------------------------

```
// d3d11.h error codes
```

// -------------------------------------------------------------

        CHK_ERR(D3D11_ERROR_TOO_MANY_UNIQUE_STATE_OBJECTS, "There are too many unique state objects.")

        CHK_ERR(D3D11_ERROR_FILE_NOT_FOUND, "File not found")

        CHK_ERR(D3D11_ERROR_TOO_MANY_UNIQUE_VIEW_OBJECTS, "Therea are too many unique view objects.")

        CHK_ERR(D3D11_ERROR_DEFERRED_CONTEXT_MAP_WITHOUT_INITIAL_DISCARD, "Deferred context requires Map-Discard usage pattern")

4b. Discord integration removal

Now this part is the biggest one. All the following code blocks need to be removed or commented out:

src\xrEngine\defines.h

```
	rsDiscord = (1 << 5),
```

src\xrEngine\RenderRef\Execution & 3D\Device\device.cpp

```
extern discord::Core* discord_core;
```
```
extern bool use_discord;
```

src\xrEngine\RenderRef\Execution & 3D\Device\device.cpp

```
	//Discord
```

	if (use_discord && psDeviceFlags2.test(rsDiscord))

```
	{
```
```
		discord_core->RunCallbacks();
```
```
 
```
```
		static float last_update;
```
```
		if (!last_update)
```
```
		{
```
```
			updateDiscordPresence();
```
```
			last_update = Device.fTimeGlobal;
```
```
		}
```

		else if ((Device.fTimeGlobal - last_update) > discord_update_rate)

```
		{
```
```
			updateDiscordPresence();
```
```
			last_update = Device.fTimeGlobal;
```
```
		}
```
```
	}
```

src\xrEngine\General\x_ray.cpp

```
#import discord\discord.h
```

src\xrEngine\General\x_ray.cpp

```
//Discord
```
```
discord::Core* discord_core{};
```
```
discord::Activity discordPresence{};
```
```
static int64_t StartTime;
```
```
bool use_discord = true;
```

#pragma comment(lib, "discord_game_sdk.lib")

```
rpc_info discord_gameinfo;
```
```
rpc_strings discord_strings;
```
```
float discord_update_rate = .5f;
```

src\xrEngine\General\x_ray.cpp

//Discord Rich Presence - Rezy ------------------------------------------------

```
 
```

void DiscordLog(discord::LogLevel level, std::string message)

```
{
```

	Msg("[Discord RPC]: %s", message.c_str());

```
}
```
```
 
```
```
void updateDiscordPresence()
```
```
{
```
```
	if (!use_discord)
```
```
		return;
```
```
 
```
```
	static char details_buffer[128];
```
```
	static char state_buffer[128];
```
```
 
```
```
	// Main Menu
```
```
	if (discord_gameinfo.mainmenu)
```
```
	{
```

		snprintf(state_buffer, 128, discord_strings.mainmenu);

		discordPresence.GetAssets().SetLargeImage("gamelogo");

		discordPresence.GetAssets().SetLargeText("");

		discordPresence.GetAssets().SetSmallImage("");

		discordPresence.GetAssets().SetSmallText("");

```
 
```
```
		// Pause Menu
```
```
		if (discord_gameinfo.ingame)
```

			snprintf(state_buffer, 128, discord_strings.paused);

```
		else
```
```
			discordPresence.SetDetails("");
```
```
	}	
```
```
 
```
```
	// Loading
```
```
	else if (discord_gameinfo.loadscreen)
```
```
	{
```

		snprintf(state_buffer, 128, discord_strings.loading);

```
		discordPresence.SetDetails("");
```

		discordPresence.GetAssets().SetLargeImage("gamelogo");

		discordPresence.GetAssets().SetLargeText("");

		discordPresence.GetAssets().SetSmallImage("");

		discordPresence.GetAssets().SetSmallText("");

```
		discord_gameinfo.ex_update = true;
```
```
	}
```
```
 
```
```
	// In Game
```
```
	else if (discord_gameinfo.ingame)
```
```
	{
```
```
		// Time + Level Name
```
```
		char levelname_time[128];
```

		if (discord_gameinfo.level_name && discord_gameinfo.currenttime)

```
		{
```

			snprintf(levelname_time, 128, "%s | %s", discord_gameinfo.level_name, discord_gameinfo.currenttime);

			discordPresence.GetAssets().SetLargeText(levelname_time);

```
		}
```
```
		else if (discord_gameinfo.level_name)
```
```
		{
```

			snprintf(levelname_time, 128, discord_gameinfo.level_name);

			discordPresence.GetAssets().SetLargeText(levelname_time);

```
		}
```
```
		else
```
```
			discord_gameinfo.ex_update = true;
```
```
 
```
```
		//Faction, Rank, Rep
```

		if (discord_gameinfo.faction && discord_gameinfo.faction_name)

```
		{
```

			discordPresence.GetAssets().SetSmallImage(discord_gameinfo.faction);

```
			char rank_faction_rep[128];
```

			if (discord_gameinfo.rank_name && discord_gameinfo.reputation)

				snprintf(rank_faction_rep, 128, "%s | %s", discord_gameinfo.rank_name, discord_gameinfo.reputation);

```
			else
```

				snprintf(rank_faction_rep, 128, discord_gameinfo.faction_name);

			discordPresence.GetAssets().SetSmallText(rank_faction_rep);

```
		}
```
```
 
```
```
		// GameMode + Active Task
```
```
		if (discord_gameinfo.gamemode)
```
```
		{
```

			if (discord_gameinfo.task_name && 0 != xr_strcmp(discord_gameinfo.task_name, ""))

				snprintf(details_buffer, 128, "%s | %s", discord_gameinfo.gamemode, discord_gameinfo.task_name);

```
			else
```

				snprintf(details_buffer, 128, discord_gameinfo.gamemode);

			discordPresence.SetDetails(details_buffer);

```
		}
```
```
 
```
```
		// God Mode
```
```
		if (discord_gameinfo.godmode)
```

			snprintf(state_buffer, 128, discord_strings.godmode);

```
 
```
```
		// Health
```
```
		else if (discord_gameinfo.health)
```
```
		{
```
```
			// Iron Man
```

			if (discord_gameinfo.ironman && discord_gameinfo.lives_left)

```
			{
```

				if (discord_gameinfo.lives_left == 0 || discord_gameinfo.lives_left > 1)

					snprintf(state_buffer, 128, "%s: %i | %i %s", discord_strings.health, discord_gameinfo.health,

					        discord_gameinfo.lives_left, discord_strings.livesleft);

```
				else
```

					snprintf(state_buffer, 128, "%s: %i | %i %s", discord_strings.health, discord_gameinfo.health,

					        discord_gameinfo.lives_left, discord_strings.livesleftsingle);

```
			}
```
```
 
```
```
			// Azazel
```

			else if (discord_gameinfo.possessed_lives)

```
			{
```

				if (discord_gameinfo.possessed_lives == 0 || discord_gameinfo.possessed_lives > 1)

					snprintf(state_buffer, 128, "%s: %i | %i %s", discord_strings.health, discord_gameinfo.health,

					        discord_gameinfo.possessed_lives, discord_strings.livespossessed);

```
				else
```

					snprintf(state_buffer, 128, "%s: %i | %i %s", discord_strings.health, discord_gameinfo.health,

					        discord_gameinfo.possessed_lives, discord_strings.livespossessedsingle);

```
			}
```
```
 
```
```
			// No Iron Man or Azazel
```
```
			else
```

				snprintf(state_buffer, 128, "%s: %i", discord_strings.health, discord_gameinfo.health);

```
 
```

			discordPresence.SetState(state_buffer);

```
		}
```
```
		else
```
```
		{
```
```
			// Iron Man
```

			if (discord_gameinfo.ironman && discord_gameinfo.lives_left)

```
			{
```

				int real_lives = discord_gameinfo.lives_left - 1;

				if (real_lives == 0 || real_lives > 1)

					snprintf(state_buffer, 128, "%s | %i %s", discord_strings.dead, real_lives, discord_strings.livesleft);

```
				else
```

					snprintf(state_buffer, 128, "%s | %i %s", discord_strings.dead, real_lives,

```
						discord_strings.livesleftsingle);
```
```
			}
```
```
 
```
```
 
```
```
			// Azazel
```

			else if (discord_gameinfo.possessed_lives)

```
			{
```

				if (discord_gameinfo.possessed_lives == 0 || discord_gameinfo.possessed_lives > 1)

					snprintf(state_buffer, 128, "%s | %i %s", discord_strings.dead, discord_gameinfo.possessed_lives,

```
						discord_strings.livespossessed);
```
```
				else
```

					snprintf(state_buffer, 128, "%s | %i %s", discord_strings.dead, discord_gameinfo.possessed_lives,

						discord_strings.livespossessedsingle);

```
			}
```
```
 
```
```
			// No Iron Man or Azazel
```
```
			else
```

				snprintf(state_buffer, 128, "%s", discord_strings.dead);

```
 
```

			discordPresence.SetState(state_buffer);

```
		}
```
```
 
```
```
		// Level Icon
```

		if (discord_gameinfo.level && discord_gameinfo.level_icon_index)

```
		{
```
```
			char icon_buffer[32];
```

			snprintf(icon_buffer, 32, "%s_%i", discord_gameinfo.level, discord_gameinfo.level_icon_index);

			discordPresence.GetAssets().SetLargeImage(icon_buffer);

```
		}
```
```
	}
```
```
 
```

	discordPresence.SetState(state_buffer);

	discord_core->ActivityManager().UpdateActivity(discordPresence, [](discord::Result result) {});

```
}
```
```
 
```
```
void Init_Discord()
```
```
{
```

	auto result = discord::Core::Create(477910171964801060, DiscordCreateFlags_NoRequireDiscord, &discord_core);

```
 
```
```
	if (result != discord::Result::Ok)
```
```
	{
```

		Msg("[Discord RPC] Failed to create Discord RPC");

```
		use_discord = false;
```
```
		return;
```
```
	}
```
```
 
```

	discord_core->SetLogHook(discord::LogLevel::Error, DiscordLog);

	Msg("[Discord RPC] Created successfully!");

```
 
```
```
	//Set up basic RPC
```
```
	StartTime = time(0);
```

	discordPresence.SetType(discord::ActivityType::Playing);

	discordPresence.GetTimestamps().SetStart(StartTime);

	discordPresence.GetAssets().SetLargeImage("gamelogo");

	discord_core->ActivityManager().UpdateActivity(discordPresence, [](discord::Result result) {});

```
}
```
```
 
```
```
void clearDiscordPresence()
```
```
{
```
```
	if (discord_core)
```

		discord_core->ActivityManager().ClearActivity([](discord::Result result) {});

```
}
```

src\xrEngine\General\x_ray.cpp

```
	//Discord Rich Presence - Rezy
```
```
	Init_Discord();
```

src\xrEngine\General\x_ray.cpp

```
	// Discord
```
```
	clearDiscordPresence();
```

src\xrGame\Core\Common\console_commands.cpp

class CCC_DiscordStatus : public CCC_Mask

```
{
```
```
public:
```

	CCC_DiscordStatus(LPCSTR N, Flags32* V, u32 M) :

```
		CCC_Mask(N, V, M){};
```
```
 
```
```
	virtual void Execute(LPCSTR args)
```
```
	{
```
```
		if (EQ(args, "on") || EQ(args, "1"))
```
```
		{
```
```
			value->set(mask, TRUE);
```
```
			discord_gameinfo.ex_update = true;
```
```
		}
```

		else if (EQ(args, "off") || EQ(args, "0"))

```
		{
```
```
			value->set(mask, FALSE);
```
```
			clearDiscordPresence();
```
```
		}
```
```
		else InvalidSyntax();
```
```
	}
```
```
};
```

src\xrGame\Core\Common\console_commands.cpp

```
	//Discord
```
```
	psDeviceFlags2.set(rsDiscord, TRUE);
```

	CMD3(CCC_DiscordStatus, "discord_status", &psDeviceFlags2, rsDiscord);

	CMD4(CCC_Float, "discord_update_rate", &discord_update_rate, .5f, 5.f);

src\xrGame\Core\Common\StringTable\string_table.cpp

```
	//Discord
```

	snprintf(discord_strings.mainmenu, 128, xr_ToUTF8(*CStringTable().translate("st_main_menu")));

	snprintf(discord_strings.paused, 128, xr_ToUTF8(*CStringTable().translate("st_pause_menu")));

	snprintf(discord_strings.loading, 128, xr_ToUTF8(*CStringTable().translate("st_loading")));

	snprintf(discord_strings.health, 128, xr_ToUTF8(*CStringTable().translate("st_ui_health_sensor")));

	snprintf(discord_strings.dead, 128, xr_ToUTF8(*CStringTable().translate("st_player_dead")));

	snprintf(discord_strings.livesleft, 128, xr_ToUTF8(*CStringTable().translate("st_hardcore_lives_left")));

	snprintf(discord_strings.livesleftsingle, 128, xr_ToUTF8(*CStringTable().translate("st_hardcore_lives_left_single")));

	snprintf(discord_strings.livespossessed, 128, xr_ToUTF8(*CStringTable().translate("st_azazel_lives_possessed")));

	snprintf(discord_strings.livespossessedsingle, 128, xr_ToUTF8(*CStringTable().translate("st_azazel_lives_possessed_single")));

	snprintf(discord_strings.godmode, 128, xr_ToUTF8(*CStringTable().translate("st_godmode")));

```
 
```
```
	discord_gameinfo.ex_update = true;
```

src\xrGame\UI\Common\MainMenu\MainMenu.cpp

```
	//Discord
```
```
	discord_gameinfo.mainmenu = bActivate;
```

	if (bActivate && psDeviceFlags2.test(rsDiscord))

```
		updateDiscordPresence();
```

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

```
	//Discord
```
```
	discord_gameinfo.ingame = true;
```

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

```
	//Discord
```
```
	discord_gameinfo.ingame = false;
```

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

```
//Discord
```
```
	if (psDeviceFlags2.test(rsDiscord))
```
```
	{
```
```
		//God
```

		bool isGodmode = psActorFlags.test(AF_GODMODE);

```
		discord_gameinfo.godmode = isGodmode;
```
```
 
```
```
		if (!isGodmode)
```
```
		{
```

			int disc_cur_health = roundf(GetfHealth() * 100);

```
			if (disc_cur_health <= 0)
```
```
				discord_gameinfo.health = NULL;
```
```
			else
```

				discord_gameinfo.health = disc_cur_health;

```
		}
```
```
 
```
```
		//Current Time
```

		str_c current_time = InventoryUtilities::GetGameTimeAsString(InventoryUtilities::etpTimeToMinutes).c_str();

		discord_gameinfo.currenttime = current_time;

```
 
```
```
		// Update once after a loadscreen
```

		if (!discord_gameinfo.loadscreen && discord_gameinfo.ex_update)

```
		{
```
```
			//Update Iron Man state and lives
```
```
			luabind::functor ironman_enabled;
```

			if (ai().script_engine().functor("_g.IsHardcoreMode", ironman_enabled))

```
			{
```

				if (ironman_enabled && ironman_enabled())

```
					discord_gameinfo.ironman = true;
```
```
				else
```
```
					discord_gameinfo.ironman = false;
```
```
			}
```
```
 
```
```
			if (discord_gameinfo.ironman)
```
```
			{
```
```
				//Lives left
```
```
				luabind::functor ironman_lives;
```

				if (ai().script_engine().functor("ironman_manager.get_lives_left", ironman_lives))

```
				{
```
```
					if (ironman_lives)
```
```
					{
```
```
						int lives_left = ironman_lives();
```

						discord_gameinfo.lives_left = lives_left;

```
					}
```
```
				}
```
```
			}
```
```
 
```
```
			//Level
```

			if (g_pGameLevel && g_pGameLevel->name() != NULL)

```
			{
```

				snprintf(discord_gameinfo.level_name, 128, xr_ToUTF8(*CStringTable().translate(g_pGameLevel->name())));

```
				srand(time(0));
```
```
				int level_icon_id = rand() % 3 + 1;
```

				discord_gameinfo.level_icon_index = level_icon_id;

				discord_gameinfo.level = g_pGameLevel->name().c_str();

```
			}
```
```
 
```
```
			//Story Mode
```
```
			luabind::functor game_mode;
```

			if (ai().script_engine().functor("_g.IsStoryMode", game_mode) && game_mode())

				snprintf(discord_gameinfo.gamemode, 128, xr_ToUTF8(*CStringTable().translate("st_cap_check_story")));

```
 
```
```
			//Warfare
```

			else if (ai().script_engine().functor("_g.IsWarfare", game_mode) && game_mode())

				snprintf(discord_gameinfo.gamemode, 128, xr_ToUTF8(*CStringTable().translate("st_cap_check_warfare")));

```
 
```
```
			//Azazel Mode
```

			else if (ai().script_engine().functor("_g.IsAzazelMode", game_mode) && game_mode())

```
			{
```

				snprintf(discord_gameinfo.gamemode, 128, xr_ToUTF8(*CStringTable().translate("st_cap_check_azazel_mode")));

```
 
```
```
				luabind::functor possessed_lives;
```

				if (ai().script_engine().functor("azazel_mode.get_possessed_lives", possessed_lives))

```
				{
```

					int lives_possessed = possessed_lives();

					discord_gameinfo.possessed_lives = lives_possessed;

```
				}
```
```
			}
```
```
 
```
```
			//Survival Mode
```

			else if (ai().script_engine().functor("_g.IsSurvivalMode", game_mode) && game_mode())

				snprintf(discord_gameinfo.gamemode, 128, xr_ToUTF8(*CStringTable().translate("st_cap_check_survival")));

```
 
```
```
			//Freeplay Mode
```
```
			else
```

				snprintf(discord_gameinfo.gamemode, 128, xr_ToUTF8(*CStringTable().translate("st_cap_check_freeplay")));

```
 
```
```
			//Update Active Task
```

			Level().GameTaskManager().RPC_UpdateTaskName();

```
 
```

			//Update Faction, Rank and Reputation

```
			RPC_UpdateFaction();
```
```
			RPC_UpdateRank();
```
```
			RPC_UpdateReputation();
```
```
 
```
```
			discord_gameinfo.ex_update = false;
```
```
		}
```
```
	}
```

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

			discord_gameinfo.faction = faction_name;

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

			snprintf(discord_gameinfo.faction_name, 128, xr_ToUTF8(*CStringTable().translate(buffer)));

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

			snprintf(discord_gameinfo.rank_name, 128, xr_ToUTF8(*CStringTable().translate(rank_name)));

src\xrGame\Core\Client\Objects\actor\base\Actor.cpp

```
					if (reputation_name)
```

						snprintf(discord_gameinfo.reputation, 128, xr_ToUTF8(*CStringTable().translate(reputation_name)));

src\xrGame\Core\Client\Objects\tasks_info_dialogs\game tasks\GametaskManager.cpp

```
	//Discord
```
```
	if (psDeviceFlags2.test(rsDiscord))
```
```
		RPC_UpdateTaskName();
```

src\xrGame\Core\Client\Objects\tasks_info_dialogs\game tasks\GametaskManager.cpp

void CGameTaskManager::RPC_UpdateTaskName()

```
{
```
```
	CGameTask* tr = ActiveTask();
```
```
	if (tr)
```

		snprintf(discord_gameinfo.task_name, 128, xr_ToUTF8(*CStringTable().translate(tr->m_Title)));

```
}
```

src\xrGame\Core\Client\Level\level_script.cpp

```
	if (psDeviceFlags2.test(rsDiscord))
```
```
	{
```
```
		Actor()->RPC_UpdateFaction();
```
```
		Actor()->RPC_UpdateRank();
```
```
		Actor()->RPC_UpdateReputation();
```
```
 
```

		Level().GameTaskManager().RPC_UpdateTaskName();

```
	}
```

The next part handles editing a file of the xrRender_R2 layer. Technically it doesn’t matter whether you edit it for xrRender_R1, xrRender_R2, xrRender_R3 or xrRender_R4, as removing the following code for one of them will affect all four. I just picked xrRender_R2 arbitrarily:

src\Layers\xrRender_R2\Interfase Implementations\ApplicationRender\dxApplicationRender.cpp

```
	//Discord
```
```
	discord_gameinfo.loadscreen = true;
```
```
 
```
```
	if (psDeviceFlags2.test(rsDiscord))
```
```
		updateDiscordPresence();
```

src\Layers\xrRender_R2\Interfase Implementations\ApplicationRender\dxApplicationRender.cpp

```
	//Discord
```
```
	discord_gameinfo.loadscreen = false;
```
```
 
```
```
	if (psDeviceFlags2.test(rsDiscord))
```
```
	{
```
```
		discord_gameinfo.ex_update = true;
```
```
		updateDiscordPresence();
```
```
	}
```

Aaand that’s it. Minus all the stuff I forgot to document of course. But even if this doesn’t work a 100% out of the box, it should at least give you about 90% of the idea.

If you your compiler does die with a heap memory exhaustion error caused by the precompiled header memory allocation, you can just open the affected projects’ “Configuration Properties \ C/C++ \ All Options” and add the option /Zm256 to the “Additional Options” field. Or if one is already set there, e.g. /Zm112, just increase the value to 256, then rebuild.

Once the build completes successfully, you will find the game executable here: _build\_game\bin_dbg\AnomalyDX9.exe. Additionally to that you’ll need to copy the companion libraries tbb.dll, soft_oal.dll, icuuc65.dll and icudt65.dll from the sdk\binaries\ subfolder. The discord_game_sdk.dll also sitting there can be ignored, as all Discord support is now gone with this patched version of S.T.A.L.K.E.R. Anomaly.

Edit: Actually, only almost, as I just found out. There is one Discord artifact left in the game’s main menu: The switch for enabling Discord status reporting on the game’s channel. It’s a dud though, so while you can ostensibly enable it and save that setting, the next time to open that submenu, it’ll be deactivated again. No crashes or anything. See here:

S.T.A.L.K.E.R. Anomaly "Discord status" option in the main menu

“Discord status” option found in the main menu under “Settings \ Others” (click to enlarge)

This concludes all necessary build system & source code modifications to build the game for a Windows XP Professional x64 Edition target.

And with that: Cheers!

Step by step SSL/TLS certificate change around here

Security, Server No Responses »

Jun 132022

In May 2021, I had [mentioned] that XIN.at had been starting to use Let’s Encrypt certificates with it’s root and intermediate certificates cross-signed by the expired IdenTrust DST Root CA X3 certificate authority. This was done mostly for compatibility reasons, as despite its expiry, some platforms could still trust DST Root CA X3, but not the newer ISRG Root X1. Let’s Encrypt themselves mention Windows XP and Android 7 as such platforms on their [Chain of Trust page]. Recently, some programs, e.g. FileZilla on RedHat Enterprise Linux and probably other systems or R2Mail2 on Android have stopped trusting this cross-signing construct and require constant user intervention to remain operable, which is quite cumbersome for those end users with modern platforms and software.

Let’s Encrypt chain of trust since August 2021; The DST Root CA X3 is optional (click to enlarge)

On top of that I believe that most Windows XP clients still accessing any services on this server will quite likely be using software integrating their own, much newer cryptographic libraries such as OpenSSL, and their own certificate stores. Examples would include Java, all forks of Mozilla web browsers with XP compatibility and current security fixes, the Thunderbird eMail client, Python and many others.

Hence, I have decided to slowly phase out the DST Root CA X3 cross-signed ISRG Root X1 as well as the equally cross-signed R3 intermediate certificates. From now on, services will be adapted to using this chain:

ISRG Root X1 ―signs→ R3 Intermediate ―signs→ XIN.at subscriber/server certificate

…as opposed to this one:

DST Root CA X3 (expired)―signs―┐
            │          ↓
          signs        R3 Intermediate ―signs→ XIN.at subscriber/server certificate
            ↓                ↑
       ISRG Root X1―――signs――――┘

For now, only the FTP+TLS service has been altered as such. The next ones will be the eMail server, the XIN.at webchat frontend, and the main web server, in no particular order.

By now there should be virtually zero clients running into problems as a result of this cleanup!

Also: Please note that while TLS v1.2 and modern cipher suites have been backported for most services on my ancient server, this was so far not possible for the main web server, which will remain using TLS v1.0 and older ciphers. If you cannot access this weblog because of that limitation (e.g. you’re using a modern Google Chrome instead of a reconfigured FireFox), you’ll have to revert to plain, unencrypted HTTP. For most of what few readers stumble over this page that should be fine anyway, given the nature of this site.

Edit 2022-06-22: The certificate chain of the secure IRC server listening at [ircs://xin.at:6697] has now been updated. This does not yet affect the webchat interface at [https://xin.at:8080], which will come next. For very old clients that do not know the ISRG Root X1 CA, you may still be able to connect if your client allows you to enable untrusted certificates, e.g. as X-Chat / HexChat do. Legacy cryptographic protocols implemented for backwards compatibility (e.g. with Windows 9x, 2000, XP, or old Linux 2.x-based systems) will remain in place, namely SSLv3 and TLS v1.0. Aside from having to allow untrusted certificates on very old machines, this should not have any negative effect for users, at least not a prohibitive one.

Edit 2022-06-23: Today, the following services have been updated in the same way: Web mail, web mail administration, SMTPS, IMAPS, POP3S and the [web chat interface]. While doing so, an issue was discovered for the web chat interface while testing it with [testssl]; The server had only provided the subscriber certificate, but no full chain including intermediate and root CA certificates. This was simply a configuration error that has now been corrected!

Older Entries