Jun 072015

Kung Fury: Street Rage logoSo after the release of that crazy crowdfunded (and free of charge) movie [Kung Fury], there is also a game! Now that was fast. Made by the Swedish game developers of [Hello There], the game is basically a clone of [One Finger Death Punch], as many gamers have already pointed out. Not that anybody seems to mind that – me included. It’s a superficially very simple 2-button street fighting game, where one button means “punch/kick/whatever to the left” and the other “punch/kick/whatever to the right”. Don’t let the seeming simplicity fool you though. There is more skill involved than you might think…

So let’s have a look at the intro of the game, which strongly resembles an 80s arcade machine style:

Kung Fury: Street Rage; That's our Hero!

Kung Fury: Street Rage; That’s our Hero!

So with the use of some Direct3D 9.0c shaders, the game simulates the look of an old CRT monitor, just like the arcade machines of old had! At the press of a button or after waiting for a bit we’re greeted with this:

Kung Fury: Street Rage - Insert Coin

Insert Coin!

Another button press and we can hear our virtual player throwing a coin into the machine, which gives us three lives (after being hit three times, we’d go down for good). And then, whenever any enemy approaches us from either side, you just press left or right to punch, kick, shoot or electrify the guy. It’s ok, they’re all Nazis anyway. We do this with our pals Barbarianna, Triceracop and Hackerman standing around in the background – all three as seen in the movie of course, just like all the enemies we’re beating up:

Kung Fury: Street Rage; Beat 'em up!

There are splatter effects even!

That screenshot is from the very beginning of the game, where we can only see our lowest-end Nazi foes. There are some Swedish Aryans too, which can take two hits, then that clone chick with Kung Fury essence infused into her, which needs a more advanced left/right combo to put down, and more. Like the kicking machine and the mysterious Ninja, all as seen in the movie. As long as you don’t miss too much (you have limited range) or get hit, you’ll build up a score bonus too. Not sure if there are more enemies than that, I haven’t really gotten that far yet.

Actually, I did reach a new High Score while doing those screenshots accidentally, leaving both chicks behind me, pretty neat:

Kung Fury: Street Rage; New High Score!

A new High Score!

Now Thor might still be doable, but Hackerman will be one tough nut to crack. I don’t think I’ll ever make it to the top though, the game is pretty damn hard. As it progresses, it starts speeding up more and more, and it’ll also throw more of the harder enemies at you, which will require quick reaction and sharp perception to get the combos right. “Just mash two buttons” may sound easy as said, but don’t underestimate it! Like with “One Finger Death Punch”, only the most skillful players will have a chance to reach the top!

When you’ve got enough, just press <Esc> (on the PC), and you’re asked whether you really want to quit. In an interesting way:

Kung Fury: Street Rage; Quitting the game.

Quitting/bluescreening the game.

If you confirm to quit the game, you’ll get another shader-based CRT effect thrown in:

Kung Fury: Street Rage; Quitting the game

It’s shutting down…

I haven’t really managed to play this for more than 5 minutes in a row, which sounds like very little, but this game is extremely fast-paced, so I can’t take much more in one go. ;) It’s quite a lot of fun though, and while not as sophisticated as “One Finger Death Punch”, it’s awesome in its own right, given the Kung Fury cheesiness, the CRT look and the chiptune-like soundtrack of the game.

The game is available in both paid and partially also free editions on several platforms now, and while I’ve read that the free versions do have ads, the paid ones definitely don’t, as I can vouch for on the PC platform at least. So here are the links:

  • [PC version] @ Steam for 1.99€ / $2.50. Supports >=Windows XP, >=MacOS X 10.6 and SteamOS plus regular Linux on x86_32/x86_64.
  • [PC+ARM version] @ Windows Store for $2.29. Supports Windows >=8.0 on x86_32/x86_64 and ARM architectures.
  • [Android version] @ Google Play for free or for 2.46€ with ads removed. Also available as a [separate APK file]. Supports Android >=2.0.1.
  • [iOS version] @ iTunes for free or for $1.99 with ads removed. Supports iOS >=6.0 on the iPhone 5/6, iPad and iPod Touch.

Now if you’ll excuse me, I’ll take another shot at number 3! :)

Jan 152015

4Kn logoWhile I’ve been planning to build myself a new RAID-6 array for some time (more space, more speed), I got interested in the latest and greatest of hard drive innovations, which is the 4Kn Advanced Format. Now you may now classic hard drives with 512 byte sectors and the regular Advanced Format also known as 512e, which uses 4kiB physical sector sizes, but emulates 512 byte sectors for compatibility reasons. Interestingly, [Microsoft themselves state], that “real” 4Kn harddrives, which expose their sector size to the operating system with no glue layers in between are only supported in Windows 8 and above. So even Windows 7 has no official support.

On top of that, Intel [has stated], that their SATA controller drivers do not support 4Kn, so hooking one such drive up to your Intel chipsets’ I/O controller hub (ICH) or platform controller hub (PCH) will not work. Quote:

“Intel® Rapid Storage Technology (Intel® RST) version 9.6 and newer supports 4k sector disks if the device supports 512 byte emulation (512e). Intel® RST does not support 4k native sector size devices.”

For clarity, to make 4Kn work in a clean fashion, it must be supported on three levels, from lowest to highest:

  1. The firmware: For mainboards, this means your system BIOS/UEFI. For dedicated storage controllers, the controller BIOS itself.
  2. The kernel driver of the storage controller, so that’s your SATA AHCI/RAID drivers or SAS drivers.
  3. Any applications above it performing raw disk access, whether kernel or user space. File system drivers, disk cloning software, low level benchmarks, etc.

Granted, 4Kn drives are extremely new and still very rare. There is basically only the 6TB Seagate enterprise drives available ([see here]) and then some Toshiba drives, also enterprise class. But, to protect my future investments in that RAID-6, I got myself a [Toshiba MG04ACA300A] 3TB drive, which was the only barely affordable 4Kn disk I could get, basically also the only one available right now besides the super expensive 6TB Seagates. That way I can check for 4Kn compatibility relatively cheaply (click to enlarge images):

If you look closely, you can spot the nice 4Kn logo right there. In case you ask yourselves “Why 4Kn?”, well, mostly cost and efficiency. 4kiB sectors are 8 times as large as classic 512 byte ones. Thus, for the same data payload you need 8 times less sector gaps, 8 times less synchronization markers and 8 times less address markers. Also, a stronger checksum can be used for data integrity. See this picture from [Wikipedia]:


Sector size comparison (Image is © Dougolsen under the CC-BY 3.0 unported license)

Now this efficiency is already there with 512e drives. 512e Advanced Format was supposedly invented, because more than half the programs working with raw disks out there can’t handle variable sector sizes and are hardcoded for 512n. That also includes system firmwares, so your mainboards’ BIOS/UEFI. To solve those issues, they used 4kiB sectors, then let a fast ARM processor translate them into 512 byte sectors on the fly to give legacy software something it could understand.

4Kn on the other hand is the purist, “right” approach. No more emulation, no more cheating. No more 1GHz ARM dual core processor in your hard drive just to be able to serve data fast enough.

Now we already know that Intel controllers won’t work. For fun, I hooked it up to my ASUS P6T Deluxes’ secondary SATA controller though, a Marvell 88SE6120. Plus, I gave the controller the latest possible driver, the quite hard-to-get version You can download that [here] for x86 and x64.  To forestall the result: It doesn’t work. At all. This is what the systems’ log has to say about it (click to enlarge):

So that’s a complete failure right there. Even after the “plugged out” message, the timeouts would still continue to happen roughly every 30 seconds, accompanied by the whole operating system freezing for 1-2 seconds every time. I cannot say for any other controllers like the Marvell 9128 or Silicon Image chips and others, but I do get the feeling that none of them will be able to handle 4Kn.

Luckily, I do already have the controller for my future RAID-6 right here, an Areca ARC-1883ix-12, the latest and greatest tech with PCIe x8 3.0 and SAS/12Gbit ports with SATA/6Gbit encapsulation. Its firmware and driver supports 4Kn fully as you can see in Arecas [specifications]. The controller features an out-of-band management system via its own ethernet port and integrated web server for browser-based administration, even if the system doesn’t even have any OS booted up. All that needs to be installed on the OS then is a very tiny driver (click to enlarge):

Plus, Areca gives us one small driver for many Windows operating systems. Only for the Windows XP 32-Bit NT5.1 kernel you’ll get a SCSI Miniport driver exclusively, while all newer systems (WinXP x64, Windows Vista, 7, 8) get a more efficient StorPort driver. So, plugged the controller in, installed the driver, hooked up the disk, and it seems we’re good to go:

The 4Kn drive is being recognized

The 4Kn drive is being recognized (click to enlarge)

Now, any legacy master boot record (MBR) partition table has a 32-bit address field. That means, it can address 232 elements. With each element being 512 bytes large, you reach 2TiB. So that’s where the 2TiB limit comes from. With 4Kn however, the smallest addressable atom is now eight times as large: 4096 bytes! So we should be able to reach 16TiB due to the larger sector size. Supposedly, some USB hard drive manufacturers have used this trick (by emulating 4Kn) to make their larger drives work easily on Windows XP. When trying to partition the Toshiba drive however, I hit a wall, as it seems Windows disk management is about as stupid as was the FAT32 formatter on Windows 98:

MBR initialization failed

MBR initialization failed (click to enlarge)

That gets me thinking. On XP x64, I can still just switch from MBR to the GPT partitioning scheme to be able to partition huge block devices. But what about Windows XP 32-bit? I don’t know how the USB drive manufacturers do it, so I can only presume they ship the drives pre-partitioned if its one of those that don’t come with a special mapping tool for XP. In my case, I just switch to GPT and carry on (click to enlarge):

Now I guess I am the first person in the world to be able to look at this, and potentially the last too:

fsutil.exe showing a 4Kn drive on XP x64

fsutil.exe showing a native SATA 4Kn drive on XP x64, encapsulated in SAS. Windows 7 would show the physical and logical sector size separately due to its official 512e support. Windows XP always reports the logical sector size (click to enlarge)

So far so good. The very first and most simple test? Just copy a file onto the newly formatted file system. I picked the 4k (no pun intended) version of the movie “Big Buck Bunny”:

Copying a first file onto the 4Kn disks NTFS file system

Copying a first file onto the 4Kn disks NTFS file system

Hidden files and folders are shown here, but Windows doesn’t seem to want to create a System Volume Information\ folder for whatever reason. Other than that it’s very fast and seems to work just nicely. Since the speed is affected by the RAID controllers write back cache, I thought I’d try HD Tune 2.55 for a quick sequential benchmark. Or in other words: “Let’s hit our second legacy software wall” (click to enlarge):

Yeah, so… HD Tune never detects anything above 2TiB, but this? At first glance, 375GB might sound quite strange for a 3TB drive. But consider this: 375 × 8 = 3000. What happened here is that HD Tune got the correct sector count of the drive, but misinterpreted each sectors’ size as 512 bytes. Thus, it reports the devices’ size as eight times as small. Reportedly, this is also the exact way how Intels RST drivers fail when trying to address a 4Kn drive. HD Tune 2.55 is thus clearly hardcoded for 512n. There is no way to make this work. Let’s try the paid version of the tool which is usually quite ahead of its free and legacy counterpart (click to enlarge):

Indeed, HD Tune Pro 5.00 works just as it should when accessing the raw drive. Users who don’t want to pay are left dead in the water here. Next, I tried HDTach, also an older tool. HDTach however reads from a formatted file system instead of from a raw block device. The file system abstracts the device to a higher level, so HDTach doesn’t know and doesn’t need to know anything about sectors. As a result, it also just works:

HD Tach benchmarking NTFS on a 4Kn drive

HD Tach benchmarking NTFS on a 4Kn drive (click to enlarge)

Next, let’s try an ancient benchmark, that again accesses drives on the sector level: The ATTO disk benchmark. It is here where we learn that 4Kn, or generally variable sector sizes aren’t space magic. This tool was written well before the times of 512e or 4Kn, and look at that (click to enlarge):

Now what does that tell us? It tells us, that hardware developers feared the chaotic ecosystem of tools and software that accesses disks at low levels. Some might be cleanly programmed, where most may not. That doesn’t just include operating systems’ built-in toolsets, but also 3rd party software, independently from the operating system itself. Maybe it also affects disk cloning software like from Acronis? Volume shadow copies? Bitlocker? Who knows. Thing is, to be sure, you need to test that stuff. And I presume that to go as far as hard drive manufacturers did with 512e, they likely found one abhorrent hell of crappy software during their tests. Nothing else will justify ARM processors at high clock rates on hard drives just to translate sector sizes plus all the massive work that went into defining the 512e Advanced Format standard before 4Kn Advanced Format.

Windows 8 might now fully support 4Kn, but that doesn’t say anything about the 3rd party software you’re going to run on that OS either. So we still live in a Windows world where a lot of fail is waiting for us. Naturally, Linux and certain UNIX systems have adapted much earlier or have never even made the mistake of hardcoding sector sizes into their kernels and tools.

But now, to the final piece of my preliminary tests: Truecrypt. A disk encryption software I still trust despite the project having been shut down. Still being audited without any terrible security hole discoveries so far, it’s my main choice for cross-platform disk encryption, working cleanly on at least Windows, MacOS X and Linux.

Now, 4Kn is disabled for MacOS X in Truecrypts source code, but seemingly, this [can be fixed]. I also discovered that TC will refuse to use anything other than 512n on Linux if Linux kernel crypto is unavailable or disabled by the user in TC, see this part of Truecrypts’ CoreUnix.cpp:

#if defined (TC_LINUX)
if (volume-&gt;GetSectorSize() != TC_SECTOR_SIZE_LEGACY)
  if (options.Protection == VolumeProtection::HiddenVolumeReadOnly)
    throw UnsupportedSectorSizeHiddenVolumeProtection();
  if (options.NoKernelCrypto)
    throw UnsupportedSectorSizeNoKernelCrypto();

Given that TC_SECTOR_SIZE_LEGACY equals 512, it becomes clear that hidden volumes are unavailable as a whole with 4Kn on Linux, and encryption is completely unavailable altogether if kernel crypto isn’t there. So I checked out the Windows specific parts of the code, but couldn’t find anything suspicious in the source for data volume encryption. It seems 4Kn is not allowed for bootable system volumes (lots of “512’s” there), but for data volumes it seems TC is fully capable of working with variable sector sizes.

Now this code has probably never been run before on an actual SATA 4Kn drive, so let’s just give it a shot (click to enlarge):

Amazingly, Truecrypt, another software written and even abandoned by its original developers before the advent of 4Kn works just fine. This time, Windows does create the System Volume Information\ folder on the file system within the Truecrypt container, and fsutil.exe once again reports a sector size of 4096 bytes. This shows clearly that TC understands 4Kn and passes the sector size on to any layers above itself in the kernel I/O stack flawlessly (The layer beneath it should be either the NT I/O scheduler or maybe the storage controller driver directly and the layer above it the NTFS file system driver, if my assumptions are correct).

Two final tests for data integrities’ sake:

Both a binary diff and SHA512 checksums prove, that the data copied from a 512n medium to the 4Kn one is still intact

Both a binary diff and SHA512 checksums prove, that the data copied from a 512n medium to the 4Kn one is still intact

So, my final conclusion? Anything that needs to work with a raw block device on a sector-by-sector level needs to be checked out before investing serious money in such hard drives and storage arrays. It might be cleanly programmed, with some foresight. It also might not.

Anything that sits above the file system layer though (anything that reads and writes folders and files instead of raw sectors) will always work nicely, as such software does not need to know anything about sectors.

Given the possibly enormous amount of software with hardcoded routines for 512 byte sectors, my assumption would be that the migration to 4Kn will be quite a sluggish one. We can see that the enterprise sector is adapting first, clearly because Linux and UNIX systems adapt much faster. The consumer market however might not see 4Kn drives anytime soon, given 512 byte sectors have been around for about 60 years (!) now.

Update 2014-01-16 (Linux): I just couldn’t let it go, so I took the Toshiba 4Kn drive to work with me, and hot plugged it into an Intel ICH10R. So that’s the same chipset as the one I ran the Windows tests on, an Intel X58. Only difference is, that now we’re on CentOS 6.6 Linux running the 2.6.32-504.1.3.el6.x86_64 kernel.  This is what dmesg had to say about my hotplugging:

ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-8: TOSHIBA MG04ACA300A, FP2A, max UDMA/100
ata3.00: 732566646 sectors, multi 2: LBA48 NCQ (depth 31/32), AA
ata3.00: configured for UDMA/100
ata3: EH complete
scsi 2:0:0:0: Direct-Access     ATA      TOSHIBA MG04ACA3 FP2A PQ: 0 ANSI: 5
sd 2:0:0:0: Attached scsi generic sg7 type 0
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
sd 2:0:0:0: [sdf] Write Protect is off
sd 2:0:0:0: [sdf] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
sd 2:0:0:0: [sdf] Attached SCSI disk

Looking good so far, also the Linux kernel typically cares rather less about the systems BIOS, bypassing whatever crap it’s trying to tell the kernel. Which is usually a good thing. Let’s verify with fdisk:

Note: sector size is 4096 (not 512)
WARNING: The size of this disk is 3.0 TB (3000592982016 bytes).
DOS partition table format can not be used on drives for volumes
larger than (17592186040320 bytes) for 4096-byte sectors. Use parted(1) and GUID 
partition table format (GPT).

Now that’s more like it! fdisk is warning me, that it will be limited to addressing 16TiB on this disk. A regular 512n or 512e drive would be limited to 2TiB as we know. Awesome. So, I created a classic MBR style partition on it, formatted it using the EXT4 file system, and mounted it. And what we get is this:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdf1             2.7T   73M  2.6T   1% /mnt/sdf1

And Intel is telling us that they don’t manage to give us any Windows drivers that can do 4Kn? Marvell doesn’t even comment on their inabilities? Well, suck this: Linux’ free driver for an Intel ICH10R south bridge (or any other that has a driver coming with the Linux kernel for that matter) seems to have no issues with that whatsoever. I bet it’s the same with BSD. Just weak, Intel. And Marvell. And all you guys who had so much time to prepare and yet did nothing!

Update 2014-01-20 (Windows XP 32-Bit): So what about regular 32-Bit Windows XP? There are stories going around that some USB drives with 3-4TB capacity would use a 4Kn emulation (or real 4Kn, bypassing the 512e layer by telling the drive firmware to do so?), specifically to enable XP compatibility without having to resort to special mapping tools.

Today, I had the time to install XP SP3 on a spare AMD machine (FX9590, 990FX), which is pretty fast thanks to a small, unused testing SSD I still had lying around. Before that I wiped all GPT partition tables from the 4Kn drive, both the one at the start as well as the backup copy at the end of the drive using dd. Again, for this test, the Areca ARC-1883ix-12 was used, now with its SCSI miniport driver, since XP 32-Bit does not support StorPort.

Please note, that this is a German installation of Windows XP SP3. I hope the screenshots are still understandable enough for English speakers.

Recognition and MBR initialization seems to work just fine this time, unlike on XP x64:

The 4Kn Toshiba as detected by Windows XP Pro 32-Bit SP3, again on an Areca ARC-1883ix-12

The 4Kn Toshiba as detected by Windows XP Pro 32-Bit SP3, again on an Areca ARC-1883ix-12 (click to enlarge)

Let’s try to partition it:

Partitioning the drive once more, MBR style

Partitioning the drive once more, MBR style

Sure looks good! And then, we get this:

A Master Boot Record, Windows XP and 4Kn: It does work after all

A Master Boot Record, Windows XP and 4Kn: It does work after all (click to enlarge)

So why does XP x64 not allow for initialization and partitioning of a 4Kn drive using MBR? Maybe because it’s got GPT for that? So in any case, it’s usable on both systems, the older NT 5.1 (XP 32-Bit) as well as the newer NT 5.2 (XP x64, Server 2003). Again, fsutil.exe confirms proper recognition of our 4Kn drive:

fsutil.exe reporting a 4kiB sector size, just like on XP x64

fsutil.exe reporting a 4kiB sector size, just like on XP x64

So all you need – just like on XP x64 – is a proper controller with proper firmware and drivers!

There is one hard limit here though that XP 32-Bit users absolutely need to keep in mind; Huge RAID volumes using LUN carving/splitting and software JBOD/disk spanning using Microsofts Dynamic Volumes are no longer possible when using 4Kn drives. Previously, you could tell certain RAID controllers to just serve huge arrays to the OS in 2TiB LUN slices (e.g. best practice for 3ware controllers on XP 32-Bit). Then, in Windows, you’d just make those slices Dynamic Volumes and span a single NTFS file system over all of them, thus pseudo-breaking the 2TiB barrier.

This can no longer be done, as Dynamic Volumes seemingly do not work with 4Kn drives on Microsoft operating systems before Windows 8, or at least not on XP 32-Bit. The option for converting the volume from MBR to GPT is simply greyed out in Windows disk management.

That means that the absolute maximum volume size using 4Kn disks on 32-Bit Windows XP is 16TiB! On XP x64 – thanks to GPT – it’s just a few blocks short of 256TiB, a limit imposed on us by the NTFS file systems’ 32-bit address field and 64kiB clusters, as 232 * 64KiB × 1024 × 1024 × 1024 = 256TiB.

And that concludes my tests, unless I have time and an actual machine to try FreeBSD or OpenBSD UNIX. Or maybe Windows 7. The likelihood for that is not too high at the moment though.

Jan 222014

clink logoTypically, any avid Linux/UNIX user would sneer at the default Windows command line shell, cmd.exe. I do use it from time to time though, for tools like [eac3to] or [x264]. By Bash and other UNIX shells’ standards, cmd.exe is simply weak, powerless and inefficient though. On top of that, its scripting language – simply called Batch – is quite awkward and nothing compared to say Bash. You can go the classic “do one thing well” [1] route and extend the Batch functionality with specialized tools like grep, awk and sed for Windows, or even by alternative scripting languages like MS VBScript or Perl, which I sometimes do. But oh well, we’re not gonna fix that.

But using the awkward interactive shell? We can fix that! Say hello to [clink]!

Welcome to cmd+clink

Welcome to cmd+clink

So what’s clink? A way of enhancing cmd.exe (and partly even the newer [Windows PowerShell]) with the [GNU readline] libraries’ functionality and with other upgrades. “Aha” you might think, “so how’s that gonna help?”. Well, it’s going to help by adding tons of stuff you would normally only see on a UNIX-style shell, I’ll give you a few examples:

  • A better auto-completion that is not limited to auto-completing local file names:
    • cmd can still auto-complete file and folder names in your current directory by pressing <TAB>. It will now do so incrementally though!
    • cmd can now also auto-complete all executable program names in your %PATH%! You wanna launch notepad? Type “not”, press <TAB>, <ENTER>, done. If there are ambiguous results, double-<TAB> will show you a list of eligible file names or other auto-completable objects!
    • It also works for environment variables. You want to go to your users home directory in %USERPROFILE%? Type “cd %u”, press <TAB>, it will auto-complete to “cd %user”, as there are three variables called %USERPROFILE%, %USERDOMAIN% and %USERNAME%. Extend that by adding a p: “cd %userp”, press <TAB> again and you got the whole variable, then <ENTER>. All that can greatly speed up your shell usage!
    • If all that’s not enough for you, you can extend and modify the auto-completion behavior using the Lua scripting language.
  • The command history is no longer “Doskey” style. It will stay persistent across sessions, and is now incrementally searchable (<CTRL>+<R> / <CTRL>+<S>)!
  • The clipboard integration is now far better. Simply paste stuff from the clipboard to the shell using <CTRL>+<V> like everywhere else, or use the right mouse button.
  • Powerful line editing features like delete last word with <CTRL>+<W>, blank screen (<CTRL>+<L>), undo (<CTRL>+<Z>)  or other stuff like quit the shell (<CTRL>+<D>), scroll around without having to use the mouse (<SHIFT>+<PgUp> and <SHIFT>+<PgDn>) and much, much more.

A few sample shots of the auto-completion:

Auto-completing a program in the search path

Auto-completing a program name in the search path

Auto-completing an environment variable

Auto-completing an environment variable

One feature I didn’t get to work yet was environment variable expansion using <CTRL>+<ALT>+<E>, which works on bash in UNIX. Maybe I did something wrong there, but in case I didn’t, let’s see how fast [new bug reports] (link removed due to project migration to github, the issue described remains unfixed, but exists only on XP, XP x64 & Server 2003 systems) are attended to in the clink project.

But even without that, it’s pretty powerful already, and I love it, so much more convenient. And there I was, just looking for a way to quit the shell with <CTRL>+<D> and got so much more instead! :)

[1] Arnold Robbins, Nelson H. F. Beebe. Classic Shell Scripting, page 4. O’Reilly Media, Inc., 2005.

Jun 132013

Wine LogoSure there are ways to compile the components of my x264 benchmark on [almost any platform]. But you never get the “reference” version of it. The one originally published for Microsoft Windows and the one really usable for direct comparisons. A while back I tried to run that Windows version on Linux using [Wine], but it wouldn’t work because it needs a shell. It never occurred to me that I could maybe just copy over a real cmd.exe from an actual Windows. A colleague looked it up in the Wine AppDB, and it seems the cmd.exe only has [bronze support status] as of Wine version 1.3.35, suggesting some major problems with the shell.

Nevertheless, I just tried using my Wine 1.4.1 on CentOS 6.3 Linux, and it seems support has improved drastically. All cmd.exe shell builtins seem to work nicely. It was just a few tools that didn’t like Wines userspace Windows API, especially timethis.exe, which also had problems talking to ReactOS. I guess it wants something from the Windows NT kernel API that Wine cannot provide in its userspace reimplementation.

But: You can make cmd.exe just run one subcommand and then terminate using the following syntax:

cmd.exe /c <command to run including switches>

Just prepend the Unix time command plus the wine invocation and you’ll get a single Windows command (or batch script) run within cmd.exe on Wine, and get the runtime out of it at the end. Somewhat like this:

time wine cmd.exe /c <command to run including switches>

Easy enough, right? So does this work with the Win32 version of x264? Look for yourself:

So as you can see it does work. It runs, it detects all instruction set extensions (SSE…) just as if it was 100% native, and as you can see from the htop and Linux system monitor screens, it utilizes all four CPU cores or all eight threads / logical CPUs to be more precise. By now this runs at around 3fps+ on a Core i7 950, so I assume it’s slower than on native Windows.

Actually, the benchmark publication itself currently knows several flags for making results “not reference / not comparable”. One is the flag for custom x264 versions / compilations, one is for virtualized systems and one for systems below minium specifications. The Wine on Linux setup wouldn’t fit into any of those. Definitely not a custom version, running on a machine that satisfies my minimum system specs, leaving the VM stuff to debate. Wine is per definition a runtime environment, not an emulator, not a VM hypervisor or paravirtualizer. It just reimplements the Win32/64 API, mapping certain function calls to real Linux libraries or (where the user configures it as such) to real Microsoft or 3rd party DLLs copied over. That’s not emulation. But it’s not quite the same as running on native Windows either.

I haven’t fully decided yet, but I think I will mark those results as “green” in the [results list], extending the meaning of that flag from virtual machines to virtual machines AND Wine, otherwise it doesn’t quite seem right.


May 292013

Metro Last Light logoAnd here’s another one. There were games that did not support Windows XP at all, like [Dishonored] or [XCOM]. Some were just without all official support, but working flawlessly nonetheless, and others just carelessly compiled and linked, which could be fixed by a [rather simple function call redirection] to a proper backport DLL that Microsoft actually offers officially. So in many cases, XP and XP x64 support is not impossible at all. So why does it disappear? Simply because of the one factor that rules all and everything: Money.

Supporting an operating system may seem simple, just see whether your game runs and be done with it. It never is that simple though. So my assumption is, that official support is simply vanishing because of cost. Cost of testing, quality assurance and ongoing after sales support. That stuff is probably more expensive than I thought, and I cannot think of any other reason why this would be the case also for 4A games’ latest release: [Metro Last Light]. If the cost of testing, QA and support is greater than return of investment expectations (=sales on XP platforms), then they won’t go there.

So, Metro Last Light is kind of an in-between thing. If you look at its [Steam page], you’ll see that it doesn’t completely rule out XP, but it limits support to the regular 32-Bit NT 5.1 version. The reasons are simple. While XP x64 may seem like a better option for modern games due to its greater RAM support and 4GB userspace windows for 32-Bit applications, it is simple not as wide-spread as good old 32-Bit XP. So from a businessmans perspective, it may seem like a good idea to throw out XP x64 support and keep XP x86 in, as about 22x as many people are using XP x86 when compared to XP x64 according to the [latest Steam survey] from April 2013 (If you click on this link later, you’ll probably get different figures as the link’ll point to a more recent survey page).

So, 0.38% of all users are on XP x64. Actually, a +0.01% rise since last month, pffh. And in comparison, 8.25% of all Steam users are on XP x86, or 32-Bit. Since Metro Last Light still has that Direct3D 9 renderer that Metro 2033 also featured, they might have thought “we can still squeeze some cash out of XP, but not XP x64”.

Still, the good thing is, that business decisions not always have to affect technical details in every single aspect. Sure, there ain’t no support whatsoever, but who cares, if you can get this nonetheless, on Windows XP x64 Edition:

Metro Last Light on XP x64

So yeah, it just works. No hiccups, no nothing. It’s still sad that developer 4A games would state upon inquiry that they do not support XP x64. It would be nice if they could just say something like “We cannot officially support this, if you’re running into issues, you’re on your own, but yes, it will usually run, but don’t take it for granted.”. That would be cool. Because in this case I had to buy the game (there is no demo!) to run the test. I will admit I actually tried to evaluate the situation using cracked copies, but it seems the cracks are actually less compatible to XP x64 than the actual game. ‘Cause the cracked binaries all crashed. The legit game does not! It was risky to buy without knowing, but at least now I can share this with you!

The configuration of the successful test:

  • ASUS P6T Deluxe mainboard, X58 chipset
  • Intel Core i7 980X @ 4.0GHz
  • 48GB of DDR-III/1456 CL10 memory
  • 2 x EVGA GeForce GTX 580 3GB Classified Ultras (running Metro Last Light in Single card mode, need to update the driver for SLI)
  • 314.07 GeForce driver w.o. Metro Last Light support (the nVidia game ready driver for the game is version [320.18])

So there you go! Another desperate win for the dying breed! ;)

Apr 022013

Frog ASPIToday I would like to discuss a few useful tricks for Windows XP x64. Yeah. Again. But this time around, it’s at least partially about stuff that you can’t easily find on the Internet anymore, whether it’s the information or the actual software part that’s missing. There are several software components that are sometimes hard to use or find for XP x64. Some are even hard to set up for regular 32-bit Windows XP. The following solutions will be discussed:

  • 1.) The ASPI Layer (digital media access)
  • 2.) UDF 2.5 (for Blu-Ray and HD-DVD access)
  • 3.) exFAT (some modern cameras etc. need this file system)
  • 4.) EXT2/3/4 (for Linux and UNIX compatibility)
  • 5.) Universal SSD TRIM (keep your SSD clean, tidy and fast, like on modern systems)

So, let’s start with #1:

1.) The ASPI layer:

Edit: Note that in the meantime, the FrogASPI solution described here is no longer the only option. Another one that works even for 64-bit applications accessing the ASPI layer has been found, see further below!

1.a) FrogASPI:

One of those things that have been abandoned by Microsoft is the ASPI layer, the Advanced SCSI Programming Interface. Meant as a way for digital storage media access it was most prominently used to read from optical drives digitally (also on ATA, not just SCSI), so you didn’t need to rip audio CDs via MSCDEX on Win98 or via the crappy analog link from your drive to the sound card. ASPI can also be used to access other types of storage devices, but this is the most important part. Some older software, like my beloved Xing AudioCatalyst (an ancient CD ripper including the Fraunhofer mp3 encoder) might still need ASPI to access audio CDs.

However, Adaptec stopped shipping ASPI drivers for Microsoft Windows after Microsoft had abandoned the API and introduced its own replacement called SPTI, the SCSI PassThrough Interface. As a matter of fact, you can still install Adaptecs ASPI layer on 32-Bit Windows XP, as it includes a working 32-Bit kernel driver. So for 32-Bit XP, it’s still fine. However, there is no such driver for XP x64 (and also not for 32/64-Bit Vista/7/8). So, no ASPI at all?

For a loong time I indeed had to live without it, until I found that french guy named Millenod (That’s his nick name, I will not disclose his real name) who had written a 100% userspace ASPI layer, that would work on any newer Windows operating system on x86, no matter the bitness. This is called FrogASPI, and unfortunately, Millenods website for it is down by now. In its core, it is actually just an SPTI wrapper. Back in those days, I even wrote the guy an email, thanking him for his work. Here is a part of his reply:

“FrogAspi is effectively an SPTI wrapper. I decided to work in “user” mode, instead of kernel ones, for many reasons.. It was for me the fastest way to implement a generic ASPI layer, which is not OS specific as drivers.”

-Millenod, Developer of FrogASPI

After renaming the FrogAspi.dll to the proper name wnaspi32.dll and putting it into %WINDIR%\SysWOW64\ for 64-Bit Windows, it can be used by any ASPI-enabled application. For 32-Bit Windows, please use %WINDIR%\system32! See, what Adaptecs own aspichk.exe has to say about what we just did:

Adaptec ASPI Check

You’ll see that some files are reported as missing. You do not have to care about that though, ASPI32.SYS would’ve been the 32-Bit kernel driver, WOWPOST.EXE is a 16-Bit Windows ASPI helper tool, WINASPI.DLL the corresponding 16-Bit Windows driver. None of those are needed. Now, that FrogASPI is mapping all ASPI calls to SPTI, we can begin to actively use it even on 64-Bit Windows. See AudioCatalyst for instance, with ASPI being available:

AudioCatalyst using FrogASPI

AudioCatalyst reading the Postal Original Soundtrack CD using FrogASPI

As you can see, everything works just fine. Oh, and besides, in case you want AudioCatalysts CDDB feature back (as shown in this screenshot), just add the following lines to your AudioCatalyst.ini, sitting in the programs installation folder, this’ll switch from the now-broken CDDB to FreeDB:

  1. CDDBServer=freedb.org
  2. CDDBHTTPPath=/~freedb/cddb.cgi
  3. CDDBLocation=Random FreeDB Site
  4. CDDBTCPIPPort=8880
  5. CDDBConnectVia=TCP/IP

You can download FrogASPI right here:

Unfortunately I cannot give you a download link for Xing AudioCatalyst, as it’s commercial software even today.

1.b) StarBurn ASPI (An update from 2016-12-05):

In the meantime, user Steve Sybesma has [commented] about a different solution provided by [StarBurn Software]. With their recording software comes a native ASPI layer for both 32-bit and 64-bit systems with full compatibility for even 64-bit programs that want to use ASPI. I decided to fetch the necessary DLLs from their SDK and release them in the form of a standalone InnoSetup installer for Windows 2000 and newer. The installer will auto-detect your operating systems’ bitness and will install the 32-bit ASPI layer on 32-bit systems as well as both the 32-bit as well as the 64-bit one on 64-bit systems.

Here it is:

It has been successfully tested on Windows 2000 Pro SP4, Windows XP Pro SP3, Windows XP Pro x64 Edition SP2 as well as Windows 10 Pro x64. If you don’t want to trust my installer, that’s fine as well of course. You can just install the StarBurn software from its original source, it should give you the same level of ASPI support, just with more additional stuff being installed!

Now, to #2, the UDF 2.5 file system for HD-DVDs and Blu-Rays:

2.) UDF 2.5:

Now this one was a real bitch. It took me months to finally get it nailed! UDF 2.5 is a file system. And it’s used for basically two type of media: HD-DVD discs and Blu-Ray discs, including the most modern 3D Blu-Rays. It just bugged me like hell, that I couldn’t access the discs as I normally would in Windows Explorer or on the command line. Windows XP simply does not have the proper file system kernel driver. And while it’s relatively easy to find one for 32-Bit WinXP, it’s immeasurably harder to find the single one existing driver for Windows XP x64. I’m not even sure if it still exists on the web..

One day I came across a person whose name I forgot, but that guy had searched the web for months just like me, and he found the driver just before it went offline. So he re-uploaded it, with his single file sharing link in some post on some unknown website being the only thing that saved my bacon. Maybe one day, somebody will find the XP x64 UDF 2.5 driver here on my site after desperate searching? Who knows.

So, the only existing driver has been developed by Panasonic/Matsushita, a company that also builds optical drives. It works on Windows XP x64 as well as Server 2003 x64. And here it is, together with the Toshiba-made UDF 2.5 driver for regular 32-Bit Windows XP just for completeness:

Again, I would like to emphasize that it took me months to find that freaking 64-Bit XP driver, I’m not even sure what Panasonic/Matsushita developed it for, but here it is for your enjoyment. After installation and reboot, you can browse a HD-DVD or Blu-Ray just like any CD or DVD disc, see here:

UDF 2.5 at work in XP x64

The Blu-Ray folder structure of the german uncut version of “The Boondock Saints”, viewed using the Panasonic/Matsushita UDF 2.5 driver for XP x64

And now, #3, the exFAT file system:

3.) exFAT:

exFAT is a successor to the older FAT32 file system that uses a 64-bit address field instead, allowing for larger than 4GB files, the most notorious limitation of regular FAT. exFAT is now being used on newer cameras and other handheld devices for memory card formatting. Since the comparably heavy-weight NTFS is typically not used on such devices, exFAT is the replacement we get. Also, exFAT is the official file system for SDXC and Memory Stick XC cards, so they may very well come exFAT-preformatted! The only sour part is, that exFAT is kind of proprietary (Microsoft), like NTFS. That’s a bit crappy, but well.

I have however already written an article about exFAT on XP x64, so I will just link to it. exFAT is also very easy to get for both 32-Bit and 64-Bit flavors of Windows XP:

And now, finally, some free file systems from the Linux world:

4.) EXT2/3/4:

EXT is one of the most widely if not the most widely used file system in the entire world of Linux. It has also been ported to other systems like BSD, Solaris or ReactOS. And also Windows actually, although the implementation is not the most seamless you can dream up. But still, you can get read/write access to all currently used EXT versions under Windows, even the fast new EXT4.

The software that I found most useful for this is called [ext2fsd], short for EXT2 file system driver. It comes with the actual driver plus a helper tool for mounting and for enabling write access. It would be nice to just be able to mount EXT filesystems seamlessly from the start, without having to use that mounting tool, but we’re not that far yet it seems. However, drives can be configured to auto-mount later, so you need the tool only once for each drive. Write access has to be enabled by hand.

Currently, ext2fsd supports the following systems: NT 5.0 (Windows 2000), NT 5.1 (Windows XP 32-Bit), NT 5.2 (Windows XP x64, Server 2003), NT 6.0 (Windows Vista), NT 6.1 (Windows 7, Server 2008). Where not explicitly mentioned, it supports both 32-Bit and 64-Bit versions of said operating systems. And here is what it looks like in action:

EXT4 mounted on XP x64

A Linux-created EXT4 filesystem mounted on XP x64 for read/write using ext2fsd

The only strange thing is, that ext2fsd calls the EXT4 file system “EXT3”. It shouldn’t. But other than that, it works just fine. I haven’t tested a lot of r/w, but so far it worked fine, without any corruption or anything. The helper tool also gives you a function for flushing the write cache, and a nice statistics window for monitoring the file system usage while mounted. It also allows the user to manually unmount of course. Even mkfs is there, so you can format drives as EXT under Windows, the only useful tool missing would be fsck for checking & repairing such filesystems. But you can’t have everything I guess.

So, usage is not as seamless as it would be with native NTFS/FAT32/exFAT drivers, but it’s ok and it greatly boosts the Linux interoperability of XP x64 and other Windows systems. Also, EXT filesystems can easily be used in other UNIX-style systems like Solaris, BSD or MacOS X, so this allows a lot of different systems to exchange data easily using removable sticks or harddrives. Since it of course supports large files (>4GB) and it’s more easy to handle across a wide spectrum of operating systems, unlike exFAT it might one day become the #1 choice for data exchange, replacing FAT32 and NTFS in that field. However, today, NTFS on Linux is probably a lot more widespread than this EXT driver on Windows, so there is still a long way to go.

Of course, there is a downside to this too: All the POSIX permissions like ownership, read/write/execute bits etc. just do not work on Windows. You can neither read nor write any such meta data using ext2fsd. I have neither checked the umask that ext2fsd uses yet, nor what ownership it sets, but I would guess it’s root:root with permissions 777 or rwxrwxrwx. But whatever it is, you will most likely need to take care of that when mounting on your UNIX-style system.

I hope this was helpful, if you have any comments about these helpful add-ons, just post away!

5.) Universal SSD TRIM:

This is a 2014-11-03 edit of the original article.

Usually, I would tell users who would like to use an SSD on Windows XP or XP x64 to re-align their partitions with a [gparted ISO] after installation to get proper alignment, and more importantly to use either Intel, Corsair or Samsung SSDs so that they can TRIM their SSDs properly using their respective manufacturers’ XP-compatible software tools, maintaining full write speed over the entire life time of the SSD drive.

It seems that I can now finally confirm, that the latter criterion no longer needs to be met. Basically, I found an easy way to TRIM any SSD on Windows XP or Windows XP x64, as long as the following logical criteria are met:

  • The SSD itself needs to support TRIM. Well, that was obvious.
  • The controller driver needs to support TRIM. Now I can’t speak for just any driver, but I have tested this on Intels ICH10-R and an ancient ICH7-M, both worked just fine with AHCI drivers installed. Usually, AHCI is what you want. Avoid IDE/legacy PATA modes. You may need to configure this properly in your systems BIOS and you may need to retrofit AHCI drivers in XP if your system was installed using IDE mode to avoid bluescreens on boot. If you need help doing that, please ask away in the comments.

So, what’s the solution? It’s called the ADATA SSD Toolbox! Basically, ADATA developers seem to have forgot or were actually never instructed to install an artificial vendor obstruction inside their software. So what they did was to write an SSD toolbox that just complies to the SATA standard, implementing its TRIM command for any compatible SSD, no-holds-barred! I have tested this on an Intel 320 SSD, as well as on a Crucial m500 SSD now, despite no official XP support being present:

So there you go! Now pretty much any TRIM capable SSD can be made to accept TRIM on Windows XP and XP x64! Together with gparted for partition alignment, we have a full-featured SSD solution for XP, leaving nothing to be desired!

Download the ADATA SSD Toolbox here: [ADATA SSD Toolbox v2.0.1].

Dec 202012

TimestampRecently, I got a request to determine and publish a list of latest created and/or modified files/folders on a filesystem and get that published in HTML form. Now since my scripting as of late has been cross-platform (Linux & Windows), I was aiming at achieving that again. So I started scripting Perl 5 on Linux, naturally using Unix style epoch time stamps, which can be done with a Perl internal set of functions or with the module Time::localtime.

So the idea is, you give the script a base folder, and it will scan all folders (or files if you wish) beneath, and output a list of the most recently added or changed ones. I decided to limit the output to what I may see fit, like the latest 20 or latest 100 added and modified folders. Very helpful: A folders timestamp gets updated when you add files to it or even when you change files in it.

So, here is some code for that (note that this might not exactly be executable as-is, because it may be incomplete, these are just extracted parts of the full code):

expand/collapse source code
  1. #!/usr/bin/perl
  3. use strict;
  4. use File::stat;
  6. my $basepath  = "X:/basefolder";  # Base folder.
  7. my @subfolders  = <$basepath/*>;  # Read subfolders to check for modifications within.
  9. my $printcount  = 20;             # Amount of folders to display in final output.
  11. my $outfile  = "C:/outputfile";   # File to write list to.
  13. my $subfolder  = "";       # Variable to use for iteration through subfolders.
  14. my $folder  = "";          # Single folder name.
  15. my $mtime  = "";           # UNIX-style epoch-related time stamp of a single folder.
  16. my %mtimestamps  =   ;     # Hash with folder names as keys and time stamps as values.
  17. my $i  = 0;                # Arbitrary counter.
  18. my $key  = "";             # Arbitrary key for iteration through sorted timestamp hash 
  20. open(OUTFILE, "<:encoding(UTF-8)", $outfile); # Open output file.
  22. # Go through subfolders and check timestamps of folders within them (level 2):
  23. foreach $folder (@subfolders) {         # Iterate through subfolders.
  24.   if (-d $folder) {                     # Check if element is truly a directory.
  25.     $mtime = stat($folder)->mtime;      # Read modification time stamp of subfolder.
  26.     $mtimestamps{$folder} = $mtime;     # Add timestamp as a value to the hash for the
  27.   }                                     # current folder name as key.
  28. }
  30. # Iterate through sorted hash, highest values (=newest folders) first:
  31. foreach $key (reverse sort {$mtimestamps{$a} <=> $mtimestamps{$b}} keys %mtimestamps) {
  32.   printf(OUTFILE $key . "\n");      # Print current folder name to file.
  33.   $i++;                             # Increment counter.
  34.   if ($i >= $printcount) { last; }  # If specified maximum of folders has been listed,
  35. }                                   # break and quit.
  37. close(OUTFILE);  # Close output file, we're all done.

So, this is it. It does definitely work on both Windows NT 5.2 (using NTFS) as well as Linux using any Posix-compliant file system. You can use this to publish the latest changed files/directories on a server, or modify it to show you the latest accessed files even, just replace $mtime = stat($folder)->mtime; with something like $atime = stat($folder)->atime; to get access times instead of modification times. Of course, your file system needs to be mounted with enabled access time stamps, which should be the default. Both Windows and Linux however allow for the disabling of that feature to speed up the file system (noatime mount option in *nix or HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate registry key on Windows). Also, you can do the same with ctime to get creation time stamps.

So that is one way to monitor and visualize changes in your directory and file structure. I use this personally to write the data into an HTML file which i can then check on my web server easily. That way I can even monitor changes to certain directories and files online, which can be extremely useful. Another nice little something in Perl from a beginner. ;)

Nov 112012

Dishonored LogoDeveloped by Arkane Studios and published by Bethesda, “Dishonored” is supposed to be a really cool new stealth / assassination style game with a deep story and a tense atmosphere amplified by really nice artwork. If you want to get an impression of this game, please watch the [debut trailer] and the [gameplay trailer]. Meanwhile, the cool song from the gameplay trailer has gotten so much attention, that it has been released as an [mp3] including its own [remix contest].

Basically, the game is set in a steampunkish world with swords, muzzle-loader pistols but also some Tesla high tech stuff. And in the dark corners of the plague-ridden, rotting city there is even magic. So, everything seems cool, and even though this is a console port, it seems to be one well-done at least. Yeah, the textures aren’t fly, even if you choose to fully [tune the engine], but the watercolor style is still more than nice enough to look at in my opinion.

Now, the one question that is interesting to me of course: Why does Bethesda insist this game needs Windows Vista/7 as a minimum? The renderer is DirectX 9.0c, sitting in an Unreal Engine 3, like many modern games. So why? Maybe some special function calls like in XCOM that need to be hacked to work? Maybe some modern API calls that do not exist on XP? Or something else? No. None of those. It was simply laziness in the software testing process I assume. Or maybe Bethesda just didn’t want to care to have to offer official XP support. Whatever the reasons are, it works just fine anyway, see here (click to enlarge):

Dishonored running on Windows XP Professional x64 Edition

So there we go! Another Vista/7 only game, that works just fine on Windows XP. Make sure though that you’re running your XP with SP3, and your XP x64 with SP2. You should really run the latest patchlevels anyway. So if you have everything up to date, you should be just fine. No modifications or hacks required for this one!

Jun 292012

I have once written an FTP logfile analyzer in Perl. That analyzer collects data like transferred megabytes and connections, groups it per user and publishes it on the web in somewhat simple, neat HTML. Sounds nice? Well, it is! There is only one problem that bugged me for ages now. The logfile is growing and growing, and since my script is optimized for constant non-growing memory consumption, it’s somewhat slow. Still faster AND far smaller as well as far more scalable than [sustat] ever was, but it’s running on a Pentium PRO, i mean…

Actually, I am updating the web statistics every hour and with my ~60MB logfile processing takes almost 10 minutes when doing the full logfile. At some point it will take 30 minutes, then the full hour etc., unacceptable.

So, my idea to solve it: Do it incrementally! Just process the whole file once, generate an intermediate data file and a byte address index of every line in the log file (necessary for efficient seeking), and the next time continue where we left off. And here is where the problem lies. I was trying to generate a binary index file consisting of 4 byte large packed data elements. Each element can hold a 32-bit unsigned integer, and that number is the corresponding line number in the log file, the one you would want to seek to, when updating the stats in an incremental fashion.

So I can just say, “I want to seek to line number 1.000.000”. Then the code will seek to the (1.000.000 × 4)th byte in the index file, and get the 1.000.000th data element. In that element, – after calling unpack() on it – lies the byte address for line number 1.000.000 in the log file. Now you can seek there and process any newly added information, after which the index file would be updated with the added new line indexes.

I have been developing this on both Linux and Windows, and on Windows the index file consisted partly of valid packed line numbers, and partly of bogus. Mixed. Why? I always failed to understand the reason for this. Now here it comes: I have opened the index file as a TEXT file, which works just fine on all Linux systems. But on Windows, bytes appearing as certain characters like "\n" might be magically transformed when written or read. For instance, if you write the line break byte "\n", Perl on Windows automagically transforms it into a Windows line break, which is 2 bytes large, "\r\n". Now that makes one of those index data elements 5 bytes large which breaks the alignment! At some point, 4 such shifts will have occurred, triggered by special bytes, pushing the alignment back into the correct one, which explains why later in the file there were valid numbers appearing again, just in the wrong position (shifted by 4×4 bytes).

It took me MONTHS to locate and understand that problem, hey I’m no programmer! Thank god there is a function called binmode() that you can call on a file handler/descriptor with an additional argument to force Perl into reading and writing only raw bytes from/to a file. Like:

open(FH, "<", $filename);
binmode(FH, ":raw");

Now finally, my index file got read and written correctly. Note that stuff like this can also happen on non-UTF-8 Linux systems running Perl 5.6.x, but most prominently this happens on Win32, even with Perl 5.14.x+. When Perl uses Win32 APIs for text mode file access, it’s trying to be a little bit too smart. Which is ok most of the time, unless you run into something like this with no idea what Perl is actually fucking up in the background.

So kids, remember: Whenever you’re handling a BINARY file, always call binmode() on your file handler right after opening your file! If you don’t do this, your code will puke on your screen if you try to port it from Linux to Windows or if you just don’t know about this stuff, like me. It most likely WILL corrupt whatever binary data you’re handling if you don’t do this!

If you want to know how to build a binary index for fast low-memory seeking in huge text files (millions of lines) in Perl, please [check this out]. Just don’t forget to add binmode() when you take that code! Preferrably with the ":raw" argument!