May 092015
 

NetworkJust so everybody knows, UPC – my Internet provider – will be doing routine maintenance work on Wednesday, the 13th of May in 2015, because they have to do… yeah, “stuff”. That’s how specific they are, technically. In any case, the time window for this maintenance will be 01:00am – 06:00am UTC+1 DST, so if you cannot reach any XIN.at service during that time frame, you’ll know why. Naturally, that’ll affect this weblog just as well as all other services I’m offering, like eMail, FTP+ES, IRC, etc. Please note that given my past experience it may very well happen that this maintenance window will be… uhm… let’s say “accidentally extended” by UPC due to reasons nobody will talk about. So if you still can’t reach any of my services at 07:00am UTC+1 DST, then yeah… chances are it’s not me to blame, at least not this time around.

So let’s just hope it’ll just be only a few minutes of blackness this time, eh?

Apr 182015
 

The Hoff - True Survivor (Kung Fury)Think about David Hasselhoff what you will, but one thing’s fore sure: He’s definitely an icon of the 80s of the past millennium, there’s no denying that. And now he’s really pulled one off, as he did the official intro song for the crowd funded movie “Kung Fury“, that set sail to collect all the 80s charm and weave it into a crazy action comedy of yet unseen proportions. Let’s give you a few buzz words right there: Cops, Kung Fu, hacking time itself, Kung Führer (yes, Hitler), ancient female viking warriors with guns, dinosaurs, evil robots. That’s enough for now I’m guessing.

The thing here is, that it’s not just the Hoff making some PR for Kung Fury – though it’ll most definitely work quite well in that regard – the song is actually catchy and surprisingly good, capturing the very soul of the 80s perfectly!

"True Survivor" single cover

“True Survivor (from Kung Fury)” single cover (click to enlarge)

Now, before I show you the music video, you should probably take a look at the official Kung Fury web site or maybe [here] first, to read about what Kung Fury is supposed to be, and to watch the official trailer. Whet that’s done, take a look at the madness The Hoff has created here, together with a lot of scenes from the actual movie:


As of 2015-04-18, 11:00 UTC+1 DST, this video has had about 2.1 million views on YouTube, having been online for roughly 2 days. Never doubt The Hoff!

Now if you want fun, and over-the-top 80s cheesiness, you’ll love this piece of trash (I just went ahead and bought the song, and I rarely do that), and you should probably go watch Kung Fury in any way you can when it’s being released next month! It will even come as a free download, heh, talk about awesome!

Awe..

..some.

Mar 062015
 

F6 logoThis is a problem I’ve wanted to solve for quite some time now: How to install older Windows on systems with an AHCI SATA or maybe SCSI or SAS controller, when there is no floppy drive and Windows setup won’t recognize any hard drive? Usually, you’ll have to press F6 during the early setup sequence, and then provide the drivers with either a real floppy, or a USB floppy drive that’s on the hardware compatibility list (not all USB floppies work). While adding non-supported USB floppies to any installation disc is possible when using [nLite] (see farther below), sometimes you may not want to build your own image or maybe you don’t have any floppies available, USB included.

What I always wanted to do was to find a way to just use some USB pendrive for this, like it’s possible with newer Windows systems. And yesterday, I finally found the solution. My original attempts to modify the Windows CD itself were all unsuccessful, and today I believe that it is hardly possible to just talk setup into accepting USB pendrives as floppies, unless your systems BIOS can actively mask the pendrive as such. The solution is much more weird than any such modification actually.

But this time around, I won’t dive into the technical details up front, but simply give you the solution first. If you’re interested in how this really works, just read the lower part 3.)!

1.) How to use any USB drive as a F6 floppy medium for storage driver loading during legacy Windows setup

1.a.) Let’s go step by step. You’ll need to be on Microsoft Windows for this. First we need nu2’s bfi.exe[1] plus a few additional files. I have packaged them all together for you in this file: [buildf6]. Please unpack the archive, and enter the newly created folder buildf6\. In that folder you’ll see a subfolder f6\. Put the storage drivers you wish to use to support your target system in that folder. Please keep in mind, that it has to be F6 capable drivers. Usually these are small and consist of a txtsetup.oem file plus some *.inf and *.sys, maybe *.dll files. When you’ve placed all necessary drivers in there, run the floppy image builder script: buildf6.bat. This will create a 1.44MiB image file called f6.img.

In case you put too many drivers in the f6\ folder and it’s growing too large to build the image file, you can also alter the script to have it build a larger 2.88MiB floppy image instead of the 1.44MiB one. Just open buildf6.bat with your favorite text editor, it’ll show this:

@echo off
cls
.\bfi\bfi.exe -v -t=144 -b=boot\boot_sector.bin -f=.\f6.img -o=BCDL.BIN -o=BCDL-IDE.SYS -o=VIDE-CDD.SYS .\boot\chainload_cd .\f6
pause

Simply change the -t option to -t=288 to get a larger floppy image which can hold more drivers.

1.b.) Next thing we need is [Syslinux] to prepare ourselves a bootable and extremely minimalistic Linux system. Actually, it’s not even Linux at all, it’s just a bootloader. The latest version 6.xx can be obtained directly from [here] at kernel.org. Unpack it, launch cmd.exe and enter the following subdirectory of Syslinux on the shell: bios\win32\ or maybe bios\win64\ in case you’re on a 64-bit Windows machine. Plug your desired USB key into the machine. I’ll assume drive letter F: here. Run the following command to prepare your USB drive, and beware – all data stored on it will be lost:

syslinux.exe --mbr --active --install F:

Please copy your previously created image buildf6\f6.img over to F:\, and also copy Syslinux’ bios\memdisk\memdisk file directly to F:\. Don’t create any subfolders, all files belong in the root directory. Now, create a new text file F:\syslinux.cfg, open it with an editor and enter the following:

default 1

label 1
kernel memdisk
append initrd=f6.img floppy

Save the file. The contents of the USB drive should now look like this:

Our F6 USB sticks content

Our bootable F6 USB sticks’ content

2.) Run it

Now, you need to put the Windows 2000/XP/2003 CD into the target machines optical drive, plug the USB drive in, and set the systems BIOS to boot from USB. If you’re on a UEFI-based machine, make sure to boot in legacy/BIOS mode. The machine will boot from the USB drive, and then go on and continue booting from the CD, as usual. When the prompt for pressing F6 comes up, please do so. When Windows setup asks you to provide a floppy for the intended drivers, confirm that you’ve already put it in, and setup will magically pick it up. See the following screenshots:

XP x64 accessing the virtual F6 floppy

XP x64 accessing the virtual F6 floppy

XP x64 will now load the Areca SAS controller driver - no real floppies involved

XP x64 will now load the Areca SAS controller driver – no real floppies involved

As said, this even works with Windows 2000. Possibly even with Windows NT 4.0 and older systems capable of booting from a CD, but I haven’t tested systems that old. I tried it with Windows 2000 Professional though, and it just works:

Windows 2000 accessing the virtual floppy drive

Windows 2000 accessing the virtual floppy drive (German)

Windows 2000 will load a Mylex AcceleRAID 352 driver for installation

Windows 2000 will load a Mylex AcceleRAID 352 driver for installation (German)

Just keep booting from the USB stick until setup is complete, so that the virtual floppy drive is present throughout all stages of the installation process. As soon as Windows is completely installed, shut down the machine, remove USB drive and CD, and you’re good to go!

Now, for the addendum…

3.) What’s really going on here

I mentioned that I believe that Windows can’t be tricked into accepting any arbitrary USB drive as a F6 floppy. If possible at all, it’d require a low-level binary hack, as Windows setup is calling the hardware interrupt 13hex to access the floppy drive, which is about as low-level as it gets. And that’s hardcoded… somewhere. For the USB floppy part, well, you can widen the field of supported drives (see even [further below]), but according to my attempts you can’t tell Windows to treat a USB device of class 08, subclass 06 and interface 80 – like a USB pendrive – just like a class 08, subclass 04 and interface 00 device – like a real USB floppy. For more information please refer to the [USB mass storage specifications].

So what we’re doing instead is something much more freakish. The exact process that this solution goes through looks like this:

  1. The system BIOS executes the boot code within the boot sector of the USB drive.
  2. The boot code boots Syslinux’ ldlinux.sys loader.
  3. Syslinux continues to higher stages by loading the Syslinux core module, ldlinux.c32.
  4. Syslinux now pulls our floppy image f6.img into a ramdisk, and executes it as if it was an operating system kernel as specified in syslinux.cfg, effectively chainloading it (It also redirects INT13hex to the virtual floppy, so when any code calls the interrupt, it’ll get to the Ramdisk instead of any real floppy!).
  5. This works, because the floppy image itself has a boot sector that the x86 host processor can execute directly as it’s x86 machine code!
  6. Instead of a real operating system, the virtual floppies’ boot sector now executes BCDL.BIN, a pure machine code executable, which was embedded into the image by buildf6.bat from buildf6\boot\chainload_cd\ together with its CD drivers BCDL-IDE.SYS and VIDE-CDD.SYS which support real IDE as well as SATA optical drives.
  7. BCDL.BIN loads its drivers and chainloads the CD by employing standard El Torito optical drive booting.
  8. The Ramdisk stays where it is, posing as a real A: INT13hex floppy drive until reboot.
  9. Windows setup boots from CD.
  10. Besides BCDL, we also have our required storage drivers on that floppy image. Windows 2000/XP/2003 may now load them by locating and reading txtsetup.oem just as it always did.

It may sound complicated, but once you know what to do and how, it’s a pretty easy and fast process. With nothing but your USB key, the drivers and the installation CD you can set up ancient Windows pretty much anywhere, as long as you still have an optical drive at least. I’d imagine this to be specifically useful for notebooks.

4.) The USB floppy thing – How to add non-supported drives

This is just for completeness’ sake. Should you wish to just keep using a USB floppy, but unfortunately have a newer unsupported one, you can add it to your setup CDs HCL. This requires copying the entire CD off to another Windows machine, modifying a small file and rebuilding a new ISO using [nLite] though.

You can find the file on your installation CD in \I386\ for 32-bit Windows (2000/XP/2003) and in \AMD64\ for 64-bit Windows (XP x64/2003 x64), it’s called TXTSETUP.SIF. Open it with your favorite text editor and locate the following part in the Section [HardwareIdsDatabase]:

txtsetup.sif, USB part of the [HardwareIdsDatabase] section

TXTSETUP.SIF, USB part of the [HardwareIdsDatabase] section

Depending on your operating system, this may be found on totally different line numbers. Now you need to determine the USB vendor ID and device ID of your USB floppy drive. I usually do this on Linux or BSD UNIX by running lsusb (FreeBSD needs the usbutils package for that, # pkg install usbutils). Here is an example with some USB device, not a floppy, but just so you know what to look for, it’s the two 4-letter hexadecimal codes separated by a colon, the first is the vendor ID, the second the device ID:

lsusb showing USB devices on FreeBSD

lsusb showing USB devices on FreeBSD (click to enlarge)

You can also get it on Windows though, just go to “Start\Settings\Control Panel”, and then “System\Hardware\Device Manager”. Locate your USB drive by exploding the USB part of the device tree on the very bottom, right click the correct device and choose “Properties”, then “Details”. Et voilà:

Getting a USB devices vendor ID (VID) and device ID (PID) on Windows

Getting a USB devices vendor ID (VID_****) and device ID (PID_****) on Windows

Patching it in is a simple process of adding an according line to TXTSETUP.SIF, like this:

Adding an unsupported USB floppy device in TXTSETUP.SIF (shown with an invalid non-floppy device here, but it works like that with any unsupported real USB floppy drive).

Adding an unsupported USB floppy device in TXTSETUP.SIF (shown with an invalid non-floppy device here, but it works exactly like that with any unsupported real USB floppy drive).

As said, you’ll have to build yourself a new ISO from that using nLite and burn it to CD. But with that, you can now use any previously unsupported USB floppy drive for F6 driver installations.

5.) Profit

So that’s how it can be achieved after all, and I always thought it to be impossible. It’s quite the obscure method, and its gain in usefulness over nLite driver embedding is maybe debatable, but I believe this can be faster and more flexible. That may be especially true if you have to deal with lots of different machines with different SATA AHCI driver requirements or other storage controllers. Building new ISOs using nLite over and over again can be time consuming after all.

[1] Lagerweij, B. BFI is © 2000-2015 Nu2 productions. All rights reserved.

Feb 272015
 

Ren'Py on FreeBSD logo[1] While not exactly “games” in the common sense of the word, visual novels are often classified as such. I’d call them a crossover between novels/manga and games, much like interactive books with storyline branching options for kids (in case you know those). Most of these visual novels – all which I know at least – are based on the [Ren’Py] engine, which itself sits on top of the Python scripting language and I believe the [pygame] library in all cases. This makes the stuff great for portability, since Python exists on a wide variety of operating systems.  While traditionally a bit more at home on Linux/UNIX, it has long since been ported to Microsoft Windows too.

Now to the interesting part: While the portability has been recognized by the Ren’Py team and also the visual novel developers, what you’ll usually get is a Windows, a MacOS X and a Linux version. Then I thought: Why? If it’s just Python, I should be able to just copy it anywhere I like, as long as I have Python itself plus the necessary modules and the Ren’Py engine installed there. My target once more: FreeBSD 10.1 UNIX.

Ren'Py launcher

The Ren’Py SDK including the engine runtime can easily be installed on FreeBSD too. The complications don’t end there however.

It turned out to be harder than expected though. Ren’Py does have dependencies on certain system libraries (for video decoding, image processing, etc.) as well as on the Python and modules versions themselves. That’s why each and every Ren’Py novel is (partly unfortunately) distributed with the whole of Python, the Ren’Py engine itself and different system libraries pre-compiled and statically linked together. Naturally, those libraries are written in C/C++, so there you have your operating system dependency.

I tried this with the novel “Sunrider – Mask of Arcadius” first. Just directly copied out of my Steam folder on Windows, straight to CentOS 6 Linux (where it worked out of the box) and then to FreeBSD 10.1 UNIX. Now the game may use small binary executables to launch, like an EXE file on Windows, but all those do really is launch the packaged Python properly. For Linux, it’s launched just by a shell script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
#!/bin/sh
 
SCRIPT="$0"
 
# Resolve the chain of symlinks leading to this script.
while [ -L "$SCRIPT" ] ; do
    LINK=$(readlink "$SCRIPT")
 
    case "$LINK" in
        /*)
            SCRIPT="$LINK"
            ;;
        *)
            SCRIPT="$(dirname "$SCRIPT")/$LINK"
            ;;
    esac
done
 
# The directory containing this shell script - an absolute path.
ROOT=$(dirname "$SCRIPT")
ROOT=$(cd "$ROOT"; pwd)
 
# The name of this shell script without the .sh on the end.
BASEFILE=$(basename "$SCRIPT" .sh)
 
if [ -z "$RENPY_PLATFORM" ] ; then
    case "$(uname -s)-$(uname -m)" in
        Darwin-*)
            RENPY_PLATFORM="darwin-x86_64"
            ROOT1="$ROOT/../Resources/autorun"
            ROOT2="$ROOT/../../.."
                        ;;
        *-x86_64|amd64)
            RENPY_PLATFORM="linux-x86_64"
            ROOT1="$ROOT"
            ROOT2="$ROOT"
                        ;;
        *-i*86)
            RENPY_PLATFORM="linux-i686"
            ROOT1="$ROOT"
            ROOT2="$ROOT"
            ;;
        *)
            echo "Ren'Py could not detect that platform it's running on. Please set"
            echo "the RENPY_PLATFORM environment variable to one of \"linux-i686\" or"
            echo "\"linux-x86_64\", or \"darwin-x86_64\" and run this command again."
            exit 1
            ;;
    esac
fi
 
 
for BASE in "$ROOT" "$ROOT1" "$ROOT2"; do
    LIB="$BASE/lib/$RENPY_PLATFORM"
                if test -d "$LIB"; then
                    break
                fi
done
 
exec $RENPY_GDB "$LIB/$BASEFILE" $RENPY_PYARGS -EO "$BASE/$BASEFILE.py" "$@"
#!/bin/sh

SCRIPT="$0"

# Resolve the chain of symlinks leading to this script.
while [ -L "$SCRIPT" ] ; do
    LINK=$(readlink "$SCRIPT")

    case "$LINK" in
        /*)
            SCRIPT="$LINK"
            ;;
        *)
            SCRIPT="$(dirname "$SCRIPT")/$LINK"
            ;;
    esac
done

# The directory containing this shell script - an absolute path.
ROOT=$(dirname "$SCRIPT")
ROOT=$(cd "$ROOT"; pwd)

# The name of this shell script without the .sh on the end.
BASEFILE=$(basename "$SCRIPT" .sh)

if [ -z "$RENPY_PLATFORM" ] ; then
    case "$(uname -s)-$(uname -m)" in
        Darwin-*)
            RENPY_PLATFORM="darwin-x86_64"
            ROOT1="$ROOT/../Resources/autorun"
            ROOT2="$ROOT/../../.."
                        ;;
        *-x86_64|amd64)
            RENPY_PLATFORM="linux-x86_64"
            ROOT1="$ROOT"
            ROOT2="$ROOT"
                        ;;
        *-i*86)
            RENPY_PLATFORM="linux-i686"
            ROOT1="$ROOT"
            ROOT2="$ROOT"
            ;;
        *)
            echo "Ren'Py could not detect that platform it's running on. Please set"
            echo "the RENPY_PLATFORM environment variable to one of \"linux-i686\" or"
            echo "\"linux-x86_64\", or \"darwin-x86_64\" and run this command again."
            exit 1
            ;;
    esac
fi


for BASE in "$ROOT" "$ROOT1" "$ROOT2"; do
    LIB="$BASE/lib/$RENPY_PLATFORM"
                if test -d "$LIB"; then
                    break
                fi
done

exec $RENPY_GDB "$LIB/$BASEFILE" $RENPY_PYARGS -EO "$BASE/$BASEFILE.py" "$@"

I decided to bypass that entirely. After installing the Ren’Py engine on FreeBSD system-wide by running # pkg install renpy, I entered the games directory and ran $ python ./SunriderMaskofArcadius.py. That works, but only out of sheer luck. I tried others, like Analogue: A Hate Story or Katawa Shoujo (warning: contains erotic content) with no such luck. The space-opera Sunrider however, no problem:

The Sunrider assault carrier takes off - on FreeBSD 10.1

The Sunrider assault carrier takes off – on FreeBSD 10.1 just as well as anywhere else

So: Why? The answer is: Because Python. Sunrider is one of the rarer gems that are written for a Ren’Py version new enough (6.18.1.670) to ship with Python 2.7. And that’s exactly the version that comes with FreeBSD 10.x, and also the version its renpy-6.18.3 package is built for. Most visual novels however were developed using older engines that come with a pre-built Python 2.6 or 2.5 for Windows/OSX/Linux.

Now I tried compiling Python 2.5 from source, which was successful. It doesn’t end there however, as you then also need to use Python 2.5 for Ren’Pys build system to compile its own platform-specific code too and to get the engine “linked” with Python 2.5. I used an older engine for that to ensure compatibility, but the fun still ends there very quickly, as I ran into severe dependency issues, some of which [this Debian bug report] also describes. One would need to build themselves an old libpng, old libav and god knows what else to be able to get the whole thing up and running. And just running old novels with Python 2.7? Ha! No:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Traceback (most recent call last):
  File "./Katawa Shoujo.py", line 132, in 
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/bootstrap.py", line 242, in bootstrap
    renpy.import_all()
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/__init__.py", line 61, in import_all
    import renpy.display.scale # Must be before module.
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/display/scale.py", line 31, in 
    import _renpy_font
ImportError: No module named _renpy_font
Exception TypeError: "poll() got an unexpected keyword argument '_deadstate'" in > ignored
Traceback (most recent call last):
  File "./Katawa Shoujo.py", line 132, in 
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/bootstrap.py", line 170, in bootstrap
    renpy.display.presplash.show(options.presplash)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/display/presplash.py", line 101, in show
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "./Katawa Shoujo.py", line 132, in 
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/bootstrap.py", line 242, in bootstrap
    renpy.import_all()
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/__init__.py", line 61, in import_all
    import renpy.display.scale # Must be before module.
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/display/scale.py", line 31, in 
    import _renpy_font
ImportError: No module named _renpy_font
Exception TypeError: "poll() got an unexpected keyword argument '_deadstate'" in > ignored
Traceback (most recent call last):
  File "./Katawa Shoujo.py", line 132, in 
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/bootstrap.py", line 170, in bootstrap
    renpy.display.presplash.show(options.presplash)
  File "/usr/home/<myuser>/games/Katawa Shoujo-linux-x86/renpy/display/presplash.py", line 101, in show
    sys.stdout.flush()
IOError: [Errno 32] Broken pipe

Ren’Py for Python 2.5 cannot run on 2.7, some module import issues. And 2.7 trying to run a novel based on Python 2.6 gives you this:

1
2
3
4
5
6
7
8
9
10
11
Traceback (most recent call last):
  File "./Analogue.py", line 144, in 
    main()
  File "./Analogue.py", line 141, in main
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Analogue A Hate Story/renpy/bootstrap.py", line 279, in bootstrap
    renpy.import_all()
  File "/usr/home/<myuser>/games/Analogue A Hate Story/renpy/__init__.py", line 139, in import_all
    import renpy.display.render # Most display stuff depends on this. @UnresolvedImport
  File "render.pyx", line 324, in init renpy.display.render (gen/renpy.display.render.c:22252)
AttributeError: 'module' object has no attribute 'Sentinel'
Traceback (most recent call last):
  File "./Analogue.py", line 144, in 
    main()
  File "./Analogue.py", line 141, in main
    renpy.bootstrap.bootstrap(renpy_base)
  File "/usr/home/<myuser>/games/Analogue A Hate Story/renpy/bootstrap.py", line 279, in bootstrap
    renpy.import_all()
  File "/usr/home/<myuser>/games/Analogue A Hate Story/renpy/__init__.py", line 139, in import_all
    import renpy.display.render # Most display stuff depends on this. @UnresolvedImport
  File "render.pyx", line 324, in init renpy.display.render (gen/renpy.display.render.c:22252)
AttributeError: 'module' object has no attribute 'Sentinel'

I stopped there.

I’m not saying it can’t be done, but compiling myself all those dependencies? I didn’t want to go that far. Who knows what kind of ever-growing dependency tree horror awaits there.

So yeah, Ren’Py visual novels can be played on FreeBSD 10.x, but you’ll need to make absolutely sure that it’s a game coming with a new enough version of Python. If you only have FreeBSD systems, there are still several ways to verify this by running the Windows binary using wine, the Linux binary using FreeBSDs linuxulator, or by checking the strings on the Linux binary. While sitting in the games main directory, try these (you may need to adjust the paths accordingly):

  • $ wine ./lib/windows-i686/python.exe --version | grep 2\.7 should give back some insignificant error messages and a string like “Pyton 2.7.3″. Requires wine obviously. Don’t get anything? Change the grep part to look for 2\.6 or 2\.5 instead.
  • $ LD_LIBRARY_PATH="./lib/linux-i686/" ./lib/linux-i686/python --version should give back a string like “Python 2.7.3″. Requires linuxulator.
  • strings ./lib/linux-i686/python | grep 2\.7 should give back a Python library the binary was linked against, like libpython2.7.so.1.0. No results? Change the grep part to look for 2\.6 or 2\.5 instead.

If the Python shipped with the novel in question is too old, I’d say just leave it or run it on Windows/OSX/Linux instead. Or you can play hero and try to build yourself all the required libraries, then compile and link an older Python, pygame module and Ren’Py engine against them. Would love to see how hard it’d really be. ;)

[1] Original drawing is © by ASK (Pixiv profile).【PFFK】琉璃, Pixiv Fantasia: Fallen Kings series. Altered and used with express permission.

Feb 202015
 

Hard disk logoSince we have had hard drives, we’ve been trying to make them larger and larger, just like with any other data storage medium. I believe that for mechanical disks, no single parameter is more significant than sheer size. Recently Seagate blessed us with its first SMR (“Shingled magnetic recording”) hard drive, giving us a technology bringing more capacity again, but with a few drawbacks. Drawbacks significant enough to make me want to talk about SMR and its competing technologies today, and also about why SMR is here already, and the others are not.

1.) Steamroller tactics

As I said, we’ve always been trying to increase the storage space of disks. There are usually two ways of achieving this, of which some are particularly challenging. The easy method? Just cramp in more platters. And with the platters, more read/write heads. This clearly has its limits of course. Traditionally, the maximum number of platters that could be operated safely next to each other in a regular half-height 3.5″ HDD was 5. Keep in mind that for a 7200rpm drive, we have 120 platter rotations within a single second! Smaller 2.5″ platters as used in certain enterprise disks or the WD Raptor drive can spin even faster, at 10.000rpm or 15.000rpm which means 166 and 250 rotations per second respectively.

Rotational speeds so high mean there’s going to be a lot of air turbulence in there, and air is essential to keep the heads floating at very low altitudes (freaking nanometers for Christs sake!) over the platters. Too much disturbance and you get instabilities and potentially fatal head crashes.

A Hitachi 2TB 5-platter disk

A Hitachi DeskStar 7K2000 2TB, 5-platter, 10-head disk (click to enlarge)

In recent days, Hitachi Global Storage dared to replace air-based designs with low-density helium-filled drives, thus enabling them to pack an unbelievable amount of 7 platters into a normal 3.5″ drive, accessed by 14 heads, all made possible by lower gas turbulence and resistance. This also enabled them to use lower-powered, lighter motors to spin the platters. Seagate stroke back, by using shingled magnetic recording for inexpensive disks and six platters for enterprise disks – despite the conventional air filling. The reason why Seagate didn’t introduce SMR to the enterprise markets yet are said drawbacks. But there is no way we can pack more and more platters and heads into disks of the same volume – there is only so much space, and 7 platters is insane already.

2.) The easiest way out may not always be a smooth ride…

So what is SMR, and what made it appear on the markets well before its competing technologies, like HAMR (heat-assisted magnetic recording), MAMR (microwave-assisted magnetic recording) and BPMR (bit-patterned media recording)?

Basically, price made SMR happen.

But let’s just dive into the tech first.

As you may have heard, the term “shingled” is derived from roof shingles and the way they actually overlap when being put on a roof. That’s for gabled roofs of course, not flat ones. ;) Regular hard drives have sectors sitting in line on what we call a track. Then, there is a slight gap, and next to the track there is another track and so on. Like rings sitting on the disc. The sectors tend to be of the same physical size (in square area covered), which is why data can be read the fastest on the outermost parts of the platter – more sectors passing by the head for an equal amount of angular movement.

This is considered to waste space though. Modern read heads can read much more narrow tracks than what write heads are able to store safely on the disks magnetic film. Individual bits are stored using roughly [20-30 grains of magnetized material] on the disks film right now. So, ~20 grains with the same magnetic orientation, distributed amongst a few nanometers. It seems read heads can cope with less though, so the industry (or academia?!) came up with this:

SMR structual comparison

A structural comparison: a normal disk to the left, and shingles to the right. As one can see, the shingled magnetic recording disk allows for packing more data into the same space, despite individual tracks being wider when written.

So, yeah. No gap anymore, no more wasted space, right? The gap was actually there because the edges of the tracks are not too well defined by classic write heads, so reading safely could be tricky when you pack them too close together. The ones used on SMR disks aren’t any different technologically. They’re just wider, writing fatter tracks, which enables the head to write more well-defined track edges. Now the read head can pick up the narrower tracks even without any gaps between the tracks. Thus, we can pack stuff tighter and still read back without corruption. But…  how do we modify written data?! Each track looks like it’s been partially overwritten by its fat-ass successors? Lets see what would happen, if we attempted to write “the regular way” to a part of filled-up shingled disk surface, as compared to a normal one:

Writing to an SMR surface is a problem

Writing to an SMR surface is a problem. Writes within the structure overwrite adjacent tracks, because write heads are wider than read heads to be able to create strong enough magnetic fields for writing.

On the normal drive (to the left) we just write. No problem. On the SMR disk we write. And…  oops. Its not actually a write, it’s a write+destroy. In the example density above, by writing three sectors in directly sequential order (maybe a small 1.5kiB file), we effectively overwrite six additional sectors “to the right”, because the write head is too wide. For writing 1.5kiB, we potentially corrupt 3kiB more. Maybe even more when using 4kiB sectors instead of 512 byte ones. The effective amount of destroyed data depends on how much overlap there actually is of course – but there will always be data loss!

So how do we rewrite?! Well, we do it like SSDs do it in such a case – which is why SSDs also need the TRIM command and/or garbage collection algorithms. First, we need to read all affected data, which is basically everything down-cylinder from the write location (called “downstream” in SMR lingo). Reason is that if we rewrite just the six additional sectors right next to the affected ones, we lose the next six etc.

So?

We read everything downstream into a cache memory in as many cycles as there are tracks downstream, then we write downstream in as many cycles as there are downstream, including the original write. This is known in a less extreme form from solid-state disks, as a “read-modify-write” cycle:

Classic HDD writes vs. read-modfiy-write on SMR disks

Classic HDD writes vs. read-modify-write on SMR disks

So, what does that mean? Lets sum up the operations:

Regular hard drive:

  • Wait for platter to rotate and seek head to first target sector in track
  • Write three sectors in direct succession

SMR hard drive:

  • Wait for platter to rotate and seek head to target track + 1
  • Read three sectors in direct succession, store in cache
  • Wait for platter to rotate and seek head to target track + 2
  • Read three sectors in direct succession, store in cache
  • Wait for platter to rotate and seek head to target track + n
  • Read three sectors in direct succession, store in cache
  • (Repeat until we hit end of medium* or band)
  • Seek head to target track
  • Write original three sectors
  • Wait for platter to rotate and seek head to target track + 1
  • Rewrite three previously stored sectors, recalled from cache
  • Wait for platter to rotate and seek head to target track + 2
  • Rewrite three previously stored sectors, recalled from cache
  • Wait for platter to rotate and seek head to target track + n
  • Rewrite three previously stored sectors, recalled from cache
  • (Repeat until we hit end of medium* or band)

*As you can see, this is crazy. If we write to the up-most sector, we have to rewrite all the way downstream, this could be millions of sectors and seeks for a small file, affecting the entire radius of the platter! This is why SMR doesn’t completely do away with track gaps. It’s just that tracks are now being grouped into bands of arbitrary size to limit the read-modify-write impact. Let’s have a look at two side-by-side 7-tracks-wide SMR “bands”, both being written to:

Shingled media - organized into bands

Shingled media – organized into bands 7 tracks wide

From this we can learn two things: Bands can mitigate the severity of the issue. Also, the amount of work depends on where within a band we write to. The farther downstream, the less latency hit we will have to endure, the less seeks and write overhead we’ll have. Bands can’t be too wide, as write performance would deteriorate too much. Bands can’t be too narrow either, or we’ll lose too much of the density advantage because we’d have more band gaps using up platter real estate.

Let’s look at an overview regarding band width efficiency:

SMD band width efficiency

SMD band width efficiency[1] (click to enlarge)

I won’t go into all the detail about what this means, but especially the part about reserved non-shingled storage on the disk is pretty much unusable in todays scenarios I believe.  So, please pay attention to the green lines, where f=0. The number r is simply the track width in nanometers. By looking at the graph we can learn, that the sweet spot for the number of tracks per band is maybe around 10 to 25 or so. Beyond that we don’t gain much by saving us band gaps, and below that, data per square area isn’t packed with enough increase in density.

This makes me think that Seagate went with a rather “low” band width with their current SMR drive (the [Seagate Archive HDD v2]), as the platter size increase was only +250GB, so from 1TB to 1.25TB for the first SMR generation, and then +333GB with 1.33GB platters in the final generation hitting the market. So they got to an areal density increase factor of just 1.33×, which may correspond to 6 tracks per band, maybe 8 or 10 depending on track width in nanometers (I do not have solid data on track widths of any modern drives, especially not Seagates SMR disks). Some rumors are saying “5 – 10 bands” which does seem right considering my math.

Probably bad enough, but hey.

As said, SMR disks – like SSDs – are not showing their inner structure to the operating system as-is, as this would require new I/O schedulers, file systems, applications[2] and so on. Instead, they’re going for a fully firmware-abstracted[3] approach, showing only a “normal hard drive” to the OS, like any SSD would do too. All the nasty stuff happens on the drive itself, implemented 100% in the drive firmware.

File systems also need to be considered. A file system that fragments quickly will scatter larger files all across the disk, potentially across a multitude of bands. Rewriting such a file on a file system that can’t do [copy-on-write] and is fragmented will likely be painful even with firmware optimizations and cached/delayed writes in place. Exposing SMR to the file system would help a lot, but would also mean a lot of work for the file system developers side, and I just don’t see that happening at least outside of seriously expensive large-scale systems. Current file systems like FAT, exFAT, NTFS, ReFS, EXT2/3/4, XFS, btrfs, ZFS, UFS/FFS and so on simply don’t understand SMR bands. To my knowledge there is no file system that would. It’s likely going to be handled like 512e Advanced format – all the magic happens below the fake layer the operating system is being presented.

2a.) Pricing

Now, in the beginning I said the main reason for SMR to appear right now is price. Thing is, with SMR you can still use the same platters, and mostly also the same heads as before, with just minor modifications like the wider write head. It’s much less of a radical hardware design challenge, and more of a data packing and organization solution saving a lot of money. To give you an idea, let’s compare some actual prices from the EU region as of 2015-02-20, (only drives actually in stock), source: [Geizhals]German flag. Drives compared are roughly targeting the same market or at least share as many properties as possible, like warranty periods, 24/7 qualification, URE ratings, etc.:

Some regular drives:

  • Western Digital Purple 6TB: 251,52€ (Price per GB: 4,19¢)
  • Western Digital Red 6TB: 263,75€ (Price per GB: 4,40¢)
  • Hitachi GST Deskstar NAS 6TB: 273,32€ (Price per GB: 4,56¢)
  • Hitachi GST Ultrastar He8 8TB: 660,95€ (Price per GB: 8,26¢. This is an Enterprise helium drive with 5 years warranty and an URE rating at <1 bit read error in 1015 reads, so it’s hardly comparable. It’s the only other 8TB drive available though, which is why I’d like to show it here too.)
  • Seagate Surveillance HDD 7200rpm 6TB: 395,14€ (Price per GB: 6,59¢)
  • Seagate NAS HDD 6TB: 394,44€ (Price per GB: 6,57¢)

SMR Drive:

  • Seagate Archive HDD v2 8TB: 259,–€ (Price per GB: 3,24¢)

So as you can see, the price per GB of the shingled disk is simply unmatched here. First of all, it has no direct competitors to run against it, and it’s much cheaper per GB than the 6TB disks of the competition! Heck, even its absolute price is lower in most cases. Seagate does state, that the drive is meant for specific work loads only though. In essence, the optimal way to use it would be to treat it as a WORM (write once, read many) medium, as read performance is not impacted by SMR. But how bad is it really?

2b.) Actual performance numbers

Those are extremely hard to come by at this stage, as there are no real reviews yet. All I could find are some inconclusive tests [here]German flag. Inconclusive, because no re-writes or overwrites were tested. So far all that can be said is that the disks seems to show more progressive write caching, which does make some sense for grouping data together so whole bands can be written at once. We’ll have to wait a bit longer for any in-depth analysis though.

If anything comes up, I will add the links here.

For now, let’s continue:

3.) SMR super-charged: 2-dimensional reading (TDMR)

I’ll be brief about this, as the basic ideas of 2D reading are “relatively” simple. Here we’re trying to make the tracks even narrower, up to a level where a single read head might run into issues when it comes to staying on track and maintaining data integrity on reading. The idea is to put two more read heads on the whole head, one that would read at position shifted slightly in direction of track n-1 and one that would read shifted to the opposite side, track n+1. Like this:

Multiple read heads

Multiple read heads for two-dimensional magnetic reading[4]

From the differences between the read heads’ acquired data, a more precise 2D map can be constructed, making it easier to decide what the actual data must be, and what’s just interference from nearby tracks. The end result of course being an even higher density and increased data integrity.

To save money, one could also retain the single read head setup and read the adjacent tracks in additional passes. Naturally, this would be much slower and possibly less safe.

TDMR readback

Two-dimensional magnetic readback[4]

From our current standpoint it is hard to tell how much more density can be gained by TDMR-enhanced SMR, and at what cost exactly. The basic problems of SMR aren’t solved by TDMR at all however, as the data is still being organized in shingled bands. For regular drives, TDMR doesn’t make much sense, as the written tracks are more than wide enough for a single read head anyway. This could be considered useful for a second or third generation SMR technology, if ever released. It may have other uses too however, see 5.

4.) The future #1: Heat/microwave-assisted magnetic recording (HAMR/MAMR)

There are technologies in the making that are aiming at increasing density by employing whole new technologies for reading and writing without necessarily tampering with the data organization on the platters. HAMR or the less-known MAMR are one of them.

I mentioned before, that current drives store a single bit by magnetizing about 20-30 individual grains on the actual surface of the disk. This number can not be reduced easily any longer for data integrity reasons, and the grain size can’t be reduced either, which is because of a magnetic limitation known as the super-paramagnetism wall[4]. In essence, you need a certain amount of energy to change the polarity of a bunch of grains. Due to this effect, the smaller the grains you have, the more energy you must concentrate on a smaller spot to make a write operation work.

As electromagnetic energy increases on extremely dense small grain tracks, the field becomes too strong and will affect nearby tracks, data might be lost. Also, the heads must shrink together with the tracks, and it becomes harder and harder and eventually impossible to even generate fields that strong with downscaled write heads.

The super-paramagnetic wall – or rather the grains themselves – have an interesting property though. The wall shifts with the materials’ temperature, because temperature affects the grains coercivity. The colder the surface becomes, the more energy is required to write to it. The hotter it becomes, the easier writing will be, with less powerful magnetic fields required.

Given that the wall can be pushed around at will by changing the materials coercivity, researchers came up with a few solutions based on Lasers and Microwave emitters. So basically, shoot the surface with a Laser or with Microwaves to heat it up with pinpoint accuracy, then write with a small heads weak field, and profit!

HAMR head with Laser emitter

HAMR head with Laser emitter[4]

MAMR head with microwave emitter

MAMR head with Microwave emitter[4]

Clearly, to make that happen, several challenges must be mastered. First of all, you’d want a disk surface which doesn’t mess with the Microwaves or get messed up by them. Alternatively, the surface must have properties that do not limit the effectiveness of the Laser. In both cases, it needs to house smaller grains. And on top of that, we need highly miniaturized Laser or Microwave emitters, and we’re talking about nanometer level here. Additionally, robustness has to be ensured, which might be an issue with the head permanently shooting Laser beams around or heating itself up with Microwaves.

This is why HAMR/MAMR development is extremely hard and extremely expensive. And not just development; The entire hard drive manufacturing process would need to change considerably, creating additional costs. None of this is true for SMR.

Naturally, a working HAMR/MAMR solution doesn’t need to mean the end of SMR. It may give us a way to keep pushing out large disks without SMR implications for certain professional markets and even larger ones using SMR for regular end users. Currently, it seems that HAMR is getting the most attention, and MAMR is likely not going to ever see the light of day.

Seagate HAMR prototype

Seagate has already demonstrated a working HAMR prototype. MAMR is nowhere to be seen. 20TB is the goal here according to Seagate.

5.) The future #2: BPMR (bit-patterned magnetic recording)

Another approach towards dealing with the density issue is to deposit the magnetic film layer and have the grains sit on nanolithographically created, elevated “islands”. When doing so, the grains show a strong coupling of their exchange energy, which is a quantum-physical effect. That coupling means, that grains will follow the magnetic orientation of their neighbors more willingly, and also all together stay in the same orientation. By doing this, the energy required for altering the magnetic orientation is proportional to the islands volume, not the volume of individual grains representing a single bit. So what we could do is use smaller grains. Or maybe even make the “1 grain per bit” dream a reality by putting a single grain onto each island of a bit-patterned medium.

Bit-patterned media

Bit-patterned medium[5]

So far it’s been said, that staggered BPMR would be the easiest to manufacture and might enable manufacturers to pack data even tighter together (think: hexagonal pattern alignment), although this might require two-dimensional magnetic reading again, so: three read heads to eliminate cross-read interference. Since TDMR wouldn’t imply SMR in this case as there is no shingling with BPMR, the two could be used together without any issues for the end user.

Staggered bit-patterned surfaces

Staggered bit-patterned surfaces[6]

Also, servo patterns are making progress, which is equally important as are the actual data islands to maintain proper head positioning:

A BPMR servo pattern to the right with non-staggered data tracks/islands to the left

A BPMR servo pattern to the right with non-staggered data tracks/islands to the left[7]

Even with all that goodness, the super-paramagnetic limit will still apply here, albeit in a reduced fashion. A BPMR head could however once more be equipped with a Laser, thus combining BPMR with HAMR technology to further downscale the islands. Throw staggered island tracks and triple read heads for two-dimensional reading into the mix, and the density possibilities grow even further.

Needless to say, this costs a truckload of money. BPMR would require even more drastic changes in how hard drives are being manufactured, as we’d need 20 – 10nm nanolithography technology, a different platter composition, and new write and read heads plus the software – or rather firmware – to properly control all of that.

6.) Conclusion

I don’t believe we’ll really see HAMR anytime soon. On top of that, SMR might not really be just as intermediate a technology, only filling the gap until we can do better and bring the Lasers to the storage battle. When you look at HAMR, it becomes clear that SMR or SMR+TDMR are still feasible in conjunction with HAMR. I believe what we’ll see there will be an even stronger segmentation of hard drive markets, where fast writers might be available only in certain expensive enterprise segments, and slower-(re)writing SMR based media might serve other markets like cold data / cloud storage and data archiving. HAMR would be enhancing both of them, whether SMR is on board or not. Anything with a considerably higher need for reads could be using SMR, for a long time to come.

HAMR or not, any disk featuring SMR will definitely be larger and also feature the implications discussed here.

Then comes BPMR, likely after all other technologies described here. Now this baby is something else. Shingling tracks is pretty much out of the question here, as grains and the bits they represent can no longer overlap, becoming essentially atomic. If BPMR ever hits the markets – and I’m guessing it will in 10-15 years – the time for SMR to disappear will have come. But thats so far into the future that calling SMR “intermediate” now might be a bit premature. I’m guessing it may stay for a decade or more after all, even if I consider it quite ugly by design.

People needed to pay increasing attention to what drives they were going to buy over the past years, and with SMR that will just intensify. Users will need to have a proper idea of their I/O workloads and each and every storage technology available to them to make good decisions so that nobody runs off and buys a bunch of SMR disks to build a random-write-heavy mail server with them.

Good thing I’m building my new RAID-6 soon enough to still get SMR-free drives that I can consider “huge” despite them being non-shingled. ;)

[1] Gibson, G.; Ganger, G. 2011. (Carnegie Mellon University Parallel Data Lab). “Principles of Operation for Shingled Disk Devices“.

[2] Dunn, M.; Feldman, T. 2014. (Seagate, Storage Networking Industry Association). “Shingled Magnetic Recording Models, Standardization and Applications“.

[3] Feldman, T.; Gibson, G. 2013. “Shingled Magnetic Recording“. Usenix ;login: vol. 38 no. 3, 2013-06.

[4] Wood, R. 2010. (Hitachi GST). “Shingled Magnetic Recording and Two-Dimensional Magnetic Recording“. IEEE SCV MagSoc, 2011-10-19.

[5] Toshiba. 2010. “Bit-Patterned Media for High-Density HDDs“. The 21st Magnetic Recording Conference (TMRC) 2010.

[6] A*Star. “Packing in six times more storage density with the help of table salt“.

[7] Original unscaled image is © Kim Lee, Seagate Technology.

Feb 062015
 

Network[1] Everybody hates servers going offline. Especially email servers. Or web servers. Or MY SERVER! Now I prepared for a lot of things with my home server, I prepared for power failures, storage failures, operating system kernel crashes, everything. I thought I can recover from almost any possible breakdown even remotely, all but one: My four bonded G.SHDSL lines all failing at once. Which is what just happened. After lots of calls and even a replacement Paradyne/Zhone SNE2040G-S network extender having been brought to me within the time allowed by my SLA, all four lines still remained dark.

Now, today the telecommunication company which is responsible for the national network fixed the issue in the local automatic exchange. I tried to find out what had happened exactly, but ran into walls there. My Internet provider UPC got no information feedback from the telecommunication company A1 either, or at least nothing besides “it’s been fixed at the digital exchange”. Plus, as I am not an A1 customer exactly, so they won’t answer me directly. The stack is: UPC (internet provider) <=> Kapsch (field technicians handling UPC branded Internet access hardware, via outsourcing by UPC) <=> A1 (field technicians regarding the whole telecommunications infrastructure), while UPC may also communicate with A1 directly to handle outages. Communication seems to be kept to a minimum though. :(

Bad thing is, for a “business class” line, an outage of almost two days or 47 hours is a bit extreme. In such a case, more efficient communication could easily fix it faster. But it is what it is, I guess. And now I have to send one of the two Paradyne/Zhone G.SHDSL extenders back to UPC, this little bugger here:

The actively cooled Zhone 2040 G.SHDSL extender

The actively cooled Zhone SNE2040G-S G.SHDSL extender (click to enlarge)

There is actually a HSDPA (3G) fallback option, which works by implementing an OSI layer 2 coupling between the G.SHDSL line and the 3G access, keeping all IP addresses and domains the same and the services reachable during times of complete DSL failure. But I won’t order that upgrade, because it’s a steep 39€ before tax per month, or 46.80€ after tax. That’s just too expensive on top of what that connection’s already draining from my wallet.

All in all, this greatly endangers my usual, self-imposed yearly service availability of >=99%. 47 hours is a lot after all. So to maintain 99%, the server cannot go offline for more than 3 days, 15 hours and 36 minutes per regular year, and now I already have 1 day and 23 hours on the clock, and it’s just the beginning of the year! Let’s hope it runs more smoothly for the rest of 2015.

[1] Logo image is © Kyle Wickert, Do You Really Understand The Applications Flowing Through Your Network?

Jan 262015
 

FreeBSD battery logo[1] While I’m still using the free HP/Compaq nx6310 notebook a friend has given me, I ran into a little problem during actual productive use with FreeBSD 10.1 UNIX. A problem that also affects more modern HP notebooks like the EliteBook or ProBook series as well as certain Macbooks as I learned only later. The issue lies within the battery status reporting. While the very first state transition will be recognized, as in say you disconnect the AC cable, or the charge drops by 1% or whatever, subsequent changes are completely ignored.

An example: You want to join a meeting with 100% charge. You disconnect the AC cable, an event that is still recognized by the system. The charge may then stand at 99%. It would however not update further, but just stay there until either a reboot or until the lights just go dark. Not good if you intend to actually use the system on the go. The user just needs to be aware of the battery level!

1.) The right direction, but still a fruitless attempt

So I read that HP has quite some share of trouble with Windows-specific and also buggy ACPI BIOS code, and thus, this has been a pain for some time, not just on FreeBSD, but also Linux and in some cases even Windows.

At first I [reported the issue in the FreeBSD forums], after which I was asked to report the problem on the [freebsd-acpi@freebsd.org] mailing list. Problem only was that one can’t post anonymously and the registration never worked for me. So I kept digging in the dirt, and my first attempt was to fix the ACPI code of course, or DSDT as it’s called – the Differentiated System Description Table, written in ASL – the ACPI Source Language. Basically a pretty creepy byte code language providing a bidirectional interface between the operating system and the systems BIOS/UEFI, which in turn controls certain aspects of the hardware. Think display brightness adjustments, WiFi on/off switches, audio volume control via keyboard, sleep buttons etc. – all that is ACPI. You can find the specs [here] if you really want to take a look.

HP compiled the code using [Microsofts ASL compiler], which is known to be rather sloppy and ignore tons of bugs in the code when compiling. So my first step was to dump the active DSDT from the systems memory using acpidump and tell the tool to disassemble it using Intels ASL compiler iasl in the same step:

# acpidump -d > ~/nx6310-stock.asl

Then, I attempted to recompile using the Intel ASL compiler again:

# iasl ~/nx6310-stock.asl

And then came the errors and warnings. Plus some nasty OS-specific code. Since an operating system can and will identify itself to the BIOS via ACPI, one can unfortunately filter by operating system on the BIOS level. Very bad stuff. Like this, taken from the ASL code directly, as written by HP:

1
2
3
4
5
6
Name (C015, Package (0x08)
{
     "Microsoft Windows",
     "Microsoft WindowsME: Millennium Edition",
     "Microsoft Windows NT"
})
Name (C015, Package (0x08)
{
     "Microsoft Windows",
     "Microsoft WindowsME: Millennium Edition",
     "Microsoft Windows NT"
})

…and:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
If (LOr (LEqual (C014, 0x00), LEqual (C014, 0x03)))
{
     If (CondRefOf (\_OSI, Local0))
     {
          If (\_OSI ("Windows 2001"))
          {
               Store (0x04, C014)
          }
 
          If (\_OSI ("Windows 2001 SP1"))
          {
               Store (0x04, C014)
          }
 
          If (\_OSI ("Windows 2001 SP2"))
          {
               Store (0x05, C014)
          }
 
          If (\_OSI ("Windows 2006"))
          {
               Store (0x06, C014)
          }
     }
}
If (LOr (LEqual (C014, 0x00), LEqual (C014, 0x03)))
{
     If (CondRefOf (\_OSI, Local0))
     {
          If (\_OSI ("Windows 2001"))
          {
               Store (0x04, C014)
          }

          If (\_OSI ("Windows 2001 SP1"))
          {
               Store (0x04, C014)
          }

          If (\_OSI ("Windows 2001 SP2"))
          {
               Store (0x05, C014)
          }

          If (\_OSI ("Windows 2006"))
          {
               Store (0x06, C014)
          }
     }
}

Next logical step was to fake the OS FreeBSD was reporting to the ACPI subsystem. You can either do that by changing and embedding the ASL (non-MS OS strings are for instance “FreeBSD” or “Linux”), or luckily also by adding something like the following line to /boot/loader.conf, after which you need to reboot:

hw.acpi.osname="Windows 2006"

That didn’t do the trick for me however, so I tried to run my modified ACPI code after all. To do that you need to recompile the ASL, and then have loader pick it up in the early boot sequence of the kernel. First, I compiled the fixed code:

# iasl -tc ~/nx6310.asl

The result will be /tmp/acpidump.aml (for whatever reason). I placed it in /boot/ and added the following to /boot/loader.conf:

acpi_dsdt_load="YES"
acpi_dsdt_name="/boot/nx6310.aml"

Now another reboot. To verify the AML gets loaded, boot the kernel in verbose mode, you can choose that option in the boot loader. It’ll show something like Preloaded acpi_dsdt "/boot/nx6310.aml" at 0xc18dcc34. in /var/log/messages.

In my case however, while the AML was indeed being loaded, the BIOS’ tables were strangely not overwritten. I never found out why. Dumping them again would just give me the bugged code by HP. So that seemed to be dead end.

2.) The solution

Then I discovered that this might actually be a problem with the FreeBSD kernel itself by stumbling over [problem report 162859] in FreeBSDs bugzilla. It seems there was a problematic commit to the kernels ACPI subsystem written by Jung-uk Kim, initially for FreeBSD 9.0 in 2011: [r216942]. By now, he had provided two patches for it, of which I tried the [newer one] unsuccessfully, making the problem even worse. You can see my replies under my real name “Michael Lackner” in that PR.

Luckily, it was seemingly my complaint on top of the others that made Jung-uk Kim revert the [original patch], all back to how it was in FreeBSD 8.0, which some people reported to work just fine. So I got the now reverted/fixed code from [r277579] – a file named acpi_ec.c, and replaced /usr/src/sys/dev/acpica/acpi_ec.c with it. Then, I recompiled the FreeBSD kernel [as described in the FreeBSD handbook] once more (fear not, it’s actually very easy).

Reboot, and everything just worked! No need for a DSDT replacement or operating system faking even. I tested charge, discharge, AC cable dis-/reconnect and full charge states. All ok now! Finally, that book is now 100% ready for production use! :)

I mean, let’s not make any mistake here, the ACPI code from HP is still buggy. But like the Linux kernel, the FreeBSD kernel knows how to sail around the most severe bugs, including Microsoft-specific code in certain cases, at least now that this issue is fixed.

Thanks fly out to Jung-uk Kim for reverting the problematic patch! It should be merged from current any day now. If you want to get it done faster than that, just fetch acpi_ec.c, place it in the right folder and recompile the kernel and with it all of its modules, just like I did.

[1] Original battery icon used for the logo is © Freepik from Flaticon and is licensed under CC BY 3.0

Jan 152015
 

4Kn logoWhile I’ve been planning to build myself a new RAID-6 array for some time (more space, more speed), I got interested in the latest and greatest of hard drive innovations, which is the 4Kn Advanced Format. Now you may now classic hard drives with 512 byte sectors and the regular Advanced Format also known as 512e, which uses 4kiB physical sector sizes, but emulates 512 byte sectors for compatibility reasons. Interestingly, [Microsoft themselves state], that “real” 4Kn harddrives, which expose their sector size to the operating system with no glue layers in between are only supported in Windows 8 and above. So even Windows 7 has no official support.

On top of that, Intel [has stated], that their SATA controller drivers do not support 4Kn, so hooking one such drive up to your Intel chipsets’ I/O controller hub (ICH) or platform controller hub (PCH) will not work. Quote:

“Intel® Rapid Storage Technology (Intel® RST) version 9.6 and newer supports 4k sector disks if the device supports 512 byte emulation (512e). Intel® RST does not support 4k native sector size devices.”

For clarity, to make 4Kn work in a clean fashion, it must be supported on three levels, from lowest to highest:

  1. The firmware: For mainboards, this means your system BIOS/UEFI. For dedicated storage controllers, the controller BIOS itself.
  2. The kernel driver of the storage controller, so that’s your SATA AHCI/RAID drivers or SAS drivers.
  3. Any applications above it performing raw disk access, whether kernel or user space. File system drivers, disk cloning software, low level benchmarks, etc.

Granted, 4Kn drives are extremely new and still very rare. There is basically only the 6TB Seagate enterprise drives available ([see here]) and then some Toshiba drives, also enterprise class. But, to protect my future investments in that RAID-6, I got myself a [Toshiba MG04ACA300A] 3TB drive, which was the only barely affordable 4Kn disk I could get, basically also the only one available right now besides the super expensive 6TB Seagates. That way I can check for 4Kn compatibility relatively cheaply (click to enlarge images):

If you look closely, you can spot the nice 4Kn logo right there. In case you ask yourselves “Why 4Kn?”, well, mostly cost and efficiency. 4kiB sectors are 8 times as large as classic 512 byte ones. Thus, for the same data payload you need 8 times less sector gaps, 8 times less synchronization markers and 8 times less address markers. Also, a stronger checksum can be used for data integrity. See this picture from [Wikipedia]:

Sectors

Sector size comparison (Image is © Dougolsen under the CC-BY 3.0 unported license)

Now this efficiency is already there with 512e drives. 512e Advanced Format was supposedly invented, because more than half the programs working with raw disks out there can’t handle variable sector sizes and are hardcoded for 512n. That also includes system firmwares, so your mainboards’ BIOS/UEFI. To solve those issues, they used 4kiB sectors, then let a fast ARM processor translate them into 512 byte sectors on the fly to give legacy software something it could understand.

4Kn on the other hand is the purist, “right” approach. No more emulation, no more cheating. No more 1GHz ARM dual core processor in your hard drive just to be able to serve data fast enough.

Now we already know that Intel controllers won’t work. For fun, I hooked it up to my ASUS P6T Deluxes’ secondary SATA controller though, a Marvell 88SE6120. Plus, I gave the controller the latest possible driver, the quite hard-to-get version 1.2.0.8400. You can download that [here] for x86 and x64.  To forestall the result: It doesn’t work. At all. This is what the systems’ log has to say about it (click to enlarge):

So that’s a complete failure right there. Even after the “plugged out” message, the timeouts would still continue to happen roughly every 30 seconds, accompanied by the whole operating system freezing for 1-2 seconds every time. I cannot say for any other controllers like the Marvell 9128 or Silicon Image chips and others, but I do get the feeling that none of them will be able to handle 4Kn.

Luckily, I do already have the controller for my future RAID-6 right here, an Areca ARC-1883ix-12, the latest and greatest tech with PCIe x8 3.0 and SAS/12Gbit ports with SATA/6Gbit encapsulation. Its firmware and driver supports 4Kn fully as you can see in Arecas [specifications]. The controller features an out-of-band management system via its own ethernet port and integrated web server for browser-based administration, even if the system doesn’t even have any OS booted up. All that needs to be installed on the OS then is a very tiny driver (click to enlarge):

Plus, Areca gives us one small driver for many Windows operating systems. Only for the Windows XP 32-Bit NT5.1 kernel you’ll get a SCSI Miniport driver exclusively, while all newer systems (WinXP x64, Windows Vista, 7, 8) get a more efficient StorPort driver. So, plugged the controller in, installed the driver, hooked up the disk, and it seems we’re good to go:

The 4Kn drive is being recognized

The 4Kn drive is being recognized (click to enlarge)

Now, any legacy master boot record (MBR) partition table has a 32-bit address field. That means, it can address 232 elements. With each element being 512 bytes large, you reach 2TiB. So that’s where the 2TiB limit comes from. With 4Kn however, the smallest addressable atom is now eight times as large: 4096 bytes! So we should be able to reach 16TiB due to the larger sector size. Supposedly, some USB hard drive manufacturers have used this trick (by emulating 4Kn) to make their larger drives work easily on Windows XP. When trying to partition the Toshiba drive however, I hit a wall, as it seems Windows disk management is about as stupid as was the FAT32 formatter on Windows 98:

MBR initialization failed

MBR initialization failed (click to enlarge)

That gets me thinking. On XP x64, I can still just switch from MBR to the GPT partitioning scheme to be able to partition huge block devices. But what about Windows XP 32-bit? I don’t know how the USB drive manufacturers do it, so I can only presume they ship the drives pre-partitioned if its one of those that don’t come with a special mapping tool for XP. In my case, I just switch to GPT and carry on (click to enlarge):

Now I guess I am the first person in the world to be able to look at this, and potentially the last too:

fsutil.exe showing a 4Kn drive on XP x64

fsutil.exe showing a native SATA 4Kn drive on XP x64, encapsulated in SAS. Windows 7 would show the physical and logical sector size separately due to its official 512e support. Windows XP always reports the logical sector size (click to enlarge)

So far so good. The very first and most simple test? Just copy a file onto the newly formatted file system. I picked the 4k (no pun intended) version of the movie “Big Buck Bunny”:

Copying a first file onto the 4Kn disks NTFS file system

Copying a first file onto the 4Kn disks NTFS file system

Hidden files and folders are shown here, but Windows doesn’t seem to want to create a System Volume Information folder for whatever reason. Other than that it’s very fast and seems to work just nicely. Since the speed is affected by the RAID controllers write back cache, I thought I’d try HD Tune 2.55 for a quick sequential benchmark. Or in other words: “Let’s hit our second legacy software wall” (click to enlarge):

Yeah, so… HD Tune never detects anything above 2TiB, but this? At first glance, 375GB might sound quite strange for a 3TB drive. But consider this: 375 × 8 = 3000. What happened here is that HD Tune got the correct sector count of the drive, but misinterpreted each sectors’ size as 512 bytes. Thus, it reports the devices’ size as eight times as small. Reportedly, this is also the exact way how Intels RST drivers fail when trying to address a 4Kn drive. HD Tune 2.55 is thus clearly hardcoded for 512n. There is no way to make this work. Let’s try the paid version of the tool which is usually quite ahead of its free and legacy counterpart (click to enlarge):

Indeed, HD Tune Pro 5.00 works just as it should when accessing the raw drive. Users who don’t want to pay are left dead in the water here. Next, I tried HDTach, also an older tool. HDTach however reads from a formatted file system instead of from a raw block device. The file system abstracts the device to a higher level, so HDTach doesn’t know and doesn’t need to know anything about sectors. As a result, it also just works:

HD Tach benchmarking NTFS on a 4Kn drive

HD Tach benchmarking NTFS on a 4Kn drive (click to enlarge)

Next, let’s try an ancient benchmark, that again accesses drives on the sector level: The ATTO disk benchmark. It is here where we learn that 4Kn, or generally variable sector sizes aren’t space magic. This tool was written well before the times of 512e or 4Kn, and look at that (click to enlarge):

Now what does that tell me? It tells me that hardware developers feared the chaotic landscape of tools and softwares that access disks at low levels. Some might be cleanly programmed, where most may not. That doesn’t just include operating systems’ built-in toolsets, but also 3rd party software, independently from the operating system itself. Maybe it also affects disk cloning software like from Acronis? Volume shadow copies? Bitlocker? Who knows. Thing is, to be sure, you need to test that stuff. And I presume that to go as far as hard drive manufacturers did with 512e, they likely found one abhorrent hell of crappy software during their tests. Nothing else will justify ARM processors at high clock rates on hard drives just to translate sector sizes plus all the massive work that went into defining the 512e Advanced Format standard before 4Kn Advanced Format.

Windows 8 might now fully support 4Kn, but that doesn’t say anything about the 3rd party software you’re going to run on that OS either. So we still live in a Windows world where a lot of fail is waiting for us. Naturally, Linux and certain UNIX systems have adapted much earlier or have never even made the mistake of hardcoding sector sizes into their kernels and tools.

But now, to the final piece of my preliminary tests: Truecrypt. A disk encryption software I still trust despite the project having been shut down. Still being audited without any terrible security hole discoveries so far, it’s my main choice for cross-platform disk encryption, working cleanly on at least Windows, MacOS X and Linux.

Now, 4Kn is disabled for MacOS X in Truecrypts source code, but seemingly, this [can be fixed]. I also discovered that TC will refuse to use anything other than 512n on Linux if Linux kernel crypto is unavailable or disabled by the user in TC, see this part of Truecrypts CoreUnix.cpp:

1
2
3
4
5
6
7
8
9
10
#if defined (TC_LINUX)
if (volume->GetSectorSize() != TC_SECTOR_SIZE_LEGACY)
{
  if (options.Protection == VolumeProtection::HiddenVolumeReadOnly)
    throw UnsupportedSectorSizeHiddenVolumeProtection();
 
  if (options.NoKernelCrypto)
    throw UnsupportedSectorSizeNoKernelCrypto();
}
#endif
#if defined (TC_LINUX)
if (volume->GetSectorSize() != TC_SECTOR_SIZE_LEGACY)
{
  if (options.Protection == VolumeProtection::HiddenVolumeReadOnly)
    throw UnsupportedSectorSizeHiddenVolumeProtection();

  if (options.NoKernelCrypto)
    throw UnsupportedSectorSizeNoKernelCrypto();
}
#endif

Given that TC_SECTOR_SIZE_LEGACY equals 512, it becomes clear that hidden volumes are unavailable as a whole with 4Kn on Linux, and encryption is completely unavailable altogether if kernel crypto isn’t there. So I checked out the Windows specific parts of the code, but couldn’t find anything suspicious in the source for data volume encryption. It seems 4Kn is not allowed for bootable system volumes (lots of “512’s” there), but for data volumes it seems TC is fully capable of working with variable sector sizes.

Now this code has probably never been run before on an actual SATA 4Kn drive, so let’s just give it a shot (click to enlarge):

Amazingly, Truecrypt, another software written and even abandoned by its original developers before the advent of 4Kn works just fine. This time, Windows does create the System Volume Information folder on the file system within the Truecrypt container, and fsutil.exe once again reports a sector size of 4096 bytes. This shows clearly that TC understands 4Kn and passes the sector size on to any layers above itself in the kernel I/O stack flawlessly (The layer beneath it should be either the NT I/O scheduler or maybe the storage controller driver directly and the layer above it the NTFS file system driver, if my assumptions are correct).

Two final tests for data integrities’ sake:

Both a binary diff and SHA512 checksums prove, that the data copied from a 512n medium to the 4Kn one is still intact

Both a binary diff and SHA512 checksums prove, that the data copied from a 512n medium to the 4Kn one is still intact

So, my final conclusion? Anything that needs to work with a raw block device on a sector-by-sector level needs to be checked out before investing serious money in such hard drives and storage arrays. It might be cleanly programmed, with some foresight. It also might not.

Anything that sits above the file system layer though (anything that reads and writes folders and files instead of raw sectors) will always work nicely, as such software does not need to know anything about sectors.

Given the possibly enormous amount of software with hardcoded routines for 512 byte sectors, my assumption would be that the migration to 4Kn will be quite a sluggish one. We can see that the enterprise sector is adapting first, clearly because Linux and UNIX systems adapt much faster. The consumer market however might not see 4Kn drives anytime soon, given 512 byte sectors have been around for about 60 years (!) now.

Update 2014-01-16 (Linux): I just couldn’t let it go, so I took the Toshiba 4Kn drive to work with me, and hot plugged it into an Intel ICH10R. So that’s the same chipset as the one I ran the Windows tests on, an Intel X58. Only difference is, that now we’re on CentOS 6.6 Linux running the 2.6.32-504.1.3.el6.x86_64 kernel.  This is what dmesg had to say about my hotplugging:

ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: ATA-8: TOSHIBA MG04ACA300A, FP2A, max UDMA/100
ata3.00: 732566646 sectors, multi 2: LBA48 NCQ (depth 31/32), AA
ata3.00: configured for UDMA/100
ata3: EH complete
scsi 2:0:0:0: Direct-Access     ATA      TOSHIBA MG04ACA3 FP2A PQ: 0 ANSI: 5
sd 2:0:0:0: Attached scsi generic sg7 type 0
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
sd 2:0:0:0: [sdf] Write Protect is off
sd 2:0:0:0: [sdf] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
 sdf:
sd 2:0:0:0: [sdf] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
sd 2:0:0:0: [sdf] Attached SCSI disk

Looking good so far, also the Linux kernel typically cares rather less about the systems BIOS, bypassing whatever crap it’s trying to tell the kernel. Which is usually a good thing. Let’s verify with fdisk:

Note: sector size is 4096 (not 512)

WARNING: The size of this disk is 3.0 TB (3000592982016 bytes).
DOS partition table format can not be used on drives for volumes
larger than (17592186040320 bytes) for 4096-byte sectors. Use parted(1) and GUID 
partition table format (GPT).

Now that’s more like it! fdisk is warning me, that it will be limited to addressing 16TiB on this disk. A regular 512n or 512e drive would be limited to 2TiB as we know. Awesome. So, I created a classic MBR style partition on it, formatted it using the EXT4 file system, and mounted it. And what we get is this:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdf1             2.7T   73M  2.6T   1% /mnt/sdf1

And Intel is telling us that they don’t manage to give us any Windows drivers that can do 4Kn? Marvell doesn’t even comment on their inabilities? Well, suck this: Linux’ free driver for an Intel ICH10R south bridge (or any other that has a driver coming with the Linux kernel for that matter) seems to have no issues with that whatsoever. I bet it’s the same with BSD. Just weak, Intel. And Marvell. And all you guys who had so much time to prepare and yet did nothing!

Update 2014-01-20 (Windows XP 32-Bit): So what about regular 32-Bit Windows XP? There are stories going around that some USB drives with 3-4TB capacity would use a 4Kn emulation (or real 4Kn, bypassing the 512e layer by telling the drive firmware to do so?), specifically to enable XP compatibility without having to resort to special mapping tools.

Today, I had the time to install XP SP3 on a spare AMD machine (FX9590, 990FX), which is pretty fast thanks to a small, unused testing SSD I still had lying around. Before that I wiped all GPT partition tables from the 4Kn drive, both the one at the start as well as the backup copy at the end of the drive using dd. Again, for this test, the Areca ARC-1883ix-12 was used, now with its SCSI miniport driver, since XP 32-Bit does not support StorPort.

Please note, that this is a German installation of Windows XP SP3. I hope the screenshots are still understandable enough for English speakers.

Recognition and MBR initialization seems to work just fine this time, unlike on XP x64:

The 4Kn Toshiba as detected by Windows XP Pro 32-Bit SP3, again on an Areca ARC-1883ix-12

The 4Kn Toshiba as detected by Windows XP Pro 32-Bit SP3, again on an Areca ARC-1883ix-12 (click to enlarge)

Let’s try to partition it:

Partitioning the drive once more, MBR style

Partitioning the drive once more, MBR style

Sure looks good! And then, we get this:

A Master Boot Record, Windows XP and 4Kn: It does work after all

A Master Boot Record, Windows XP and 4Kn: It does work after all (click to enlarge)

So why does XP x64 not allow for initialization and partitioning of a 4Kn drive using MBR? Maybe because it’s got GPT for that? So in any case, it’s usable on both systems, the older NT 5.1 (XP 32-Bit) as well as the newer NT 5.2 (XP x64, Server 2003). Again, fsutil confirms proper recognition of our 4Kn drive:

fsutil.exe reporting a 4kiB sector size, just like on XP x64

fsutil.exe reporting a 4kiB sector size, just like on XP x64

So all you need – just like on XP x64 – is a proper controller with proper firmware and drivers!

There is one hard limit here though that XP 32-Bit users absolutely need to keep in mind; Huge RAID volumes using LUN carving/splitting and software JBOD/disk spanning using Microsofts Dynamic Volumes are no longer possible when using 4Kn drives. Previously, you could tell certain RAID controllers to just serve huge arrays to the OS in 2TiB LUN slices (e.g. best practice for 3ware controllers on XP 32-Bit). Then, in Windows, you’d just make those slices Dynamic Volumes and span a single NTFS file system over all of them, thus pseudo-breaking the 2TiB barrier.

This can no longer be done, as Dynamic Volumes seemingly do not work with 4Kn drives on Microsoft operating systems before Windows 8, but at least on XP 32-Bit. The option for converting the volume from MBR to GPT is simply greyed out in Windows disk management.

That means that the absolute maximum volume size using 4Kn disks on 32-Bit Windows XP is 16TiB! On XP x64 – thanks to GPT – it’s just a few blocks short of 256TiB, a limit imposed on us by the NTFS file systems’ 32-bit address field and 64kiB clusters, as 232 * 64KiB * 1024 * 1024 * 1024 = 256TiB.

And that concludes my tests, unless I have time and an actual machine to try FreeBSD or OpenBSD UNIX. Or maybe Windows 7. The likelihood for that is not too high at the moment though.

Jan 052015
 

DirectX logo[1] Quite some time ago – I was probably playing around with some DirectShow audio and video codec packs on my Windows system – I hit a wall in my favorite media player, which is [Mediaplayer Classic Home Cinema], in its 64-Bit version to be precise. I love the player, because it has its own built-in splitters and decoders, its small, light-weight, and it can actually use DXVA1 video acceleration on Windows XP / XP x64 with AMD/ATi and nVidia graphics cards. So yeah, Blu-Ray playback with very little CPU load is possible as long as you deal with the ACSS encryption layer properly. Or decode files from your hard disk instead. But I’m wandering from the subject here.

One day, I launched my 64-Bit MPC-HC, tried to decode a new Blu-Ray movie I got, and all of a sudden, this:

MPC-HC "Failed to query the needed interfaces for playback"

MPC-HC failing to render any file – whether video or audio, showing only a cryptic error message

I tried to get to the bottom of this for weeks. Months later I tried again, but I just couldn’t solve it. A lack of usable debug modes and log files didn’t help either. Also, I failed to properly understand the error message “Failed to query the needed interfaces for playback”. Main reason for my failure was that I thought MPC-HC had it all built in – container splitters, A/V decoders, etc. But still, the 64-Bit version failed. Interestingly, the 32-Bit version still worked fine, on XP x64 in this specific case. Today, while trying to help another guy on the web who had issues with his A/V decoding using the K-Lite codec pack, I launched Microsofts excellent [GraphEdit] tool to build a filter graph to show him how to debug codec problems with Microsofts DirectShow system. You can download the tool easily [here]. It can visualize the entire stack of system-wide DirectShow splitters and decoders on Windows, and can thus help you understand how this shit really works. And debug it.

Naturally, I launched the 32-Bit version, as I’ve been working with 32-Bit A/V tools exclusively since that little incident above – minus the x264 encoder maybe, which has its own 64-Bit libav built in. Out of curiosity, I started the 64-Bit version of GraphEdit, and was greeted with this:

GraphEdit x64 failure due to broken DirectShow core

GraphEdit x64 failure due to broken DirectShow core

“DirectShow core components failed to initialize.”, eh? Now this piqued my interest. Immediately the MPC-HC problem from a few years ago came to my mind, and I am still using the very same system today. So I had an additional piece of information now, which I used to search the web for solutions with. Interestingly, I found that this is linked to the entire DirectShow subsystem being de-registered and thus disabled on the system. I had mostly found people who had this problem for the 32-Bit DirectShow core on 64-Bit Windows 7. Also, I learned the name of the DirectShow core library.

quartz.dll.

On any 64-Bit system, the 32-Bit version of this library would sit in %WINDIR%\SysWOW64\ with its 64-Bit sibling residing in %WINDIR%\system32\. I thought: What if I just tried to register the core and see what happens? So, with my 64-Bit DirectShow core broken, I just opened a shell with an administrative account, went to %WINDIR%\system32\ and ran regsvr32.exe quartz.dll. And indeed, the libary wasn’t registered/loaded. See here:

Re-registering quartz.dll

Re-registering the 64-Bit version of quartz.dll (click to enlarge)

Fascinating, I thought. Now I don’t know what kind of shit software would disable my entire 64-Bit DirectShow subsystem. Maybe one of those smart little codec packs that usually bring more problems that solutions with them? Maybe it was something else I did to my system? I wouldn’t know what, but it’s not like I can remember everything I did to my systems’ DLLs. Now, let’s try to launch the 64-Bit version of GraphEdit again, with a 64-Bit version of [LAVfilters] installed. That’s basically [libav] on Windows with a DirectShow layer wrapped around and a nice installer shipped with it. YEAH, it’s a codec pack alright. But in my opinion, libav and ffmpeg are the ones to be trusted, just like on Linux and UNIX too. And Android. And iOS. And OSX. Blah. Here we go:

64-Bit GraphEdit being fed the Open Source movie "Big Buck Bunny"

64-Bit GraphEdit being fed the Open Source movie “Big Buck Bunny” (click to enlarge)

And all of a sudden, GraphEdit launches just fine, and presents us a properly working filter graph after having loaded the movie [Bick Buck Bunny] in its 4k/UHD version. We can see the container being split, video and audio streams being picked up by the pins of the respective libav decoders, which feed the streams to a video renderer and a DirectSound output device – which happens to be an X-Fi card in this case. All pins are connected just fine, so this looks good. Now what does it look like in MPC-HC now? Like this (warning – large image, 5MB+, this will take time to load from my server!):

64-Bit MPC-HC playing the 4k/UHD version of "Big Buck Bunny"

64-Bit MPC-HC playing the 4k/UHD version of “Big Buck Bunny” (click to enlarge)

So there we go. It seems MPC-HC does rely on DirectShow after all, at least in that it tries to initialize the subsystem. It can after all also use external filters, or in other words system-wide DirectShow codecs too, where its internal codec suite wouldn’t suffice, if that’s ever the case. So MPC-HC seems to want to talk to DirectShow at play time in any case, even if you haven’t even allowed it to use any external filters/codecs. Maybe its internal codec pack even is DirectShow-based too? And if DirectShow is simply not there, it won’t play anything. At all. And I failed to solve this for years. And today I launch one program in a different bitness than usual, and 5 minutes later, everything works again. For all it takes is something like this:

regsvr32.exe %WINDIR%\system32\quartz.dll

Things can be so easy sometimes, if only you know what the fuck really happened…

The importance and significance of error handling and reporting to the user can never be understated!

But hey! Where there is a shell, there is a way, right? Even on Windows. ;)

[1] © Siegel, K. “DirectX 9 logo design“. Kamal Siegel’s Artwork.

Dec 112014
 

FreeBSD + WineSince I’ve abandoned OpenBSD for FreeBSD in my recent attempts to actually use a ‘real’ UNIX system, and all that just because of Wine so I can use some small Windows tools I need (yeah…), I was a bit fed up with Wines’ font anti-aliasing not working. I remember having a similar problem on CentOS Linux some while back, but I can’t remember how I solved it on that OS. Thing is, I just want to document my solution here briefly, so I don’t forget it, if I have to re-do this any time soon. The problem seems to originate from the X11 render extension that is being used for compositing on X11 without any additional compositing engine like Compiz. Wines usage of the extension is actually controlled by its system registry, and the key that seems to need editing is HKEY_CURRENT_USER\Software\Wine\X11 Driver\ClientSideWithRender.

Interestingly, people suggested to switch the use of the X render extension off instead of on, but when inspecting my setup by running wine regedit, I found I didn’t even have the key, and its default is off! So what saved me was:

[HKEY_CURRENT_USER\Software\Wine\X11 Driver]
"ClientSideWithRender"="Y"

Setting this back to “N” is basically the same thing as just deleting the key altogether, at least with the current default configuration of my Wine version on FreeBSD, which is wine-1.6.2. See how it looks like with no anti-aliasing (click to enlarge):

Wine running its regedit on FreeBSD 10 with no font AA

Wine running its regedit on FreeBSD 10 with no font AA

And here with proper AA, again, please click to enlarge so you can see the effect applied:

Wine with proper font anti-aliasing

Wine with proper font anti-aliasing

To demonstrate the effect more prominently, let’s look at some zoomed in versions, click to enlarge again for all the glory or so:

No anti-aliased fonts here, all jagged

No anti-aliased fonts here, all jagged

And:

Wine font anti-aliasing up close, nice and smooth

Wine font anti-aliasing up close, nice and smooth

Now I heard that some people actually prefer the clearer look of jagged fonts, or they may prefer pure gray smoothing, but I actually like the look of this smoothing a lot more. This is with medium font hinting turned on in Xfce4, and it appears very smooth and nice to my eyes.

If you want to try this in case you’re offended by jagged fonts using wine, just take the following:

[HKEY_CURRENT_USER\Software\Wine\X11 Driver]
"ClientSideWithRender"="Y"

Save it into a file like wine-Xrender-fontAA.reg, and then import it into your wines registry database by opening a terminal, going into the directory where your reg file sits, and by running: wine regedit ./wine-Xrender-fontAA.reg. Then restart your wine application and it should work for 99% of the fonts. I’ve encountered one font in Bruce Schneiers [PasswordSafe] that wouldn’t anti-alias, ever. But it’s the same on Windows, so I’m guessing that’s ok.

Switching it off is just as easy, just edit the file, change “Y” to “N” and re-run the wine regedit command. But as I said, I’ll keep it, no more eye cancer when starting Win32 applications on FreeBSD. :)