Nov 222016
 

FreeBSD IBM ServeRAID Manager logoAnd yet another FreeBSD-related post: After [updating] the IBM ServeRAID manager on my old Windows 2000 server I wanted to run the management software on any possible client. Given it’s Java stuff, that shouldn’t be too hard, right? Turned out not to be too easy either. Just copying the .jar file over to Linux and UNIX and running it like $ java -jar RaidMan.jar wouldn’t do the trick. Got nothing but some exception I didn’t understand. I wanted to have it work on XP x64 (easy, just use the installer) and Linux (also easy) as well as FreeBSD. But there is no version for FreeBSD?!

The ServeRAID v9.30.21 manager only supports the following operating systems:

  • SCO OpenServer 5 & 6
  • SCO Unixware 7.1.3 & 7.1.4
  • Oracle Solaris 10
  • Novell NetWare 6.5
  • Linux (only certain older distributions)
  • Windows (2000 or newer)

I started by installing the Linux version on my CentOS 6.8 machine. It does come with some platform-specific libraries as well, but those are for running the actual RAID controller management agent for interfacing with the driver on the machine running the ServeRAID controller. But I only needed the user space client program, which is 100% Java stuff. All I needed was the proper invocation to run it! By studying IBMs RaidMan.sh, I came up with a very simple way of launching the manager on FreeBSD by using this script I called serveraid.sh (Java is required naturally):

  1. #!/bin/sh
  2.  
  3. # ServeRAID Manager launcher script for FreeBSD UNIX
  4. # written by GAT. http://www.xin.at/archives/3967
  5. # Requirements: An X11 environment and java/openjdk8-jre
  6.  
  7. curDir="$(pwd)"
  8. baseDir="$(dirname $0)/"
  9.  
  10. mkdir ~/.serveraid 2>/dev/null
  11. cd ~/.serveraid/
  12.  
  13. java -Xms64m -Xmx128m -cp "$baseDir"RaidMan.jar com.ibm.sysmgt.raidmgr.mgtGUI.Launch \
  14. -jar "$baseDir"RaidMan.jar $* < /dev/null >> RaidMan_StartUp.log 2>&1
  15.  
  16. mv ~/RaidAgnt.pps ~/RaidGUI.pps ~/.serveraid/
  17. cd "$curDir"

Now with that you probably still can’t run everything locally (=in a FreeBSD machine with ServeRAID SCSI controller) because of the Linux libraries. I haven’t tried running those components on linuxulator, nor do I care for that. But what I can do is to launch the ServeRAID manager and connect to a remote agent running on Linux or Windows or whatever is supported.

Now since this server/client stuff probably isn’t secure at all (no SSL/TLS I think), I’m running this through an SSH tunnel. However, the Manager refuses to connect to a local port because “localhost” and “127.0.0.1” make it think you want to connect to an actual local RAID controller. It would refuse to add such a host, because an undeleteable “local machine” is always already set up to begin with, and that one won’t work with an SSH tunnel as it’s probably not running over TCP/IP. This can be circumvented easily though!

Open /etc/hosts as root and enter an additional fantasy host name for 127.0.0.1. I did it like that with “xin”:

::1			localhost localhost.my.domain xin
127.0.0.1		localhost localhost.my.domain xin

Now I had a new host “xin” that the ServeRAID manager wouldn’t complain about. Now set up the SSH tunnel to the target machine, I put that part into a script /usr/local/sbin/serveraidtunnel.sh. Here’s an example, 34571 is the ServeRAID agents’ default TCP listen port, 10.20.15.1 shall be the LAN IP of our remote machine hosting the ServeRAID array:

#!/bin/bash
ssh -fN -p22 -L34571:10.20.15.1:34571 mysshuser@www.myserver.com

You’d also need to replace “mysshuser” with your user name on the remote machine, and “www.myserver.com” with the Internet host name of the server via which you can access the ServeRAID machine. Might be the same machine or a port forward to some box within the remote LAN.

Now you can open the ServeRAID manager and connect to the made-up host “xin” (or whichever name you chose), piping traffic to and from the ServeRAID manager through a strongly encrypted SSH tunnel:

IBM ServeRAID Manager on FreeBSD

It even detects the local systems’ operating system “FreeBSD” correctly!

And:

IBM ServeRAID Manager on FreeBSD

Accessing a remote Windows 2000 server with a ServeRAID II controller through an SSH tunnel, coming from FreeBSD 11.0 UNIX

IBM should’ve just given people the RaidMan.jar file with a few launcher scripts to be able to run it on any operating system with a Java runtime environment, whether Windows, or some obscure UNIX flavor or something else entirely, just for the client side. Well, as it stands, it ain’t as straight-forward as it may be on Linux or Windows, but this FreeBSD solution should work similarly on other systems as well, like e.g. Apple MacOS X or HP-UX and others. I tested this with the Sun JRE 1.6.0_32, Oracle JRE 1.8.0_112 and OpenJDK 1.8.0_102 for now, and even though it was originally built for Java 1.4.2, it still works just fine.

Actually, it works even better than with the original JRE bundled with RaidMan.jar, at least on MS Windows (no more GUI glitches).

And for the easy way, here’s the [package]! Unpack it wherever you like, maybe in /usr/local/. On FreeBSD, you need [archivers/p7zip] to unpack it and a preferably modern Java version, like [java/openjdk8-jre], as well as X11 to run the GUI. For easy binary installation: # pkg install p7zip openjdk8-jre. To run the manager, you don’t need any root privileges, you can execute it as a normal user, maybe like this:

$ /usr/local/RaidMan/serveraid.sh

Please note that my script will create your ServeRAID configuration in ~/.serveraid/, so if you want to run it as a different user or on a different machine later on, you should recursively copy that directory to the new user/machine. That’ll retain the local client configuration.

That should do it! :)

Nov 192016
 

FreeBSD GMABoost logoRecently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
 
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call gmaboost.sh. I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  2.  
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  13.  
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  16.  
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # gmaboost.sh 200 or # gmaboost.sh 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Nov 142016
 

HP/Compaq nx6310/nc6320 logoA good while back, I got a free notebook from [The_Plague]German flag, a HP/Compaq nx6310[1][2] which he kinda pulled out of the trash at his company. It’s not exactly “Thinkpad T23” material, but it’s a pretty solid, well-built machine with a good keyboard. I’ve been using the thing as an operating system testbed for a while (Linux, ReactOS, Haiku OS, OpenBSD, Dragonfly BSD, and finally: FreeBSD UNIX). After settling for FreeBSD the machine clearly showed its limitations though, the most problematic being imposed by the very low-end i940GML chipset. That one has limited the machine to a single processor core and a 533MHz data rate FSB.

I did give the machine a Core Duo T2450, but switching dual core on in the BIOS results in a lockup at POST time. Also, the chipset cannot use dual-channel DDR-II and limits the user to 2GiB of memory, making the use of a 64-bit processor rather pointless. Which turned out to be bad, because some code doesn’t even provide full functionality for 32-bit anymore, like x265, which dropped deep color support on 32-bit architectures.

But now, The_Plague pulled another one out of the trash, it’s basically the exact same machine, but a higher-end model, the nc6320. This one has an i945GM chipset, which means dual core support, FSB667 and 4GiB dual-channel RAM capability! It came with a Core 2 Duo T5600 @ 1.83GHz with 2MiB L2 cache. I ordered the largest possible chip for this box from ebay Hong Kong, so now it has a Core 2 Duo T7600 @ 2.33GHz with 4MiB L2 cache. Also, 2×2=4GiB of DDR-II/667 CL4 are on their way already, together with a 12-cell secondary monster battery!

And of course, FreeBSD UNIX again, in its brand new version 11.0-RELEASE:

HP/Compaq nc6320 running FreeBSD 11.0 UNIX

HP/Compaq nc6320 running FreeBSD 11.0 UNIX (click to enlarge)

The CPU upgrade is actually even noticeable when browsing the web, lots of resource-hungry Javascript and CSS3, you know. Luckily, Chromium supports hardware acceleration on the Intel GMA950 GPU on FreeBSD, as the OS comes with a kernel modesetting compliant driver for almost all integrated Intel graphics chips. It’s too slow to do the rasterization stage on the GPU, but it still helps.

Once again, it shall serve mostly as a meeting and sysadmin machine, with a little bit of private-use-fun added on top. Let’s have a look at the software! Oh and by the way, I decided to make the screenshots 8-bit .png images, so some of them will look a bit bad. But still better+smaller than JPEG for that purpose:

Running screenfetch on the nc6320

Running screenfetch on the nc6320 (click to enlarge)

$ screenfetch is showing us some details about the machine, which also makes it clear that everything is “Tokisaki Kurumi”-themed. Since there’s a lot of red color on that girls’ garments it seems at least somewhat fitting for a FreeBSD machine.

Chromium with FVD Speed Dial

Chromium with FVD Speed Dial (click to enlarge)

I’m a [Vivaldi] fan personally, but that browser isn’t available on any BSD yet, so I installed a few extensions to make Chromium work somewhat like Vivaldi; The most important part being the static FVD speed dial you can see above. What you can’t see here are the other extensions that followed it: AdBlockPlus and Ghostery. I hear there are better/faster solutions than ABP for ad blocking these days however, so maybe I’ll revise that.

IBM Lotus Notes via wine 1.8

IBM Lotus Notes 6.5.1 via 32-bit wine 1.8.4 (click to enlarge)

Also, for work I would sometimes need IBM Lotus Notes, as it’s our Universities’ groupware solution (think of that what you will). While I couldn’t get the Linux version to run, our Domino servers still accept connections from older clients, so it’s Lotus Notes 6.5.1 running under a 32-bit [wine], which is a solution IBM officially recommended for running the software on Linux/UNIX a few years ago. And yeah, it still works. And if you have Windows software wine can’t cope with?

XP x64 via VirtualBox on FreeBSD

XP x64 via VirtualBox on FreeBSD (click to enlarge)

For anything that wine can’t handle, the VirtualBox port kicks in, as we can see here. Together with the CPUs VT-x extension and the guest tools, virtualizing Windows on FreeBSD UNIX works relatively well. Not all features are there (like USB passthrough), but it works ok for me. Will need a Windows 7 VM as well I think.

More stuff:

Communicating on FreeBSD

Communicating on FreeBSD (parts are censored, click to enlarge)

One important part is communication! Luckily, there is a version of licq in the ports tree now. It builds well together with its Qt4 UI, so no complaints there. Hexchat for IRC access is also available, but the tricky part was Skype; Not that I really need it, but I wanted to have the linuxulator up and running as well! For those of you who don’t know what the “linuxulator” is: It’s a series of kernel modules that extend FreeBSDs kernel with parts of the Linux kernel interface. On top of that, you can pull parts of Fedora 10 or CentOS 6.8 or some CentOS 7 Linux userspace components from the package servers. Together with the kernel modules those form a kind of runtime environment for executing Linux programs – Skype 4.3 in this case! So I have both wine and linuxulator ready for action, and with it access to ICQ, Jabber, MSN, IRC and Skype. Now, what about multimedia?

Multimedia on FreeBSD

smplayer and xmms on FreeBSD, unfortunately the 8-bit color is a bit too noticeable for this screenshot, my apologies (click to enlarge)

This is a part where the upgraded processor also helps. Here we can see (s)mplayer play the last episode of the Anime Hanayamata in taxing 2.5Mbit H.265/HEVC encoding, paired with AAC-LC audio. The Core 2 Duo T5600 had some issues with this, but the faster T7600 shows now problems. Additionally, xmms is playing a Commodore 64 SID tune using libsidplay2 and the reSID engine. xmms comes with a lot of funny plugins from the FreeBSD ports tree for Gameboy tunes or NES tunes, but the C64 one you need to compile for yourself. Not too hard though, you can fetch libsidplay2 and reSID from packages beforehand to make things easier! What else?

ioquake3

ioquake3, a cleaned up version of the Quake III Arena source code, here in its 64-bit FreeBSD build (click to enlarge)

A pretty fun part: Playing the native Quake3 port [ioquake3] in 64-bit, for whenever you just need to shoot something to blow off some steam. ;) I have to say, I had to tweak it quite a bit to run fluently on the WVA 1400×1050 display of this book given the weak GMA950 GPU, but it runs “rather ok” now. ioquake3 is also available for Windows, OSX and Linux by the way, including a more advanced OpenGL 2 renderer, which gives users access to some advanced graphical effects. And if I get bored by that…

HakuNeko Manga ripper and qComicbook

HakuNeko Manga ripper and qComicbook showing some sweet girls love! (click to enlarge)

Once again, fixing up HakuNekos’ build system and C++ code to work with FreeBSD properly took some time. Unfortunately there is no port for it yet (and I’m too stupid/lazy to create one), so you have to fix it by hand. Lots of replacing sed invocations with gsed, find with gfind etc. and the OS #ifdef parts, which need to be changed in several .cpp files, here’s an example from MangaConnector.cpp:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

Something like that needs to turn into this to compile on FreeBSD, otherwise you’ll end up with a HakuNeko that can’t do shit (it’ll still compile and run, but like I said, it’d be devoid of function):

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

This is true for the latest version 1.4.1 as well. I guess the modifications should also apply to other operating systems by adding things like __OpenBSD__ or similar.

Now all that’s left is to wait for that massive 12C battery, the RAM capacity+speed upgrade and some FreeBSD case sticker that I ordered from [unixstickers.com] (hint: That’s a referral URL, it’s supposed to give you some $5 coupon upon ordering, I hope it works). Upon my order, a small part was donated to the LLVM project – very fitting, given that I’ve used clang/llvm a lot to compile stuff on FreeBSD as of late. :)

FreeBSD case sticker (preview)

This is what it’s supposed to look like, and it’s going to replace the current Windows XP+Vista sticker

I hope it’ll look as good in real life! :) Ah, I think I’m gonna have a lot of fun with that old piece of junk. ;)

Ah, and thanks fly out to The_Plague, who saved this laptop from the trash bin and gave it to me for free! Prost!

Edit: And the memory is here, two G.Skill “performance” modules doing 4-4-4 latencies at 667MHz data rate, replacing a single Samsung module running 5-5-5. Now I was interested in how much going from single channel CL5 to dual channel CL4 would really affect performance. Let’s just say, it didn’t do too much for CPU processes. However, the effect on the integrated GMA950 GPU (using shared system memory!) was amazing. It seems the graphics chip was held back a lot by the memory interface! Let’s have a quick look at Quake III Arena performance using a quickly recorded demo just for this purpose (ioquake3 can’t play old Quake III Arena demos like the “001” demo):

  • ioquake3 1.36, single channel DDR-II/667 CL5:
  • 30.6fps
  • ioquake3 1.36, dual channel DDR-II/667 CL4:
  • 41.2fps

Roughly +35%!!

Tests were run three times, then three more times after a reboot. After that, an average was taken. For ioquake3 this wouldn’t even have been necessary though, as the results were extremely consistent. It’s amazing how much the added memory speed really affects the game engine! I rebooted and re-ran the tests several times because I couldn’t believe in that massive boost in performance, but it’s actually true and fully reproducible! This reminds me of how well modern AMD APU graphics chips scale with main memory speed and it explains why people were asking for quad-channel DDR4 on those Kaveri APU chips. Its built-in Radeons would’ve probably loved the added bandwidth!

I also kinda felt that browsing web sites got a lot more smooth using Chromium with most of its GPU acceleration turned on. So I tried the graphics-centric browser test [Motionmark] to put that to the test. Parts of the results were inconclusive, but let’s have a look first:

  • Motionmark 1.0 (medium screen profile), single channel DDR-II/667 CL5:
  • Overall result: 13.85 ±22.24%
  • Multiply: 119.26 ±2.95%
  • Canvas Arcs: 19.04 ±68.48%
  • Leaves: 3.00 ±133.33%
  • Paths: 85.30 ±6.57%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 1.76 ±5.22%
  • Images: 40.58 ±2.56%
  • Design: 18.89 ±8.00%
  • Suits: 24.00 ±37.50%
  • Motionmark 1.0 (medium screen profile), dual channel DDR-II/667 CL4:
  • Overall result: 22.47 ±15.93%
  • Multiply: 124.55 ±1.60%
  • Canvas Arcs: 26.00 ±138.46%
  • Leaves: 65.90 ±16.93%
  • Paths: 37.00 ±16.89%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 2.00 ±50.00%
  • Images: 41.58 ±3.59%
  • Design: 24.49 ±2.35%
  • Suits: 90.65 ±13.55%

Now first things first: This was just my first pick for any kind of graphics-heavy browser benchmark. I thought I needed something that would make the browser do a lot of stuff on the GPU, given that hardware acceleration was almost fully enabled on FreeBSD UNIX + Chromium + GMA950. However, after repeated runs it showed that the variance was just far too high on the following tests: Leaves, Paths, Suits. Those would also mess up the overall score. The ones that showed consistent performance were: Multiply, Canvas Arcs, Canvas Lines, Focus, Images, Design, so we should focus on those. Well, not all of those tests show promising results (Multiply, Canvas Lines), but some clearly do. It seems my feeling that parts of CSS3 etc. had gotten faster after the memory upgrade was spot-on!

Not bad, not bad at all! And tomorrow morning, the [x264 benchmark] will also have finished, showing how much a classic CPU-heavy task would profit from that upgrade (probably not much, but we’ll see tomorrow).

Edit 2: And here is the rest. Like I thought, the memory upgrade had only minimal impact on CPU performance:

  • x264 benchmark, single channel DDR-II/667 CL5:
  • Runtime: 04:40:08.621
  • x264 benchmark, dual channel DDR-II/667 CL4:
  • Runtime: 04:38:23.851

So yeah it’s faster. But only by a meager +0.62%. Completely negligible. But it’s still a good upgrade given the GPU performance boost and the fact that I can now use more memory for virtual machines. :)

Ah, and here’s the 12-cell ultra capacity battery, which gives me a total of 18 cells in conjunction with the 6-cell primary battery:

Nice hardware actually, you can check it’s charge (roughly) with a button and a 4-LED display, and it has it’s own charging plug. What surprised me most though was this:

$ hwstat | grep -i -e "serial number" -i -e battery
[ACPI Battery (sysctl)]
[battery0]
        Serial number:                  00411 2006/10/12
[battery1]
        Serial number:                  00001 2016/07/29

That probably explains how a still sealed battery could come with a ~25% pre-charge. Manufactured in July 2016, wow. And that for a notebook that’s 10 years old? Ok, it’s an aftermarket battery by [GRS], but that’s just damn fine still! With that I’ll surely have enough battery runtime to make it through longer meetings as well! :)

Edit 3: And today I used the notebook for a sysadmin task, helping our lead developer in debugging a weird problem in a Java-based student exam submission and evaluation system of ours at work. I suspected that the new CuPPIX (=KNOPPIX derivative) distribution I built for this was to blame, but it turned out to be a faulty Java library handling MySQL database access, hence crashing our server software under high parallel loads. In any case, I had the nc6320 with me during the entire morning up until 12:30 or so, walking away with a total charge of 49% left after the developer had fixed the problem. Not stellar given a total of 18 cells, but definitely good enough for me! :)

Edit 4: And my FreeBSD sticker from unixstickers is finally here! They even gave me a bunch of random free stickers to go with it! I gave those to some colleagues for their kids. ;) And here it is:

FreeBSD sticker from unixstickers.com

There was a Windows Vista/XP sticker before, now it shows some UNIX love! (click to enlarge)

The sticker shows some pretty good quality as well, nice stuff! :)

Jul 272016
 

x264 logoSince I’ve been doing a bit of Anime batch video transcoding with x264 and x265 in the last few months, I thought I’d document this for myself here. My goal was to loop over an arbitrary amount of episodes and just batch-transcode them all at once. And that on three different operating systems: Windows (XP x64), Linux (CentOS 6.8 x86_64) and FreeBSD 10.3 UNIX, x86_64. Since I’ve started to split the work across multiple machines, I lost track of what was where and which machine finished what, and when.

So I thought, why not let the loop send me a small notification email upon completion? And that’s what I did. On Linux and UNIX this relies on the bash shell and the mailx command. Please note that I’m talking about [Heirloom mailx], not some other mail program by the same name! I’m mentioning this, because there is a different default mailx on FreeBSD, that won’t work for this. That’s why I put alias mailx='/usr/local/bin/mailx' in my ~/.bash_profile on that OS after installing the right program to make it the default for my user.

On Windows, my loops depend on my own [colorecho] command (you can replace that with cmds’ ECHO if you want) as well as the command line mailer [blat]. Note that, if you need to use SSL/TLS encryption when mailing, blat can’t do that. A suitable replacement could be [mailsend]. Please note, that mailsend does not work on Windows XP however.

In the x265 case, avconv (from the [libav] package) is required on all platforms. You can get my build for Windows [here]. If you don’t like it, the wide-spread [ffmpeg] can be a suitable drop-in replacement.

Now, when setting up blat on Windows, make sure to run blat -help first, and learn the details about blat -install. You need to run that with certain parameters to set it up for your SMTP mail server. For whatever reason, blat reads some of that data from the registry (ew…), and blat -install will set that up for you.

Typically, when I transcode, I do so on the elementary streams rather than .mkv files directly. So I’d loop through some source files and extract the needed streams. Let’s say we have “A series – episode 01.mkv” and some more, all the way up to “A series – episode 13.mkv”, then, assuming track #0 is the video stream…

On Windows:

FOR %I IN (01,02,03,04,05,06,07,08,09,10,11,12,13) DO mkvextract tracks "A series - episode %I.mkv" ^
 0:%I\video.h264

On Linux/UNIX:

for i in {01..13}; do mkvextract tracks "A series - episode $i.mkv" 0:$i/video.h264; done

mkvextract will create the non-existing subfolder for us, and a x264 transcoding loop would then look like this on Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps ^
 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f ^
 -2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats %I\v.stats -t 2 --no-fast-pskip ^
 --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o %I\pass1.264 & colorecho "Pass 1 ^
 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & ^
 "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps 24000/1001 --preset veryslow --tune animation ^
 --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats ^
 %I\v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o ^
 %I\pass2.264 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10) & echo !SERIES! ^
 transcoding complete | blat - -t "myself@another.mailhost.com" -c "myself@mailhost.com" -s "x264 ^
 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET SERIES="

Note that I always write all the iteration out in full here. That’s because cmd can’t do loops with leading zeroes in the iterator. The reason for this is that those source files usually have them in their lower episode numbers. If it wasn’t 01,02, … ,12,13, but 1,2, … ,12,13 instead, you could do FOR /L %I IN (1,1,13) DO. But this isn’t possible in my case. Even if elements need alphanumeric names like here,  FOR %I IN (01,02,03,special1,special2,ova1,ova2) DO, you still won’t need that syntax on Linux/UNIX because the bash can have iterator groups like for i in {{01..13},special1,special2,ova1,ova2}; do. Makes me despise the cmd once more. ;)

Edit:

Ah, according to [this], you can actually do something like cmd /V /C "FOR /L %I IN (1,1,13) DO (SET "fI=00%I" & echo "!fI!:~-2")", holy shit. It actually works and gives you leading zeroes. :~-2 for 2 digits, :~-3 for three. Expand fI for more in this example. I mean, what is this even? Some number formatting magic? I probably don’t even wanna know… Couldn’t find any way of having several groups for the iterator however. Meh. Still don’t like it.

So, well, it’s like this on Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do nice -n19 x264 --fps \
24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f \
-2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats $i/v.stats -t 2 --no-fast-pskip \
--cqm flat --non-deterministic --demuxer lavf $i/video.h264 -o $i/pass1.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && nice -n19 \
x264 --fps 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid \
normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats $i/v.stats -t 2 --no-fast-pskip --cqm flat \
--non-deterministic --demuxer lavf $i/video.h264 -o $i/pass2.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x264 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

The variable $MACHINE or %MACHINE%/!MACHINE! specifies the machines’ host name. This will be noted in the email, so I know which machine just completed something. $EPNUM – or %EPNUM%/!EPNUM! on Windows – is used for periodic updates on the shell. The output would be like “Pass 1 done for Episode 07/13 of AnimeX” in green on Windows and bold red on Linux/UNIX (just change the color to your liking).

Finally, $SERIES aka %SERIES%/!SERIES! would be the series’ name. So say, the UNIX machine named “BEAST” above is done with this loop. The email would come with the subject line “x264 notification from BEAST” and would read “AnimeX transcoding complete” in plain text. That’s all.

Please note, that cmd batch on Windows is extremely creepy. Every whitespace (especially the leading ones when doing multi-line like this for display) needs to be exactly where it is. The same goes for double quotes where you might think they aren’t needed. They are! Also, this needs delayed variable expansion once again, which is why we see variables like !EPNUM! instead of %EPNUM% and why it’s called in a subshell by running cmd /V /C.

On Linux/UNIX we don’t need to rely on some specific API like cmds’ SetConsoleTextAttribute() to print colors, as most terminals understand ANSI color codes.

And this is what it looks like for x265:

Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO avconv -r 24000/1001 -i %I\video.h264 -f yuv4mpegpipe ^
 -pix_fmt yuv420p -r 24000/1001 - 2>NUL | "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 ^
 --fps 24000/1001 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate ^
 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 ^
 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
 --pass 1 --slow-firstpass --stats %I\v.stats --sar 1 --range full -o %I\pass1.h265 & colorecho ^
 "Pass 1 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & avconv -r 24000/1001 -i ^
 %I\video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>;NUL | ^
 "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 --fps 24000/1001 -p veryslow --pmode ^
 --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 ^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2 --stats %I\v.stats --sar ^
 1 --range full -o %I\pass2.h265 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" ^
 10) & echo !SERIES! transcoding complete | blat - -t "myself@another.mailhost.com" -c ^
 "myself@mailhost.com" -s "x265 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET ^
 SERIES="

Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do avconv -r 24000/1001 -i \
$i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice -19 x265 - --y4m \
-D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 \
--rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq \
5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 \
--slow-firstpass --stats $i/v.stats --sar 1 --range full -o $i/pass1.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && avconv -r \
24000/1001 -i $i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice \
-19 x265 - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid \
--bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd \
1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 \
--pass 2 --stats $i/v.stats --sar 1 --range full -o $i/pass2.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x265 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

And that’s it. The loops for audio transcoding are simpler, as that part is so fast, it doesn’t need email notifications. Runs for minutes rather than days. When all is done, I’d usually fire up the MKVToolnix GUI, and prepare a mux for the first episode. There is a nice “copy command line to clipboard” function there when you click on “Muxing” after everything is set up. With that I can build another loop that muxes everything to final .mkv files. On Windows that part is more complicated if you want Unicode support, so I needed to create input files by using a generator I wrote in Perl for that, but that’s for another day… :)

Oh, and if you wanna ssh into your Linux or UNIX boxes from afar to check on your transcoders, consider launching them on a GNU screen] session. It’s immensely useful! Too bad it won’t work on the Windows cmd. :(

Jan 262016
 

HakuNeko logoSince I’ve started using FreeBSD as a Linux and Windows replacement, I’ve naturally always been looking at porting my “known good” software over to the UNIX OS, or at replacing it by something that gets the job done without getting on my nerves too much at the same time. For most parts other than TrueCrypt, that was quite achievable, even though I had to endure varying degrees of pain getting there. Now, my favorite Manga / Comic ripper on Windows, [HakuNeko] was the next piece of software on the list. It’s basically just a more advanced Manga website parser and downloader based on stuff like [cURL], [OpenSSL] or the [wxWidgets] GUI libraries.

I didn’t even know this until recently (shame on me for never looking closely), but HakuNeko is actually free software licensed under the MIT license. Unfortunately, the source code and build system are quite Linux- and Windows-centric, and there exist neither packages nor ports of it for FreeBSD UNIX. Actually, the code doesn’t even build on my CentOS 6.7 Linux right now (I have yet to figure out the exact problem), but I managed to fix it up so it can compile and work on FreeBSD! And here’s how, step by step:

1.) Prerequisites

Please note that from here on, terminal commands are shown in this form: $ command or # command. Commands starting with a $ are to be executed as a regular user, and those starting with # have to be executed as the superuser root.

Ok, this has been done on FreeBSD 10.2 x86_32 using HakuNeko 1.3.12, both are current at the time of writing. I guess it might work on older and future releases of FreeBSD with different releases of HakuNeko as well, but hey, who knows?! That having been said, you’ll need the following software on top of FreeBSD for the build system to work (I may have missed something here, if so, just install the missing stuff like shown below):

  • cURL
  • GNU sed
  • GNU find
  • bash
  • OpenSSL
  • wxWidgets 2.8.x

Stuff that’s not on your machine can be fetched and installed as root from the official package repository, Like e.g.: # pkg install gsed findutils bash wx28-gtk2 wx28-gtk2-common wx28-gtk2-contrib wx28-gtk2-contrib-common

Of course you’ll need the HakuNeko source code as well. You can get it from official sources (see the link in first paragraph) or download it directly from here in the version that I’ve used successfully. If you take my version, you need 7zip for FreeBSD as well: # pkg install p7zip.

Unpack it:

  • $ 7z x hakuneko_1.3.12_src.7z (My version)
  • $ tar -xzvf hakuneko_1.3.12_src.tar.gz (Official version)

The insides of my archive are just vanilla as well however, so you’ll still need to do all the modifications by yourself.

2.) Replace the shebang lines in all scripts which require it

Enter the unpacked source directory of HakuNeko and open the following scripts in your favorite text editor, then replace the leading shebang lines #!/bin/bash with #!/usr/local/bin/bash:

  • ./configure
  • ./config_clang.sh
  • ./config_default.sh
  • ./res/parsers/kissanime.sh

It’s always the first line in each of those scripts, see config_clang.sh for example:

  1. #!/bin/bash
  2.  
  3. # import setings from config-default
  4. . ./config_default.sh
  5.  
  6. # overwrite settings from config-default
  7.  
  8. CC="clang++"
  9. LD="clang++"

This would have to turn into the following (I also fixed that comment typo while I was at it):

  1. #!/usr/local/bin/bash
  2.  
  3. # import settings from config-default
  4. . ./config_default.sh
  5.  
  6. # overwrite settings from config-default
  7.  
  8. CC="clang++"
  9. LD="clang++"

3.) Replace all sed invocations with gsed invocations in all scripts which call sed

This is needed because FreeBSDs sed and Linux’ GNU sed aren’t exactly that compatible in how they’re being called, different options and all.

In the text editor vi, the expression :%s/sed /gsed /g can do this globally over an entire file (mind the whitespaces, don’t omit them!). Or just use a convenient graphical text editor like gedit or leafpad for searching and replacing all occasions. The following files need sed replaced with gsed:

  • ./configure
  • ./res/parsers/kissanime.sh

4.) Replace all find invocations with gfind invocations in ./configure

Same situation as above with GNU find, like :%s/find /gfind /g or so, but only in one file:

  • ./configure

5.) Fix the make check

This is rather cosmetic in nature as $ ./configure won’t die if this test fails, but you may still wish to fix this. Just replace the string make --version with gmake --version (there is only one occurrence) in:

  • ./configure

6.) Fix the DIST variables’ content

I don’t think that this is really necessary either, but while we’re at it… Change the DIST=linux default to DIST=FreeBSD in:

  • ./configure

Again, only one occurrence.

7.) Run ./configure to create the Makefile

Enough with that, let’s run the first part of the build tools:

  • $ ./configure --config-clang

Notice the --config-clang option? We could use GCC as well, but since clang is FreeBSDs new and default platform compiler, you should stick with that whenever feasible. It works for HakuNeko, so we’re gonna use the default compiler, which means you don’t need to install the entire GCC for just this.

There will be error messages looking quite intimidating, like the basic linker test failing, but you can safely ignore those. Has something to do with different function name prefixes in FreeBSDs libc (or whatever, I don’t really get it), but it doesn’t matter.

However, there is one detail that the script will get wrong, and that’s a part of our include path. So let’s handle that:

8.) Fix the includes in the CFLAGS in the Makefile

Find the line containing the string CFLAGS = -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread or similar in the newly created ./Makefile. After the option -O2 add the following: -I/usr/local/include. So it looks like this: CFLAGS = -c -Wall -O2 -I/usr/local/include -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread. That’s it for the Makefile.

9.) Fix the Linux-specific conditionals across the C++ source code

And now the real work starts, because we need to fix up portions of the C++ code itself as well. While the code would build and run fine on FreeBSD, those relevant parts are hidden behind some C++ preprocessor macros/conditionals looking for Linux instead. Thus, important parts of the code can’t even compile on FreeBSD, because the code only knows Linux and Windows. Fixing that isn’t extremely hard though, just a bit of copy, paste and/or modify. First of all, the following files need to be fixed:

  • ./src/MangaConnector.cpp
  • ./src/Logger.cpp
  • ./src/MangaDownloaderMain.cpp
  • ./src/MangaDownloaderConfiguration.cpp

Now, what you should look for are all conditional blocks which look like #ifdef __LINUX__. Each will end with an #endif line. Naturally, there are also #ifdef __WINDOWS__ blocks, but those don’t concern us, as we’re going to use the “Linux-specific” code, if you can call it that. Let me give you an example right out of MangaConnector.cpp, starting at line #20:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Now given that the Linux code builds just fine on FreeBSD, the most elegant and easier version would be to just alter all those #ifdef conditionals to inclusive #if defined ORs, so that they trigger for both Linux and FreeBSD. If you do this, the block from above would need to change to this:

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Should you ever want to create different code paths for Linux and FreeBSD, you can also just duplicate it. That way you could later make changes for just Linux or just FreeBSD separately:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif
  4. #ifdef __FreeBSD__
  5. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  6. #endif

Whichever way you choose, you’ll need to find and update every single one of those conditional blocks. There are three in Logger.cpp, three in MangaConnector.cpp, two in MangaDownloaderConfiguration.cpp and again three in MangaDownloaderMain.cpp. Some are more than 10 lines long, so make sure to not make any mistakes if duplicating them.

Note that you can maybe extend compatibility even further with additional directives like __OpenBSD__ or __NetBSD__ for additional BSDs or __unix__ for a wide range of UNIX systems like AIX or HP-UX. None of which has been tested by me of course.

When all of that is done, it’s compile and install time:

10.) Compile and install

You can compile as a regular user, but the installation needs root by default. I’ll assume you’ll want to install HakuNeko system-wide, so, we’ll leave the installation target directories on their defaults below /usr/local/. While sitting in the unpacked source directory, run:

  • $ gmake
  • # gmake install

If nothing starts to crash and burn, this should compile and install the code. clang will show some warnings during compilation, but you can safely ignore that.

11.) Start up the white kitty

The installation procedure will also conveniently update your window manager as well, if you’re using panels/menus. Here it’s Xfce4:

HakuNeko is showing up as an "Internet" tool

HakuNeko (“White Cat”) is showing up as an “Internet” tool. Makes sense.

With the modifications done properly it should fire up just fine after initializing its Manga connectors:

HakuNeko with the awesomeness that is "Gakkou Gurashi!" being selected from the HTTP source "MangaReader"

HakuNeko with the awesomeness that is “Gakkou Gurashi!” being selected from the HTTP source [MangaReader].

Recently the developers have also added [DynastyScans] as a source, which provides access to partially “rather juicy” Yuri Dōjinshi (self-published amateur and sometimes semi-professional works) of well-known Manga/Anime, if you’re into that. Yuri, that is (“girls love”). Mind you, not all, but a lot of the stuff on DynastyScans can be considered NSFW and likely 18+, just as a word of warning:

HakuNeko fetching a Yuru Yuri Dōjinshi from DynastyScans, bypassing their download limits by not fetching packaged ZIPs - it works perfectly!

HakuNeko fetching a Yuru Yuri Dōjinshi called “Secret Flowers” from DynastyScans, bypassing their download limits by not fetching packaged ZIPs – it works perfectly!

Together with a good comic book reader that can read both plain JPEG-filled folders and stuff like packaged .cbz files, HakuNeko makes FreeBSD a capable comic book / Manga reading system. My personal choice for a reader to accompany HakuNeko would be [QComicBook], which you can easily get on FreeBSD. There are others you can fetch from the official package repository as well though.

Final result:

HakuNeko and QComicBook make a good team on FreeBSD UNIX

HakuNeko and QComicBook make a good team on FreeBSD UNIX – I like the reader even more than ComicRack on Windows.

And, at the very end, one more thing, even though you’re likely going to be aware of this already: Just like Anime fansubs, fan-translated Manga or even Dōjinshi are sitting in a legal grey zone, as long as the book in question hasn’t been licensed in your country. It’s being tolerated, but if it does get licensed, ownership of a fan-translated version will likely become illegal, which means you should actually buy the stuff at that point in time.

Just wanted to have that said as well.

Should you have trouble building HakuNeko on FreeBSD 10 UNIX (maybe because I missed something), please let me know in the comments!

Jan 262015
 

FreeBSD battery logo[1] While I’m still using the free HP/Compaq nx6310 notebook a friend has given me, I ran into a little problem during actual productive use with FreeBSD 10.1 UNIX. A problem that also affects more modern HP notebooks like the EliteBook or ProBook series as well as certain Macbooks as I learned only later. The issue lies within the battery status reporting. While the very first state transition will be recognized, as in say you disconnect the AC cable, or the charge drops by 1% or whatever, subsequent changes are completely ignored.

An example: You want to join a meeting with 100% charge. You disconnect the AC cable, an event that is still recognized by the system. The charge may then stand at 99%. It would however not update further, but just stay there until either a reboot or until the lights just go dark. Not good if you intend to actually use the system on the go. The user just needs to be aware of the battery level!

1.) The right direction, but still a fruitless attempt

So I read that HP has quite some share of trouble with Windows-specific and also buggy ACPI BIOS code, and thus, this has been a pain for some time, not just on FreeBSD, but also Linux and in some cases even Windows.

At first I [reported the issue in the FreeBSD forums], after which I was asked to report the problem on the [freebsd-acpi@freebsd.org] mailing list. Problem only was that one can’t post anonymously and the registration never worked for me. So I kept digging in the dirt, and my first attempt was to fix the ACPI code of course, or DSDT as it’s called – the Differentiated System Description Table, written in ASL – the ACPI Source Language. Basically a pretty creepy byte code language providing a bidirectional interface between the operating system and the systems BIOS/UEFI, which in turn controls certain aspects of the hardware. Think display brightness adjustments, WiFi on/off switches, audio volume control via keyboard, sleep buttons etc. – all that is ACPI. You can find the specs [here] if you really want to take a look.

HP compiled the code using [Microsofts ASL compiler], which is known to be rather sloppy and ignore tons of bugs in the code when compiling. So my first step was to dump the active DSDT from the systems memory using acpidump and tell the tool to disassemble it using Intels ASL compiler iasl in the same step:

# acpidump -d &gt; ~/nx6310-stock.asl

Then, I attempted to recompile using the Intel ASL compiler again:

# iasl ~/nx6310-stock.asl

And then came the errors and warnings. Plus some nasty OS-specific code. Since an operating system can and will identify itself to the BIOS via ACPI, one can unfortunately filter by operating system on the BIOS level. Very bad stuff. Like this, taken from the ASL code directly, as written by HP:

expand/collapse source code
  1. Name (C015, Package (0x08)
  2. {
  3.      "Microsoft Windows",
  4.      "Microsoft WindowsME: Millennium Edition",
  5.      "Microsoft Windows NT"
  6. })

…and:

  1. If (LOr (LEqual (C014, 0x00), LEqual (C014, 0x03)))
  2. {
  3.      If (CondRefOf (\_OSI, Local0))
  4.      {
  5.           If (\_OSI ("Windows 2001"))
  6.           {
  7.                Store (0x04, C014)
  8.           }
  9.  
  10.           If (\_OSI ("Windows 2001 SP1"))
  11.           {
  12.                Store (0x04, C014)
  13.           }
  14.  
  15.           If (\_OSI ("Windows 2001 SP2"))
  16.           {
  17.                Store (0x05, C014)
  18.           }
  19.  
  20.           If (\_OSI ("Windows 2006"))
  21.           {
  22.                Store (0x06, C014)
  23.           }
  24.      }
  25. }

Next logical step was to fake the OS FreeBSD was reporting to the ACPI subsystem. You can either do that by changing and embedding the ASL (non-MS OS strings are for instance “FreeBSD” or “Linux”), or luckily also by adding something like the following line to /boot/loader.conf, after which you need to reboot:

hw.acpi.osname="Windows 2006"

That didn’t do the trick for me however, so I tried to run my modified ACPI code after all. To do that you need to recompile the ASL, and then have loader pick it up in the early boot sequence of the kernel. First, I compiled the fixed code:

# iasl -tc ~/nx6310.asl

The result will be /tmp/acpidump.aml (for whatever reason). I placed it in /boot/ and added the following to /boot/loader.conf:

  1. acpi_dsdt_load="YES"
  2. acpi_dsdt_name="/boot/nx6310.aml"

Now another reboot. To verify the AML gets loaded, boot the kernel in verbose mode, you can choose that option in the boot loader. It’ll show something like Preloaded acpi_dsdt "/boot/nx6310.aml" at 0xc18dcc34. in /var/log/messages.

In my case however, while the AML was indeed being loaded, the BIOS’ tables were strangely not overwritten. I never found out why. Dumping them again would just give me the bugged code by HP. So that seemed to be dead end.

2.) The solution

Then I discovered that this might actually be a problem with the FreeBSD kernel itself by stumbling over [problem report 162859] in FreeBSDs bugzilla. It seems there was a problematic commit to the kernels ACPI subsystem written by Jung-uk Kim, initially for FreeBSD 9.0 in 2011: [r216942]. By now, he had provided two patches for it, of which I tried the [newer one] unsuccessfully, making the problem even worse. You can see my replies under my real name “Michael Lackner” in that PR.

Luckily, it was seemingly my complaint on top of the others that made Jung-uk Kim revert the [original patch], all back to how it was in FreeBSD 8.0, which some people reported to work just fine. So I got the now reverted/fixed code from [r277579] – a file named acpi_ec.c, and replaced /usr/src/sys/dev/acpica/acpi_ec.c with it. Then, I recompiled the FreeBSD kernel [as described in the FreeBSD handbook] once more (fear not, it’s actually very easy).

Reboot, and everything just worked! No need for a DSDT replacement or operating system faking even. I tested charge, discharge, AC cable dis-/reconnect and full charge states. All ok now! Finally, that book is now 100% ready for production use! :)

I mean, let’s not make any mistake here, the ACPI code from HP is still buggy. But like the Linux kernel, the FreeBSD kernel knows how to sail around the most severe bugs, including Microsoft-specific code in certain cases, at least now that this issue is fixed.

Thanks fly out to Jung-uk Kim for reverting the problematic patch! It should be merged from current any day now. If you want to get it done faster than that, just fetch acpi_ec.c, place it in the right folder and recompile the kernel and with it all of its modules, just like I did.

[1] Original battery icon used for the logo is © Freepik from Flaticon and is licensed under CC BY 3.0

Dec 112014
 

FreeBSD + WineSince I’ve abandoned OpenBSD for FreeBSD in my recent attempts to actually use a ‘real’ UNIX system, and all that just because of Wine so I can use some small Windows tools I need (yeah…), I was a bit fed up with Wines’ font anti-aliasing not working. I remember having a similar problem on CentOS Linux some while back, but I can’t remember how I solved it on that OS. Thing is, I just want to document my solution here briefly, so I don’t forget it, if I have to re-do this any time soon. The problem seems to originate from the X11 render extension that is being used for compositing on X11 without any additional compositing engine like Compiz. Wines usage of the extension is actually controlled by its system registry, and the key that seems to need editing is HKEY_CURRENT_USER\Software\Wine\X11 Driver\ClientSideWithRender.

Interestingly, people suggested to switch the use of the X render extension off instead of on, but when inspecting my setup by running wine regedit, I found I didn’t even have the key, and its default is off! So what saved me was:

  1. [HKEY_CURRENT_USER\Software\Wine\X11 Driver]
  2. "ClientSideWithRender"="Y"

Setting this back to “N” is basically the same thing as just deleting the key altogether, at least with the current default configuration of my Wine version on FreeBSD, which is wine-1.6.2. See how it looks like with no anti-aliasing (click to enlarge):

Wine running its regedit on FreeBSD 10 with no font AA

Wine running its regedit on FreeBSD 10 with no font AA

And here with proper AA, again, please click to enlarge so you can see the effect applied:

Wine with proper font anti-aliasing

Wine with proper font anti-aliasing

To demonstrate the effect more prominently, let’s look at some zoomed in versions, click to enlarge again for all the glory or so:

No anti-aliased fonts here, all jagged

No anti-aliased fonts here, all jagged

And:

Wine font anti-aliasing up close, nice and smooth

Wine font anti-aliasing up close, nice and smooth

Now I heard that some people actually prefer the clearer look of jagged fonts, or they may prefer pure gray smoothing, but I actually like the look of this smoothing a lot more. This is with medium font hinting turned on in Xfce4, and it appears very smooth and nice to my eyes.

If you want to try this in case you’re offended by jagged fonts using wine, just take the following:

  1. [HKEY_CURRENT_USER\Software\Wine\X11 Driver]
  2. "ClientSideWithRender"="Y"

Save it into a file like wine-Xrender-fontAA.reg, and then import it into your wines registry database by opening a terminal, going into the directory where your reg file sits, and by running: wine regedit ./wine-Xrender-fontAA.reg. Then restart your wine application and it should work for 99% of the fonts. I’ve encountered one font in Bruce Schneiers [PasswordSafe] that wouldn’t anti-alias, ever. But it’s the same on Windows, so I’m guessing that’s ok.

Switching it off is just as easy, just edit the file, change “Y” to “N” and re-run the wine regedit command. But as I said, I’ll keep it, no more eye cancer when starting Win32 applications on FreeBSD. :)