Nov 192016
 

FreeBSD GMABoost logoRecently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
 
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call gmaboost.sh. I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  2.  
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  13.  
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  16.  
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # gmaboost.sh 200 or # gmaboost.sh 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Sep 072016
 

TeamViewer on Linux logoI’m not exactly a big fan of TeamViewer, since you’ll never know what’s going to happen with that traffic of yours, so I prefer VNC over SSH instead. A few weeks ago I got TeamViewer access to a remote workstation machine for the purpose of processing A/V files however. Basically, it was about video and audio transcoding on said machine.

Since the stream meta data (like the language of an audio stream) wasn’t always there, I wanted to check it by playing back the files remotely in foobar2000 or MPC-HC. TeamViewer does offer a feature to relay the audio from a remote machine to your local box, as long as the remote server has some kind of soundcard / sound chip installed. I was using TeamViewer 11 – the newest version at the time of writing – to connect from CentOS 6.8 Linux to a Windows 7 Professional machine. Playing back audio yielded nothing but silence though.

Now, TeamViewer is actually not native Linux software. Both its Linux and MacOS X versions come with a bundled Wine 1.6 distribution preconfigured to run the 32-bit TeamViewer Windows binary. It was thus logical to assume that the configuration of TeamViewers’ built-in Wine was broken. This may happen in cases where you upgrade TeamViewer from previous releases (which is what I had done, 7 -> 8 -> 9 -> 11).

There are a multitude of proposed solutions to fix this, and since none of them worked for me as-is, I’d like to add my own to the mix. The first useful hint came from [here]. You absolutely need a working system-wide Wine setup for this. I already had one that I needed for work anyway, namely Wine 1.8.6 from the [EPEL] repository, configured using [winetricks]. We’re going to take some files from that installation and essentially replace TeamViewers’ own Wine with the one distributed by EPEL.

So I had TeamViewer 11 installed in /opt/teamviewer/ and some important configuration files for it in ~/.local/share/teamviewer11/ and ~/.config/teamviewer/. First, we backup the wine files of TeamViewer and replace them with the platform ones (the paths may vary depending on your Linux distribution, but the file names should not):

# mv /opt/teamviewer/tv_bin/wine/bin/wine /opt/teamviewer/tv_bin/wine/bin/wine.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wineserver /opt/teamviewer/tv_bin/wine/bin/wineserver.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/wine-preloader.BACKUP
# mv /opt/teamviewer/tv_bin/wine/lib/libwine.so.1.0 /opt/teamviewer/tv_bin/wine/lib/libwine.so.1.0.BACKUP
# mv /opt/teamviewer/tv_bin/wine/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/wine.BACKUP/
# cp /usr/bin/wine /usr/bin/wineserver /usr/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/
# cp /usr/lib/libwine.so.1.0 /opt/teamviewer/tv_bin/wine/lib/
# cp -r /usr/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/

This will replace all the binaries and libraries, in my case shoving Wine 1.8.6 underneath TeamViewer. This isn’t all that’s needed however. We’ll also need the system registry hive of your working Wine installation (with sound). That should be stored in ~/.wine/system.reg! Let’s replace TeamViewers’ own hive with this one:

$ mv ~/.local/share/teamviewer11/system.reg ~/.local/share/teamviewer11/system.reg.BACKUP
$ cp ~/.wine/system.reg ~/.local/share/teamviewer11/

Ok, and the final part is adding the proper Linux audio backend to this Wines’ configuration. That part is stored in ~/.wine/user.reg. Replacing the whole file didn’t work for me though, as TeamViewer would crash upon launch, probably missing some keys from its own user.reg. So, let’s just edit its file instead, open ~/.local/share/teamviewer11/system.reg with your favorite text editor and add the following line in a proper location (it’s sorted alphabetically):

[Software\\Wine\\Drivers\\winepulse.drv] 1473239241

The corresponding file should be found within TeamViewers’ replaced Wine distribution now by the way, in my case it’s /opt/teamviewer/tv_bin/wine/lib/wine/fakedlls/winepulse.drv.

Now, run the TeamViewer profile updater (Some people say it’s required to make this work, it wasn’t for me, but it didn’t hurt either): $ /opt/teamviewer/tv_bin/TeamViewer --update-profile and then its’ Wine configuration: $ /opt/teamviewer/tv_bin/TeamViewer --winecfg. After that, you should be greeted with this:

TeamViewer 11 running its own winecfg

TeamViewer 11 running its own copy of winecfg.

Before the modifications, the configuration window would show “None” as the driver, without any way to change it. So no audio, whereas we have Pulseaudio now. Press “Test Sound” if you want to check whether it truly works. I haven’t tested the ALSA backend by the way. In my case, as soon as the registry was fixed, Wine just autoselected Pulseaudio, which is fine for me.

Now launch TeamViewer and check out the audio options in this submenu:

TeamViewer preferences

The TeamViewer 11 preferences can be found here.

It should look like this:

TeamViewers audio options

Make sure “Play computer sounds and music” is checked! (click to enlarge)

Now, after having connected and logged in, you may also wish to verify the conference audio settings in TeamViewers’ top menu:

TeamViewer 11 conference audio settings

TeamViewer 11 conference audio settings, make sure “Computer sound” is checked!

When you play a sound file on the remote computer, you should hear it on your local one as well. With that, I can finally test the audio files I’m supposed to use on that remote machine for their actual language (which is a rather important detail) where meta data isn’t available.

This seems to be a problem of TeamViewers installation / update procedure which hasn’t been addressed for several major released now. I presume just removing all traces of TeamViewer and installing it from scratch might also do the trick, but I didn’t try it for myself.

Ah, and one more thing: If you can’t launch TeamViewer on CentOS 6.x because you’re getting the following error…

teamviewerd error

TeamViewer Daemon not running…

…forget about the solutions on the web on top of what this message is telling you. TeamViewer 11 uses a systemd-style script for launching its daemon on Linux now, and that won’t do on SysV init systems. Just become root and launch the crap manually: # /opt/teamviewer/tv_bin/teamviewerd &, then press <CTRL>+<d> and it works!

Let’s hope that daemon isn’t doing anything evil while running as root. :roll:

Aug 052016
 

VirtualDimension logoShort story: It’s [VirtualDimension].

Long story… It’s most definitely not what Microsoft added in Windows 10. Besides it being limited to Windows 10, it just sucks for a multitude of reasons. And there I was, having hopes for it as well… If you’ve ever used multiple desktops on a graphical, X11-based Linux or UNIX window manager / desktop environment, you’d know what I’m talking about. Usually, what you’d get on those systems, whether KDE, Gnome, Xfce4, LXDE, or whatever is just one small, configurable panel which allows you to control multiple virtual desktops. On my Gnome 2 on CentOS 6.8 Linux, it looks like this (others are very similar):

Virtual desktops on Gnome 2

Virtual desktops on Gnome 2

The leftmost desktop is my usual “Internet” environment, here I chat, read emails, browse the web for anything work-related and so on. The second one is a Linux distribution development desktop. Here I’m building a derivative of Klaus Knoppers’ [Knoppix] distro. Then comes the testing environment for said distribution on desktop #3. Usually some shells and one VMware Player instance. Next to that are two more VM desktops for software testing and for writing user guides for software installation on different operating systems. At the moment that’s MacOS X and a Windows XP x64 software build VM. Usually there’s also a Windows 7 one. One is empty (for arbitrary stuff), then comes the server administration desktop with 9 open shells, one for each server. And the last one is my private desktop with yet another web browser, and some shells for spawning screen sessions for running software compilations, encoding runs and the likes.

Now, I have a 30″ screen both at home and at work, resolution is 2560×1600. But it’s just never enough screen real estate. So I wanted well-integrated virtual desktops for Windows as well, but last time I tried out some software, I couldn’t find a good one. Recently, I tried again for some reason, like “let’s give this one last shot”. And I tried a lot of programs!

Among the software tested were [Dexpot], [Finestra], [VirtuaWin], [WindowsPager], Xilisoft [Multiple Desktops] and the Windows PowerToy predecessor of [Desktops 2.0] written by Windows Hacker Mark Russinovich and Bryce Cogswell. And finally, [VirtualDimension]. Some of those are free and open source software, others are not.

One of my primary requirements was compatibility to Windows XP x64. Of course it’d be nice if it worked on Windows 10 as well. But most of the above had important features missing or severely misbehaving on XP. Some were just very, very sluggish when switching desktops. Others had missing features to begin with, like previews on the desktop tiles. A blank desktop tile doesn’t help at all, as I need to see roughly what’s running where at a glance.

I’m not gonna make this a lengthy top list or anything, I’m just gonna show you what the software of my choice – VirtualDimension – could do for me, let’s look at the tiles first:

VirtualDimension on XP x64

VirtualDimension on XP x64

We’ll start with my good old XP x64 first. Here you can see my system tray, and Miranda being open. VirtualDimension cannot be embedded into the taskbar properly (damn), but it has an “always on top” feature. Since the contact list in my docked and always-launched Miranda doesn’t go all the way down, there is free and unused space there. Perfect for VirtualDimension! And since it’s always on top, it doesn’t disappear when clicking on Miranda for chatting.

Given the source code is definitely coming from a UNIX or Linux user (given he built it with GCC/Mingw), some features immediately ring a bell. Like “mouse warp”, where you switch desktops by moving your mouse to the border of the screen. I disabled that, don’t like it. But yeah, it’s there.

Important: While it doesn’t give you live window geometry previews, it does give you iconized previews, so you can always identify any desktop quickly by seeing what’s running there. The desktops can also be named, and there is an OSD that you can have pop up on you when switching, like so:

VirtualDimension OSD

OSD showing right after a desktop switch

In this case I had just switched to desktop #2, which is for A/V processing exclusively. This is just the top left part of the screen, where one of my eight transcoding shells was running a x265 benchmark prototype test. Color, display duration, transparency to mouse clicks on the OSD part, font and size are configurable.

Also, you can freely define keyboard shortcuts for switching desktops as well. I chose CTRL+Shift+Right as well as CTRL+Shift+Left for switching desktops and Alt+Right / Alt+Left for pushing a window to the next/previous desktop as those don’t conflict with other shortcuts I’m using.

What else can it do? Let’s right click on one of the icons in the preview tiles:

VirtualDimension iconized window right click

Clicking on a program icon in VirtualDimensions’ preview tiles gives you this menu

The first five options from the top are global ones. However, the ones below are specific to the icon you right-clicked. With “Activate”, you’d switch to the target desktop and put focus on that programs’ window. The others are pretty self-explanatory as well I guess. We also get a graceful “Close” option, and a brutal “Kill” option that’s equivalent to murdering the process in task manager. Maybe useful since it’s faster that way.

And if we click on the free area?

VirtualDimension, right click on the free area of a preview tile

Right-clicking on the free area of a preview tile gives you a list of all programs on that desktop.

Ok, not sure how useful that is, but at least it may help with identifying the windows on a desktop in more detail, as you get the window titles here. For my encoding shells I could get very quick glance at the progress, but not exactly in great detail. So the helpfulness of this is limited.

What else?

Well, it’s extremely fast! That’s one major plus for VirtualDimension, as several of the other solutions (open source ones as well) were abysmally slow, at least on XP x64. But damn, VirtualDimension just flies! And its memory footprint is minimal. I saw less than 12MiB of consumption here. Even if you add a truckload of Desktops (there seems to be no upper limit), it just won’t slow down unless you spawn like 50 CPU intensive processes all over the place killing your CPU or maxing out your RAM. But that wouldn’t have been VirtualDimensions fault then. Its memory footprint will linearly grow by spawning more desktops, so with eight you may see around 20MiB. Still neat.

And what’s bad about it?

Well, sometimes, if you have a lot of windows on one desktop, the icons are’t cut off in the right spot at the bottom of the preview tile, so they overflow just a little bit. Just a cosmetic issue. Also, you should maybe deactivate the shell integration. With this, VirtualDimension hooks itself into all windows (such a DLL hook means entering another processes’ memory area). With that, you can get its functions via right clicking on a windows’ title bar, like on UNIX.

Nice, but dangerous! This can trigger anti-cheat systems in online games, because they really don’t like you stepping into their processes’ memory areas! That’s what cheating tools do to modify a games’ parameters on the fly as well. You don’t wanna be banned because of such a thing!

In my case, I managed to lock myself out of Mechwarrior Online because of this. I wasn’t banned, but the login process wouldn’t even let me launch its window. Disabling the feature, launching MWO, then re-enabling it and trying to log in caused a pretty abnormal process termination:

Mechwarrior Online really doesn't like VirtualDimensions' shell integration feature

Mechwarrior Online really doesn’t like VirtualDimensions’ shell integration feature! And no, there was no “update available”… (click to enlarge)

There is an exception list for this feature, and I added all of MWOs’ .exe files to it, to no avail. Better to stay away from this one.

Now, well, this otherwise beautiful piece of software was dropped by its developer around 2005. About the time my XP x64 came out. Latest alpha build is from some time in 2006. So this is ancient! It even supports Windows 98 and NT 4.0, I mean… So, how about Windows 10 then? I mean, Windows 10 doesn’t even have a GDI UI anymore, this is like one completely different world. Since I do have a Windows 10 machine (yeah, ew), let’s check it out:

VirtualDimension on Windows 10

VirtualDimension on Windows 10 – hey, it really works!?

Miranda seemingly can’t dock properly on Windows 10 anymore. It kinda… floats near the desktop border when docked. It’s strange and it wastes space, but well, I don’t know how to fix that yet. But anyway, I embedded VirtualDimension into Miranda (by just moving the window there, removing its title bar and resizing it properly again). And guess what?

It just works™!

I launched some Metro / Modern UI apps in a window as well, and while those aren’t shown in the preview tiles, they can be controlled with keyboard shortcuts, just like regular windows. Also, it’s just as blazing fast as it is on XP. Ah, and… yeah, it actually does work on all 64-bit x86 Windows versions it seems! It’s amazing, but an ancient piece of 32-bit software that does alter a Windows systems’ usage pattern quite fundamentally still works fine on Windows 10 64-bit. I gotta say, I’m pretty relieved, because Windows 10s’ own solution just sucks – where is my live preview? – and I don’t want to change my usage paradigm too much when switching operating systems (even from Linux/UNIX to Windows and back).

Some of the other solutions like Dexpot or Finestra may be faster on Windows 10 then they are on a just half-supported XP x64, but nah, don’t need them.

VirtualDimension is as perfect as it gets, despite its age! Or maybe because of it?

Still, anyone interested in picking up that [VirtualDimension project on SourceForge] and in continuing its development? ;) I guess I can’t touch that code, would probably just mess everything up. Ah, it’s C++ by the way…

A few things could use some fixing, like the icon overflowing issue, Modern UI window detection, certain, rare windows being sticky on all Desktops even if nobody told them to do so (Miranda, X-Chat DCC windows) and that exclusion list for the shell integration, which doesn’t hook into all windows properly when active either anyway.

Would be so nice if somebody could continue working this! :)

Until that happens (I know, it never will), I’ll just continue using v0.95 alpha. ;)

Feb 252016
 

SlySoft logoIt’s over – after 13 years of being almost constantly under pressure by US-based companies, SlySoft finally had to close its doors. Most notably known for software such as CloneCD or AnyDVD, the Antiguan-based company has provided people all over the world with ways to quickly and easily circumvent disc-based copy protection mechanisms such as Sony ArcCos, CSS, ACSS or BD+ and many others for years.

The companys’ founder, a certain Mr. Giancarla Bettini had already been sued – and successfully so – before an Antiguan court. While it was strictly up to Antiguan Authorities to actually sue SlySoft (because the AACS-LA could not do so themselves due to some legal constraints), this did finally happen in 2012, fining Mr. Bettini for a sum of USD $30.000. That didn’t result in SlySoft closing down however.

What it was that happened exactly a few days ago is unclear, as SlySoft seems to be under NDA or maybe legal pressure as to not release any statement regarding the reasons for the shutdown, quote, “We were not allowed to respond to any request nor to post any statement”. The only thing that we have besides a forum thread with next to zero information is the statement on the official website, which is rather concise as well:

“Due to recent regulatory requirements we have had to cease all activities relating to SlySoft Inc.
We wish to thank our loyal customers/clients for their patronage over the years.”

It should be relatively clear however, that this has to have something to do with the AACS-LA and several movie studios as well as software and hardware companies “reminding” the United States government of SlySofts illicit activities just recently. This would’ve resulted in Antigua being put onto the US priority [watchlist] of countries violating US/international copyright laws. Ultimately, being put onto that list can result in trade barriers being put up within a short time, hurting a countrys’ economy, thus escalating the whole SlySoft thing to an international incident. More information [here].

AnyDVD HD

This little program and its little brothers made it all the way to the top and became an international incident! Quite the career…

It seems – and here is where my pure speculations start – that there was some kind of agreement found between SlySofts’ founder and the AACS-LA and/or the Antiguan and US governments resulting in the immediate shutdown of SlySoft without further consequences for either its founder or other members of the company. If true then SlySoft will surely also have to break their promise of releasing a “final” version of AnyDVD HD including all the decryption keys from the online database in case they have to close their doors forever. This is, what “[…] we have had to cease all activities relating to SlySoft Inc. […]” means after all.

So what are the consequences, technically?

I can only say for AnyDVD HD as according to the forums over at SlySoft, but the latest version 7.6.9.0 supposedly includes some 130.000 AACS keys and should still be able to decrypt a lot of Blu-Rays, even if not all of them.

In the end however, the situation can only deteriorate as time passes and new versions of AACS keys and BD+ certificates are being released, even if you bypass the removed DNS A-Records of key.slysoft.com and access one of the key servers by resolving the IP address locally (via your hosts file). Thing is, nobody can tell when SlySoft will be forced to implement more effective methods of making their services inaccessible, like by just switching off the machines themselves.

But even if they stay online for years to come, no new keys or certificates are going to be added, so it’s probably safe to say that the red fox is truly dead.

AddendumJust to be clear for those of you who are scared of even accessing any SlySoft machines with their real IPs any longer; According to a SlySoft employee (you can read it in their forums), all of the servers are still 100% under SlySofts physical control, and their storage backends are encrypted. They were not raided or anything. So it seems you do not have to fear “somebody else listening” on SlySofts key servers.

PS.: A sad day if you ask me, a victorious one if you ask the movie industry. Maybe somebody should just walk over and tell them, that cheaper, DRM-free media actually work a lot better on the market, when compared to jailing users into some “trusted” (by them) black boxes with forced software updates and closed software. Yeah, I actually want to play my BD movies on the PC (legally!!), and on systems based on free software like Linux and BSD UNIX as well, not on some blackboxed HW player, so go suck it down, Hollywood. I mean, I’m even BUYING your shit, for Christs’ sake…

Oh, by the way, China is actually sitting on that copyright watchlist (I mean, obviously), and they gave us DVDFab. Also, there are MakeMKV and [others as well]. We’ll see whether the AACS-LA can hunt them all down… And even if they can… Will it really make them more money? Debatable at best…

Red Fox logoUpdate: Those guys work fast! While SlySoft is gone, several of the developers have grabbed the software and moved the servers to Belize, the discussion forums have already been migrated and a new version 7.6.9.1 of AnyDVD HD has been released, including new keys and reconfigured to access the new key servers as well. The company is now called “Red Fox” and the forums can be accessed via [forum.redfox.bz].

By now, AnyDVD HD respects the old licenses as well, and this will stay this way for the transition period. Ultimately however, according to posts on the forums, people will have to buy new licenses, even if they had a lifetime license before. They also said they’ll cook up “something nice” for people who bought licenses just recently. Probably some kind of discount I presume.

Still, if I may quote one of the developers: “SlySoft is dead, long live RedFox!”

Feb 202016
 

H.265/HEVC logoRecently, after [successfully compiling] the next generation x265 H.265/HEVC video encoder on Windows, Linux and FreeBSD, I decided to ask for guidance when it comes to compressing Anime (live action will follow at a later time) in the Doom9 forums, [see here]. Thing is, I didn’t understand all of the knobs x265 has to offer, and some of the convenient presets of x264 didn’t exist here (like --tune film and --tune animation). So for a newbie it can be quite hard to make x265 perform well without sacrificing far too much CPU power, as x265 is significantly more taxing on the processor than x264.

Thanks to [Asmodian] and [MeteorRain]/[LittlePox] I got rid of x265s’ blurring issues, and I took their settings and turned them up to achieve more quality while staying within sane encoding times. My goal was to be able to encode 1080p ~24fps videos on an Intel Xeon X5690 hexcore @ 3.6GHz all-core boost clock at >=1fps for a target bitrate of 2.5Mbit.

In this post, I’d like to compare 7 scenes from the highly opulent Anime [The Garden of Words] (言の葉の庭) by [Makoto Shinkai] (新海 誠) at three different average bitrates, 1Mbit, 2.5Mbit (my current x264 default) and 5Mbit. The Blu-Ray source material is H.264/AVC at roughly 28Mbit on average. Also, both encoders are running in 10-bit color depth mode instead of the common 8-bit, meaning that the internal arithmetic precision is boosted from 8- to 16-bit integers as well. While somewhat “non-standard” for H.264/AVC, this is officially supported by H.265/HEVC for Blu-Ray 4K. The mode of operation is 2-pass to aim for comparable file sizes and bitrates. The encoding speed penalty for switching from x264 to x265 at the given settings is around a factor of 8. Somewhat.

The screenshots below are losslessly compressed 1920×1080 images. Since this is all about compression, I chose to serve the large versions of the images in WebP format to all browsers which support it (Opera 11+, Chromium-based Browsers like Chrome, Iron, Vivaldi, the Android Browser or Pale Moon as the only Gecko browser). This is done, because at maximum level, WebP does lossless compression much more efficiently, so the pictures are smaller. This helps, because my server only has 8Mbit/s upstream. If your browser doesn’t support WebP (like Firefox, IE, Edge, Safari), it’ll be fed lossless PNG instead. All of this happens automatically, you don’t need to do anything!

Now, let’s start with the specs.

Specifications:

Here are the source material encoding settings according to the video stream header:

cabac=1 / ref=4 / deblock=1:1:1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=0.40:0.00 /
mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 /
fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=1 / sliced_threads=0 / slices=4 /
nr=0 / decimate=1 / interlaced=0 / bluray_compat=1 / constrained_intra=0 / bframes=3 / b_pyramid=1 /
b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=1 / weightp=1 / keyint=24 / keyint_min=1 /
scenecut=40 / intra_refresh=0 / rc_lookahead=24 / rc=2pass / mbtree=1 / bitrate=28229 /
ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / cplxblur=20.0 / qblur=0.5 /
vbv_maxrate=31600 / vbv_bufsize=30000 / nal_hrd=vbr / filler=0 / ip_ratio=1.40 / aq=1:0.60

x264 10-bit encoding settings (pass 1 & pass 2), 2.5Mbit example:

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats v.stats -t 2 --no-fast-pskip --cqm flat
--non-deterministic

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 2500 --aq-mode 1 -p 2 --stats v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic

x265 10-bit encoding settings (pass 1 & pass 2), 2.5Mbit example:

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 2500 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1
--slow-firstpass --stats v.stats --sar 1 --range full

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 2500 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2
--stats v.stats --sar 1 --range full

Since x265 can only read raw YUV and Y4M, the source video is being fed to it via [libavs’] avconv tool, piping it into x265. The avconv commandline for that looks as follows:

$ avconv -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null

If you want to do something similar, but you don’t like avconv, you can use [ffmpeg] as a replacement, the options are completely the same. Note that you should always specify the correct frame rates (-r) for input and output, or the bitrate setting of the encoder will be applied wrongly!

x264 on the other hand was linked against libav directly, using its decoding capabilities without any workarounds.

Let’s compare:

“The Garden of Words” has a lot of rain. This is a central story element of the 46 minute movie, and it’s hard on any encoder, because a lot of stuff is moving on screen all the time. Let’s take a look at such a scene for our first side-by-side comparison. Each comparison is done in two rows: H.264/AVC first (including the source material), and below that H.265/HEVC, also including the source.

Let’s go:

Scene 1, H.264/AVC encoded by x264 0.148.x:

Scene 1, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

It has been said that x265 performs specifically well at two things: Very high resolutions (which we don’t have here) and low bitrates. And yep, it shows. When comparing the 1Mbit shots, it becomes clear pretty quickly that x265 manages to preserve more detail for the parts with lots of motion. x264 on the other hand starts to wash out the scene pretty severely, smearing out some raindrops, spray water and parts of the foliage. Also, it’s pretty bad around the outlines as well, but that’s true for both encoders. You can spot that easily with all the aliasing artifacts on the raindrops.

Moving up a notch, it becomes very hard to distinguish between the two. When zooming in you can still spot a few minor differences (note the kids umbrella, even if it’s not marked), but it’s quite negligible. Here I’m already starting to think x265 might not be worth it in all cases. There are still differences between the two 2.5Mbit shots and the original however, see the red areas of the umbrella and the most low-contrast, dark parts of the foliage.

At 5Mbit, I really can’t see any difference anymore. Maybe the colors are a little off or something, but when seen in motion, distinguishing between the two and the original becomes virtually impossible. Given that we just threw a really difficult scene at x264 and x265, this should be a trend to continue throughout the whole test.

Now, even more rain:

Scene 2, H.264/AVC encoded by x264 0.148.x:

Scene 2, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Now this is extreme at 1Mbit! Looking at H.264, the spray water on top of the cable can’t even be told apart from the cloud in the background anymore. Detail loss all over the scene is catastrophic in comparison to the original. Tons of raindrops are simply gone entirely, and the texture details on the tower and the angled brick wall of the house to the left? Almost completely washed out and smeared.

Now, let’s look at H.265 @ 1Mbit. The spray water is also pretty bad, but it’s amazing how much more detail was preserved overall. Sure, there are still parts of the raindrops missing, but it’s much, much closer to the original. We can now see details on the walls as well, even the steep angle one on the left. The only serious issue is the red light bleeding at the tower. There is very little red there in the original, so I’m not sure what happened there. x264 does this as well, but x265 is a bit worse.

At the next level, the differences are less pronounced again, but there is still a significant enough improvement when going from x264 to x265 at 2.5Mbit: The spray water on the cable becomes more well-defined, and more rain is being preserved. Also, the textures on the walls are a tiny little bit more detailed and crisp. Once again though, x265 is bleeding too much red light at the tower.

Since it’s noticeably not fully on the level of the source still, let’s look at 5Mbit briefly. x265 is able to preserve a tiny little bit more rain detail here, coming extremely close to the original. In motion, you can’t really see the difference however.

Now, let’s get steamy:

Scene 3, H.264/AVC encoded by x264 0.148.x:

Scene 3, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

1Mbit first again: Let me just say: It’s ugly. x264 pretty much messes up the steam coming from the iron. We get lots of block artifacts now. Some of the low-contrast patterns on the ironing board are being smeared out at a pretty terrible level. Also, the bokeh background partly shows block artifacts and banding. x265 produces quite a lot of banding here itself, but no blocks. Also, outlines and sharp contrasts are more well-defined, and the low contrast part is done noticeably better.

At 2.5Mbit, the patterns repeat themselves now. The steam is only slightly better with x265, outlines are slightly more well-defined, and the low-contrast patterns are slightly more visible. For some of the blurred parts, x265 seems to be a tiny little bit to prone to banding though, in a very few spots, x264 might be just that 1% better. Overall however, x265 wins this, and even if it’s just for the better outlines.

At 5Mbit, you really need to zoom in and analyze very small details, e.g. around the outer border of the steam. Yes, x265 does better again. But you’d not really be able to notice this when watching.

How about we go cry a little bit:

Scene 4, H.264/AVC encoded by x264 0.148.x:

Scene 4, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Cutting onions would be a classic fun part in a slice-of-life anime. Here, it’s just kitchen work. And quite the bad looking one for H.264 at 1Mbit. The letters on the knife are partly lost completely, becoming unreadable. The onion parts that fly off are visibly worse than when encoded with x265 at the same bitrate. Also, x264 produced block artifacts in the blurred bokeh areas again, that simply aren’t there with x265.

On the next level, the two come much closer to each other. However, x265 simply does the outlines better. Less artifacts and sharper, just like with the writing on the knifes’ blade as well. The issues with the bokeh are nearly gone. What’s left is a negligible amount of blocking for x264 and banding for x265. Not really noticeable however.

Finally, at 5Mbit, x265 shows those ever so slightly more well-done outlines. But that’s about it, the rest looks nice for both, and pretty much identical to the source.

Now, please, dig in:

Scene 5, H.264/AVC encoded by x264 0.148.x:

Scene 5, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Let’s keep this short: x264 does blocking, and bad transitions/outlines. x265 does it better. Plain and simple.

At 2.5Mbit, x265 nearly reaches quality transparency when compared to the original, something x264 falls short of, just a bit. While x265 does the outlines and the steam part quite like in the original frame, x264 rips the outlines apart a bit too much, and slight block artifacts can again be seen for the steam part.

At 5Mbit, x264 still shows some blocking artifacts in a part that most lossy image/video compression algorithms traditionally suck at: The reds. While not true for all human beings, most eyes perceive much finer gradients for greens, then blues, and do worst with reds. Meaning, our eyes have an unequal sensitivity distribution when it comes to colors. So image and video codecs try to save bitrate in the reds first, because supposedly it’d be less noticeable. To me subjectively, x265 achieves transparency here, meaning it looks just like the original. x264 doesn’t manage entirely. Close, but not not entirely.

Next:

Scene 6, H.264/AVC encoded by x264 0.148.x:

Scene 6, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

This is a highly static scene, with only few moving parts, so there is some rain again, and some shadow cast by raindrops as well. Now, for the static parts, incremental B frames really work wonders here. Most detail is being preserved by both encoders. What’s supposed to be happening is happening: The encoders save bit rate where the human eye can’t easily tell: In the parts where stuff is moving around very quickly. That’s how we lose a lot of raindrop shadows and some drops as well. x264 seems to have trouble separating the scene into even smaller macro blocks though? Not sure if that’s the reason, but a lot of mesh detail for the basket on the balcony on the top right is lost – x265 does better there! This is maybe because x264 couldn’t distinct the moving drops from the static background so well anymore?

At 2.5Mbit, the scenes become almost indistinguishable. The more static content we have, the easier it gets of course, so the transparency threshold becomes lower. And if you ask me, both of them reach perfect quality at 5Mbit.

Let’s throw another hard one at them for the last round:

Scene 7, H.264/AVC encoded by x264 0.148.x:

Scene 7, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Enough rain already? Pffh! Here we have a lot of foliage and low contrast added to the mix. And it gets smeared a lot by x264, rain detail lost, fine details of the bushes turning into green mud, that’s how it goes. x265 also loses too much detail here (I mean, 1Mbit is really NOT much), but again, it fares quite a bit better.

At 2.5Mbit, both encoders do very well. Somehow, this scene doesn’t seem to penalize x264 that much at the medium level. You’d really need your magnifying glass to find the spots where x265 still does better, which surprises me a bit for this scene. And finally, at 5Mbit – if you ask me – visual transparency is reached for both x264 and x265.

Final thoughts:

Clearly it’s true what a lot of people have been saying. x265 rocks at low bitrates, if configured correctly. But that isn’t gonna give me perfect quality or anything. Yeah, it sucks less – much less – than x264 in that department, but at a higher 2.5Mbit, where both start looking quite decent, x265 having just a slight edge… it becomes hardly justifiable to use it, simply because it’s that much slower to run it at decent settings.

Also, you need to take device compatibility into account. Sure, a powerful PC can always play the stuff. No matter if it’s some UNIX, Linux, MacOS X or Windows. But how about video game consoles? Older TVs? That kind of thing. Most of those can only play H.264/AVC. Or course, if you’re only using your PC and you have a lot of time and electricity to burn, then hey – why not?

But I’ll have to think really long and really hard about whether I want to replace x264 with x265 at this given point in time. Overall, it might not be practical enough on my current hardware yet. Maybe I’d need an AVX/AVX2-capable processor, as x265 has tons of optimizations for those instruction set extensions. But I’m gonna stay on my Xeon X5690 for quite a while, so SSE 4.2 is the latest I have.

I’d say, if you can stand some quality degradation, then x265 might be the way to go, as it can give you much smaller file sizes at lower bitrates with slight degradation.

If you’re aiming for high bitrates and quality, it might not be worth it right now, at least for 1080p. It’s been said the tables are turning once more when going up to 4K and UHD, but I haven’t tested that yet, as all my content – both Anime and live action movies – are “low resolution” *cough* 1080p or 720p.

Addendum:

Screenshots were taken using the latest stable mplayer 1.3.0 on Linux. Thank god they’re bundling it with ffmpeg now, making things much easier. This choice was made because mplayer can easily grab screenshots from specific spots in a video in an automated fashion. I used framesteps for this, like this:

$ mplayer ./TEST-H.265/HEVC-1mbit.mkv -nosound -vf framestep=24 \
 -vo png:z=9:outdir=./screenshots/1mbit/H.265/HEVC/:prefix=H.265/HEVC-1mbit-

This will decode every 24th frame of the video file TEST-H.265/HEVC-1mbit.mkv, and grab it into a .png file with maximum lossless compression as supported by mplayer. The .png files will be prefixed with a user-defined string and numbered, like H.265/HEVC-1mbit-00000001.pngH.265/HEVC-1mbit-00000002.png and so on, until the end of file is reached.

To encode the full size screenshots to WebP, the most recent [libwebp-0.5.0], or rather one of its companion tools – cwebp – was used as follows:

$ cwebp -z 9 -lossless -q 100 -noalpha -mt ./input.png -o ./output.webp

Now… somebody wanna grant me remote access to some quad socket Haswell-EX Xeon box for free?

No?

Meh… :roll:

Jan 272016
 

HakuNeko logoJust yesterday I’ve showed you how to modify and compile the [HakuNeko] Manga Ripper so it can work on FreeBSD 10 UNIX, [see here] for some background info. I also mentioned that I couldn’t get it to build on CentOS 6 Linux, something I chose to investigate today. After flying blind for a while trying to fix include paths and other things, I finally got to the core of the problem, which lies in HakuNekos’ src/CurlRequest.cpp, where the code calls CURLOPT_ACCEPT_ENCODING from cURLs typecheck-gcc.h. This is critical stuff, as [cURL] is the software library needed for actually connecting to websites and downloading the image files of Manga/Comics.

It turns out that CURLOPT_ACCEPT_ENCODING wasn’t always called that. With cURL version 7.21.6, it was renamed to that from the former CURLOPT_ENCODING as you can read [here]. And well, CentOS 6 ships with cURL 7.19.7…

When running $ ./configure && make from the unpacked HakuNeko source tree without fixing anything, you’ll run into this problem:

g++ -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread -o obj/CurlRequest.cpp.o src/CurlRequest.cpp
src/CurlRequest.cpp: In member function ‘void CurlRequest::SetCompression(wxString)’:
src/CurlRequest.cpp:122: error: ‘CURLOPT_ACCEPT_ENCODING’ was not declared in this scope
make: *** [obj/CurlRequest.cpp.o] Error 1

So you’ll have to fix the call in src/CurlRequest.cpp! Look for this part:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Change CURLOPT_ACCEPT_ENCODING to CURLOPT_ENCODING. The rest can stay the same, as the name is all that has really changed here. It’s functionally identical as far as I can tell. So it should look like this:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Save the file, go back to the main source tree and you can do:

  • $ ./configure && make
  • # make install

And done! Works like a charm:

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux!

And now, for your convenience I fixed up the Makefile and rpm/SPECS/specfile.spec a little bit to build proper rpm packages as well. I can provide them for CentOS 6.x Linux in both 32-bit as well as 64-bit x86 flavors:

You need to unzip these first, because I was too lazy to allow the rpm file type in my blogging software.

The naked rpms have also been submitted to the HakuNeko developers as a comment to their [More Linux Packages] support ticket which you’re supposed to use for that purpose, so you can get them from there as well. Not sure if the developers will add the files to the projects’ official downloads.

This build of HakuNeko has been linked against the wxWidgets 2.8.12 GUI libraries, which come from the official CentOS 6.7 package repositories. So you’ll need to install wxGTK to be able to use the white kitty:

  • # yum install wxGTK

After that you can install the .rpm package of your choice. For a 64-bit system for instance, enter the folder where the hakuneko_1.3.12_el6_x86_64.rpm file is, run # yum localinstall ./hakuneko_1.3.12_el6_x86_64.rpm and confirm the installation.

Now it’s time to have fun using HakoNeko on your Enterprise Linux system! Totally what RedHat intended you to use it for! ;) :roll:

Jan 262016
 

HakuNeko logoSince I’ve started using FreeBSD as a Linux and Windows replacement, I’ve naturally always been looking at porting my “known good” software over to the UNIX OS, or at replacing it by something that gets the job done without getting on my nerves too much at the same time. For most parts other than TrueCrypt, that was quite achievable, even though I had to endure varying degrees of pain getting there. Now, my favorite Manga / Comic ripper on Windows, [HakuNeko] was the next piece of software on the list. It’s basically just a more advanced Manga website parser and downloader based on stuff like [cURL], [OpenSSL] or the [wxWidgets] GUI libraries.

I didn’t even know this until recently (shame on me for never looking closely), but HakuNeko is actually free software licensed under the MIT license. Unfortunately, the source code and build system are quite Linux- and Windows-centric, and there exist neither packages nor ports of it for FreeBSD UNIX. Actually, the code doesn’t even build on my CentOS 6.7 Linux right now (I have yet to figure out the exact problem), but I managed to fix it up so it can compile and work on FreeBSD! And here’s how, step by step:

1.) Prerequisites

Please note that from here on, terminal commands are shown in this form: $ command or # command. Commands starting with a $ are to be executed as a regular user, and those starting with # have to be executed as the superuser root.

Ok, this has been done on FreeBSD 10.2 x86_32 using HakuNeko 1.3.12, both are current at the time of writing. I guess it might work on older and future releases of FreeBSD with different releases of HakuNeko as well, but hey, who knows?! That having been said, you’ll need the following software on top of FreeBSD for the build system to work (I may have missed something here, if so, just install the missing stuff like shown below):

  • cURL
  • GNU sed
  • GNU find
  • bash
  • OpenSSL
  • wxWidgets 2.8.x

Stuff that’s not on your machine can be fetched and installed as root from the official package repository, Like e.g.: # pkg install gsed findutils bash wx28-gtk2 wx28-gtk2-common wx28-gtk2-contrib wx28-gtk2-contrib-common

Of course you’ll need the HakuNeko source code as well. You can get it from official sources (see the link in first paragraph) or download it directly from here in the version that I’ve used successfully. If you take my version, you need 7zip for FreeBSD as well: # pkg install p7zip.

Unpack it:

  • $ 7z x hakuneko_1.3.12_src.7z (My version)
  • $ tar -xzvf hakuneko_1.3.12_src.tar.gz (Official version)

The insides of my archive are just vanilla as well however, so you’ll still need to do all the modifications by yourself.

2.) Replace the shebang lines in all scripts which require it

Enter the unpacked source directory of HakuNeko and open the following scripts in your favorite text editor, then replace the leading shebang lines #!/bin/bash with #!/usr/local/bin/bash:

  • ./configure
  • ./config_clang.sh
  • ./config_default.sh
  • ./res/parsers/kissanime.sh

It’s always the first line in each of those scripts, see config_clang.sh for example:

  1. #!/bin/bash
  2.  
  3. # import setings from config-default
  4. . ./config_default.sh
  5.  
  6. # overwrite settings from config-default
  7.  
  8. CC="clang++"
  9. LD="clang++"

This would have to turn into the following (I also fixed that comment typo while I was at it):

  1. #!/usr/local/bin/bash
  2.  
  3. # import settings from config-default
  4. . ./config_default.sh
  5.  
  6. # overwrite settings from config-default
  7.  
  8. CC="clang++"
  9. LD="clang++"

3.) Replace all sed invocations with gsed invocations in all scripts which call sed

This is needed because FreeBSDs sed and Linux’ GNU sed aren’t exactly that compatible in how they’re being called, different options and all.

In the text editor vi, the expression :%s/sed /gsed /g can do this globally over an entire file (mind the whitespaces, don’t omit them!). Or just use a convenient graphical text editor like gedit or leafpad for searching and replacing all occasions. The following files need sed replaced with gsed:

  • ./configure
  • ./res/parsers/kissanime.sh

4.) Replace all find invocations with gfind invocations in ./configure

Same situation as above with GNU find, like :%s/find /gfind /g or so, but only in one file:

  • ./configure

5.) Fix the make check

This is rather cosmetic in nature as $ ./configure won’t die if this test fails, but you may still wish to fix this. Just replace the string make --version with gmake --version (there is only one occurrence) in:

  • ./configure

6.) Fix the DIST variables’ content

I don’t think that this is really necessary either, but while we’re at it… Change the DIST=linux default to DIST=FreeBSD in:

  • ./configure

Again, only one occurrence.

7.) Run ./configure to create the Makefile

Enough with that, let’s run the first part of the build tools:

  • $ ./configure --config-clang

Notice the --config-clang option? We could use GCC as well, but since clang is FreeBSDs new and default platform compiler, you should stick with that whenever feasible. It works for HakuNeko, so we’re gonna use the default compiler, which means you don’t need to install the entire GCC for just this.

There will be error messages looking quite intimidating, like the basic linker test failing, but you can safely ignore those. Has something to do with different function name prefixes in FreeBSDs libc (or whatever, I don’t really get it), but it doesn’t matter.

However, there is one detail that the script will get wrong, and that’s a part of our include path. So let’s handle that:

8.) Fix the includes in the CFLAGS in the Makefile

Find the line containing the string CFLAGS = -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread or similar in the newly created ./Makefile. After the option -O2 add the following: -I/usr/local/include. So it looks like this: CFLAGS = -c -Wall -O2 -I/usr/local/include -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread. That’s it for the Makefile.

9.) Fix the Linux-specific conditionals across the C++ source code

And now the real work starts, because we need to fix up portions of the C++ code itself as well. While the code would build and run fine on FreeBSD, those relevant parts are hidden behind some C++ preprocessor macros/conditionals looking for Linux instead. Thus, important parts of the code can’t even compile on FreeBSD, because the code only knows Linux and Windows. Fixing that isn’t extremely hard though, just a bit of copy, paste and/or modify. First of all, the following files need to be fixed:

  • ./src/MangaConnector.cpp
  • ./src/Logger.cpp
  • ./src/MangaDownloaderMain.cpp
  • ./src/MangaDownloaderConfiguration.cpp

Now, what you should look for are all conditional blocks which look like #ifdef __LINUX__. Each will end with an #endif line. Naturally, there are also #ifdef __WINDOWS__ blocks, but those don’t concern us, as we’re going to use the “Linux-specific” code, if you can call it that. Let me give you an example right out of MangaConnector.cpp, starting at line #20:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Now given that the Linux code builds just fine on FreeBSD, the most elegant and easier version would be to just alter all those #ifdef conditionals to inclusive #if defined ORs, so that they trigger for both Linux and FreeBSD. If you do this, the block from above would need to change to this:

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif

Should you ever want to create different code paths for Linux and FreeBSD, you can also just duplicate it. That way you could later make changes for just Linux or just FreeBSD separately:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. #endif
  4. #ifdef __FreeBSD__
  5. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  6. #endif

Whichever way you choose, you’ll need to find and update every single one of those conditional blocks. There are three in Logger.cpp, three in MangaConnector.cpp, two in MangaDownloaderConfiguration.cpp and again three in MangaDownloaderMain.cpp. Some are more than 10 lines long, so make sure to not make any mistakes if duplicating them.

Note that you can maybe extend compatibility even further with additional directives like __OpenBSD__ or __NetBSD__ for additional BSDs or __unix__ for a wide range of UNIX systems like AIX or HP-UX. None of which has been tested by me of course.

When all of that is done, it’s compile and install time:

10.) Compile and install

You can compile as a regular user, but the installation needs root by default. I’ll assume you’ll want to install HakuNeko system-wide, so, we’ll leave the installation target directories on their defaults below /usr/local/. While sitting in the unpacked source directory, run:

  • $ gmake
  • # gmake install

If nothing starts to crash and burn, this should compile and install the code. clang will show some warnings during compilation, but you can safely ignore that.

11.) Start up the white kitty

The installation procedure will also conveniently update your window manager as well, if you’re using panels/menus. Here it’s Xfce4:

HakuNeko is showing up as an "Internet" tool

HakuNeko (“White Cat”) is showing up as an “Internet” tool. Makes sense.

With the modifications done properly it should fire up just fine after initializing its Manga connectors:

HakuNeko with the awesomeness that is "Gakkou Gurashi!" being selected from the HTTP source "MangaReader"

HakuNeko with the awesomeness that is “Gakkou Gurashi!” being selected from the HTTP source [MangaReader].

Recently the developers have also added [DynastyScans] as a source, which provides access to partially “rather juicy” Yuri Dōjinshi (self-published amateur and sometimes semi-professional works) of well-known Manga/Anime, if you’re into that. Yuri, that is (“girls love”). Mind you, not all, but a lot of the stuff on DynastyScans can be considered NSFW and likely 18+, just as a word of warning:

HakuNeko fetching a Yuru Yuri Dōjinshi from DynastyScans, bypassing their download limits by not fetching packaged ZIPs - it works perfectly!

HakuNeko fetching a Yuru Yuri Dōjinshi called “Secret Flowers” from DynastyScans, bypassing their download limits by not fetching packaged ZIPs – it works perfectly!

Together with a good comic book reader that can read both plain JPEG-filled folders and stuff like packaged .cbz files, HakuNeko makes FreeBSD a capable comic book / Manga reading system. My personal choice for a reader to accompany HakuNeko would be [QComicBook], which you can easily get on FreeBSD. There are others you can fetch from the official package repository as well though.

Final result:

HakuNeko and QComicBook make a good team on FreeBSD UNIX

HakuNeko and QComicBook make a good team on FreeBSD UNIX – I like the reader even more than ComicRack on Windows.

And, at the very end, one more thing, even though you’re likely going to be aware of this already: Just like Anime fansubs, fan-translated Manga or even Dōjinshi are sitting in a legal grey zone, as long as the book in question hasn’t been licensed in your country. It’s being tolerated, but if it does get licensed, ownership of a fan-translated version will likely become illegal, which means you should actually buy the stuff at that point in time.

Just wanted to have that said as well.

Should you have trouble building HakuNeko on FreeBSD 10 UNIX (maybe because I missed something), please let me know in the comments!

Jun 272015
 

Corsair Logo #2This is just a minor update after [part 2], but anyway. My old workstation (the one I’m migrating away from) just broke down a few days ago, so I had to do something, and quickly. Since I still don’t have any disks for my new RAID-6, I had to pull the existing RAID array from my old box and attach it to my new workstation in a hurry. It does look quite ugly too, with the RAID lying around on the table beside an open Lian Li PC-A79B. This is not how it was supposed to be, but well… In the meantime I found out that it was my Tagan Piperock 1300W power supply which broke down (Built by Topower by the way). Sad, because I liked it for its sturdy metal screw-on modular plugs, but well. So the machine now sits in its final location, it just doesn’t look too good at the moment:

"Helios" RAID-6 array emergency migration

Now the new machine has to host the old “Helios” RAID-6 array. Not quite as planned (click to enlarge).

In any case, I wanted to play around with that new Corsair “Professional Series Platinum AX1200i” of mine, which is a fully digital power supply unit featuring an I²C port. With that, you can hook it up to Corsairs Link [Commander], or you can use the dongle provided with the unit and hook it up to an internal USB header on your mainboard. Here’s a crop of a photo previously shown, this is the dongle:

The Corsair Link dongle

The Corsair Link dongle.

Now what this actually is, is a [Silicon Labs] – or Silabs in short – I²C to [USBXPress] bridge chip. So it’s not using the same USB HID device class of the Link Commander, but a completely different protocol, which is also why we’re tied to using the Corsair Link software. The free software project [CorsairLinkPlusPlus] won’t work with it at all as it supports only the Link Commander itself.

Having said that, I can’t use the Corsair Link software – which uses .Net 4.5 – on XP x64, it just won’t work on the old OS. The drivers that come with the device though are from Silabs and DO support XP and XP x64. The USB vendor ID was changed from Silabs to Corsair though, so it’s not 10CE:1C00, but 1B1C:1C00, making it impossible to install original Silabs drivers. But that’s ok, what Corsair’s shipping with the power supply works just as well.

You may not wish to install the whole Corsair Link software on XP just to get the drivers though. So I have isolated the drivers from the package for you to install them separately. The Hydro water cooler driver is also provided, but you don’t need it if it’s just for a power supply like in my case:

But, while you can install the dongle, you can’t talk to it, lacking the userland software for that. Now when I said “how to run Corsair Link on XP x64” in the title, I have to admit I was lying a bit. Because what I did was to virtualize the dongle using Oracle VirtualBox 4.2.26 and then run the Corsair Link software on a Windows 7 x64 virtual machine. Now, before launching that, the XP x64 host systems device manager will show this:

Corsair/Silabs dongle installed on XP x64

Corsair/Silabs dongle installed on XP x64.

Just so it’s handled automatically for every boot of my Windows 7 VM, I created a USB device filter in the virtual machines’ settings:

VirtualBox Corsair Link Filter

VirtualBox USB filter for the Corsair Link dongle.

Now when you start up the VM, VirtualBox will grab the device and replace it with a device called “VirtualBox USB”, thus making it unavailable on the host machine. Just install Corsair Link in the VM, and everything will work nicely:

Corsair Link, virtualized

With the USBXPress dongle virtualized properly, we can run Corsair Link on a Windows 7 VM, controlling the host machines’ power supply (Click to enlarge).

Many have described the software as buggy and crappy, but for me it gets the job done. All I wanted was to change the behavior of the unit, disabling its passive mode at low loads. While a nice feature, the internal thermal probe reports temperatures of up to 60°C at 300W load with the fan sitting still, and I don’t quite like that. I don’t see why it is needed to artificially accelerate the aging process of the PSUs electrolytic capacitors like that, so I set the fan speed to 40%, resulting in slightly short of 800rpm. Very silent, and good enough even for high loads. I now get down to 28-35°C depending on ambient temperatures without perceiving any additional noise. It may reach 40°C on really hot days I guess, but that’s a lot better than 60°C.

Just sad that we can’t define a complete custom fan curve for this unit, based on load or temperature readings. It’s possible with system fans when working with the Link Commander, but not for this one.

Naturally, virtualization also works if you’re on Linux or BSD UNIX or Solaris or whatever. It’s cumbersome, yes, but if you need it only to tell the PSUs firmware to keep the fan spinning, it’s ok. You don’t need to keep the software running, that’s the sweet part. The settings will be stored in the power supplys’ firmware directly.

Only downside is: You need a Windows Vista/7 or newer license for that of course. But maybe we’ll see some free software in the future, people are working on it, that much’s for sure!

Now let’s hope part 4 of this log will be my new hard disks, because I’m really starting to run low on disk space already…

Edit: Part 4 should now be ready, because my new 6TB SAS drives are here. However, instead it turned out to be quite the disaster, which is why [it’s part 3½ instead]. There are some preliminary benchmarks for you to see however, at least something. ;)

Mar 062015
 

F6 logoThis is a problem I’ve wanted to solve for quite some time now: How to install older Windows on systems with an AHCI SATA or maybe SCSI or SAS controller, when there is no floppy drive and Windows setup won’t recognize any hard drive? Usually, you’ll have to press F6 during the early setup sequence, and then provide the drivers with either a real floppy, or a USB floppy drive that’s on the hardware compatibility list (not all USB floppies work). While adding non-supported USB floppies to any installation disc is possible when using [nLite] (see farther below), sometimes you may not want to build your own image or maybe you don’t have any floppies available, USB included.

What I always wanted to do was to find a way to just use some USB pendrive for this, like it’s possible with newer Windows systems. And yesterday, I finally found the solution. My original attempts to modify the Windows CD itself were all unsuccessful, and today I believe that it is hardly possible to just talk setup into accepting USB pendrives as floppies, unless your systems BIOS can actively mask the pendrive as such. The solution is much more weird than any such modification actually.

But this time around, I won’t dive into the technical details up front, but simply give you the solution first. If you’re interested in how this really works, just read the lower part 3.)!

1.) How to use any USB drive as a F6 floppy medium for storage driver loading during legacy Windows setup

1.a.) Let’s go step by step. You’ll need to be on Microsoft Windows for this. First we need nu2’s bfi.exe[1] plus a few additional files. I have packaged them all together for you in this file: [buildf6]. Please unpack the archive, and enter the newly created folder buildf6\. In that folder you’ll see a subfolder f6\. Put the storage drivers you wish to use to support your target system in that folder. Please keep in mind, that it has to be F6 capable drivers. Usually these are small and consist of a txtsetup.oem file plus some *.inf and *.sys, maybe *.dll files. When you’ve placed all necessary drivers in there, run the floppy image builder script: buildf6.bat. This will create a 1.44MiB image file called f6.img.

In case you put too many drivers in the f6\ folder and it’s growing too large to build the image file, you can also alter the script to have it build a larger 2.88MiB floppy image instead of the 1.44MiB one. Just open buildf6.bat with your favorite text editor, it’ll show this:

  1. @echo off
  2. cls
  3. .\bfi\bfi.exe -v -t=144 -b=boot\boot_sector.bin -f=.\f6.img -o=BCDL.BIN -o=BCDL-IDE.SYS -o=VIDE-CDD.SYS .\boot\chainload_cd .\f6
  4. pause

Simply change the -t option to -t=288 to get a larger floppy image which can hold more drivers.

1.b.) Next thing we need is [Syslinux] to prepare ourselves a bootable and extremely minimalistic Linux system. Actually, it’s not even Linux at all, it’s just a bootloader. The latest version 6.xx can be obtained directly from [here] at kernel.org. Unpack it, launch cmd.exe and enter the following subdirectory of Syslinux on the shell: bios\win32\ or maybe bios\win64\ in case you’re on a 64-bit Windows machine. Plug your desired USB key into the machine. I’ll assume drive letter F: here. Run the following command to prepare your USB drive, and beware – all data stored on it will be lost:

syslinux.exe --mbr --active --install F:

Please copy your previously created image buildf6\f6.img over to F:\, and also copy Syslinux’ bios\memdisk\memdisk file directly to F:\. Don’t create any subfolders, all files belong in the root directory. Now, create a new text file F:\syslinux.cfg, open it with an editor and enter the following:

default 1
 
label 1
kernel memdisk
append initrd=f6.img floppy

Save the file. The contents of the USB drive should now look like this:

Our F6 USB sticks content

Our bootable F6 USB sticks’ content

2.) Run it

Now, you need to put the Windows 2000/XP/2003 CD into the target machines optical drive, plug the USB drive in, and set the systems BIOS to boot from USB. If you’re on a UEFI-based machine, make sure to boot in legacy/BIOS mode. The machine will boot from the USB drive, and then go on and continue booting from the CD, as usual. When the prompt for pressing F6 comes up, please do so. When Windows setup asks you to provide a floppy for the intended drivers, confirm that you’ve already put it in, and setup will magically pick it up. See the following screenshots:

XP x64 accessing the virtual F6 floppy

XP x64 accessing the virtual F6 floppy

XP x64 will now load the Areca SAS controller driver - no real floppies involved

XP x64 will now load the Areca SAS controller driver – no real floppies involved

As said, this even works with Windows 2000. Possibly even with Windows NT 4.0 and older systems capable of booting from a CD, but I haven’t tested systems that old. I tried it with Windows 2000 Professional though, and it just works:

Windows 2000 accessing the virtual floppy drive

Windows 2000 accessing the virtual floppy drive (German)

Windows 2000 will load a Mylex AcceleRAID 352 driver for installation

Windows 2000 will load a Mylex AcceleRAID 352 driver for installation (German)

Just keep booting from the USB stick until setup is complete, so that the virtual floppy drive is present throughout all stages of the installation process. As soon as Windows is completely installed, shut down the machine, remove USB drive and CD, and you’re good to go!

Now, for the addendum…

3.) What’s really going on here

I mentioned that I believe that Windows can’t be tricked into accepting any arbitrary USB drive as a F6 floppy. If possible at all, it’d require a low-level binary hack, as Windows setup is calling the hardware interrupt 13hex to access the floppy drive, which is about as low-level as it gets. And that’s hardcoded… somewhere. For the USB floppy part, well, you can widen the field of supported drives (see even [further below]), but according to my attempts you can’t tell Windows to treat a USB device of class 08, subclass 06 and interface 80 – like a USB pendrive – just like a class 08, subclass 04 and interface 00 device – like a real USB floppy. For more information please refer to the [USB mass storage specifications].

So what we’re doing instead is something much more freakish. The exact process that this solution goes through looks like this:

  1. The system BIOS executes the boot code within the boot sector of the USB drive.
  2. The boot code boots Syslinux’ ldlinux.sys loader.
  3. Syslinux continues to higher stages by loading the Syslinux core module, ldlinux.c32.
  4. Syslinux now pulls our floppy image f6.img into a ramdisk, and executes it as if it was an operating system kernel as specified in syslinux.cfg, effectively chainloading it (It also redirects INT13hex to the virtual floppy, so when any code calls the interrupt, it’ll get to the Ramdisk instead of any real floppy!).
  5. This works, because the floppy image itself has a boot sector that the x86 host processor can execute directly as it’s x86 machine code!
  6. Instead of a real operating system, the virtual floppies’ boot sector now executes BCDL.BIN, a pure machine code executable, which was embedded into the image by buildf6.bat from buildf6\boot\chainload_cd\ together with its CD drivers BCDL-IDE.SYS and VIDE-CDD.SYS which support real IDE as well as SATA optical drives.
  7. BCDL.BIN loads its drivers and chainloads the CD by employing standard El Torito optical drive booting.
  8. The Ramdisk stays where it is, posing as a real A: INT13hex floppy drive until reboot.
  9. Windows setup boots from CD.
  10. Besides BCDL, we also have our required storage drivers on that floppy image. Windows 2000/XP/2003 may now load them by locating and reading txtsetup.oem just as it always did.

It may sound complicated, but once you know what to do and how, it’s a pretty easy and fast process. With nothing but your USB key, the drivers and the installation CD you can set up ancient Windows pretty much anywhere, as long as you still have an optical drive at least. I’d imagine this to be specifically useful for notebooks.

4.) The USB floppy thing – How to add non-supported drives

This is just for completeness’ sake. Should you wish to just keep using a USB floppy, but unfortunately have a newer unsupported one, you can add it to your setup CDs HCL. This requires copying the entire CD off to another Windows machine, modifying a small file and rebuilding a new ISO using [nLite] though.

You can find the file on your installation CD in \I386\ for 32-bit Windows (2000/XP/2003) and in \AMD64\ for 64-bit Windows (XP x64/2003 x64), it’s called TXTSETUP.SIF. Open it with your favorite text editor and locate the following part in the Section [HardwareIdsDatabase]:

txtsetup.sif, USB part of the [HardwareIdsDatabase] section

TXTSETUP.SIF, USB part of the [HardwareIdsDatabase] section

Depending on your operating system, this may be found on totally different line numbers. Now you need to determine the USB vendor ID and device ID of your USB floppy drive. I usually do this on Linux or BSD UNIX by running lsusb (FreeBSD needs the usbutils package for that, # pkg install usbutils). Here is an example with some USB device, not a floppy, but just so you know what to look for, it’s the two 4-letter hexadecimal codes separated by a colon, the first is the vendor ID, the second the device ID:

lsusb showing USB devices on FreeBSD

lsusb showing USB devices on FreeBSD (click to enlarge)

You can also get it on Windows though, just go to “Start\Settings\Control Panel”, and then “System\Hardware\Device Manager”. Locate your USB drive by exploding the USB part of the device tree on the very bottom, right click the correct device and choose “Properties”, then “Details”. Et voilà:

Getting a USB devices vendor ID (VID) and device ID (PID) on Windows

Getting a USB devices vendor ID (VID_****) and device ID (PID_****) on Windows

Patching it in is a simple process of adding an according line to TXTSETUP.SIF, like this:

Adding an unsupported USB floppy device in TXTSETUP.SIF (shown with an invalid non-floppy device here, but it works like that with any unsupported real USB floppy drive).

Adding an unsupported USB floppy device in TXTSETUP.SIF (shown with an invalid non-floppy device here, but it works exactly like that with any unsupported real USB floppy drive).

As said, you’ll have to build yourself a new ISO from that using nLite and burn it to CD. But with that, you can now use any previously unsupported USB floppy drive for F6 driver installations.

5.) Profit

So that’s how it can be achieved after all, and I always thought it to be impossible. It’s quite the obscure method, and its gain in usefulness over nLite driver embedding is maybe debatable, but I believe this can be faster and more flexible. That may be especially true if you have to deal with lots of different machines with different SATA AHCI driver requirements or other storage controllers. Building new ISOs using nLite over and over again can be time consuming after all.

[1] Lagerweij, B. BFI is © 2000-2015 Nu2 productions. All rights reserved.

May 282014
 

YaCy logoJust recently I have published my [vision for our networked society], claiming that freedom, self-determination and independence can be reached through decentralization, putting control over our services and data back into the hand of the users. My idea was to use distributed hash tables on a lower level to power search engines or social networks, distributed across a wide field of user-operated home servers. I thought my idea was pure utopia. Something we’d need to work hard for years to accomplish.

After I published it, users approached me via various channels, pointing out already existing software that deals with the centralization problems and dangers of today, like for instance the decentralized social network [Diaspora*] or more significantly even, [YaCy], which is a DHT-based search engine just like I envisioned it.

Let me show you the simple way the YaCy developers chose to show us, what they’re doing exactly. If you’ve read my article about decentralization linked at the beginning, you’ll immediately recognize what’s going on here (Images taken from YaCy):

So you can see, where this is going? In the right direction is where! And how is it implemented? Basically a Java server built on top of the [Apache Solr] / [Lucene] full text search engine well known in certain enterprises with a web interface on top. The web interface can be used for administration and as a simple web search, like we know it already. The Java code works with both Oracle Java 1.7 (Windows, MacOS X) as well as OpenJDK 1.7, which is preferred on Linux. I haven’t tested it, but I presume it might also work on BSD UNIX, as some BSD systems do support OpenJDK 1.7 too. Could also work on OpenSolaris I guess, and it can run with user privileges.

If you want to go the Oracle route on Linux, this also seems to work, at least for me, despite the YaCy developers asking for OpenJDK. But then again, if you wanna stay clear of any even remotely fishy software licenses, just go with OpenJDK!

In case you haven’t noticed yet, I have already joined the YaCy DHT network as a node, and the search using my node as an entry point into the DHT superstructure is embedded in this weblog already, look at the top right and you’ll find it! Mind you, it ain’t the fastest thing on the track, and the quality of its results won’t yet match Google or anything, but we’re getting there! I may also choose to embed it at [http://www.xin.at], not just here. But we’ll see about that.

Also, the web interface has a few nice monitoring gadgets, let’s have a look at those for my own fresh node, too:

Now, YaCy doesn’t provide data all by itself. Like in my original idea, the network needs to pull in data from outside the superstructure, from the regular Internet and make it searchable. For that, YaCys administration web features a simple crawler that you can send off to index everything on a certain domain via HTTP/HTTPS, like “http://wp.xin.at/”, or from a Samba/Windows share server, or from local files, or FTP etc. There is also a more complex, extremely configurable and powerful crawler, but I’ve only used the simple one so far. And it also visualizes what it does, look here:

So the web interface is pretty cool, and it actually works, too! The Crawler also has parsers for tons of file types, like PDF, Office documents (Microsoft and Open-/LibreOffice), media files, ZIP files etc., so it can index the contents and/or meta data of such files too, full text!

While I do not need it, you may actually also disconnect YaCy from the “freeworld” network entirely and use it as an intranet search engine. There is even professional support for that if you’d like to use it in that context within your company.

So there we go, a free, decentralized search engine, that lies not in the hand of some opaque megacorporation, but in our very own hands. How could I’ve missed this?! I have no idea. But it seems even I have walked the world in blindness too, for the three pillars of my vision are more real than I’d have thought; Independence, Self-determination, Freedom. It’s all right there!

And you don’t even need a home server for that. You can just run it on your desktop or laptop too. Not perfect, but this works due to the massively fail-proof nature of the DHT network, as described in my earlier publication.

Seriously, this is pretty damn awesome! We’re far closer to my “stage II” than I would’ve believed just 2 weeks ago. All we need to do now is to solve the social problem and make people actually migrate to freedom! Yeah, it’s the hardest of problems, but at least we have the tech sitting there, ready to be used.

So now it’s your turn (and mine actually): Let’s inform people, lets educate whomever we can, and help ourselves lose the chains!!