Nov 222016
 

FreeBSD IBM ServeRAID Manager logoAnd yet another FreeBSD-related post: After [updating] the IBM ServeRAID manager on my old Windows 2000 server I wanted to run the management software on any possible client. Given it’s Java stuff, that shouldn’t be too hard, right? Turned out not to be too easy either. Just copying the .jar file over to Linux and UNIX and running it like $ java -jar RaidMan.jar wouldn’t do the trick. Got nothing but some exception I didn’t understand. I wanted to have it work on XP x64 (easy, just use the installer) and Linux (also easy) as well as FreeBSD. But there is no version for FreeBSD?!

The ServeRAID v9.30.21 manager only supports the following operating systems:

  • SCO OpenServer 5 & 6
  • SCO Unixware 7.1.3 & 7.1.4
  • Oracle Solaris 10
  • Novell NetWare 6.5
  • Linux (only certain older distributions)
  • Windows (2000 or newer)

I started by installing the Linux version on my CentOS 6.8 machine. It does come with some platform-specific libraries as well, but those are for running the actual RAID controller management agent for interfacing with the driver on the machine running the ServeRAID controller. But I only needed the user space client program, which is 100% Java stuff. All I needed was the proper invocation to run it! By studying IBMs RaidMan.sh, I came up with a very simple way of launching the manager on FreeBSD by using this script I called serveraid.sh (Java is required naturally):

  1. #!/bin/sh
  2.  
  3. # ServeRAID Manager launcher script for FreeBSD UNIX
  4. # written by GAT. http://www.xin.at/archives/3967
  5. # Requirements: An X11 environment and java/openjdk8-jre
  6.  
  7. curDir="$(pwd)"
  8. baseDir="$(dirname $0)/"
  9.  
  10. mkdir ~/.serveraid 2>/dev/null
  11. cd ~/.serveraid/
  12.  
  13. java -Xms64m -Xmx128m -cp "$baseDir"RaidMan.jar com.ibm.sysmgt.raidmgr.mgtGUI.Launch \
  14. -jar "$baseDir"RaidMan.jar $* < /dev/null >> RaidMan_StartUp.log 2>&1
  15.  
  16. mv ~/RaidAgnt.pps ~/RaidGUI.pps ~/.serveraid/
  17. cd "$curDir"

Now with that you probably still can’t run everything locally (=in a FreeBSD machine with ServeRAID SCSI controller) because of the Linux libraries. I haven’t tried running those components on linuxulator, nor do I care for that. But what I can do is to launch the ServeRAID manager and connect to a remote agent running on Linux or Windows or whatever is supported.

Now since this server/client stuff probably isn’t secure at all (no SSL/TLS I think), I’m running this through an SSH tunnel. However, the Manager refuses to connect to a local port because “localhost” and “127.0.0.1” make it think you want to connect to an actual local RAID controller. It would refuse to add such a host, because an undeleteable “local machine” is always already set up to begin with, and that one won’t work with an SSH tunnel as it’s probably not running over TCP/IP. This can be circumvented easily though!

Open /etc/hosts as root and enter an additional fantasy host name for 127.0.0.1. I did it like that with “xin”:

::1			localhost localhost.my.domain xin
127.0.0.1		localhost localhost.my.domain xin

Now I had a new host “xin” that the ServeRAID manager wouldn’t complain about. Now set up the SSH tunnel to the target machine, I put that part into a script /usr/local/sbin/serveraidtunnel.sh. Here’s an example, 34571 is the ServeRAID agents’ default TCP listen port, 10.20.15.1 shall be the LAN IP of our remote machine hosting the ServeRAID array:

#!/bin/bash
ssh -fN -p22 -L34571:10.20.15.1:34571 mysshuser@www.myserver.com

You’d also need to replace “mysshuser” with your user name on the remote machine, and “www.myserver.com” with the Internet host name of the server via which you can access the ServeRAID machine. Might be the same machine or a port forward to some box within the remote LAN.

Now you can open the ServeRAID manager and connect to the made-up host “xin” (or whichever name you chose), piping traffic to and from the ServeRAID manager through a strongly encrypted SSH tunnel:

IBM ServeRAID Manager on FreeBSD

It even detects the local systems’ operating system “FreeBSD” correctly!

And:

IBM ServeRAID Manager on FreeBSD

Accessing a remote Windows 2000 server with a ServeRAID II controller through an SSH tunnel, coming from FreeBSD 11.0 UNIX

IBM should’ve just given people the RaidMan.jar file with a few launcher scripts to be able to run it on any operating system with a Java runtime environment, whether Windows, or some obscure UNIX flavor or something else entirely, just for the client side. Well, as it stands, it ain’t as straight-forward as it may be on Linux or Windows, but this FreeBSD solution should work similarly on other systems as well, like e.g. Apple MacOS X or HP-UX and others. I tested this with the Sun JRE 1.6.0_32, Oracle JRE 1.8.0_112 and OpenJDK 1.8.0_102 for now, and even though it was originally built for Java 1.4.2, it still works just fine.

Actually, it works even better than with the original JRE bundled with RaidMan.jar, at least on MS Windows (no more GUI glitches).

And for the easy way, here’s the [package]! Unpack it wherever you like, maybe in /usr/local/. On FreeBSD, you need [archivers/p7zip] to unpack it and a preferably modern Java version, like [java/openjdk8-jre], as well as X11 to run the GUI. For easy binary installation: # pkg install p7zip openjdk8-jre. To run the manager, you don’t need any root privileges, you can execute it as a normal user, maybe like this:

$ /usr/local/RaidMan/serveraid.sh

Please note that my script will create your ServeRAID configuration in ~/.serveraid/, so if you want to run it as a different user or on a different machine later on, you should recursively copy that directory to the new user/machine. That’ll retain the local client configuration.

That should do it! :)

Nov 192016
 

FreeBSD GMABoost logoRecently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
 
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call gmaboost.sh. I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  2.  
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  13.  
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  16.  
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # gmaboost.sh 200 or # gmaboost.sh 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Nov 142016
 

HP/Compaq nx6310/nc6320 logoA good while back, I got a free notebook from [The_Plague]German flag, a HP/Compaq nx6310[1][2] which he kinda pulled out of the trash at his company. It’s not exactly “Thinkpad T23” material, but it’s a pretty solid, well-built machine with a good keyboard. I’ve been using the thing as an operating system testbed for a while (Linux, ReactOS, Haiku OS, OpenBSD, Dragonfly BSD, and finally: FreeBSD UNIX). After settling for FreeBSD the machine clearly showed its limitations though, the most problematic being imposed by the very low-end i940GML chipset. That one has limited the machine to a single processor core and a 533MHz data rate FSB.

I did give the machine a Core Duo T2450, but switching dual core on in the BIOS results in a lockup at POST time. Also, the chipset cannot use dual-channel DDR-II and limits the user to 2GiB of memory, making the use of a 64-bit processor rather pointless. Which turned out to be bad, because some code doesn’t even provide full functionality for 32-bit anymore, like x265, which dropped deep color support on 32-bit architectures.

But now, The_Plague pulled another one out of the trash, it’s basically the exact same machine, but a higher-end model, the nc6320. This one has an i945GM chipset, which means dual core support, FSB667 and 4GiB dual-channel RAM capability! It came with a Core 2 Duo T5600 @ 1.83GHz with 2MiB L2 cache. I ordered the largest possible chip for this box from ebay Hong Kong, so now it has a Core 2 Duo T7600 @ 2.33GHz with 4MiB L2 cache. Also, 2×2=4GiB of DDR-II/667 CL4 are on their way already, together with a 12-cell secondary monster battery!

And of course, FreeBSD UNIX again, in its brand new version 11.0-RELEASE:

HP/Compaq nc6320 running FreeBSD 11.0 UNIX

HP/Compaq nc6320 running FreeBSD 11.0 UNIX (click to enlarge)

The CPU upgrade is actually even noticeable when browsing the web, lots of resource-hungry Javascript and CSS3, you know. Luckily, Chromium supports hardware acceleration on the Intel GMA950 GPU on FreeBSD, as the OS comes with a kernel modesetting compliant driver for almost all integrated Intel graphics chips. It’s too slow to do the rasterization stage on the GPU, but it still helps.

Once again, it shall serve mostly as a meeting and sysadmin machine, with a little bit of private-use-fun added on top. Let’s have a look at the software! Oh and by the way, I decided to make the screenshots 8-bit .png images, so some of them will look a bit bad. But still better+smaller than JPEG for that purpose:

Running screenfetch on the nc6320

Running screenfetch on the nc6320 (click to enlarge)

$ screenfetch is showing us some details about the machine, which also makes it clear that everything is “Tokisaki Kurumi”-themed. Since there’s a lot of red color on that girls’ garments it seems at least somewhat fitting for a FreeBSD machine.

Chromium with FVD Speed Dial

Chromium with FVD Speed Dial (click to enlarge)

I’m a [Vivaldi] fan personally, but that browser isn’t available on any BSD yet, so I installed a few extensions to make Chromium work somewhat like Vivaldi; The most important part being the static FVD speed dial you can see above. What you can’t see here are the other extensions that followed it: AdBlockPlus and Ghostery. I hear there are better/faster solutions than ABP for ad blocking these days however, so maybe I’ll revise that.

IBM Lotus Notes via wine 1.8

IBM Lotus Notes 6.5.1 via 32-bit wine 1.8.4 (click to enlarge)

Also, for work I would sometimes need IBM Lotus Notes, as it’s our Universities’ groupware solution (think of that what you will). While I couldn’t get the Linux version to run, our Domino servers still accept connections from older clients, so it’s Lotus Notes 6.5.1 running under a 32-bit [wine], which is a solution IBM officially recommended for running the software on Linux/UNIX a few years ago. And yeah, it still works. And if you have Windows software wine can’t cope with?

XP x64 via VirtualBox on FreeBSD

XP x64 via VirtualBox on FreeBSD (click to enlarge)

For anything that wine can’t handle, the VirtualBox port kicks in, as we can see here. Together with the CPUs VT-x extension and the guest tools, virtualizing Windows on FreeBSD UNIX works relatively well. Not all features are there (like USB passthrough), but it works ok for me. Will need a Windows 7 VM as well I think.

More stuff:

Communicating on FreeBSD

Communicating on FreeBSD (parts are censored, click to enlarge)

One important part is communication! Luckily, there is a version of licq in the ports tree now. It builds well together with its Qt4 UI, so no complaints there. Hexchat for IRC access is also available, but the tricky part was Skype; Not that I really need it, but I wanted to have the linuxulator up and running as well! For those of you who don’t know what the “linuxulator” is: It’s a series of kernel modules that extend FreeBSDs kernel with parts of the Linux kernel interface. On top of that, you can pull parts of Fedora 10 or CentOS 6.8 or some CentOS 7 Linux userspace components from the package servers. Together with the kernel modules those form a kind of runtime environment for executing Linux programs – Skype 4.3 in this case! So I have both wine and linuxulator ready for action, and with it access to ICQ, Jabber, MSN, IRC and Skype. Now, what about multimedia?

Multimedia on FreeBSD

smplayer and xmms on FreeBSD, unfortunately the 8-bit color is a bit too noticeable for this screenshot, my apologies (click to enlarge)

This is a part where the upgraded processor also helps. Here we can see (s)mplayer play the last episode of the Anime Hanayamata in taxing 2.5Mbit H.265/HEVC encoding, paired with AAC-LC audio. The Core 2 Duo T5600 had some issues with this, but the faster T7600 shows now problems. Additionally, xmms is playing a Commodore 64 SID tune using libsidplay2 and the reSID engine. xmms comes with a lot of funny plugins from the FreeBSD ports tree for Gameboy tunes or NES tunes, but the C64 one you need to compile for yourself. Not too hard though, you can fetch libsidplay2 and reSID from packages beforehand to make things easier! What else?

ioquake3

ioquake3, a cleaned up version of the Quake III Arena source code, here in its 64-bit FreeBSD build (click to enlarge)

A pretty fun part: Playing the native Quake3 port [ioquake3] in 64-bit, for whenever you just need to shoot something to blow off some steam. ;) I have to say, I had to tweak it quite a bit to run fluently on the WVA 1400×1050 display of this book given the weak GMA950 GPU, but it runs “rather ok” now. ioquake3 is also available for Windows, OSX and Linux by the way, including a more advanced OpenGL 2 renderer, which gives users access to some advanced graphical effects. And if I get bored by that…

HakuNeko Manga ripper and qComicbook

HakuNeko Manga ripper and qComicbook showing some sweet girls love! (click to enlarge)

Once again, fixing up HakuNekos’ build system and C++ code to work with FreeBSD properly took some time. Unfortunately there is no port for it yet (and I’m too stupid/lazy to create one), so you have to fix it by hand. Lots of replacing sed invocations with gsed, find with gfind etc. and the OS #ifdef parts, which need to be changed in several .cpp files, here’s an example from MangaConnector.cpp:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

Something like that needs to turn into this to compile on FreeBSD, otherwise you’ll end up with a HakuNeko that can’t do shit (it’ll still compile and run, but like I said, it’d be devoid of function):

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

This is true for the latest version 1.4.1 as well. I guess the modifications should also apply to other operating systems by adding things like __OpenBSD__ or similar.

Now all that’s left is to wait for that massive 12C battery, the RAM capacity+speed upgrade and some FreeBSD case sticker that I ordered from [unixstickers.com] (hint: That’s a referral URL, it’s supposed to give you some $5 coupon upon ordering, I hope it works). Upon my order, a small part was donated to the LLVM project – very fitting, given that I’ve used clang/llvm a lot to compile stuff on FreeBSD as of late. :)

FreeBSD case sticker (preview)

This is what it’s supposed to look like, and it’s going to replace the current Windows XP+Vista sticker

I hope it’ll look as good in real life! :) Ah, I think I’m gonna have a lot of fun with that old piece of junk. ;)

Ah, and thanks fly out to The_Plague, who saved this laptop from the trash bin and gave it to me for free! Prost!

Edit: And the memory is here, two G.Skill “performance” modules doing 4-4-4 latencies at 667MHz data rate, replacing a single Samsung module running 5-5-5. Now I was interested in how much going from single channel CL5 to dual channel CL4 would really affect performance. Let’s just say, it didn’t do too much for CPU processes. However, the effect on the integrated GMA950 GPU (using shared system memory!) was amazing. It seems the graphics chip was held back a lot by the memory interface! Let’s have a quick look at Quake III Arena performance using a quickly recorded demo just for this purpose (ioquake3 can’t play old Quake III Arena demos like the “001” demo):

  • ioquake3 1.36, single channel DDR-II/667 CL5:
  • 30.6fps
  • ioquake3 1.36, dual channel DDR-II/667 CL4:
  • 41.2fps

Roughly +35%!!

Tests were run three times, then three more times after a reboot. After that, an average was taken. For ioquake3 this wouldn’t even have been necessary though, as the results were extremely consistent. It’s amazing how much the added memory speed really affects the game engine! I rebooted and re-ran the tests several times because I couldn’t believe in that massive boost in performance, but it’s actually true and fully reproducible! This reminds me of how well modern AMD APU graphics chips scale with main memory speed and it explains why people were asking for quad-channel DDR4 on those Kaveri APU chips. Its built-in Radeons would’ve probably loved the added bandwidth!

I also kinda felt that browsing web sites got a lot more smooth using Chromium with most of its GPU acceleration turned on. So I tried the graphics-centric browser test [Motionmark] to put that to the test. Parts of the results were inconclusive, but let’s have a look first:

  • Motionmark 1.0 (medium screen profile), single channel DDR-II/667 CL5:
  • Overall result: 13.85 ±22.24%
  • Multiply: 119.26 ±2.95%
  • Canvas Arcs: 19.04 ±68.48%
  • Leaves: 3.00 ±133.33%
  • Paths: 85.30 ±6.57%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 1.76 ±5.22%
  • Images: 40.58 ±2.56%
  • Design: 18.89 ±8.00%
  • Suits: 24.00 ±37.50%
  • Motionmark 1.0 (medium screen profile), dual channel DDR-II/667 CL4:
  • Overall result: 22.47 ±15.93%
  • Multiply: 124.55 ±1.60%
  • Canvas Arcs: 26.00 ±138.46%
  • Leaves: 65.90 ±16.93%
  • Paths: 37.00 ±16.89%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 2.00 ±50.00%
  • Images: 41.58 ±3.59%
  • Design: 24.49 ±2.35%
  • Suits: 90.65 ±13.55%

Now first things first: This was just my first pick for any kind of graphics-heavy browser benchmark. I thought I needed something that would make the browser do a lot of stuff on the GPU, given that hardware acceleration was almost fully enabled on FreeBSD UNIX + Chromium + GMA950. However, after repeated runs it showed that the variance was just far too high on the following tests: Leaves, Paths, Suits. Those would also mess up the overall score. The ones that showed consistent performance were: Multiply, Canvas Arcs, Canvas Lines, Focus, Images, Design, so we should focus on those. Well, not all of those tests show promising results (Multiply, Canvas Lines), but some clearly do. It seems my feeling that parts of CSS3 etc. had gotten faster after the memory upgrade was spot-on!

Not bad, not bad at all! And tomorrow morning, the [x264 benchmark] will also have finished, showing how much a classic CPU-heavy task would profit from that upgrade (probably not much, but we’ll see tomorrow).

Edit 2: And here is the rest. Like I thought, the memory upgrade had only minimal impact on CPU performance:

  • x264 benchmark, single channel DDR-II/667 CL5:
  • Runtime: 04:40:08.621
  • x264 benchmark, dual channel DDR-II/667 CL4:
  • Runtime: 04:38:23.851

So yeah it’s faster. But only by a meager +0.62%. Completely negligible. But it’s still a good upgrade given the GPU performance boost and the fact that I can now use more memory for virtual machines. :)

Ah, and here’s the 12-cell ultra capacity battery, which gives me a total of 18 cells in conjunction with the 6-cell primary battery:

Nice hardware actually, you can check it’s charge (roughly) with a button and a 4-LED display, and it has it’s own charging plug. What surprised me most though was this:

$ hwstat | grep -i -e "serial number" -i -e battery
[ACPI Battery (sysctl)]
[battery0]
        Serial number:                  00411 2006/10/12
[battery1]
        Serial number:                  00001 2016/07/29

That probably explains how a still sealed battery could come with a ~25% pre-charge. Manufactured in July 2016, wow. And that for a notebook that’s 10 years old? Ok, it’s an aftermarket battery by [GRS], but that’s just damn fine still! With that I’ll surely have enough battery runtime to make it through longer meetings as well! :)

Edit 3: And today I used the notebook for a sysadmin task, helping our lead developer in debugging a weird problem in a Java-based student exam submission and evaluation system of ours at work. I suspected that the new CuPPIX (=KNOPPIX derivative) distribution I built for this was to blame, but it turned out to be a faulty Java library handling MySQL database access, hence crashing our server software under high parallel loads. In any case, I had the nc6320 with me during the entire morning up until 12:30 or so, walking away with a total charge of 49% left after the developer had fixed the problem. Not stellar given a total of 18 cells, but definitely good enough for me! :)

Edit 4: And my FreeBSD sticker from unixstickers is finally here! They even gave me a bunch of random free stickers to go with it! I gave those to some colleagues for their kids. ;) And here it is:

FreeBSD sticker from unixstickers.com

There was a Windows Vista/XP sticker before, now it shows some UNIX love! (click to enlarge)

The sticker shows some pretty good quality as well, nice stuff! :)

Jul 272016
 

x264 logoSince I’ve been doing a bit of Anime batch video transcoding with x264 and x265 in the last few months, I thought I’d document this for myself here. My goal was to loop over an arbitrary amount of episodes and just batch-transcode them all at once. And that on three different operating systems: Windows (XP x64), Linux (CentOS 6.8 x86_64) and FreeBSD 10.3 UNIX, x86_64. Since I’ve started to split the work across multiple machines, I lost track of what was where and which machine finished what, and when.

So I thought, why not let the loop send me a small notification email upon completion? And that’s what I did. On Linux and UNIX this relies on the bash shell and the mailx command. Please note that I’m talking about [Heirloom mailx], not some other mail program by the same name! I’m mentioning this, because there is a different default mailx on FreeBSD, that won’t work for this. That’s why I put alias mailx='/usr/local/bin/mailx' in my ~/.bash_profile on that OS after installing the right program to make it the default for my user.

On Windows, my loops depend on my own [colorecho] command (you can replace that with cmds’ ECHO if you want) as well as the command line mailer [blat]. Note that, if you need to use SSL/TLS encryption when mailing, blat can’t do that. A suitable replacement could be [mailsend]. Please note, that mailsend does not work on Windows XP however.

In the x265 case, avconv (from the [libav] package) is required on all platforms. You can get my build for Windows [here]. If you don’t like it, the wide-spread [ffmpeg] can be a suitable drop-in replacement.

Now, when setting up blat on Windows, make sure to run blat -help first, and learn the details about blat -install. You need to run that with certain parameters to set it up for your SMTP mail server. For whatever reason, blat reads some of that data from the registry (ew…), and blat -install will set that up for you.

Typically, when I transcode, I do so on the elementary streams rather than .mkv files directly. So I’d loop through some source files and extract the needed streams. Let’s say we have “A series – episode 01.mkv” and some more, all the way up to “A series – episode 13.mkv”, then, assuming track #0 is the video stream…

On Windows:

FOR %I IN (01,02,03,04,05,06,07,08,09,10,11,12,13) DO mkvextract tracks "A series - episode %I.mkv" ^
 0:%I\video.h264

On Linux/UNIX:

for i in {01..13}; do mkvextract tracks "A series - episode $i.mkv" 0:$i/video.h264; done

mkvextract will create the non-existing subfolder for us, and a x264 transcoding loop would then look like this on Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps ^
 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f ^
 -2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats %I\v.stats -t 2 --no-fast-pskip ^
 --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o %I\pass1.264 & colorecho "Pass 1 ^
 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & ^
 "c:\Program Files\VFX\x264cli\x264-10b.exe" --fps 24000/1001 --preset veryslow --tune animation ^
 --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats ^
 %I\v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic --demuxer lavf %I\video.h264 -o ^
 %I\pass2.264 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10) & echo !SERIES! ^
 transcoding complete | blat - -t "myself@another.mailhost.com" -c "myself@mailhost.com" -s "x264 ^
 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET SERIES="

Note that I always write all the iteration out in full here. That’s because cmd can’t do loops with leading zeroes in the iterator. The reason for this is that those source files usually have them in their lower episode numbers. If it wasn’t 01,02, … ,12,13, but 1,2, … ,12,13 instead, you could do FOR /L %I IN (1,1,13) DO. But this isn’t possible in my case. Even if elements need alphanumeric names like here,  FOR %I IN (01,02,03,special1,special2,ova1,ova2) DO, you still won’t need that syntax on Linux/UNIX because the bash can have iterator groups like for i in {{01..13},special1,special2,ova1,ova2}; do. Makes me despise the cmd once more. ;)

Edit:

Ah, according to [this], you can actually do something like cmd /V /C "FOR /L %I IN (1,1,13) DO (SET "fI=00%I" & echo "!fI!:~-2")", holy shit. It actually works and gives you leading zeroes. :~-2 for 2 digits, :~-3 for three. Expand fI for more in this example. I mean, what is this even? Some number formatting magic? I probably don’t even wanna know… Couldn’t find any way of having several groups for the iterator however. Meh. Still don’t like it.

So, well, it’s like this on Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do nice -n19 x264 --fps \
24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid normal -f \
-2:0 --bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats $i/v.stats -t 2 --no-fast-pskip \
--cqm flat --non-deterministic --demuxer lavf $i/video.h264 -o $i/pass1.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && nice -n19 \
x264 --fps 24000/1001 --preset veryslow --tune animation --open-gop -b 16 --b-adapt 2 --b-pyramid \
normal -f -2:0 --bitrate 2500 --aq-mode 1 -p 2 --stats $i/v.stats -t 2 --no-fast-pskip --cqm flat \
--non-deterministic --demuxer lavf $i/video.h264 -o $i/pass2.264 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x264 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

The variable $MACHINE or %MACHINE%/!MACHINE! specifies the machines’ host name. This will be noted in the email, so I know which machine just completed something. $EPNUM – or %EPNUM%/!EPNUM! on Windows – is used for periodic updates on the shell. The output would be like “Pass 1 done for Episode 07/13 of AnimeX” in green on Windows and bold red on Linux/UNIX (just change the color to your liking).

Finally, $SERIES aka %SERIES%/!SERIES! would be the series’ name. So say, the UNIX machine named “BEAST” above is done with this loop. The email would come with the subject line “x264 notification from BEAST” and would read “AnimeX transcoding complete” in plain text. That’s all.

Please note, that cmd batch on Windows is extremely creepy. Every whitespace (especially the leading ones when doing multi-line like this for display) needs to be exactly where it is. The same goes for double quotes where you might think they aren’t needed. They are! Also, this needs delayed variable expansion once again, which is why we see variables like !EPNUM! instead of %EPNUM% and why it’s called in a subshell by running cmd /V /C.

On Linux/UNIX we don’t need to rely on some specific API like cmds’ SetConsoleTextAttribute() to print colors, as most terminals understand ANSI color codes.

And this is what it looks like for x265:

Windows:

expand/collapse source code
cmd /V /C "ECHO OFF & SET MACHINE=NOVASTORM& SET EPNUM=13& SET SERIES="AnimeX"& (FOR %I IN ^
 (01,02,03,04,05,06,07,08,09,10,11,12,13) DO avconv -r 24000/1001 -i %I\video.h264 -f yuv4mpegpipe ^
 -pix_fmt yuv420p -r 24000/1001 - 2>NUL | "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 ^
 --fps 24000/1001 -p veryslow --pmode --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate ^
 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 ^
 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 ^
 --pass 1 --slow-firstpass --stats %I\v.stats --sar 1 --range full -o %I\pass1.h265 & colorecho ^
 "Pass 1 done for Episode %I/"!EPNUM!" of "!SERIES!"" 10 & ECHO. & avconv -r 24000/1001 -i ^
 %I\video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>;NUL | ^
 "C:\Program Files\VFX\x265cli-mb\x265.exe" - --y4m -D 10 --fps 24000/1001 -p veryslow --pmode ^
 --pme --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 ^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1 ^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2 --stats %I\v.stats --sar ^
 1 --range full -o %I\pass2.h265 & colorecho "Pass 2 done for Episode %I/"!EPNUM!" of "!SERIES!"" ^
 10) & echo !SERIES! transcoding complete | blat - -t "myself@another.mailhost.com" -c ^
 "myself@mailhost.com" -s "x265 notification from !MACHINE!" & SET MACHINE= & SET EPNUM= & SET ^
 SERIES="

Linux/UNIX:

expand/collapse source code
(export MACHINE=BEAST EPNUM=13 SERIES='AnimeX'; for i in {01..13}; do avconv -r 24000/1001 -i \
$i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice -19 x265 - --y4m \
-D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 \
--rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq \
5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1 \
--slow-firstpass --stats $i/v.stats --sar 1 --range full -o $i/pass1.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 1 done for episode $i/$EPNUM of $SERIES\e[0m" && echo && avconv -r \
24000/1001 -i $i/video.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null | nice \
-19 x265 - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6 --bframes 16 --b-pyramid \
--bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd \
1.6 --psy-rdoq 5.0 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 \
--pass 2 --stats $i/v.stats --sar 1 --range full -o $i/pass2.h265 && echo && echo -e \
"\e[1;31m`date +%H:%M`, pass 2 done for episode $i/$EPNUM of $SERIES\e[0m" && echo; done && echo \
"$SERIES transcoding complete" | mailx -s "x265 notification from $MACHINE" -r \
"myself@mailhost.com" -c "myself@another.mailhost.com" -S smtp-auth="login" -S \
smtp="smtp.mailhost.com" -S smtp-auth-user="myuser" -S smtp-auth-password="mysecurepassword" \
myself@mailhost.com)

And that’s it. The loops for audio transcoding are simpler, as that part is so fast, it doesn’t need email notifications. Runs for minutes rather than days. When all is done, I’d usually fire up the MKVToolnix GUI, and prepare a mux for the first episode. There is a nice “copy command line to clipboard” function there when you click on “Muxing” after everything is set up. With that I can build another loop that muxes everything to final .mkv files. On Windows that part is more complicated if you want Unicode support, so I needed to create input files by using a generator I wrote in Perl for that, but that’s for another day… :)

Oh, and if you wanna ssh into your Linux or UNIX boxes from afar to check on your transcoders, consider launching them on a GNU screen] session. It’s immensely useful! Too bad it won’t work on the Windows cmd. :(

Nov 112013
 

PC-BSD LogoSince I got to know some BSD UNIX operating systems on my journeys across the sea of systems for my x264 benchmark, I was fascinated with those UNICES, with BSD even more so than with Solaris. Besides OpenBSD, NetBSD, FreeBSD and Dragonfly BSD, one was particularly interesting, and that was [PC-BSD]. Yet another FreeBSD at heart, it adds quite a lot of value on top of that solid core to make it easier for users of both the Windows and Linux (and OSX I guess) worlds to start with BSD. Things that you would need to set up all by yourself on FreeBSD are already taken care of here, like e.g. giving you a built-in Linux compatibility layer ready for use, or the so-called “AppCafe” package manager, that is sinfully easy to use, a binary nVidia graphics card driver built-in, a graphical installer even…

Let me show you said AppCafe first. Now you still get to use the classic BSD binary package management with pkg_add -r, pkg_delete etc, also you can still make use of the source code based Ports system. But for the most important parts, and even more so for beginners, AppCafe is the place to start. Have a look:

As you can see, AppCafe shows you the latest additions on top, while allowing the user to either browse through packages by category or doing a full text search on the entire database of packages. The packages themselves are distributed as so-called “PBI” files, which contain the program itself plus all necessary libraries the program depends on. This might bloat packages a bit here and there, but it also makes it very easy to handle. Just click on a package, choose install, wait, done. Almost “Ubuntu”, eh?

The packages are actually built and added to the database by an automatic system from the Ports tree and (I assume) other sources. So if there is a new FireFox, chances are it’s going to appear in AppCafe rather sooner than later.

Above you may also have noticed Skype? There is no native BSD version of Skype, only Windows, OSX and Linux. Still, there is a Skype in AppCafe. This can be achieved easily, because PC-BSD ships with a fully configured Linux compatibility layer, from the kernel module linux.ko to linprocfs to other things. Typically, the user space components of that layer can be found in /compat/linux/. Because it’s already included in PC-BSD, AppCafe can just serve you certain Linux binaries where no other options are available.

This is also the way how PC-BSD included Adobe Flash by default. So yeah, despite UNIX, you still get Adobe Flash if you want. Plus OpenJDK if you need Java.

Another cool thing is the PC-BSD “Warden”. On Linux it can be quite tricky to set up chroot environments or “jails” to run ancient software with massive dependencies or to just lock in certain services. On this OS, there is a (graphical!) configuration tool to do just that very easily. Thus, jail management becomes very managable, and you can easily create new locked-in instances of different BSD systems:

PC-BSD Warden

PC-BSD Warden

Now you got the Linux compatibility layer and also a pretty modern Wine as your Windows runtime including good configuration interfaces like Swine (I even managed to get Lotus Notes Basic 8.5 to work!). But still, sometimes you just need a real Linux or Windows. And for that, PC-BSD has Oracles VirtualBox included by default, with kernel drivers and all, ready to use:

PC-BSD VirtualBox

VirtualBox

Basically, you got all you need though. Even Opera is available in its native BSD version from AppCafe, plus FireFox, Chromium, Thunderbird, Chat clients like SIM-IM, X-Chat and even Skype. Wine for Valves Steam? It’s there! Rarer software can be taken from the FreeBSD package repository using pkg_* or compiled from BSD Ports, which is not too hard to use. And for all other things you have a full GNU autoconf, make, cmake and several GCC compiler versions ranging from the system stock 4.2.1 all the way up to the 4.9.0 experimental version, so you can just compile your software from source code.

I was quite surprised by how easy BSD can be made. Sure, I had to compile my own Murrine engine to be able to use that neat Orta GTK+ theme for my UI properly, I had to compile my own libsidplay2+reSID engine to get some decent Commodore 64 SID emulation in xmms1 and so on. Some special things still do require some work. But then again, they also might on Linux.

Plus, there is ZFS. The greatest of all file systems with checksummed blocks, built-in blocklevel [data deduplication], dynamic block sizes, online compression, online file system repair, a full-blown RAID implementation and neat performance etc.

Oh, and before I forget it, this is what my PC-BSD looks like at the moment:

PC-BSD Screenshot

PC-BSD Screenshot

I do have certain long-term plans about replacing Microsoft Windows on my primary Workstation. That won’t happen anytime soon, even though I am using Windows XP Pro x64 Edition, but still. One day, a new OS will be needed. And when that day comes, it will likely not be Windows 7 or 8 or 8.1 or 9 or 17 or whatever. The way Microsoft is going just pisses me off too much. And besides Linux, I might actually go for BSD UNIX. I can’t say why really, but somehow I like this system. Plus, again, ZFS!

Dec 202012
 

TimestampRecently, I got a request to determine and publish a list of latest created and/or modified files/folders on a filesystem and get that published in HTML form. Now since my scripting as of late has been cross-platform (Linux & Windows), I was aiming at achieving that again. So I started scripting Perl 5 on Linux, naturally using Unix style epoch time stamps, which can be done with a Perl internal set of functions or with the module Time::localtime.

So the idea is, you give the script a base folder, and it will scan all folders (or files if you wish) beneath, and output a list of the most recently added or changed ones. I decided to limit the output to what I may see fit, like the latest 20 or latest 100 added and modified folders. Very helpful: A folders timestamp gets updated when you add files to it or even when you change files in it.

So, here is some code for that (note that this might not exactly be executable as-is, because it may be incomplete, these are just extracted parts of the full code):

expand/collapse source code
  1. #!/usr/bin/perl
  2.  
  3. use strict;
  4. use File::stat;
  5.  
  6. my $basepath  = "X:/basefolder";  # Base folder.
  7. my @subfolders  = <$basepath/*>;  # Read subfolders to check for modifications within.
  8.  
  9. my $printcount  = 20;             # Amount of folders to display in final output.
  10.  
  11. my $outfile  = "C:/outputfile";   # File to write list to.
  12.  
  13. my $subfolder  = "";       # Variable to use for iteration through subfolders.
  14. my $folder  = "";          # Single folder name.
  15. my $mtime  = "";           # UNIX-style epoch-related time stamp of a single folder.
  16. my %mtimestamps  =   ;     # Hash with folder names as keys and time stamps as values.
  17. my $i  = 0;                # Arbitrary counter.
  18. my $key  = "";             # Arbitrary key for iteration through sorted timestamp hash 
  19.  
  20. open(OUTFILE, "<:encoding(UTF-8)", $outfile); # Open output file.
  21.  
  22. # Go through subfolders and check timestamps of folders within them (level 2):
  23. foreach $folder (@subfolders) {         # Iterate through subfolders.
  24.   if (-d $folder) {                     # Check if element is truly a directory.
  25.     $mtime = stat($folder)->mtime;      # Read modification time stamp of subfolder.
  26.     $mtimestamps{$folder} = $mtime;     # Add timestamp as a value to the hash for the
  27.   }                                     # current folder name as key.
  28. }
  29.  
  30. # Iterate through sorted hash, highest values (=newest folders) first:
  31. foreach $key (reverse sort {$mtimestamps{$a} <=> $mtimestamps{$b}} keys %mtimestamps) {
  32.   printf(OUTFILE $key . "\n");      # Print current folder name to file.
  33.   $i++;                             # Increment counter.
  34.   if ($i >= $printcount) { last; }  # If specified maximum of folders has been listed,
  35. }                                   # break and quit.
  36.  
  37. close(OUTFILE);  # Close output file, we're all done.

So, this is it. It does definitely work on both Windows NT 5.2 (using NTFS) as well as Linux using any Posix-compliant file system. You can use this to publish the latest changed files/directories on a server, or modify it to show you the latest accessed files even, just replace $mtime = stat($folder)->mtime; with something like $atime = stat($folder)->atime; to get access times instead of modification times. Of course, your file system needs to be mounted with enabled access time stamps, which should be the default. Both Windows and Linux however allow for the disabling of that feature to speed up the file system (noatime mount option in *nix or HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate registry key on Windows). Also, you can do the same with ctime to get creation time stamps.

So that is one way to monitor and visualize changes in your directory and file structure. I use this personally to write the data into an HTML file which i can then check on my web server easily. That way I can even monitor changes to certain directories and files online, which can be extremely useful. Another nice little something in Perl from a beginner. ;)