Sep 192013
 

Perl logoI have this script here that I wrote to walk an entire tree of directories (a rather flat structure) and that generates several lists of directories sorted by last modification time. So the last modified directories are determined and then published on a web server. Sounds neat enough.

Yesterday I tried to extend it to not only generate lists of the last 30 and 100 changed directories, but also a full sorted list containing all elements of the folder structure. By doing so, I found a nasty bug that kept me busy for an hour or so. Thing is, to read the folders I used something known as Perl Globbing. According to the Internets it’s one of the nastier things you can do. Works like that:

use strict;                        # Strict code only.
 
my $basepath = "/home/myuser";     # Directory to read from.
my @folders = glob("$basepath/*"); # Reading all element names in $basepath (files+dirs).
# my @folders = <$basepath/*>;     # This is the same thing actually.
my $folder = "";                   # Single elements name.
 
foreach $folder (@folders) {       # Walking through the array of files and dirs.
  if (-d $folder) {                  # We only want to show directories, no files!
    print ($folder . "\n");          # Print directory names.
  }
}

This would read all elements in /home/myuser/* into an array and then show them on the shell. That’s UNIX style, it could also be C:/somepath on Windows with Perl auto-converting / to \ internally. This does break however as soon as $basepath contains whitespaces, because glob() from Perls CORE::glob uses whitespaces as search pattern separators, and you can NOT change or escape that. See this example:

use strict;
 
my $basepath = "/home/my little user";
my @folders = glob("$basepath/*");
my $folder = "";
 
foreach $folder (@folders) {
  if (-d $folder) {
    print ($folder . "\n");
  }
}

Now this won’t show anything, because instead of looking for /home/my little user/*, it will look for /home/my, then little, and finally user/* in the current path. Clearly, this sucks. But there are two other implementations of glob(), one in File::DosGlob and one in File::Glob. Let me just say, even when on Windows, the Microsoft-specific File::DosGlob just does the same crap with whitespaces. The one in File::Glob however is bsd_glob(), which doesn’t do crap like that. If you want to call it the same way as above, you’ll have to request it as glob() function specifically though, as CORE functions always take precedence over others. Oh and, while it’s called bsd_glob(), don’t worry, this also works on Windows.

So the following will work just fine:

use strict;
use File::Glob 'glob';
 
my $basepath = "/home/my little user";
my @folders = glob("$basepath/*");
my $folder = "";
 
foreach $folder (@folders) {
  if (-d $folder) {
    print ($folder . "\n");
  }
}

You can also just call bsd_glob() by it’s right name instead of course, which is probably more clean anyway. In that case, you can use the module just like use File::Glob;. What do we learn from this? BSD UNIX’ globbing implementation is best. And if you want to be on the safe side, just don’t use globbing at all. Everybody seems to agree that it’s dirty and shitty. But hey, it’s also QUICK and dirty. ;) Still, if you want to do it more cleanly, you can use [opendir()] and [readdir()] instead, both of which work on directory handles, which are similar to file handles, that you can also pass around, making it more flexible.

Yeah, I didn’t do that because I didn’t want to write 3-4 more lines of code… Just fine as it is now! :roll:

Aug 062013
 

WebDAV logoRecently I was reading through my web servers log file, only to find some rather alarming lines that I didn’t immediately understand. It was some HTTP PUT methods obviously trying to implant PHP scripts onto my server via WebDAV. A few years back I had WebDAV implemented on my webserver so users could run subversion repositories for software development on my server, via HTTPS+SVN.

So I did a little bit of research and found out, that there were some Apache web server distributions like XAMPP, that had WebDAV switched on by default, and that included a default user and password for – now hold on – full access!

I never allowed anonymous access to the WebDAV repositories, which were isolated from the usual web directories anyway. Still, there were some lines that made me feel a bit uncomfortable, like this one, which seemed to be an attempt at doing an anonymous, unauthenticated PUT:

199.167.133.124 - - [24/Dec/2011:01:53:20 +0100] "PUT /webdav/sPriiNt...php HTTP/1.1" 200 346

Now, the part that made my eyebrows raise was that my web server seems to have responded with HTTP 200 to this PUT method right there, which means that an existing resource has been modified, or in other words overwritten even though my server never had a path “/webdav/”. For the creation of a fresh new file it should’ve responded with HTTP 201 instead. So there were many such PUT lines in the log file, but I was unable to locate the initial uploads, the one where my server should’ve responded with “PUT /webdav/sPriiNt…php HTTP/1.1” 201 346.

In the meantime, WebDAV is long switched off and my server now responds with 405 to all PUT attempts (“Method not implemented”). But I kept all the data of all directories and could still not locate any of those PHP files, nor any DELETE method calls in the log file. Now did my server really accept the PUT or not?! It seems not, especially since those scripts were never accessed via HTTP according to the logs. Well, at least that was all before I had to reinstall the server anyway, because of a major storage fuckup.

Still, I went ahead and googled for some of the PHP script names, and I found some really interesting shit. Looking for them, I found dubious lists of hundreds of web servers with both IP addresses and hostnames and the according paths to said uploaded PHP scripts. Most of them where obsolete already, but I managed to find some vulnerable servers where the PHP script was still accessible and executable.

What I found was a DDoS script that enabled you to specify a target IP/host and a time and duration of attack. That means, all those scripts are uploaded via what I assume to be a botnet to a wide range of vulnerable web servers, and then lists would be created for the human attackers to use and attack arbitrary targets in a timed fashion.

So I continued to search around and finally found some already modified full source code plus a WebDAV vulnerability exploit all packed together in a nice RAR archive, even including a tutorial written for what seems to be “non-hackers”. With that it was extremely easy to reconstruct an even more dangerous variety of the attack procedure which was targetted at Windows servers specifically:

  • Let the bundled WebDavLinkCrawler scan for WebDAV-capable servers for as long as you please, then press “stop”.
  • Copy all IP addresses collected by the crawler and presented to the user, save them in a TXT file.
  • Open the IP scanner exploit and load that TXT file. Let it scan all WebDAV capable servers for exploitable ones.
  • After that’s done, save the list of vulnerable servers.
  • Verify an arbitrary server by surfing to http://<ip>/webdav, looking for a “WebDAV testpage”. If it’s there, it worked.
  • Install the bundled BitKinex tool, press CTRL+2, enter an arbitrary name, then enter the servers IP address. Use “wampp” as the user name, “xampp” as the password. (That’s said default credentials with full access). Enter “/webdav” as the data source.
  • Click on the Http/WebDAV tab, click browse, and upload your PHP exploit!
  • Access http://<ip>/webdav/yournicephpscript.php. A shell will come up if system calls are allowed.
  • Run the following commands in that shell: net user <username> /add, then net user <username> <newpassword>, then net localgroup “Administrators” “<username>” /add. Only works if XAMPP is running with administrative privileges of course, but on Windows machines it often does.
  • Connect to the IP via RDP (remote desktop protocol, built into pretty much any Windows OS). Welcome to your free server!

Well, now this was even more dangerous and less likely to succeed, but I can imagine there where lots of successful attacks like that. And maybe even more that aimed at providing the DDoS interface I was looking for in the beginning. Good thing that my web server runs under a user without any administrative privileges, eh?

What surprised me most is how much the exploit is not just targeted at Windows servers, but also seemingly meant to be used by regular Windows operators. And there I was, thinking that people would usually attack from Linux or BSD machines. But na-ah. All EXE files, RAR archives and easy tutorials. As if the black hatters would have a large “audience” willingly helping them pull off their attacks.

An interesting thought, don’t you think?

Also funny that parts of the tutorials were written in German, more precisely the part with the WebDavLinkCrawler which seems to be a german tool.

This does offer some interesting insights into how DDoS and server hijacking attempts are pulled off. How or why people are making those exploits newbie-proof I still do not fully understand. Maybe this is also some sort of institutionalized cracking going on? I simply do not know enough to be able to tell.

But it sure doesn’t look like what I would have expected. No UNIX stuff there at all. And I still wonder whether I ever was exploitable or not, given how I had WebDAV locked down. Who knows.

Jul 182013
 

Buffalo logoSince a colleague of mine has [rooted] our Buffalo Terastation III NAS (TS-XLC62) at work a while back, we changed the remote shell from Telnet to SSH and did a few other nice hacks. But there is one problem: The TS-XLC62 does not monitor the hard drives’ health by SMART, even though parts of the required smartmontools are installed on the tiny embedded Linux system. They’re just stitting there unused, just like the sshd before.

Today I’m going to show you how you can make this stuff work and how to enable SMART email notifications on this system, which has no standard Linux mail command, but a Buffalo-specific tmail command instead. We will enable the background smartd service, and configure it properly for this specific Terastation model. All of the steps shown here are done on a rooted TS-XLC62, so make sure you’re always root here:

Buffalo Terastation IIIThe smartmontools on the box are actually almost complete. Only the drive database and init scripts are missing, and for some reason, running update-smart-drivedb on the system would fail. So we need to get the database from another Linux/UNIX or even Windows machine running smartmontools. Usually, on Linux you can find the file here: /usr/local/share/smartmontools/drivedb.h“. Copy it onto the Terastation using scp from another *nix box: scp /usr/local/share/smartmontools/drivedb.h root@<terastation-host-or-ip>:/usr/local/share/smartmontools/“. You can use [FileZilla] or [puTTY] to copy stuff over from a Windows machine instead.

Note that this only makes sense if you have smartmontools 5.40 or newer (smartctl -V tells the version). Older releases cannot have their drive databases updated seperately, but it will most likely still work fine.

Now, log in to your Terastation using Telnet or SSH, and you can test whether it’s working by running a quick info check on one of the hard drives. We will need to specify the controller type as marvell, as the SATA controller of the Marvell Feroceon  MV78XX0 SoC in the box cannot be addressed by regular ATA/SCSI commands. Run:

smartctl -d marvell -i /dev/sda

In my case I get this, as I have already replaced the first failing Seagate hard drive with an even crappier WD one already (yeah, yeah, I know, but it was the only one available), it’s also not yet known by the smartmontools database:

smartctl version 5.37 [arm-none-linux-gnueabi] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZAL555899
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Thu Jul 18 09:54:53 2013 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Now that that’s done we should make sure that smartmontools’ daemon called smartd will be running in the background doing regular checks on the drives. But since we will need to configure email notifications for that, we need to make sure that smartd can send emails first. The Terastation has no mail command however, only some Buffalo tmail command, that is no valid drop-in replacement for mail as the syntax is different.

So we need to write some glue-code, that will then later be invoked by smartd. I call this mailto.sh, and I’ll place it in /usr/local/sbin/. It’s based on [this article], that gave me a non-working solution on my Terastation for several reasons, but that’s easily fixed up, so it shall look somewhat like this (you’ll need to fill in several of the variables with your own data of course), oh, and as always, don’t forget to do chmod 550 on it when it’s done:

expand/collapse source code
  1. #! /bin/bash
  2. ##############################################################
  3. # Written as glue code, so that smartmontools/smartd can use #
  4. # Buffalos own "tmail", as we don't have "mail" installed    #
  5. # on the Terastation.                                        #
  6. ##############################################################
  7.  
  8. # User-specific declarations:
  9.  
  10. TMP_FILE=/tmp/Smartctl.error.txt
  11. SMTP=&lt;PUT SMTP HOST NAME HERE&gt;
  12. SMTP_PORT=25
  13. FROM=&lt;PUT "FROM" EMAIL ADDRESS HERE&gt;
  14. SUBJECT="SMART Error"
  15. FROM_NAME=&lt;PUT "FROM" NAME HERE. LIKE "TERASTATION MAILER"&gt;
  16. ENCODING="UTF-8"
  17. BYTE=8
  18.  
  19. # Code:
  20.  
  21. # Write email metadata to the temp file (smartd gives us this):
  22. echo To:  $SMARTD_ADDRESS &gt; $TMP_FILE 
  23. echo Subject:  "$SMARTD_SUBJECT" &gt;&gt; $TMP_FILE 
  24. echo &gt;&gt; $TMP_FILE 
  25. echo &gt;&gt; $TMP_FILE 
  26.  
  27. # Save the email message (STDIN) to the temp file:
  28. cat &gt;&gt; $TMP_FILE 
  29.  
  30. # Append the output of smartctl -a to the message:
  31. smartctl -a -d $SMARTD_DEVICETYPE $SMARTD_DEVICE &gt;&gt; $TMP_FILE 
  32.  
  33. # Now email the message to the user using Buffalos mailer:
  34. tmail -s $SMTP -t $SMARTD_ADDRESS -f $FROM -sub $SUBJECT \
  35. -h $FROM_NAME -c $ENCODING -b $BYTE -s_port $SMTP_PORT &lt; $TMP_FILE 
  36.  
  37. # Delete temporary file
  38. rm -f $TMP_FILE

So this is our mailer script wrapping the stuff coming from smartd's invocation of mail around Buffalos own tmail. Now how do we make smartd call this? Let’s edit /usr/local/etc/smartd.conf to make it happen, fill in your email address where it says here, like you changed all the variables in mailto.sh before:

  1. # Monitor all four harddrives in the Buffalo Terastation with self-tests running
  2. # on Sunday 01:00AM for disk 1, 02:00AM for disk 2, 03:00AM for disk 3 and 04:00AM
  3. # for disk 4:
  4.  
  5. /dev/sda -d marvell -a -s L/../../7/01 -m &lt;EMAIL&gt; -M exec /usr/local/sbin/mailto.sh
  6. /dev/sdb -d marvell -a -s L/../../7/02 -m &lt;EMAIL&gt; -M exec /usr/local/sbin/mailto.sh
  7. /dev/sdc -d marvell -a -s L/../../7/03 -m &lt;EMAIL&gt; -M exec /usr/local/sbin/mailto.sh
  8. /dev/sdd -d marvell -a -s L/../../7/04 -m &lt;EMAIL&gt; -M exec /usr/local/sbin/mailto.sh

Now if you want to test the functionality of the mailer beforehand, you can use this instead:

/dev/sda -d marvell -a -s L/../../7/01 -m &lt;EMAIL&gt; -M exec /usr/local/sbin/mailto.sh -M test

To test it, just run smartd -d on the shell. This will give you debugging output including some warnings caused by a bit of unexpected output that tmail will pass to smartd. This is non-critical though, it should look similar to this:

smartd version 5.37 [arm-none-linux-gnueabi]
Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Opened configuration file /usr/local/etc/smartd.conf
Configuration file /usr/local/etc/smartd.conf parsed.
Device: /dev/sda, opened
Device: /dev/sda, is SMART capable. Adding to "monitor" list.
Monitoring 1 ATA and 0 SCSI devices
Executing test of /usr/local/sbin/mailto.sh to <EMAIL> ...
Test of /usr/local/sbin/mailto.sh to <EMAIL> produced unexpected 
output (50 bytes) to STDOUT/STDERR: 
smtp_port 25
Get smtp portnum 25
pop3_port (null)

Test of /usr/local/sbin/mailto.sh to <EMAIL>: successful

Now you can kill smartd on a secondary shell by running the following command. We will be re-using this in an init script later too, as the Terastation init functions are leaving quite a lot to be desired, so I’ll go into the details a bit:

kill `ps | grep smartd | grep -v grep | cut -f1 -d"r"`

This command will get the process id of smartd and feed it to the kill command. The delimiter “r” is used for the cut command, because whitespace won’t work in some cases where the leading character of the ps output is also a whitespace, so it’ll match the first letter of the user running smartd, which has to be root.

To understand this better, just run ps | grep smartd | grep -v grep while smartd is running. If the PID is 5-digit, the leading character will be a number from the PID, but if it is 4-digit, the leading character is a whitespace instead, which would make cut -f1 -d " " report an empty string in our case, hence cut -f1 -d"r"… Very dirty, I know… Don’t care though. ;) You may remove the -M test directive from /usr/local/etc/smartd.conf now, if you’ve played around with that, so the smart spam will stop. :roll:

Finally, to make our monitoring run as a smooth auto-starting daemon in the background, we will need to write ourselves that init script. The default smartmontools one won’t work out of the box, as a few functions like killproc or daemon are missing on the Terastations embedded Linux. Yeah, I was too lazy to port them over. So a few adaptions will make it happen in a simplified fashion. See this reduced and adapted init script called smartd sitting in /etc/init.d/:

expand/collapse source code
  1. #! /bin/sh
  2. SMARTD_BIN=/usr/local/sbin/smartd
  3.  
  4. RETVAL=0
  5. prog=smartd
  6. pidfile=/var/lock/subsys/smartd
  7. config=/usr/local/etc/smartd.conf
  8.  
  9. start()
  10. {
  11.         [ $UID -eq 0 ] || exit 4
  12.         [ -x $SMARTD_BIN ] || exit 5
  13.         [ -f $config ] || exit 6
  14.         echo -n $"Starting $prog: "
  15.         $SMARTD_BIN $smartd_opts
  16.         RETVAL=$?
  17.         echo
  18.         [ $RETVAL = 0 ] &amp;&amp; touch $pidfile
  19.         return $RETVAL
  20. }
  21.  
  22. stop()
  23. {
  24.         [ $UID -eq 0 ] || exit 4
  25.         echo -n $"Shutting down $prog: "
  26.         kill `ps | grep smartd | grep -v grep | cut -f1 -d"r"`
  27.         RETVAL=$?
  28.         echo
  29.         rm -f $pidfile
  30.         return $RETVAL
  31. }
  32.  
  33. *)
  34.         echo $"Usage: $0 {start|stop}"
  35.         RETVAL=2
  36.         [ "$1" = 'usage' ] &amp;&amp; RETVAL=0
  37.  
  38. esac
  39.  
  40. exit $RETVAL

So yeah, instead of killproc we’re making due with kill and most of the service functions have been removed, limiting the script to start and stop. Plus, it will not check for multiple start invocations in this version, so it’s possible to start multiple smartd daemons and stop will only work for one running process at a time, so you’ll need to pay attention. Could be fixed easily, but I think it’s good enough that way. To make smartd start on boot, link it properly, somewhat like that, I guess S90 should be fine:

ln -s /etc/init.d/smartd /etc/rc.d/sysinit.d/S90smartd

Also, you can start and stop smartd from the shell more conveniently now without having to run smartd in the foreground and kill it from a secondary shell as it doesn’t have CTRL+C kill it. You can now just do these two things instead, like on any other SysVinit system, only with the limitations described above:

root@TS-XLC62:~# /etc/init.d/smartd stop
Shutting down smartd: Terminated
root@TS-XLC62:~# /etc/init.d/smartd start
Starting smartd: 
root@TS-XLC62:~#

Better, eh? Now, welcome your SMART monitoring-enabled Buffalo Terastation with email notifications being sent on any upcoming hard drive problems detected by courtesy of smartmontools! :cool:

Edit: And here is a slighty more sophisticated init script, that will detect whether smartd is already running or not on start, so that multiple starts can no longer happen. It will also detect if smartd has been killed from outside the scope of the init scripts (like when it crashed or something) by looking at the PID file:

expand/collapse source code
  1. #! /bin/sh
  2. SMARTD_BIN=/usr/local/sbin/smartd
  3. RETVAL=0
  4. prog=smartd
  5. pidfile=/var/lock/subsys/smartd
  6. config=/usr/local/etc/smartd.conf
  7.  
  8. start()
  9. {
  10.   [ $UID -eq 0 ] || exit 4
  11.   [ -x $SMARTD_BIN ] || exit 5
  12.   [ -f $config ] || exit 6
  13.   if [ -f $pidfile ]; then
  14.     echo "PID file $pidfile found! Will not start,"
  15.     echo "smartd probably already running!"
  16.     PID=`ps | grep smartd | grep -v grep | grep -v "smartd start" | cut -f1 -d"r"`
  17.     if [ ${#PID} -gt 0 ]; then
  18.       echo "Trying to determine smartd PID: $PID"
  19.     elif [ ${#PID} -eq 0 ]; then
  20.       echo "No running smartd process found. You may want to"
  21.       echo "delete $pidfile and then try again."
  22.     fi
  23.     exit 6
  24.   elif [ ! -f $pidfile ]; then
  25.     echo -n $"Starting $prog: "
  26.     $SMARTD_BIN $smartd_opts
  27.     RETVAL=$?
  28.     echo
  29.     [ $RETVAL = 0 ] &amp;&amp; touch $pidfile
  30.     return $RETVAL
  31.   fi
  32. }
  33.  
  34. stop()
  35. {
  36.   [ $UID -eq 0 ] || exit 4
  37.   PID=`ps | grep smartd | grep -v grep | grep -v "smartd stop" | cut -f1 -d"r"`
  38.   if [ ${#PID} -eq 0 ]; then
  39.     echo "Error: No running smartd process detected!"
  40.     echo "Cleaning up..."
  41.     echo -n "Removing $pidfile if there is one... "
  42.     rm -f $pidfile
  43.     echo "Done."
  44.     exit 6
  45.   elif [ ${#PID} -gt 0 ]; then
  46.     echo -n $"Shutting down $prog: "
  47.     kill `ps | grep smartd | grep -v grep | grep -v "smartd stop" | cut -f1 -d"r"`
  48.     RETVAL=$?
  49.     echo
  50.     rm -f $pidfile
  51.     return $RETVAL
  52.   fi
  53. }
  54.  
  55. case "$1" in
  56.   start)
  57.     start
  58.     ;;
  59.   stop)
  60.     stop
  61.     ;;
  62.   restart)
  63.     stop
  64.     start
  65.     ;;
  66.   status)
  67.     ps | grep smartd | grep -v grep | grep -v status
  68.     RETVAL=$?
  69.     ;;
  70.   *)
  71.     echo $"Usage: $0 {start|stop|restart|status}"
  72.     RETVAL=2
  73.     [ "$1" = 'usage' ] &amp;&amp; RETVAL=0
  74. esac
  75.  
  76. exit $RETVAL

Mar 062013
 

Stereoscopy logoNow if you decide to read this post, you’ll be going through a lot of technical crap again, so to make it easier for you, click on the posts logo to the left. There you get an interesting picture to look at if you get bored by me talking, and if you apply the cross-eye technique (although you’ll have to cross your eyes up to a level where it might be a bit of a strain, just try it until you get a “lock” on the “middle” picture), you’ll see the image in sterescopic 3D. Yay, boobies in stereoscopic 3D, not real ones though, very unfortunate. Better than nothing though, eh?

Now, where was I? Ah yes, stereoscopy. These days it seems to be a big thing, not just in movie theatres, but also for the home user with 3D Blu-Rays and 3D HDTVs. I’m quite into transcoding Blu-Rays using the x264 encoder as well as a bunch of other helper tools, and I’m archiving all my discs to nice little MKV files, that I prefer to having to actually insert the BD and deal with all the DRM crap like ACSS encryption and BD+. Ew.

But, with 3D, there is an entirely new challenge to master. On a Blu-Ray, 3D information is stored as a so-called MVC stream, which is an extension to H.264/AVC, which also means, there is no Microsoft Codec involved anymore, so there is no VC-1 3D. Only H.264. This extension is for the right eye and works like a diff stream or like DTS MLP data roughly, so while the full 2D left eye stream may be 10GB in size, the right eye one (as it only contains parts that are different from the left eye stream) will be maybe around 5GB.  Usually, stereoscopic 3D works by giving you one image for your left and one for your right eye, while filtering the left eye stream for the right eye (so the right eye never sees the left eye stream) and vice-versa. There are slight differences in the streams, which trick your brain into interpreting the information presented to it as 3D depth.

A cheap version of “filtering” is the cross-eye technique, that works best for plain images, like the logo above, which is a so called side-by-side 3D image. Side-by-side is cheap, easy, and for movies it’s very compatible, because pretty much any 3D screen supports it. Also, there are free tools available to work with SBS 3D. But, before SBS 3D can be created, the method of the Blu-Ray (which is not SBS but “frame packing”, see my description of MVC above) needs to be decoded first. Since the MVC right eye stream is a diff, and not a complete one, the complete SBS right eye stream needs to be constructed from the MVC and the left-eye H.264/AVC first. That can be done using certain plugins in the AviSynth frameserver under Windows. One of those plugins is H264StereoSource, which offers the function H264StereoSource().

Using this, i successfully transcoded Prometheus 3D. However, that was the only success ever. I tried it again as I got new 3D Blu-Rays, but all of a sudden, this (click to enlarge):

x264 3D Crash

As you can see in the shell window, there is lots of weird crap instead of regular x264 output. At some point, rather sooner than later, x264 would just crash. Taking whatever output it has generated by that time, you get something that looks like this (click to enlarge):

H264StereoSource() failure

Clearly, this is bad. As an SBS, the right frame should look awfully familiar, almost mirroring the left one. As you can see, this looks awfully not very much like anything. Fragments of the shapes of the left eye stream can be recognized in the right eye one when the movie plays, but it’s totally garbled and a bit too.. well… green. Whatever. H264StereoSource() fucked up big time, so I had to look into another solution. Luckily, [Sharc], a [Doom9] user pointed me in the right direction by mentioning [BD3D2MK3D].

This is yet another GUI with some of the tools I already use in the background plus some additional ones, most significantly another AviSynth plugin called SSIFSource2.dll, offering the function ssifSource2(). That one reads an SSIF container (which is H.264/AVC and MVC linked together) directly, and can not only decode it, but also generate proper left/right eye output from it directly.

Now first, this is what my old AviSynth script looked like, the now broken one:

  1. LoadPlugin("C:\Program Files (x86)\VFX\AviSynth 2.5\plugins\DGAVCDecode.dll")
  2. LoadPlugin("C:\Program Files (x86)\VFX\BDtoAVCHD\H264StereoSource\H264StereoSource.dll")
  3. left = AVCSource("D:\left.dga")
  4. right = H264StereoSource("D:\crash.cfg", 132566).AssumeFPS(24000,1001)
  5. return StackHorizontal(HorizontalReduceBy2(left), HorizontalReduceBy2(right)).Trim(0, 132516)

This loads the H.264/AVC decoder, the stereo plugin, reads the left-eye stream index file, a dga created by DGAVCIndex, and loads a configuration file that defines both streams as they were demuxed by using eac3to. Then it returns a shrinked version of the output (compressing the 3840×1080 full SBS stream to 1920×1080 half SBS, that most TVs can handle), trimming the last 50 frames away to work around a frame alignment bug between decoder and encoder. The cfg file looks somewhat like this:

  1. InputFile = "D:\left.h264"
  2. InputFile2 = "D:\right.h264"
  3. FileFormat = 0
  4. POCScale = 1
  5. DisplayDecParams = 1
  6. ConcealMode = 0
  7. RefPOCGap = 2
  8. POCGap = 2
  9. IntraProfileDeblocking = 1
  10. DecFrmNum = 0

By playing around with it I found out that it was truly the right-eye MVC decoding that did no longer work, H264StereoSource() was just broken, so ssifSource2() then! BD3D2MK3D will allow you to use its GUI to analyze a decrypted 3D Blu-Ray folder structure, and generate an AviSynth script for you. There was only one part that did not work for me, and that was the determination of the frame count. But there’s an easy fix. The AviSynth script that BD3D2MK3D will generate will look somewhat like this:

expand/collapse source code
  1. # Avisynth script generated Wed Mar 06 18:07:23 CET 2013 by BD3D2MK3D v0.13
  2. # to convert "D:\0_ADVDWORKDIR\My Movie\3D\MYMOVIE\BDMV\STREAM\SSIF\00001.ssif"
  3. # (referenced by playlist 00001.mpls)
  4. # to Half Side by Side, Left first.
  5. # Movie title: My Movie 3D
  6. #
  7. # Source MPLS information:
  8. # MPLS file: 00001.mpls
  9. # 1: Chapters, 16 chapters
  10. # 2: h264/AVC  (left eye), 1080p, fps: 24 /1.001 (16:9)
  11. # 3: h264/AVC (right eye), 1080p, fps: 24 /1.001 (16:9)
  12. # 4: DTS Master Audio, English, 7.1 channels, 16 bits, 48kHz
  13. # (core: DTS, 5.1 channels, 16 bits, 1509kbps, 48kHz)
  14. # 5: AC3, English, 5.1 channels, 640kbps, 48kHz, dialnorm: 28dB
  15. # 6: DTS Master Audio, German, 5.1 channels, 16 bits, 48kHz
  16. # (core: DTS, 5.1 channels, 16 bits, 1509kbps, 48kHz)
  17. # 7: AC3, Turkish, 5.1 channels, 640kbps, 48kHz
  18. # 8: Subtitle (PGS), English
  19. # 9: Subtitle (PGS), German
  20. # 10: Subtitle (PGS), Turkish
  21.  
  22. #LoadPlugin("C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe\DirectShowMVCSource.dll")
  23. ## Alt method using ssifSource2 (for SBS or T&amp;B only):
  24. LoadPlugin("C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe\ssifSource2.dll")
  25.  
  26. SIDEBYSIDE_L     = 14 # Half Side by Side, Left first
  27. SIDEBYSIDE_R     = 13 # Half Side by Side, Right first
  28. OVERUNDER_L      = 16 # Half Top/Bottom, Left first
  29. OVERUNDER_R      = 15 # Half Top/Bottom, Right first
  30. SIDEBYSIDE_L     = 14 # Full Side by Side, Left first
  31. SIDEBYSIDE_R     = 13 # Full Side by Side, Right first
  32. OVERUNDER_L      = 16 # Full Top/Bottom, Left first
  33. OVERUNDER_R      = 15 # Full Top/Bottom, Right first
  34. ROWINTERLEAVED_L = 18 # Row interleaved, Left first
  35. ROWINTERLEAVED_R = 17 # Row interleaved, Right first
  36. COLINTERLEAVED_L = 20 # Column interleaved, Left first
  37. COLINTERLEAVED_R = 19 # Column interleaved, Right first
  38. OPTANA_RC        = 45 # Anaglyph: Optimised, Red - Cyan
  39. OPTANA_CR        = 46 # Anaglyph: Optimised, Cyan - Red
  40. OPTANA_YB        = 47 # Anaglyph: Optimised, Yellow - Blue
  41. OPTANA_BY        = 48 # Anaglyph: Optimised, Blue - Yellow
  42. OPTANA_GM        = 49 # Anaglyph: Optimised, Green - Magenta
  43. OPTANA_MG        = 50 # Anaglyph: Optimised, Magenta - Green
  44. COLORANA_RC      = 39 # Anaglyph: Colour, Red - Cyan
  45. COLORANA_CR      = 40 # Anaglyph: Colour, Cyan - Red
  46. COLORANA_YB      = 41 # Anaglyph: Colour, Yellow - Blue
  47. COLORANA_BY      = 42 # Anaglyph: Colour, Blue - Yellow
  48. COLORANA_GM      = 43 # Anaglyph: Colour, Green - Magenta
  49. COLORANA_MG      = 44 # Anaglyph: Colour, Magenta - Green
  50. SHADEANA_RC      = 33 # Anaglyph: Half-colour, Red - Cyan
  51. SHADEANA_CR      = 34 # Anaglyph: Half-colour, Cyan - Red
  52. SHADEANA_YB      = 35 # Anaglyph: Half-colour, Yellow - Blue
  53. SHADEANA_BY      = 36 # Anaglyph: Half-colour, Blue - Yellow
  54. SHADEANA_GM      = 37 # Anaglyph: Half-colour, Green - Magenta
  55. SHADEANA_MG      = 38 # Anaglyph: Half-colour, Magenta - Green
  56. GREYANA_RC       = 27 # Anaglyph: Grey, Red - Cyan
  57. GREYANA_CR       = 28 # Anaglyph: Grey, Cyan - Red
  58. GREYANA_YB       = 29 # Anaglyph: Grey, Yellow - Blue
  59. GREYANA_BY       = 30 # Anaglyph: Grey, Blue - Yellow
  60. GREYANA_GM       = 31 # Anaglyph: Grey, Green - Magenta
  61. GREYANA_MG       = 32 # Anaglyph: Grey, Magenta - Green
  62. PUREANA_RB       = 23 # Anaglyph: Pure, Red - Blue
  63. PUREANA_BR       = 24 # Anaglyph: Pure, Blue - Red
  64. PUREANA_RG       = 25 # Anaglyph: Pure, Red - Green
  65. PUREANA_GR       = 26 # Anaglyph: Pure, Green - Red
  66. MONOSCOPIC_L     = 1 # Monoscopic 2D: Left only
  67. MONOSCOPIC_R     = 2 # Monoscopic 2D: Right only
  68.  
  69. # Load main video (0 frames)
  70. #DirectShowMVCSource("D:\0_ADVDWORKDIR\My Movie\3D\MYMOVIE\BDMV\STREAM\SSIF\00001.ssif", seek=false, seekzero=true, stf=SIDEBYSIDE_L)
  71. ## Alt method using ssifSource2
  72. ssifSource2("D:\0_ADVDWORKDIR\My Movie\3D\MYMOVIE\BDMV\STREAM\SSIF\00001.ssif", 153734, left_view = true, right_view = true, horizontal_stack = true)
  73.  
  74. AssumeFPS(24000,1001)
  75. # Resize is necessary only for Half-SBS or Half-Top/Bottom,
  76. # or when the option to Resize to 720p is on.
  77. LanczosResize(1920, 1080)
  78. # Anaglyph and Interleaved modes are in RGB32, and must be converted to YV12.
  79. #ConvertToYV12(matrix="PC.709")

So here, when ssifSource2() is being called, there is a number “153734”, where initially there will only be “0”. I got the run time and frame rate of the movie from the file eac3to_demux.log, that BD3D2MK3D generates. Using that, you can easily calculate your total frame number to enter there, like I have already done in this example. Also enter the framerate in the AssumeFPS() function. Here, 24000,1001 means 24000/1001p or in other words roughly 23.976fps, which is the NTSC film frame rate. Now, the tricky part: For this function to operate, the libraries and binaries of BD3D2MK3D must be in the systems search path and our x264 encoder must sit in the same directory as the libs, also you will want to make sure you’re using a 32-Bit version of x264. In our example, the installation path of BD3D2MK3D is C:\Program Files (x86)\VFX\BD3D2MK3D\, and the tools and libs are in C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe\. So, open a cmd shell, copy your preferred 32-bit x264.exe to C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe\ and then add the necessary paths to the search path:

set %PATH%=C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe;C:\Program Files (x86)\VFX\BD3D2MK3D\toolset;%PATH%

Of course, you’ll need to adjust the paths to reflect your local BD3D2MK3D installation folder. Now, you can call x264 using the generated AviSynth script as its input (usually called _ENCODE_3D_MOVIE.avs), and it should work! Just make sure you call x264.exe from that folder C:\Program Files (x86)\VFX\BD3D2MK3D\toolset\stereoplayer.exe\, and not from somewhere else on the system, or you’ll get this: avs [error]: Can't create the graph!

For me, x264 using ssifSource2() sometimes talks about adding tons of duplicate frames while transcoding. I am not sure what this means and how it may affect things, but for now output seems to work just fine regardless. When it does happen, it looks like this:

ssifSource2() frame duplicates

Nonetheless, it creates half-SBS output, and using MKVtoolnix you can multiplex that into an MKV/MK3D container setting the proper stereoscopy flags (usually half SBS, left eye first) and everything will work just fine.

My method using half-SBS will cost half the horizontal resolution unfortunately, but few TVs support full SBS, plus it’s costly because files would become far larger at similar quality settings. So far I’ve been pretty surprised how good, sharp and detailed it still looks, despite the halved resolution. So far it’s the only real possibility anyway, as full-SBS (using two 1920×1080 frames besides each other) is not very wide-spread, large and bandwidth-hungry, and for the frame packed mode that Blu-Rays use, there are simply no free tools available so far, so direct MVC transcoding is not an option. :(

But, at least it works now, thanks to the [help from the Doom9 forums]!

Update: It seems the frame duplicates are a really weird bug, that makes the output very jittery and no longer as smooth as you would expect. Even stranger is the fact, that it is not deterministic. Sometimes it happens a lot, adding tens of thousands of frame duplicates. Then, abort, restart the process and the problem is magically gone. Sometimes you might have to restart 10-20 times and all of a sudden it works. How is this even possible? I do not know. Luckily, BD3D2MK3D also comes with a second Avisynth plugin that you can use instead of ssifSource2.dll, this one is called DirectShowMVCSource.dll and gives you the corresponding function DirectShowMVCSource().

For DirectShowMVCSource() you will also no longer need to determine a frame count, it will be able to do that all by itself. Also, BD3D2MK3D will automatically add it to the Avisynth script that it generates, it’s just commented out. So, uncomment it, comment the ssifSource2 lines instead, and you’re good to go. So far, the error rate seems to be far lower with DirectShowMVCSource(), although I had a problem once, with a very long movie / high frame count, here a 2nd pass encode would just stall at 0% CPU usage.

But other than that it seems to be more solid, even if it has to suboptimally filter stuff through DirectShow. There is another more significant drawback though: DirectShowMVCSource() is a lot slower than ssifSource2(). I would say, it’s a factor of 3-4 even. But at least it’s giving us another option!

Dec 202012
 

TimestampRecently, I got a request to determine and publish a list of latest created and/or modified files/folders on a filesystem and get that published in HTML form. Now since my scripting as of late has been cross-platform (Linux & Windows), I was aiming at achieving that again. So I started scripting Perl 5 on Linux, naturally using Unix style epoch time stamps, which can be done with a Perl internal set of functions or with the module Time::localtime.

So the idea is, you give the script a base folder, and it will scan all folders (or files if you wish) beneath, and output a list of the most recently added or changed ones. I decided to limit the output to what I may see fit, like the latest 20 or latest 100 added and modified folders. Very helpful: A folders timestamp gets updated when you add files to it or even when you change files in it.

So, here is some code for that (note that this might not exactly be executable as-is, because it may be incomplete, these are just extracted parts of the full code):

expand/collapse source code
  1. #!/usr/bin/perl
  2.  
  3. use strict;
  4. use File::stat;
  5.  
  6. my $basepath  = "X:/basefolder";  # Base folder.
  7. my @subfolders  = <$basepath/*>;  # Read subfolders to check for modifications within.
  8.  
  9. my $printcount  = 20;             # Amount of folders to display in final output.
  10.  
  11. my $outfile  = "C:/outputfile";   # File to write list to.
  12.  
  13. my $subfolder  = "";       # Variable to use for iteration through subfolders.
  14. my $folder  = "";          # Single folder name.
  15. my $mtime  = "";           # UNIX-style epoch-related time stamp of a single folder.
  16. my %mtimestamps  =   ;     # Hash with folder names as keys and time stamps as values.
  17. my $i  = 0;                # Arbitrary counter.
  18. my $key  = "";             # Arbitrary key for iteration through sorted timestamp hash 
  19.  
  20. open(OUTFILE, "<:encoding(UTF-8)", $outfile); # Open output file.
  21.  
  22. # Go through subfolders and check timestamps of folders within them (level 2):
  23. foreach $folder (@subfolders) {         # Iterate through subfolders.
  24.   if (-d $folder) {                     # Check if element is truly a directory.
  25.     $mtime = stat($folder)->mtime;      # Read modification time stamp of subfolder.
  26.     $mtimestamps{$folder} = $mtime;     # Add timestamp as a value to the hash for the
  27.   }                                     # current folder name as key.
  28. }
  29.  
  30. # Iterate through sorted hash, highest values (=newest folders) first:
  31. foreach $key (reverse sort {$mtimestamps{$a} <=> $mtimestamps{$b}} keys %mtimestamps) {
  32.   printf(OUTFILE $key . "\n");      # Print current folder name to file.
  33.   $i++;                             # Increment counter.
  34.   if ($i >= $printcount) { last; }  # If specified maximum of folders has been listed,
  35. }                                   # break and quit.
  36.  
  37. close(OUTFILE);  # Close output file, we're all done.

So, this is it. It does definitely work on both Windows NT 5.2 (using NTFS) as well as Linux using any Posix-compliant file system. You can use this to publish the latest changed files/directories on a server, or modify it to show you the latest accessed files even, just replace $mtime = stat($folder)->mtime; with something like $atime = stat($folder)->atime; to get access times instead of modification times. Of course, your file system needs to be mounted with enabled access time stamps, which should be the default. Both Windows and Linux however allow for the disabling of that feature to speed up the file system (noatime mount option in *nix or HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate registry key on Windows). Also, you can do the same with ctime to get creation time stamps.

So that is one way to monitor and visualize changes in your directory and file structure. I use this personally to write the data into an HTML file which i can then check on my web server easily. That way I can even monitor changes to certain directories and files online, which can be extremely useful. Another nice little something in Perl from a beginner. ;)

Nov 262012
 

SuSE LogoRecently, I have prepared a cheap HP 655 notebook for the mother of a colleague of mine. HP gives you the option of getting the book with SuSE Linux Enterprise Desktop 11 SP2 preinstalled, and I told my colleague to go for that option to minimize maintainance and potential security risks. It might be that she will have to adapt a little, but since she’s coming from Windows 98 (!), she would have had to anyways.

Now my colleague bought her a 3G mobile internet stick from the Austrian provider A1. Getting this piece of trash to work was a bit of a challenge, just plugging it in did absolutely nothing. No modem recognized, no nothing.

This is a stick built by ZTE, called the MF180 or MF190. These things are USB mass storage media when plugged in, meaning they act like a regular USB stick, providing a small storage space to give the user easy access to the Windows drivers, which are stored on the stick directly. Sounds neat, but is entirely useless in Linux. To switch from mass storage mode to modem mode, special sequences need to be sent over the USB bus to the stick, and a device id needs changing. SuSE Linux already has the proper software for that, which is udev in conjunction with “usb_modeswitch”. However, SLED 11 SP2 was missing the proper usb_modeswitch configuration for this stick, which had the vendor id 19d2 and the device id 2000 (which needed changing to 0117 for modem mode).

ZTE MF180/MF190 3.5G Stick

To make the magic happen, a file named <vendor id>:<device id> needs to be created in /etc/usb_modeswitch.d/. So in our case it’s called 19d2:2000, and should contain the following:

########################################################
# ZTE devices
 
DefaultVendor=  0x19d2
DefaultProduct= 0x2000
 
TargetVendor=   0x19d2
TargetProductList="0001,0002,0015,0016,0017,0031,0037,0052,0055,0063,0064,0066,0091,0108,0117,0128"
 
MessageContent="5553424312345678000000000000061e000000000000000000000000000000"
MessageContent2="5553424312345679000000000000061b000000020000000000000000000000"
MessageContent3="55534243123456702000000080000c85010101180101010101000000000000"
 
NeedResponse=1
 
CheckSuccess=20

That alone however doesn’t do the trick, at least not on SuSE Linux Enterprise Desktop 11. When usb_modeswitch does it’s stuff, it results in three new devices being created: /dev/ttyUSB0/dev/ttyUSB1 and /dev/ttyUSB2. The first two are an NMEA GPS device and a signal strength monitoring device, the last one is the desired modem device. Now when the kernel modem driver called option loads, it’s calling a probe() function trying to determine the device to bind to. This should always result in the driver binding to /dev/ttyUSB2, which does however sometimes not work. From time to time – in about half the cases – the driver erroneously binds to /dev/ttyUSB1, which results in the dialup not working of course.

Now, what one should do is write a udev rule that prevents the creation of the first two devices, which was too hard for me this time around (I know next to nothing about writing proper udev rules and I was being lazy!). So I went the route of quick and extremely dirty hacking. So instead of writing a proper udev rule, I just edited /etc/init.d/boot.d/S01boot.udev and added the following lines right after start):

mkdir /dev/ttyUSB0
mkdir /dev/ttyUSB1
chmod 000 /dev/ttyUSB0
chmod 000 /dev/ttyUSB1

Now this runs before the real udev starts, and by creating dummy directories in /dev/, it prohibits the device node creation of /dev/ttyUSB0 and /dev/ttyUSB1. This is extremely ugly as it may keep other USB devices from working, like maybe other 3G sticks or GPS sticks. However, the user will most likely never ever buy any such thing, so this time around it’s fine. Now the option driver can only see the one correct device to bind to: /dev/ttyUSB2. With that done and the pin code saved in Network Manager, the internet connection is established automatically whenever the stick is plugged in. No popup window, no nothing. Perfect where it needs to be as simple as possible.

Now that this works without any user interaction, and that all important software is linked to the desktop, it should be fine. To make usage of the web browser even easier than before, I chose Opera with its excellent SpeedDial functionality. It seems this is another successful implementation of Linux on the desktop!

Oct 252012
 

Sun Grid Engine LogoOk ok, I guess whoever is reading this (probably nobody anyway) will most likely already be tired of all this x264 stuff. But this one I need as documentation for myself anyway, because the experiment might be repeated later. So, the [chair for simulation and modelling of metallurgic processes] here at my university has allowed me to try and play with a distributed grid-style Linux cluster built by Supermicro. It’s basically one full rack cabinet with one Pentium 4 3.2GHz processor and 1GB RAM per node, with Hyper-Threading being disabled because it slowed down the simulation jobs that were originally being run on the cluster. Operating system for the nodes was OpenSuSE 10.3. Also, the head node was very similar to the compute nodes, which made it easy to compile libav and x264 on the head node and let the compute nodes just use those binaries.

The software installed for using it is the so called Sun GRID engine. I  have once already set up my own distributed cluster based on an OpenPBS style system called Torque, together with the Maui scheduler. When I was introduced to this Sun GRID engine, most of the stuff seemed awfully familiar, even the job submission tools and scripting system were quite the same actually. So this system uses tools like qsub, qdel, qstat plus some additional ones not found in the open source Torque system, like sns.

Now since x264 is not cluster-aware and not MPI capable, how DO we actually distribute the work across several physical machines? Lacking any more advanced approaches, I chose a very crude way to do it. Basically, I just cut the input video into n slices, where n is the number of cluster nodes. Since the cluster nodes all have access to the same storage backend via NFS, there was no need to send the files to the nodes, as access to the users home directory was a given.

Now, to make the job more easy, all slices were numbered serially, and I wrote a qsub job array script, where the array id would be used to specify the input file. So node[2] would get file[2], node[15] would get file[15] to encode etc. The job array script would then invoke the actual worker script. This is what the sliced input files look like before starting the computation:

Sliced input file

And here, the qsub job array script that I sent to the cluster, called benchmark-qsub.sh, the directory /SAS/home/autumnf is the users home directory:

#$ -N x264benchmark
#$ -t 1-19
 
export PATH=$PATH:/SAS/home/autumnf/usr/bin
echo $PATH
 
cd /SAS/home/autumnf/x264benchmark
 
time transcode.sh

And the actual worker script, transcode.sh:

#!/bin/bash
# Pass 1:
x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f -2:0 --bitrate 10000 --aq-mode 1 -p 1 --slow-firstpass --stats benchmark_slice$SGE_TASK_ID.stats -t 2 --no-fast-pskip --cqm flat slice$SGE_TASK_ID.264 -o benchmark_1stpass_slice$SGE_TASK_ID.264
 
# Pass 2:
x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f -2:0 --bitrate 10000 --aq-mode 1 -p 2 --stats benchmark_slice$SGE_TASK_ID.stats -t 2 --no-fast-pskip --cqm flat slice$SGE_TASK_ID.264 -o benchmark_2ndpass_slice$SGE_TASK_ID.264

As you can see, the worker is using the environment variable $SGE_TASK_ID as a part of the input and output file names. This variable contains the job array id passed down from the job submission system of the Sun GRID engine. The actual job submission script contains the line #$ -t 1-19 which tells the system, that the job array consists of 19 jobs, as the cluster had 19 working nodes left, the rest was already dead as the cluster was pretty much out of service and hence unmaintained. Let’s see how the Sun tool sns reports the current status of the grid cluster:

Empty Grid Cluster

So, some nodes are in “au” or “E” status. While I do not know the exact meaning of the status abbreviations, that basically means that those nodes are non-functional. Taking the broken nodes into account we have 19 working ones left. Now every node invokes its own x264 job and gives to it the proper input file from slice1.264 to slice19.264, writing correspondingly named outputs for both passes. Now let’s send the script to the Sun GRID engine using qsub ./benchmark-qsub.sh and check what sns has to say about this afterwards:

Sns reporting a grid cluster under load

Hurray! Now if you’re more used to OpenPBS style tools, we can also use qstat to report the current job status on the cluster:

Qstat showing a cluster under load

As you can see,  qstat also reports a “ja-task-ID”, which is essentially our job array id or in other words $SGE_TASK_ID. So thats basically one job with one job id, but 19 “daughter” processes, each having its own array id. Using tools like qdel or qalter you can either modify the entire job, or only subprocesses on specific nodes. Pretty handy. Now the Pentium 4 processor might suck ass, but 19 of them are still pretty damn powerful when combined, at the moment of writing you can find the cluster on [place #4 on the results list]! Here the Voodooalert style result, just under one hour:

0:58:01.600 | SMMP | 19/1/1 | Intel Pentium 4 (no HT) 3.20GHz | 1GB DDR-I/266 (per node) | SuperMicro/SGE GRID Cluster | OpenSuSE 10.3 Linux (Custom GCC Build)

To ensure that this is actually working, I have recombined the output slices of pass 2, and tried to play that file. To my surprise it worked and would also allow seeking. Pretty nice considering that there is quite some bogus data in the file, like multiple H.264/AVC headers or cut up frames. I originally tried to split the input file into slices at keyframes in a clean fashion using ffmpeg, but that just wouldn’t work for that type of input, so I had to use dd, resulting in some frames being cut up (and hence dropped), and slices 2-19 having no headers. That required very specific versions of libav and x264, as not all versions can accept garbled files like this.

Also, the output files have been recombined using dd. Luckily, mplayer using libav/ffmpeg would play that stuff nicely, but there’s simply no guarantee that every player and/or decoder would. So that’s why it cannot be considered a clean solution. Also, since motion estimation is less efficient for this setup at the cutting points, it’s not directly comparable to a non-clustered run. So there are some drawbacks. But if you would cluster x264 for productive work, you’d still do it kind of like that. Here, the final output, already containing the final concatenated file, quite a mess of files right there:

Clusterrun done

So this is it. The clustered x264. I hope to be able to test this approach on another cluster at the Metallurgy chair in the next months, a Nehalem-based machine with far more cores, so that’d be really massive. Also, access to a Sandy Bridge-E cluster is possible, although not really probable. But we’ll see. If you’re interested in using x264 in a similar approach, you might want to check out the software versions that I used, these should be able to cope with rudely cut up slices quite well:

Also, if you require some guidance building that source code on Linux, please check out my guide:

If anybody knows a better way to slice up H.264/AVC elementary video streams, by all means, let me know! I would love to be able to have slices cut at proper keyframe positions including their own header, and I would also like to be able to reconcatenate slices to one file that is clean, having only one header at the beginning of the file and no damaged / to be dropped frames at their joints. So if you know how to do that – preferrably using Linux command line tools – just tell me, I’d be happy to learn that!

Edit: Thanks to [LoRd_MuldeR] from the [Doom9 Forums] I now have a way of splitting the input stream cleanly at GOP (group of pictures) boundaries, as the Elephants Dream movie is luckily using closed GOPs. Basically, it involves the widely-used [MKVtoolnix]. With that tool, you can just take the stream, split it to n MKV slices, and then either use those or extract the H.264/AVC streams from those slices, maybe using [tsMuxer]. Just make as many as your cluster has compute nodes, and you’re done!

By the way, both MKVtoolnix and tsMuxer are available for Windows and Linux, also MacOS X.

This is clean, safe and proper, other than my dirty previous approach!

Sep 202012
 

x264 LogoI have played around with PHP a little again, and actually managed to generate PNG images with some elements rendered to them using a few basic GD functions of the scripting language. This is all still very new to me, so don’t be harsh! ;)

I thought I might use this to create some dynamic and more fancy than plain text statistics about the [x264 benchmark]. I decided to do some simple stats about operating systems and CPUs first, didn’t want to overdo it.

So I went for basic OS families and a more broken down visualization of all Windows and all UNIX derivatives. For microprocessors I went for basic architecture families (x86, RISC, VLIW) and a manufacturer breakdown. I know “x86” should probably have been “CISC” instead, but since x86 in itself is so wide-spread, I thought I should just make it its own family. See the following links:

Just so you can see how the generated images look like, I’ll link them in here. As you can see I decided to keep it very plain and simple, no fancy graphics, operating systems first:

Operating systems

Windows operating systems

Windows operating systems

And the microprocessors:

Microprocessor architectures

Microprocessor manufacturers

Not too bad for my first PHP-generated dynamic images? I would sure like to think so. ;)

Aug 232012
 

x264 LogoI always had the idea of actually visualizing the frametimes during the x264 benchmark to show where it’s fast, and where it’s not when transcoding the open source movie “Elephants Dream”. I have now finally succeeded in writing a Perl script that grabs a slightly modified x264 output. x264 can be told to output a small line of statistics for each frame transcoded. It writes that to the STDERR stream, which I decided to redirect to STDOUT and pipe into my Perl script, which in turn makes use of modern x86 systems HPET (High Precision Event Timer) using the Time::HiRes Perl module to fetch UNIX epoch timestamps for each successfully transcoded frame, and then calculates the time difference between each current and last frame. Here is the modified x264 calls for both passes, writing the frametimes to plain text files:

#!/bin/bash
# Pass 1:
x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f -2:0 --bitrate 10000 --aq-mode 1 -p 1 --slow-firstpass --stats benchmark.stats -t 2 --no-fast-pskip --cqm flat -v elephantsdream_source.264 -o /dev/null 2&gt;&amp;1 | ./writestamps.pl ./frametimes-pass1.txt
 
# Pass 2:
x264 --preset veryslow --tune film --b-adapt 2 --b-pyramid normal -r 3 -f -2:0 --bitrate 10000 --aq-mode 1 -p 2 --stats benchmark.stats -t 2 --no-fast-pskip --cqm flat -v elephantsdream_source.264 -o /dev/null 2&gt;&amp;1 | ./writestamps.pl ./frametimes-pass2.txt

And here is the source code for that “writestamps.pl” script. It’s not exactly optimal, it would probably be better to first collect the time diffs in an array and only flush to disk at the end, but I didn’t have time to do that today. Even as it is now it already gives a good idea of the frame times. First the source, but please know that the usage of HPET with Time::HiRes does not work on Windows, it’s simply not implemented there, so this is UNIX/Linux/BSD only I guess, no nanosecond precision on Windows, at least not with Time::HiRes:

#!/usr/bin/perl
 
use strict;        # Perl code is strict, no dodgy stuff!
use Time::HiRes;   # Highres Time functions (nanosecond precision)
 
my $localtime = Time::HiRes::clock_gettime(); # UNIX epoch timestamp
my $oldtime = $localtime;                     # Variable for timestamp of last frame
my $stampfile = $ARGV[0];                     # File name to write timestamp differences to
my $diff;                                     # Time difference between frames
 
# Opening timestamp file:
open(STAMPFILE, "&gt;:encoding(UTF-8)", $stampfile) || die "$stampfile could not be opened! Error: $!";
 
while (my $line = &lt;STDIN&gt;) {                 # As long as STDIN is coming from x264..
  if ($line =~ m/bytes/) {                     # Check if line is a per-frame statistic line. If so..
    $localtime = Time::HiRes::clock_gettime();   # Read current UNIX epoch timestamp
    $diff = $localtime - $oldtime;               # Calculate time difference between this frame and last frame
    $oldtime = $localtime;                       # Current frame is becoming old frame
    printf(STAMPFILE "%f", $diff);               # Printing time diff to file in fixed-point format
    printf(STAMPFILE "\n");                      # Printing UNIX line break to file
  }
}
 
close(STAMPFILE);                            # Closing timestamp statistics file.

And this is what we get from it:

Pass 1:

x264 benchmark frametimes pass 1

Pass 2:

x264 benchmark frametimes pass 2

So, I have just updated the code to not do I/O in every iteration for each line that comes piped from x264. Now, those data is being collected into an array, and only flushed to disk at the very end, which should make the data more representative without any I/O disturbing the measurements. Graphs will be updated when the run based on this new code is completed:

#!/usr/bin/perl
 
use strict;           # Perl code is strict, no dodgy stuff!
use Time::HiRes;      # Use HPET timer for nanosecond precision timestamps.
 
my $localtime = Time::HiRes::clock_gettime(); # Read current UNIX epoch timestamp.
my $oldtime = $localtime;                     # Define stamp of previous frame.
my $stampfile = @ARGV[0];                     # Fetch filename of timestamp file from cli.
my $diff;                                     # Time difference between frames.
my @diffs = ();                               # Array that holds time diffs before final flush to disk.
 
while (my $line = &lt;STDIN&gt;) {                # As long as there is STDIN coming from x264..
  if ($line =~ m/bytes/) {                    # Check if line is indeed a per-frame output line.
    $localtime = Time::HiRes::clock_gettime();  # Read UNIX epoch timestamp for current frame.
    $diff = $localtime - $oldtime;              # Calculate diff between current and previous frame.
    push (@diffs, $diff);                       # Push difference into array for later extraction.
    $oldtime = $localtime;                      # Current frame becomes old frame.
  }
}
 
# Open timestamp file:
open(STAMPFILE, "&gt;:encoding(UTF-8)", $stampfile) || die "$stampfile could not be opened! Error: $!";
 
foreach (@diffs) {             # Iterate through array of time differences per frames.
  printf(STAMPFILE "%f", $_);    # Print time diffs to file in fixed-point format, so we don't
  printf(STAMPFILE "\n");        # accidentally get scientific notation. Then print UNIX line break.
}
 
close(STAMPFILE);              # Close timestamp file.

Additionally to the graphs seen above, user “Elianda” from [Voodooalert]German flag has smoothed out the data of both passes to 500 instead of the full 15691 data points and gave me a software called “Igor 6”, which I used to create some more readable plots, the data here is already based on the new version of my Perl script, which is no longer affected by disk I/O:

Pass 1:

x264 benchmark frametimes smoothed pass 1

Pass 2:

x264 benchmark frametimes smoothed pass 2

So much for visualizing that frametime data!

Jul 092012
 

PHP & MySQL LogoServing UTF-8 can be useful. Why? For instance to display text in several languages on your website. If you want to mix different European and for instance Asian languages, more limited character sets won’t suffice anymore. Thus, UTF-8. In my case, I wanted to display a mix of European and Chinese Mandarin. However, my PHP scripts, MySQL database and even the script textfiles themselves were not prepared. So what do you need to do? First, make sure your database is encoded in UTF-8, using a proper collation too:

ALTER DATABASE `dbname` CHARACTER SET `utf8` COLLATE `utf8_general_ci`;
ALTER TABLE `tablename` CONVERT TO CHARACTER SET `utf8` COLLATE `utf8_general_ci`;

Now your raw data will be stored in UTF-8. The next step is rather easy; Ensure that your script files themselves (also any raw HTML present) are encoded in UTF-8. For that – on MS Windows – you can just use [Notepad++], it can autodetect your source character set and convert to UTF-8.

Additionally, any raw HTML that you might have or that is written by PHP needs a proper encoding definition in the <head></head> section to tell the browser “My text is UTF-8”. Like this:

<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Your Title</title>
</head>

Good. Now the hard part. Since PHP sucks a bit, it will default to ISO-8859-1 in pretty much everything it’s doing, that’s basically some modern Latin1 character set including the ‘€’ symbol. But that ain’t good enough for displaying some Japanese or Chinese, so we need to take special care here. The first important thing is to make sure that PHP will read data from MySQL in the properly encoded fashion. Otherwise it will read UTF-8 assuming it’s ISO-8859-1, hence garbling most of the text. So when defining your MySQL connection in PHP, you need to add the mysql_set_charset() function. Could look like this:

$connection = mysql_connect($sql_host,$sql_user,$sql_pass)
mysql_select_db($sql_dbname, $connection)
mysql_set_charset('utf8',$connection);

Now PHP will read and output UTF-8 by default. But be aware that there are additional caveats in PHP! Most string processing functions like e.g. substr() won’t process strings with large multibyte characters correctly. They are hardcoded for ISO-8859-1! Luckily you can load the PHP extension mb_string, which provides multibyte-capable string processing functions. They just use a mb_ prefix, so substr() simply becomes mb_substr()! Still, that means you might need to update your PHP code for UTF-8 capability, sometimes significantly. A documentation of all multibyte-related functions in PHP can be found [here]. With that, handling UTF-8 in both input, processing and output shouldn’t be a problem anymore.