Nov 142014
 

MOD Music on FreeBSD 10 with xmmsAs much as I am a WinAmp 2 lover on Microsoft Windows, I am an xmms lover on Linux and UNIX. And by that I mean xmms1, not 2. And not Audacious. Just good old xmms. Not only does it look like a WinAmp 2 clone, it even loads WinAmp 2/5 skins and supports a variety of rather obscure plugins, which I partially need and love (like for Commodore 64 SID tunes via libsidplay or libsidplay2 with awesome reSID engine support).

Recently I started playing around with a free Laptop I got and which I wanted to be my operating systems test bed. Currently, I am evaluating – and probably keeping – FreeBSD 10 UNIX. Now it ain’t perfect, but it works pretty well after a bit of work. Always using Linux or XP is just a bit boring by now. ;)

One of the problems I had was with xmms. While the package is there, its built-in tracker module file (mod, 669, it, s3m etc.) support is broken. Play one of those and its xmms-mikmod plugin will cause a segmentation fault immediately when talking to a newer version of libmikmod. Also, recompiling [multimedia/xmms] from the ports tree produced a binary with the same fault. Then I found this guy [Jakob Steltner] posting about the problem on the ArchLinux bugtracker, [see here].

Based on his work, I created a ports tree compatible patch patch-drv_xmms.c, here is the source code:

  1. --- Input/mikmod/drv_xmms.c.orig        2003-05-19 23:22:06.000000000 +0200
  2. +++ Input/mikmod/drv_xmms.c     2012-11-16 18:52:41.264644767 +0100
  3. @@ -117,6 +117,10 @@
  4.         return VC_Init();
  5.  }
  6.  
  7. +static void xmms_CommandLine(CHAR * commandLine)
  8. +{
  9. +}
  10. +
  11.  MDRIVER drv_xmms =
  12.  {
  13.         NULL,
  14. @@ -126,7 +130,8 @@
  15.  #if (LIBMIKMOD_VERSION > 0x030106)
  16.          "xmms",
  17.          NULL,
  18. -#endif
  19. +#endif
  20. +       xmms_CommandLine, // Was missing
  21.          xmms_IsThere,
  22.         VC_SampleLoad,
  23.         VC_SampleUnload

So that means recompiling xmms from source by yourself. But fear not, it’s relatively straightforward on FreeBSD 10.

Assuming that you unpacked your ports tree as root by running portsnap fetch and portsnap extract without altering anything, you will need to put the file in /usr/ports/multimedia/xmms/files/ and then cd to that directory on a terminal. Now run make && make install as root. The patch will be applied automatically.

Now one can once again run xmms and module files just work as they’re supposed to:

A patched xmms playing a MOD file on FreeBSD 10

xmms playing a MOD file on FreeBSD 10 via a patched xmms-mikmod (click to enlarge)

The sad thing is, soon xmms will no longer be with us, at least on FreeBSD. It’s now considered abandonware, as all development has ceased. The xmms port on FreeBSD doesn’t even have a maintainer anymore and it’s [scheduled for deletion] from the ports tree together with all its plugins from [ports/audio] and [ports/multimedia]. I just wish I could speak C to some degree and work on that myself. But well…

Seems my favorite audio player is finally dying, but as with all the other old software I like and consider superior to modern alternatives, I’m gonna keep it alive for as long as possible!

You can download Jakobs’ patch that I adapted for the FreeBSD ports tree right here:

Have fun! :)

Edit: xmms once again has a maintainer for its port on FreeBSD, so we can rejoice! I have [submitted the patch to FreeBSDs bugzilla] as suggested by [SirDice] on the [FreeBSD Forums], and Naddy, the ports maintainer integrated the patch promptly! It’s been committed since [revision 375356]! So now you don’t need to patch it by hand and recompile it from source code anymore – at least not on x86 machines. You can either build it from ports as-is or just pull a fully working binary version from the packages repository on FreeBSD 10.x by running pkg install xmms.

I tested this on FreeBSD 10.0 and 10.1 and it works like a charm, so thanks fly out to Jakob and Naddy! Even though my involvement here was absolutely minimal, it feels awesome to do something good, and help improving a free software project, even if only marginally so. :)

Oct 222014
 

Webchat logoXIN.at has been running an IRC chat server for some time now, but the problem always lies with people needing some client software to use it, like X-Chat or Nettalk or whatever.

People usually just don’t want to install yet another chat client software, no matter how old and well-established IRC itself may be. Alternatively, they can use some other untrusted web interface to connect to either the plain text [irc://www.xin.at:6666] or the encrypted [irc+ssl://www.xin.at:6697] server via a browser, but this isn’t optimal either. Since JavaScript cannot open TCP sockets on its own, and hence cannot connect to an IRC server directly, there are only two kinds of solutions:

  • Purely client-based as a Java Applet or Adobe Flash Applet, neither of wich are very good options.
  • JavaScript client + server backend for handling the actual communication with the IRC server.
    • Server backends exist in JavaScript/Node.js, Perl, Python, PHP etc.

Since I cannot run [Node.js] and [cgi:irc] is unportable due to its reliance on UNIX sockets, only Python and PHP remained. Since PHP was easier for me, I tried the old [WebChat2] software developed by Chris Chabot for this. To achieve connection-oriented encryption security, I wrapped SSL/TLS around the otherwise unencrypting PHP socket server of WebChat2. You can achieve this with cross-platform software like [stunnel], which can essentially wrap SSL around almost every servers connection (minus the complex FTP protocol maybe). While WebChat2’s back end is based on PHP, the front end uses JavaScript/Comet. This is what it looks like:

So that should do away with the “I don’t wanna install some chat client software” problem, especially when considering that most people these days don’t even know what Internet Relay Chat is anymore. ;) It also allows anonymous visitors on this web log to contact me directly, while allowing for a more tap-proof conversation when compared with what typical commercial solutions would give you (think WhatsApp, Skype and the likes). Well, it’s actually not more tap-proof considering the server operator can still read all communication at will, but I would like to believe that I am a more trustworthy server operator than certain big corporations. ;)

Oh, and if you finally do find it in yourself to use some good client software, check out [XChat] on Linux/UNIX and its fork [HexChat] on Windows, or [LimeChat] on MacOS X. There are mobile clients too, like for Android ([AndroIRC], [AndChat]), iOS ([SIRCL], [TurboIRC]), Windows Phone 8 ([IRC Free], [IRC Chat]), Symbian 9.x S60 ([mIRGGI]) and others.

So, all made easy now, whether client software or just web browser! Ah and before I forget it, here’s the link of course:

Edit: Currently, only the following browsers are known to work with the chat (older version may sometimes work, but are untested):

  • Mozilla FireFox 31+
  • Chromium (incl. Chrome/SRWare Iron) 30+
  • Opera 25+
  • Apple Safari 5.1.7+
  • KDE Konqueror 4.3.4+

The following browsers are known to either completely break or to make the interface practically unusable:

  • Internet Explorer <=11
  • Opera <=12.17
Jan 202014
 

ZFS logoI have been using PC-BSD UNIX for testing purposes since I think October last year. Since I only created a 30GB virtual disk image for that system in VMware Player, I started running out of space recently. As PC-BSD uses the ZFS as its root file system – originally developed by Sun – and everybody seems to think that it’s the end of all means, I thought lets try and grow my file system online!

First of all, this thing has storage pools, or so-called zpools. A zpool can have entire disks or partitions (called “slices” on BSD, as partitions are something slightly different there) as members. Then, it can make RAID arrays out of that, or add a fast SSD as a caching or log device etc.

When you’re done with your zpool, you need the actual ZFS on top as far as I understand this (I’m still very new to ZFS). There you can have extremely nice features like block-level data deduplication or transparent compression like on NTFS.

So, how to grow this thing? Since I didn’t dare to resize the slice with the BSD disklabel and BSD partition structures inside, I went ahead and just added another slice that I would then just plug into my root storage pool called “tank”. BSD doesn’t use conventional structures like other operating systems. It can use a “slice” (meaning a regular MBR/GPT partition), but it will create its own “BSD disklabel” inside, which is like another partition table, and then it will create its BSD style partitions using that disklabel. Typically, other operating systems cannot handle BSD disk structures. But they can handle “slices”, so I did add another such “slice” or partition with a bootable gparted ISO, but I believe you can reach the same goal by running this as root given the disk /dev/da0, filling the remaining disk space up completely with that new slice:

gpart add -t freebsd da0

Ok, this is the current status before modifying anything else, as you can see, I have only 3.2GB of space left on my ZFS:

[thrawn@unixbox ~]$ zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
tank                      24.7G   3.2G    31K  legacy
tank/ROOT                 11.5G   3.2G    31K  legacy
tank/ROOT/default         11.5G   3.2G  11.5G  /mnt
tank/tmp                  99.5K   3.2G  99.5K  /tmp
tank/usr                  13.0G   3.2G    31K  /mnt/usr
tank/usr/home             4.26G   3.2G    32K  /usr/home
tank/usr/home/thrawn      4.26G   3.2G  4.26G  /usr/home/thrawn
tank/usr/jails              31K   3.2G    31K  /usr/jails
tank/usr/obj                31K   3.2G    31K  /usr/obj
tank/usr/pbi              8.03G   3.2G  8.03G  /usr/pbi
tank/usr/ports             778M   3.2G   536M  /usr/ports
tank/usr/ports/distfiles   241M   3.2G   241M  /usr/ports/distfiles
tank/usr/src                31K   3.2G    31K  /usr/src
tank/var                  6.82M   3.2G    31K  /mnt/var
tank/var/audit              33K   3.2G    33K  /var/audit
tank/var/log               442K   3.2G   442K  /var/log
tank/var/tmp              6.33M   3.2G  6.33M  /var/tmp

Now, to make sure our zpool plus its ZFS file system will grow automatically when adding new devices, run the following, activating automatic expansion for our zpool:

[thrawn@unixbox] ~# zpool set autoexpand=on tank

Time to add stuff to the zpool now. Growing ZFS itself shouldn’t be necessary, as it scales with the zpool, at least that’s what I think it does. What I tried to do then was to add the new slice to the pool “tank”, but that failed initially. See what the zpool command had to say about what I attempted to do:

[thrawn@unixbox] ~# zpool add tank /dev/da0s2
cannot add to 'tank': root pool can not have multiple vdevs or separate logs

Ok so… What the hell? I googled around a bit and found out, that this seems to be a leftover limitation coming from SunOS/Solaris, where ZFS originally comes from. This problem means, we cannot add data, log or cache devices to this pool, because its the one we’re booting from. There is a dirty little trick that will make zpool believe it’s just a data pool though! Just clear the bootfs string, then retry and add bootfs after that again, as we wouldn’t have a boot volume otherwise. Like that:

[thrawn@unixbox] ~# zpool set bootfs= tank
[thrawn@unixbox] ~# zpool add tank /dev/da0s2
[thrawn@unixbox] ~# zpool set bootfs=tank tank

The first line clears the flag by setting it to an empty string for the zpool “tank”, then we’re adding the new slice to it, and finally, we’re making “tank” our bootfs again. That’s a bit scary, so you should probably make sure you have a full backup system image before trying this. For me, it did work, and immediately afterwards I got my +30GB from the added slice:

[thrawn@unixbox ~]$ zfs list
NAME                       USED  AVAIL  REFER  MOUNTPOINT
tank                      24.7G  33.2G    31K  legacy
tank/ROOT                 11.5G  33.2G    31K  legacy
tank/ROOT/default         11.5G  33.2G  11.5G  /mnt
tank/tmp                  99.5K  33.2G  99.5K  /tmp
tank/usr                  13.0G  33.2G    31K  /mnt/usr
tank/usr/home             4.26G  33.2G    32K  /usr/home
tank/usr/home/thrawn      4.26G  33.2G  4.26G  /usr/home/thrawn
tank/usr/jails              31K  33.2G    31K  /usr/jails
tank/usr/obj                31K  33.2G    31K  /usr/obj
tank/usr/pbi              8.03G  33.2G  8.03G  /usr/pbi
tank/usr/ports             778M  33.2G   536M  /usr/ports
tank/usr/ports/distfiles   241M  33.2G   241M  /usr/ports/distfiles
tank/usr/src                31K  33.2G    31K  /usr/src
tank/var                  6.82M  33.2G    31K  /mnt/var
tank/var/audit              33K  33.2G    33K  /var/audit
tank/var/log               440K  33.2G   440K  /var/log
tank/var/tmp              6.33M  33.2G  6.33M  /var/tmp

Took a deep breath, rebooted, and it worked just fine. And it works instantly and online! There is no lengthy resizing process involved for ZFS, it seems to just grow on demand, writing data to the new underlying devices as needed. It IS pretty awesome actually.

But I still have a lot to learn there, as all these tools like zpool and zfs are pretty weird, and I’m not used to BSD disk structures or the ZFS structures. For somebody like me, who has not really handled anything but MBR/GPT style primary partitions before (minus some Windows dynamic disks), it’s actually quite confusing. But I’ll get used to it I guess.

With all those other super powerful features that ZFS has, I think this was just a glimpse of what can be achieved!

Oh, and before I forget to mention it, some initial help for my hopeless self was provided by [kmoore134] on the [PC-BSD forums].

Nov 112013
 

PC-BSD LogoSince I got to know some BSD UNIX operating systems on my journeys across the sea of systems for my x264 benchmark, I was fascinated with those UNICES, with BSD even more so than with Solaris. Besides OpenBSD, NetBSD, FreeBSD and Dragonfly BSD, one was particularly interesting, and that was [PC-BSD]. Yet another FreeBSD at heart, it adds quite a lot of value on top of that solid core to make it easier for users of both the Windows and Linux (and OSX I guess) worlds to start with BSD. Things that you would need to set up all by yourself on FreeBSD are already taken care of here, like e.g. giving you a built-in Linux compatibility layer ready for use, or the so-called “AppCafe” package manager, that is sinfully easy to use, a binary nVidia graphics card driver built-in, a graphical installer even…

Let me show you said AppCafe first. Now you still get to use the classic BSD binary package management with pkg_add -r, pkg_delete etc, also you can still make use of the source code based Ports system. But for the most important parts, and even more so for beginners, AppCafe is the place to start. Have a look:

As you can see, AppCafe shows you the latest additions on top, while allowing the user to either browse through packages by category or doing a full text search on the entire database of packages. The packages themselves are distributed as so-called “PBI” files, which contain the program itself plus all necessary libraries the program depends on. This might bloat packages a bit here and there, but it also makes it very easy to handle. Just click on a package, choose install, wait, done. Almost “Ubuntu”, eh?

The packages are actually built and added to the database by an automatic system from the Ports tree and (I assume) other sources. So if there is a new FireFox, chances are it’s going to appear in AppCafe rather sooner than later.

Above you may also have noticed Skype? There is no native BSD version of Skype, only Windows, OSX and Linux. Still, there is a Skype in AppCafe. This can be achieved easily, because PC-BSD ships with a fully configured Linux compatibility layer, from the kernel module linux.ko to linprocfs to other things. Typically, the user space components of that layer can be found in /compat/linux/. Because it’s already included in PC-BSD, AppCafe can just serve you certain Linux binaries where no other options are available.

This is also the way how PC-BSD included Adobe Flash by default. So yeah, despite UNIX, you still get Adobe Flash if you want. Plus OpenJDK if you need Java.

Another cool thing is the PC-BSD “Warden”. On Linux it can be quite tricky to set up chroot environments or “jails” to run ancient software with massive dependencies or to just lock in certain services. On this OS, there is a (graphical!) configuration tool to do just that very easily. Thus, jail management becomes very managable, and you can easily create new locked-in instances of different BSD systems:

PC-BSD Warden

PC-BSD Warden

Now you got the Linux compatibility layer and also a pretty modern Wine as your Windows runtime including good configuration interfaces like Swine (I even managed to get Lotus Notes Basic 8.5 to work!). But still, sometimes you just need a real Linux or Windows. And for that, PC-BSD has Oracles VirtualBox included by default, with kernel drivers and all, ready to use:

PC-BSD VirtualBox

VirtualBox

Basically, you got all you need though. Even Opera is available in its native BSD version from AppCafe, plus FireFox, Chromium, Thunderbird, Chat clients like SIM-IM, X-Chat and even Skype. Wine for Valves Steam? It’s there! Rarer software can be taken from the FreeBSD package repository using pkg_* or compiled from BSD Ports, which is not too hard to use. And for all other things you have a full GNU autoconf, make, cmake and several GCC compiler versions ranging from the system stock 4.2.1 all the way up to the 4.9.0 experimental version, so you can just compile your software from source code.

I was quite surprised by how easy BSD can be made. Sure, I had to compile my own Murrine engine to be able to use that neat Orta GTK+ theme for my UI properly, I had to compile my own libsidplay2+reSID engine to get some decent Commodore 64 SID emulation in xmms1 and so on. Some special things still do require some work. But then again, they also might on Linux.

Plus, there is ZFS. The greatest of all file systems with checksummed blocks, built-in blocklevel [data deduplication], dynamic block sizes, online compression, online file system repair, a full-blown RAID implementation and neat performance etc.

Oh, and before I forget it, this is what my PC-BSD looks like at the moment:

PC-BSD Screenshot

PC-BSD Screenshot

I do have certain long-term plans about replacing Microsoft Windows on my primary Workstation. That won’t happen anytime soon, even though I am using Windows XP Pro x64 Edition, but still. One day, a new OS will be needed. And when that day comes, it will likely not be Windows 7 or 8 or 8.1 or 9 or 17 or whatever. The way Microsoft is going just pisses me off too much. And besides Linux, I might actually go for BSD UNIX. I can’t say why really, but somehow I like this system. Plus, again, ZFS!

Oct 102013
 

SSD alignment logoRecently, a friend from a forum asked away about what he would need to be cautious about when installing Windows 98 to a flash medium, like an SSD or in his case an IDE CompactFlash card. While this is the extreme case, people tend to ask such questions also for Windows XP and XP x64 or other older systems. So I thought I’d show you, using this Win98 + CF card case as an example. Most of the stuff would be similar for more “modern” systems like maybe Windows 2000, XP, Linux 2.4 etc. anyway.

For this, we are going to use the bootable CD version of [gparted]. In case you do not know gparted yet, you can think of it as an OpenSource PartitionMagic, that you can also boot from CD or USB. If you know other graphical partition managers, you will most likely feel right at home with gparted. Since this article is partially meant as a guide for my friend, parts of the screenshots will show german language, but it shouldn’t be too much of a problem. First, let’s boot this thing up from CD (or USB key if you have no optical drive). See the following series of screenshots, it’s pretty self-explanatory:

Now that we’re up and running you will be presented with a list of available drives accessible at “GParted / devices”. In our case we don’t need to do that as we only have one device posing as a 16GB large CompactFlash card as you will see below. Since this is a factory-new device, we shall assume that it will be unformatted and unpartitioned. If it IS pre-formatted you would need to check the given alignment first, see below to understand what i mean. But you can also just delete whatever partitions are on there if you don’t need any data from the device and proceed with this guide afterwards.

What we will do is create a properly aligned partition, so we won’t create unnecessary wear on our NAND flash and so we won’t sacrifice performance. This is critical especially for slower flash media, because if you’re going to install an operating system onto that, you’d really feel the pain with random I/O on a misaligned partition (seek performance drops, far too large reads/writes would occur etc.). Just follow these steps here to partition the device and pre-format it with FAT32. If you’re on Windows 2000 or newer, you may want to choose NTFS instead, for Linux choose whatever bootable file system your target distribution understands, like maybe JFS or something. If your flash medium supports TRIM (newer SATA SSDs do), choose EXT4, XFS or BTRFS instead, for BSD UNIX pick UFS or ZFS if you are ok with those:

This is it, our partition starts at sector number 2048 (1MiB, or 1MB as I tend to say), which will work for all known flash media. If you expect your medium to have larger blocks, you could set the start to sector 20480 instead, which would mean 10MB. I don’t think there are media out there with blocks THAT large, but by doing this you could be a 100% certain that it’ll be aligned. Time to shut gparted down:

After that, you may still re-format the partition from DOS or legacy Windows, but make sure to not re-partition the disk from within a legacy operating system, so no fdisk and no disk management partitioning stuff in Win2000/XP!

On a side note, FAT32 file systems as shown here can’t be [TRIM]’ed on Windows, so when the media gets enough write cycles, writes will permanently slow down, with maybe garbage collection helping a little bit. You could hook that FAT32-formatted disk up to a modern Linux and TRIM it there though, if it’s a new enough SATA drive. In my friends’ case that’s not an issue anyway, as he is using IDE/PATA flash that doesn’t support the TRIM command to begin with. But if you do have a modern enough SATA system and a TRIM-capable SSD, you might want to go with NTFS for Windows or EXT4/BTRFS/XFS for Linux as well as UFS/ZFS for BSD UNIX if you can, as those file systems are supported by current TRIM implementations, or yeah, FAT32 for data exchange between operating systems. Keep in mind though, to TRIM FAT32 SATA SSDs, Linux is required at the time of this writing.

And no, no IDE/PATA media can say “TRIM” as far as I know.

Also: You can re-align existing partitions with gparted in a somewhat similar fashion as shown above by just editing them. This may be useful if you messed up with DOS or WinXP and have already installed your system on a misaligned partition.  Gparted will then have to re-align the entire file system on that partition though, and that may take hours, e.g. for my Intel 320 600GB SSD (about 400GB full) it took almost 3 hours. To see which file systems gparted supports for re-alignment, [look here] (It’s mostly “shrink” that is required)!

Sep 192013
 

Perl logoI have this script here that I wrote to walk an entire tree of directories (a rather flat structure) and that generates several lists of directories sorted by last modification time. So the last modified directories are determined and then published on a web server. Sounds neat enough.

Yesterday I tried to extend it to not only generate lists of the last 30 and 100 changed directories, but also a full sorted list containing all elements of the folder structure. By doing so, I found a nasty bug that kept me busy for an hour or so. Thing is, to read the folders I used something known as Perl Globbing. According to the Internets it’s one of the nastier things you can do. Works like that:

use strict;                        # Strict code only.
 
my $basepath = "/home/myuser";     # Directory to read from.
my @folders = glob("$basepath/*"); # Reading all element names in $basepath (files+dirs).
# my @folders = &lt;$basepath/*&gt;;     # This is the same thing actually.
my $folder = "";                   # Single elements name.
 
foreach $folder (@folders) {       # Walking through the array of files and dirs.
  if (-d $folder) {                  # We only want to show directories, no files!
    print ($folder . "\n");          # Print directory names.
  }
}

This would read all elements in /home/myuser/* into an array and then show them on the shell. That’s UNIX style, it could also be C:/somepath on Windows with Perl auto-converting / to \ internally. This does break however as soon as $basepath contains whitespaces, because glob() from Perls CORE::glob uses whitespaces as search pattern separators, and you can NOT change or escape that. See this example:

use strict;
 
my $basepath = "/home/my little user";
my @folders = glob("$basepath/*");
my $folder = "";
 
foreach $folder (@folders) {
  if (-d $folder) {
    print ($folder . "\n");
  }
}

Now this won’t show anything, because instead of looking for /home/my little user/*, it will look for /home/my, then little, and finally user/* in the current path. Clearly, this sucks. But there are two other implementations of glob(), one in File::DosGlob and one in File::Glob. Let me just say, even when on Windows, the Microsoft-specific File::DosGlob just does the same crap with whitespaces. The one in File::Glob however is bsd_glob(), which doesn’t do crap like that. If you want to call it the same way as above, you’ll have to request it as glob() function specifically though, as CORE functions always take precedence over others. Oh and, while it’s called bsd_glob(), don’t worry, this also works on Windows.

So the following will work just fine:

use strict;
use File::Glob 'glob';
 
my $basepath = "/home/my little user";
my @folders = glob("$basepath/*");
my $folder = "";
 
foreach $folder (@folders) {
  if (-d $folder) {
    print ($folder . "\n");
  }
}

You can also just call bsd_glob() by it’s right name instead of course, which is probably more clean anyway. In that case, you can use the module just like use File::Glob;. What do we learn from this? BSD UNIX’ globbing implementation is best. And if you want to be on the safe side, just don’t use globbing at all. Everybody seems to agree that it’s dirty and shitty. But hey, it’s also QUICK and dirty. ;) Still, if you want to do it more cleanly, you can use [opendir()] and [readdir()] instead, both of which work on directory handles, which are similar to file handles, that you can also pass around, making it more flexible.

Yeah, I didn’t do that because I didn’t want to write 3-4 more lines of code… Just fine as it is now! :roll:

Aug 092013
 

Filezilla logoI think I can say without raising too much critisism, that [Filezilla] is one of the best if not the best cross-platform FTP client ever built. It works on many operating systems like Windows, Linux, BSD UNIX etc. Now, I’ve been stuck with an older version of Filezilla, namely 3.5.2 because of a [tedious bug] that arose when you built the client and linked it against GnuTLS 2.8.x, which is typically what you’ll have when you’re running some Enterprise Linux distribution like me (CentOS 6.4 in my case).

If you’re using SSL-enhanced FTP servers which understand FTPS and FTPES (not to be confused with the SSH-based SFTP), any Filezilla version >=3.5.3 linked against any GnuTLS 2.8.x will break when connecting to such a server using SSL/TLS. So while Filezilla 3.7.3 compiled just fine on my system, it broke at runtime when doing that. See here:

Broken Filezilla: Connection log

(click to enlarge)

Here is the dreaded GnuTLS -50 error. From what I have read, it’s a problem that comes from a bug in the way SSL handshaking is done. It appears that if a server supports low-security ciphers in its cipher suite, the negotiation will break, as Filezilla only allows high-security ciphers since 3.5.3. This is somehow linked to GnuTLS 2.8.x. So even if a server supports high-security ciphers, it still breaks if low-security ones are also supported. That’s my current assumption at least. The only way seems to be to completely remove low-sec ciphers from the servers side. Which is sometimes impossible. But then..  Why does it work on Windows? Let’s first see what the latest Filezilla 3.7.3 reports on CentOS 6 Linux when asked for version numbers (under Help / About…):

Broken Filezilla: Version info

Aha. Ok. So wxWidgets 2.8.12 which I have installed myself (the packages are called wxGTK and wxGTK-devel!), SQLite 3.6.20 which comes with CentOS 6.4 and GnuTLS 2.8.5, which is also shipped with this Linux distribution. All stock then. Let’s compare that to the Windows version, which works just fine:

Working Filezilla on Windows: Version info

So the Filezilla guys used the same wxWidgets GUI libraries as I did, a slightly newer SQLite and most importantly a far newer major release of GnuTLS. Now I have tried GnuTLS 2.8.6 too, the lastest 2.8.x branch stable release, and I even tried to build and link against an unstable 2.9.9, all bugged in the same way. So the only way is to install a GnuTLS >=3.0. However, GnuTLS 3.x requires [nettle], some low-level crypto library, which CentOS doesn’t have.

On top of that, the latest version 2.7 of nettle won’t compile on CentOS 6, you’ll just get a buttload of really nasty compiler errors, but the latest GnuTLS 3.1.13 only requires nettle 2.5, so we’ll go with that. You can still get that version from [here].

Download it, and from there it’s as easy as this (you can leave out the -O3 -ffast-math in case you’re not comfortable with aggressive compiler optimizations or you run into stability issues):

tar -xzvf ./nettle-2.5.tar.gz
cd ./nettle-2.5/
export CFLAGS="-march=native -O3 -ffast-math -pipe"
export CXXFLAGS="-march=native -O3 -ffast-math -pipe"
./configure
make
su
make install

Now we can build GnuTLS 3.1.13, which can be downloaded from [here]. On my CentOS 6.4 I managed to make it link against nettle 2.5 and build like this (-O3 and -ffast-math may be omitted again):

tar -xJvf ./gnutls-3.1.13.tar.xz
cd ./gnutls-3.1.13/
export CFLAGS="-march=native -O3 -ffast-math -I/usr/local/include -L/usr/local/lib64"
export CXXFLAGS="-march=native -O3 -ffast-math -I/usr/local/include -L/usr/local/lib64"
./configure
make
su
make install

Perfect. Now please download the [latest Filezilla source code] which is currently version 3.7.3, and you should be able to link it against our brand new GnuTLS 3.1.13 in the following way (-O3 -ffast-math may be omitted again):

tar -xjvf ./FileZilla_3.7.3_src.tar.bz2
cd ./filezilla-3.7.3/
export CFLAGS="-march=native -O3 -ffast-math -pipe -I/usr/local/include -L/usr/local/lib -L/usr/local/lib64"
export CXXFLAGS="-march=native -O3 -ffast-math -pipe -I/usr/local/include -L/usr/local/lib -L/usr/local/lib64"
export LD_LIBRARY_PATH="/usr/local/lib"
./configure --with-tinyxml=builtin --with-libgnutls-prefix=/usr/local
make
su
make install

This will make sure all the linking will work and not fall back to the systems default GnuTLS 2.8.5 or any other version of the 2.8.x branch your system might have. Now if that is done, just launch Filezilla, and look at the version numbers under Help / About… again to verify it has been linked correctly:

Working Filezilla: Version infoWorking Filezilla: Connection log

In my case, GnuTLS is now even newer than the version that has been used by the developers themselves to create their Windows build. 3.1.13 instead of 3.1.11. But in any case, a version of 3.1.x that should fix the cipher negotiation problem. Take a deep breath and try your FTPS or FTPES connection now, it should work even if the server supports low-security ciphers:

Working Filezilla: Connection log

(click to enlarge)

Space magic done! It works indeed. Note that this will probably not help you, if the SSL-enhanced FTP server you want to connect to supports no high-security ciphers. Your server will need to offer at least some of those in its cipher suite. But if it does low+high, this should fix it. And by using nettle 2.5, the latest stable version of GnuTLS can still be compiled on CentOS 6, and therefore also on Scientific Linux 6 and RedHat Enterprise Linux 6. Should be somewhat similar in Debian Squeeze and maybe SuSE Linux Enterprise Desktop.

Finally, a problem solved. Now I can use the latest version of Filezilla across all my operating systems again! Funny that nobody suggested this way in the Filezilla forums. Hm. Maybe I should.

Edit: While testing PC-BSD 9.2 UNIX I have now found out, that GnuTLS 2.12.x also seems to fix this, so you don’t absolutely need GnuTLS 3.x+. See this screenshot of FileZilla on PC-BSD 9.2 (based on FreeBSD 9.2) linked against GnuTLS 2.12.23:

Filezilla with GnuTLS 2.12.23 on BSD

So yes, this also works flawlessly!

Update 2014-04-30: And finally, I found a way to build the latest stuff on CentOS 6.5 Linux, opening up the way to newer versions of Filezilla and more importantly GnuTLS. That would include [nettle 2.7.1], [GnuTLS 3.2.13] and [Filezilla 3.8.0] as of the time of writing. Compilation and having the program work at runtime was tricky though. I am not perfectly sure if my documentation is complete (if you try this and fail, just post a comment!), but I think this should get you going, skipping the easy commands like unpacking the archives etc. here and leaving out non-essential flags like “-march=native” or “-O3”. You can still add those of course. Let’s start with nettle 2.7.1, this time disabling OpenSSL glue code:

./configure --disable-openssl
make
su
make install

Now, GnuTLS 3.2.13. For this we’ll need a bit more trickery, especially for the pkg-config library detection that GnuTLS uses to locate nettle stuff, but also, unfortunately, LD_LIBRARY_PATH:

export CFLAGS="-I/usr/local/include -L/usr/local/lib64"
export CXXFLAGS="-I/usr/local/include -L/usr/local/lib64"
export LDFLAGS="-I/usr/local/include -L/usr/local/lib64"
export PKG_CONFIG_PATH="/usr/local/lib64/pkgconfig"
export LD_LIBRARY_PATH="/usr/local/lib64"
./configure
make
su
make install

After that, assuming you’re still sitting in the same shell you built and installed GnuTLS in, thus preserving the environment variable from above, continue with Filezilla 3.8.0 exactly like this:

unset LD_LIBRARY_PATH
export CFLAGS="-I/usr/local/include -L/usr/local/lib64 -L/usr/local/lib"
export CXXFLAGS="-I/usr/local/include -L/usr/local/lib64 -L/usr/local/lib"
export LDFLAGS="-I/usr/local/include -L/usr/local/lib64 -L/usr/local/lib"
./configure --with-tinyxml=builtin --with-libgnutls-prefix=/usr/local
make
su
make install

We’re not done yet though. Since we played around with an evil LD_LIBRARY_PATH to get this built and linked, our /usr/local/bin/filezilla binary will just crash with a symbol error caused by GnuTLS unable to locate certain nettle functions. I would suggest working around that by writing yourself a small launcher script to fire up Filezilla with the proper path set for its session. I’d call this script /usr/local/sbin/filezilla.sh. You can then use that to launch the program, and the code is super simple:

#!/bin/bash
export LD_LIBRARY_PATH="/usr/local/lib64"
/usr/local/bin/filezilla

Kinda dirty, but it works. And here is the result:

Filezilla 3.8.0 working with GnuTLS 3.2.13 and nettle 2.7.1 on CentOS 6.5 Linux

Besides missing a few optimization flags, this actually seems to work pretty well so far. A few tests using encrypted connections with a modern TLS v1.2 protocol AES-256-GCM cipher suggests everything is in order. Great!

May 292013
 

Eclipse IDE logoA collegue here at work, who is developing Java projects in the Eclipse IDE on Linux has recently reported problems when tagging projects via Subversive on our Subversion server. The result was the following error message being reported by the SSH server on a project tag attempt via svn+ssh:

ssh_exchange_identification: Connection closed by remote host

So, it seemed that the builtin pure Java Subversion client “subversive” in Eclipse caused some problems on our SSH server, which is the most recent release version from RedHat Enterprise Linux 6. The problem was global though! When caused, all users including root would be locked out of that SSH server for the grace period of unsuccessful logins, typically 60-120 seconds. Unacceptable. At first I couldn’t find anything, but when we found out that the issue was somehow linked to operations on the SVN server from Eclipse, it became clearer, as my collegue found [this] and [that], showing that the problem exists on BSD also. Makes sense, since that’s where OpenSSH originally comes from. Here is an excerpt of what a ssh -vvv user@host attempt would look like after the problem had been triggered:

[user@local_host ~]$ ssh -vvv user@host
OpenSSH, OpenSSL
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to host [192.168.0.1] port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type 2
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
ssh_exchange_identification: Connection closed by remote host

The working solution seems to be to edit /etc/ssh/sshd_config and change the MaxStartups option. I chose to increase the default of 10 by a factor of 20x to 200. For some reason, Subversive seems to do a lot of unauthenticated connections at once, or maybe just too many at once before all of them had enough time to do their key-based auth. Changing MaxStartups to a higher value allows the SSH server to accept more not-yet authenticated incoming connections at once before blocking all logins for a certain grace period. Know that while older SSH daemons like ours will only take a number as input, newer ones will take input in a different format, which is not MaxStartups n, but rather MaxStartups x:y:z, where n is the number of simultaneous unauthenticated connections and x is “start”, y is “rate” and z is “full”. I do not exactly know what x:y:z mean though, as I do not have the newer SSHD documentation here, so you may need to read man sshd_config on your *nix system.

One of those guys also suggests to change some buffer parameters of the TCP kernel driver, but I just left that alone, as it seemed unrelated and our Linux kernel defaults for TCP did not seem to affect the issue. After setting MaxStartups 200 in /etc/ssh/sshd_config and restarting SSHD all worked just fine. A crosscheck with MaxStartups 2 (just to see whether we were really right) triggered the problem immediately. So this should solve the issue in a consistent fashion!

Apr 162013
 

mausmaki.netThere is this guy that one day just wrote me an eMail (I am not even remembering how it started anymore, the correspondence is really long already) that is also very much into exotic systems running his own servers at home. Actually, even rackmount stuff, pretty crazy. Guy’s called Hans-Jürgen, and he is now finally doing his own weblog!

According to him, he will be writing about technology (x86 and PPC machines, networking, audio engineering) as well as social-political topics. Note, that the blog will be in german though. HJ is also very much into open source software, so you can expect to read about Linux and maybe some server side software too.

Since I consider him a valuable source of information and a good friend, his blog will also be added to my link list. Oh yes, the link, of course, here it is:

Apr 022013
 

Frog ASPIToday I would like to discuss a few useful tricks for Windows XP x64. Yeah. Again. But this time around, it’s at least partially about stuff that you can’t easily find on the Internet anymore, whether it’s the information or the actual software part that’s missing. There are several software components that are sometimes hard to use or find for XP x64. Some are even hard to set up for regular 32-bit Windows XP. The following solutions will be discussed:

  • 1.) The ASPI Layer (digital media access)
  • 2.) UDF 2.5 (for Blu-Ray and HD-DVD access)
  • 3.) exFAT (some modern cameras etc. need this file system)
  • 4.) EXT2/3/4 (for Linux and UNIX compatibility)
  • 5.) Universal SSD TRIM (keep your SSD clean, tidy and fast, like on modern systems)

So, let’s start with #1:

1.) The ASPI layer:

Edit: Note that in the meantime, the FrogASPI solution described here is no longer the only option. Another one that works even for 64-bit applications accessing the ASPI layer has been found, see further below!

1.a) FrogASPI:

One of those things that have been abandoned by Microsoft is the ASPI layer, the Advanced SCSI Programming Interface. Meant as a way for digital storage media access it was most prominently used to read from optical drives digitally (also on ATA, not just SCSI), so you didn’t need to rip audio CDs via MSCDEX on Win98 or via the crappy analog link from your drive to the sound card. ASPI can also be used to access other types of storage devices, but this is the most important part. Some older software, like my beloved Xing AudioCatalyst (an ancient CD ripper including the Fraunhofer mp3 encoder) might still need ASPI to access audio CDs.

However, Adaptec stopped shipping ASPI drivers for Microsoft Windows after Microsoft had abandoned the API and introduced its own replacement called SPTI, the SCSI PassThrough Interface. As a matter of fact, you can still install Adaptecs ASPI layer on 32-Bit Windows XP, as it includes a working 32-Bit kernel driver. So for 32-Bit XP, it’s still fine. However, there is no such driver for XP x64 (and also not for 32/64-Bit Vista/7/8). So, no ASPI at all?

For a loong time I indeed had to live without it, until I found that french guy named Millenod (That’s his nick name, I will not disclose his real name) who had written a 100% userspace ASPI layer, that would work on any newer Windows operating system on x86, no matter the bitness. This is called FrogASPI, and unfortunately, Millenods website for it is down by now. In its core, it is actually just an SPTI wrapper. Back in those days, I even wrote the guy an email, thanking him for his work. Here is a part of his reply:

“FrogAspi is effectively an SPTI wrapper. I decided to work in “user” mode, instead of kernel ones, for many reasons.. It was for me the fastest way to implement a generic ASPI layer, which is not OS specific as drivers.”

-Millenod, Developer of FrogASPI

After renaming the FrogAspi.dll to the proper name wnaspi32.dll and putting it into %WINDIR%\SysWOW64\ for 64-Bit Windows, it can be used by any ASPI-enabled application. For 32-Bit Windows, please use %WINDIR%\system32! See, what Adaptecs own aspichk.exe has to say about what we just did:

Adaptec ASPI Check

You’ll see that some files are reported as missing. You do not have to care about that though, ASPI32.SYS would’ve been the 32-Bit kernel driver, WOWPOST.EXE is a 16-Bit Windows ASPI helper tool, WINASPI.DLL the corresponding 16-Bit Windows driver. None of those are needed. Now, that FrogASPI is mapping all ASPI calls to SPTI, we can begin to actively use it even on 64-Bit Windows. See AudioCatalyst for instance, with ASPI being available:

AudioCatalyst using FrogASPI

AudioCatalyst reading the Postal Original Soundtrack CD using FrogASPI

As you can see, everything works just fine. Oh, and besides, in case you want AudioCatalysts CDDB feature back (as shown in this screenshot), just add the following lines to your AudioCatalyst.ini, sitting in the programs installation folder, this’ll switch from the now-broken CDDB to FreeDB:

  1. CDDBServer=freedb.org
  2. CDDBHTTPPath=/~freedb/cddb.cgi
  3. CDDBLocation=Random FreeDB Site
  4. CDDBTCPIPPort=8880
  5. CDDBConnectVia=TCP/IP

You can download FrogASPI right here:

Unfortunately I cannot give you a download link for Xing AudioCatalyst, as it’s commercial software even today.

1.b) StarBurn ASPI (An update from 2016-12-05):

In the meantime, user Steve Sybesma has [commented] about a different solution provided by [StarBurn Software]. With their recording software comes a native ASPI layer for both 32-bit and 64-bit systems with full compatibility for even 64-bit programs that want to use ASPI. I decided to fetch the necessary DLLs from their SDK and release them in the form of a standalone InnoSetup installer for Windows 2000 and newer. The installer will auto-detect your operating systems’ bitness and will install the 32-bit ASPI layer on 32-bit systems as well as both the 32-bit as well as the 64-bit one on 64-bit systems.

Here it is:

It has been successfully tested on Windows 2000 Pro SP4, Windows XP Pro SP3, Windows XP Pro x64 Edition SP2 as well as Windows 10 Pro x64. If you don’t want to trust my installer, that’s fine as well of course. You can just install the StarBurn software from its original source, it should give you the same level of ASPI support, just with more additional stuff being installed!

Now, to #2, the UDF 2.5 file system for HD-DVDs and Blu-Rays:

2.) UDF 2.5:

Now this one was a real bitch. It took me months to finally get it nailed! UDF 2.5 is a file system. And it’s used for basically two type of media: HD-DVD discs and Blu-Ray discs, including the most modern 3D Blu-Rays. It just bugged me like hell, that I couldn’t access the discs as I normally would in Windows Explorer or on the command line. Windows XP simply does not have the proper file system kernel driver. And while it’s relatively easy to find one for 32-Bit WinXP, it’s immeasurably harder to find the single one existing driver for Windows XP x64. I’m not even sure if it still exists on the web..

One day I came across a person whose name I forgot, but that guy had searched the web for months just like me, and he found the driver just before it went offline. So he re-uploaded it, with his single file sharing link in some post on some unknown website being the only thing that saved my bacon. Maybe one day, somebody will find the XP x64 UDF 2.5 driver here on my site after desperate searching? Who knows.

So, the only existing driver has been developed by Panasonic/Matsushita, a company that also builds optical drives. It works on Windows XP x64 as well as Server 2003 x64. And here it is, together with the Toshiba-made UDF 2.5 driver for regular 32-Bit Windows XP just for completeness:

Again, I would like to emphasize that it took me months to find that freaking 64-Bit XP driver, I’m not even sure what Panasonic/Matsushita developed it for, but here it is for your enjoyment. After installation and reboot, you can browse a HD-DVD or Blu-Ray just like any CD or DVD disc, see here:

UDF 2.5 at work in XP x64

The Blu-Ray folder structure of the german uncut version of “The Boondock Saints”, viewed using the Panasonic/Matsushita UDF 2.5 driver for XP x64

And now, #3, the exFAT file system:

3.) exFAT:

exFAT is a successor to the older FAT32 file system that uses a 64-bit address field instead, allowing for larger than 4GB files, the most notorious limitation of regular FAT. exFAT is now being used on newer cameras and other handheld devices for memory card formatting. Since the comparably heavy-weight NTFS is typically not used on such devices, exFAT is the replacement we get. Also, exFAT is the official file system for SDXC and Memory Stick XC cards, so they may very well come exFAT-preformatted! The only sour part is, that exFAT is kind of proprietary (Microsoft), like NTFS. That’s a bit crappy, but well.

I have however already written an article about exFAT on XP x64, so I will just link to it. exFAT is also very easy to get for both 32-Bit and 64-Bit flavors of Windows XP:

And now, finally, some free file systems from the Linux world:

4.) EXT2/3/4:

EXT is one of the most widely if not the most widely used file system in the entire world of Linux. It has also been ported to other systems like BSD, Solaris or ReactOS. And also Windows actually, although the implementation is not the most seamless you can dream up. But still, you can get read/write access to all currently used EXT versions under Windows, even the fast new EXT4.

The software that I found most useful for this is called [ext2fsd], short for EXT2 file system driver. It comes with the actual driver plus a helper tool for mounting and for enabling write access. It would be nice to just be able to mount EXT filesystems seamlessly from the start, without having to use that mounting tool, but we’re not that far yet it seems. However, drives can be configured to auto-mount later, so you need the tool only once for each drive. Write access has to be enabled by hand.

Currently, ext2fsd supports the following systems: NT 5.0 (Windows 2000), NT 5.1 (Windows XP 32-Bit), NT 5.2 (Windows XP x64, Server 2003), NT 6.0 (Windows Vista), NT 6.1 (Windows 7, Server 2008). Where not explicitly mentioned, it supports both 32-Bit and 64-Bit versions of said operating systems. And here is what it looks like in action:

EXT4 mounted on XP x64

A Linux-created EXT4 filesystem mounted on XP x64 for read/write using ext2fsd

The only strange thing is, that ext2fsd calls the EXT4 file system “EXT3”. It shouldn’t. But other than that, it works just fine. I haven’t tested a lot of r/w, but so far it worked fine, without any corruption or anything. The helper tool also gives you a function for flushing the write cache, and a nice statistics window for monitoring the file system usage while mounted. It also allows the user to manually unmount of course. Even mkfs is there, so you can format drives as EXT under Windows, the only useful tool missing would be fsck for checking & repairing such filesystems. But you can’t have everything I guess.

So, usage is not as seamless as it would be with native NTFS/FAT32/exFAT drivers, but it’s ok and it greatly boosts the Linux interoperability of XP x64 and other Windows systems. Also, EXT filesystems can easily be used in other UNIX-style systems like Solaris, BSD or MacOS X, so this allows a lot of different systems to exchange data easily using removable sticks or harddrives. Since it of course supports large files (>4GB) and it’s more easy to handle across a wide spectrum of operating systems, unlike exFAT it might one day become the #1 choice for data exchange, replacing FAT32 and NTFS in that field. However, today, NTFS on Linux is probably a lot more widespread than this EXT driver on Windows, so there is still a long way to go.

Of course, there is a downside to this too: All the POSIX permissions like ownership, read/write/execute bits etc. just do not work on Windows. You can neither read nor write any such meta data using ext2fsd. I have neither checked the umask that ext2fsd uses yet, nor what ownership it sets, but I would guess it’s root:root with permissions 777 or rwxrwxrwx. But whatever it is, you will most likely need to take care of that when mounting on your UNIX-style system.

I hope this was helpful, if you have any comments about these helpful add-ons, just post away!

5.) Universal SSD TRIM:

This is a 2014-11-03 edit of the original article.

Usually, I would tell users who would like to use an SSD on Windows XP or XP x64 to re-align their partitions with a [gparted ISO] after installation to get proper alignment, and more importantly to use either Intel, Corsair or Samsung SSDs so that they can TRIM their SSDs properly using their respective manufacturers’ XP-compatible software tools, maintaining full write speed over the entire life time of the SSD drive.

It seems that I can now finally confirm, that the latter criterion no longer needs to be met. Basically, I found an easy way to TRIM any SSD on Windows XP or Windows XP x64, as long as the following logical criteria are met:

  • The SSD itself needs to support TRIM. Well, that was obvious.
  • The controller driver needs to support TRIM. Now I can’t speak for just any driver, but I have tested this on Intels ICH10-R and an ancient ICH7-M, both worked just fine with AHCI drivers installed. Usually, AHCI is what you want. Avoid IDE/legacy PATA modes. You may need to configure this properly in your systems BIOS and you may need to retrofit AHCI drivers in XP if your system was installed using IDE mode to avoid bluescreens on boot. If you need help doing that, please ask away in the comments.

So, what’s the solution? It’s called the ADATA SSD Toolbox! Basically, ADATA developers seem to have forgot or were actually never instructed to install an artificial vendor obstruction inside their software. So what they did was to write an SSD toolbox that just complies to the SATA standard, implementing its TRIM command for any compatible SSD, no-holds-barred! I have tested this on an Intel 320 SSD, as well as on a Crucial m500 SSD now, despite no official XP support being present:

So there you go! Now pretty much any TRIM capable SSD can be made to accept TRIM on Windows XP and XP x64! Together with gparted for partition alignment, we have a full-featured SSD solution for XP, leaving nothing to be desired!

Download the ADATA SSD Toolbox here: [ADATA SSD Toolbox v2.0.1].