A build of 64-bit puTTY that works on XP x64 and Server 2003 x64

 Powerful Tools, Programming, Windows  10 Responses »
Mar 132017
 

puTTY logoAnyone who has logged in to a UNIX or Linux machine remotely coming from a Windows box probably knows puTTY, which is practically “the” SSH and telnet client for Windows. In conjunction with a small X11 server like Xming you can even do Remote X. To my surprise, a new version has been released just last month, as a colleague told me! So there is version 0.68 now, and it comes in both a 32-bit and a 64-bit flavor.

Of course I had to try the 64-bit version on XP x64, and it did fail to execute:

64-bit puTTY failure on XP x64

A classic: 64-bit puTTY failure on XP x64, because it’s “not a valid Win32 application”

Out of curiosity, I fetched the puTTY source code from [here], because I thought I could just compile & link it myself. Building for 32-bit proved to be relatively easy; Just load the corresponding solution file into Microsoft Visual Studio 2010, build the whole project, done. But I wanted 64-bit! So I created a x64 build target and gave it a shot, but it couldn’t find the _addcarry_u64 intrinsic.

After a bit of searching on the web, it became clear that the intrinsics header of Visual Studio didn’t provide it. It’s too old, you need Visual Studio 2013 or newer for that. Funny part is, puTTY only comes with project files for 2010 and 2012, how are they building their x64 version? No idea. Maybe they’re linking against a different library version or something.

One attempt that I was going to make (build it with VS2013, linking against an older platform SDK) isn’t done yet, because I need to prepare my Windows 7 VM for it. I did manage to compile and run puTTY as 64-bit code on XP x64 by hacking up their program though! In the unpacked source tree, open sshbn.h and take a look at line #70:

expand/collapse source code
  1. #elif defined _MSC_VER && defined _M_AMD64
  2.  
  3.   /*
  4.    * 64-bit BignumInt, using Visual Studio x86-64 compiler intrinsics.
  5.    *
  6.    * 64-bit Visual Studio doesn't provide very much in the way of help
  7.    * here: there's no int128 type, and also no inline assembler giving
  8.    * us direct access to the x86-64 MUL or ADC instructions. However,
  9.    * there are compiler intrinsics giving us that access, so we can
  10.    * use those - though it turns out we have to be a little careful,
  11.    * since they seem to generate wrong code if their pointer-typed
  12.    * output parameters alias their inputs. Hence all the internal temp
  13.    * variables inside the macros.
  14.    */
  15.  
  16.   #include 
  17.   typedef unsigned char BignumCarry; /* the type _addcarry_u64 likes to use */
  18.   typedef unsigned __int64 BignumInt;
  19.   #define BIGNUM_INT_BITS 64
  20.   #define BignumADC(ret, retc, a, b, c) do                \
  21.       {                                                   \
  22.           BignumInt ADC_tmp;                              \
  23.           (retc) = _addcarry_u64(c, a, b, &ADC_tmp);      \
  24.           (ret) = ADC_tmp;                                \
  25.       } while (0)
  26.   #define BignumMUL(rh, rl, a, b) do              \
  27.       {                                           \
  28.           BignumInt MULADD_hi;                    \
  29.           (rl) = _umul128(a, b, &MULADD_hi);      \
  30.           (rh) = MULADD_hi;                       \
  31.       } while (0)
  32.   #define BignumMULADD(rh, rl, a, b, addend) do                           \
  33.       {                                                                   \
  34.           BignumInt MULADD_lo, MULADD_hi;                                 \
  35.           MULADD_lo = _umul128(a, b, &MULADD_hi);                         \
  36.           MULADD_hi += _addcarry_u64(0, MULADD_lo, (addend), &(rl));     \
  37.           (rh) = MULADD_hi;                                               \
  38.       } while (0)
  39.   #define BignumMULADD2(rh, rl, a, b, addend1, addend2) do                \
  40.       {                                                                   \
  41.           BignumInt MULADD_lo1, MULADD_lo2, MULADD_hi;                    \
  42.           MULADD_lo1 = _umul128(a, b, &MULADD_hi);                        \
  43.           MULADD_hi += _addcarry_u64(0, MULADD_lo1, (addend1), &MULADD_lo2); \
  44.           MULADD_hi += _addcarry_u64(0, MULADD_lo2, (addend2), &(rl));    \
  45.           (rh) = MULADD_hi;                                               \
  46.       } while (0)

I just commented out the entire codeblock using that modern _addcarry_u64 intrinsic, and replaced it with the code being used for the 32-bit version:

  1. #elif defined _MSC_VER && defined _M_AMD64
  2.  
  3.   /* 32-bit BignumInt, using Visual Studio __int64 as BignumDblInt 
  4.    * This is compatible with VS2010 & VS2012 for building a x86_64
  5.    * version of puTTY (no __int128 with those compilers).
  6.    */
  7.  
  8.   typedef unsigned int BignumInt;
  9.   #define BIGNUM_INT_BITS  32
  10.   #define DEFINE_BIGNUMDBLINT typedef unsigned __int64 BignumDblInt

I built that and it works, even though I keep thinking I should be using a wide 128-bit data type here (just like the original x64 code), but then we don’t have __int128 in MSVC before 2013, and I’m on 2010. And I don’t know how to use SSE registers in that context with things like __m128, which is why I left it alone. Looking good anyway:

puTTY 64-bit XP x64 version logged in to Windows 2000

puTTY 64-bit XP x64 version logged in to Windows 2000 using a modern SSH server (click to enlarge)

And:

puTTY 64-bit XP x64 version logged in to FreeBSD 10.3 UNIX

puTTY 64-bit XP x64 version logged in to FreeBSD 10.3 UNIX (click to enlarge)

In any case, here is a complete 64-bit build of puTTY that works on NT5.2 operating systems like Windows XP Professional x64 Edition or Windows Server 2003 x64:

  • [puTTY 0.68][1] (x64 version for NT5.2, portable without installer)

Maybe I’ll try to build a version with VS2013 on Windows 7 for the same platform target, we’ll see. But at least this works!

Oh and… No, I don’t really think anyone actually needs a 64-bit version of puTTY ;). Plus the 32-bit one works just fine on XP x64 / Server 2003 out-of-the-box anyway. But hey… You know… :roll:

[1] puTTY is © 1997-2017 Simon Tatham and is licensed under the MIT license

The final browser for Windows XP / XP x64: Firefox 52.0 ESR (And: Electrolysis / e10s multiprocessing support)

 Software, Windows  1 Response »
Mar 082017
 

Firefox ESR dark logo1.) The end is near for modern browsers on XP and Vista

After Google had stopped supporting Windows XP with Chromium 50.0 (Blink engine 537.36 & JavaScript V8 engine 5.0.71), I wondered for how much longer projects depending on it would last on the old OS. Unsurprisingly, they started dropping XP / XP x64 pretty quickly as well, like Vivaldi, Iron or the new Opera to name a few. Firefox kept support alive however, but of course, the end was looming over our (=XP users’) heads all that time, and now the decision finally has been made!

Mozilla Firefox will cease to support Windows XP, Windows XP x64 Edition and all editions of Windows Vista starting with version 53. If you’re still on 51.0.1, you’ll be provided with not just one, but two consecutive updates:

 

The reason for this is, that the first update just switches your update channel from “Release” to “ESR” (Extended Support Release). ESR versions are also typically present on Enterprise Linux systems like for instance RedHat Enterprise Linux 5 and 6, etc. After a quick restart, the browser will offer the real, final version for XP: Firefox 52.0 ESR.

After the update, the first thing you get to see is the notification of support ending for Windows XP and Windows Vista:

Firefox is notifying the user of how Vista & XP are no longer going to be supported

Firefox is notifying the user of how Vista & XP are no longer going to be supported (click to enlarge)

A quick check confirms it: This is now Firefox 52.0 ESR:

Firefox 52.0 ESR reporting is version

Firefox 52.0 ESR reporting is version (Click to enlarge)

According to the [Mozilla foundation], security updates will be provided for XP/Vista up until September 2017, and the actual, exact EOL date will be fixed mid-2017.

After that, there will no longer be any modern browser support for XP (NT 5.1), XP x64 & Server 2003 (NT 5.2) as well as Vista and the first edition of Server 2008 (NT 6.0). That is, unless somebody provides patched builds, but I don’t think that’s gonna happen…

2.) Electrolysis / e10s

On top of that, I have reevaluated the functionality of Mozillas’ multiprocessing technology on Windows XP x64 Edition. I tried that before with some 50.x version and failed to have Firefox spawn multiple processes for multiple tabs. This is a feature that makes Firefox more crash-proof and faster as well. I’m happy to announce that it does work with Firefox 52.0 ESR though!

Of course, this is not officially supported, not on XP / XP x64, nor on Vista. So force-enabling Electrolysis can only happen at your own risk! To enable the feature, open about:config, confirm the prompt, and then change the following properties as shown, create them manually if they don’t exist yet:

browser.tabs.remote.autostart     [boolean]   true
browser.tabs.remote.force-enable  [boolean]   true
extensions.e10sBloc­kedByAddons    [boolean]   false
extensions.e10sBloc­ksEnabling     [boolean]   false
dom.ipc.processCount              [integer]   16

You need to be a bit careful with dom.ipc.processCount however; Each Firefox process might consume hundreds of megabytes, and with that property set to 16, Firefox can spawn a total of 17 processes, 1 master process and 16 child processes (=tabs). So tune that value to something your machine can take! If you spawn to many processes on a machine with just 2GB or 4GB of memory, you might run into swapping rather quickly!

Also, you need to test this with your plugins and extensions! Not every plugin / extension will play nicely with e10s! In some cases the browser may even crash completely, or have individual tabs crash. You have been warned!

To test this, open about:support and look for Multiprocess Windows. Depending on how many browsers you have launched, it should show something like 1/1 (Enabled by user) or 2/2 (Enabled by user). Or, just open multiple tabs, load web sites in them (yes, you have to) and watch Firefox spawn additional processes in Task Manager:

Firefox spawning processes on XP x64 thanks to Electrolysis having been force-enabled

Firefox spawning processes on XP x64 thanks to Electrolysis having been force-enabled

And that’s it! Enjoy your security updates for a while longer, and then welcome Firefox into the mausoleum that is Windows XP! :) It might be the last browser you’ll ever use on that platform…

How to compile and install NotepadQQ on CentOS 6.x / RHEL 6.x Linux

 Linux, Powerful Tools, Programming, Scripting, Unix  No Responses »
Mar 012017
 

Notepadqq @ CentOS 6 Linux logoIt’s rather rare for me to look for a replacement of some good Windows software for Linux/UNIX instead of the other way around, but the source code editor [Notepad++] is one example of such software. The program gedit on the Gnome 2 desktop environment of my old CentOS 6 enterprise Linux isn’t bad, but it isn’t exactly good either. The thing I was missing most was a search & replace engine capable of regular expressions.

Of course, vi can do it, but at times, vi can be a bit hard to use, so I kinda looked for a Notepad++ replacement. What I found was [Notepadqq], which is basically a clone using the Qt5 UI. However, this editor is clearly made for more modern systems, but I still looked for a way to get it to compile and run on my CentOS 6.8 x86_64 Linux system. And I found one. Here are the most important prerequisites:

  • A new enough GCC (I used 6.2.0), because the v4.4.7 platform compiler won’t work with the modern C++ stuff
  • Qt5 libraries from the [EPEL] repository
  • git

First, you’ll want a new compiler. That part is surprisingly easy, but a bit time consuming. First, download a fresh GCC tarball from a server on the [mirrors list], those are in the releases/ subdirectory, so a file like gcc-6.3.0.tar.bz2 (My version is still 6.2.0). It seems Notepadqq only needs some GCC 5, but since our platform compiler won’t cut it anyway, why not just use the latest?

Now, once more, this will take time, could well be hours, so you might wanna do the compilation step over night, the last step needs root privileges:

$ tar -xzvf ./gcc-6.3.0.tar.bz2
$ cd ./gcc-6.3.0/
$ ./configure --program-suffix="-6.3.0"
$ make
# make install

And when you do this, please never forget to add a --program-suffix for the configuration step!  You might seriously fuck things up if you miss that! So double-check it!

When that’s finally done, let’s handle Qt5 next. I’ll be using a binary distribution to make things easy, and yeah, I didn’t just install the necessary packages, I got the whole Qt5 blob instead, too lazy to do the cherry picking. Oh, and if you don’t have it, add git as well:

# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
# yum install qt5* git

I assume # yum install qt5-qtwebkit qt5-qtwebkit-devel qt5-qtsvg qt5-qtsvg-devel qt5-qttools qt5-qttools-devel should also be enough according to the requirements, but I didn’t try that. Now, enter a free directory or one you generally use for source code and fetch the latest Notepadqq version (this will create a subfolder we’ll cd to):

$ git clone https://github.com/notepadqq/notepadqq.git
$ cd ./notepadqq

After that, we need to make sure that we’ll be using the correct compiler and that we’re linking against the correct libraries that came with it (like libstdc++.so.6.*). To do that, set the following environment variables, assuming you’re using the bash as your shell (use lib/ instead of lib64/ folders if you’re on 32-bit x86):

$ export CC="gcc-6.3.0"
$ export CXX="g++-6.3.0"
$ export CPP="cpp-6.3.0"
$ export CFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export CXXFLAGS="-I/usr/local/include/ -L/usr/local/lib64/"
$ export LDFLAGS="-L/usr/local/lib64/"

The C-related settings are probably not necessary as Qt5 stuff should be pure C++, but you’ll never know, so let’s play it safe.

With that we’re including and linking against the correct libraries and we’ll be using our modern compiler as well. Time to actually compile Notepadqq. To do that, we’ll still need to tell it where to find the Qt5 versions of the qmake and lrelease binaries, but luckily, we can solve that with some simple configuration options. So, let’s do this, the last step requires root privileges again, from within the notepadqq/ directory that git clone created for us:

$ ./configure --qmake /usr/bin/qmake-qt5 --lrelease /usr/bin/lrelease-qt5
$ make
# make install

Now, there are some weird linking issues that I never got fixed on CentOS (some developer please tell me how, I have the same crap when building x265!). Because of that we still can’t launch Notepadqq as-is, we need to give it an LD_LIBRARY_PATH to find the proper libraries at runtime. Let’s just create an executable launcher script /usr/local/sbin/notepadqq.sh for that. Edit it and enter the following code:

#!/bin/sh
LD_LIBRARY_PATH="/usr/local/lib64" /usr/local/bin/notepadqq "$@"

Use this as your launcher script for Notepadqq and you’re good to go with your Notepad++ replacement on good old CentOS 6.x:

Running Notepadqq on CentOS 6 Linux

Running the latest Notepadqq on CentOS 6 Linux with Qt5 version 5.6.1, state 2017-03-01

Now, let’s see whether it’s even that good actually… :roll:

Nekopara mini figures and (heh…) Illya! The fig fag ain’t dead yet…

 Anime & Manga  No Responses »
Feb 272017
 

Nekopara Chocola logo…he was just too lazy to take pictures, especially given my rather stupid photography setup. Hah, but it’s not like I stopped buying weird stuff and the figures I’m gonna show you today are just a fraction of what I got, so here we go!

1.) The Nekos / Catgirls from Nekopara

The first (as you may be able to guess from the logo) are the Nekopara mini figures, from the extremely successful and totally not steamy[1] visual novel series of the same name. Actually, the project will now even get an Anime OVA due to an equally successful [Kickstarter project] that I totally did not pledge for (I guess I still need to do something about my name showing up in the credits :roll: ).

I had planned to ask people to pledge for the extension to that – their [slacker backer] campaign – but it has already overshot the last stretch goal, so not much of a point unless to really want to get your hands on the OVA and/or the merchandise. At least we get a Nyanko bonus OVA now as well. Anyway, here are some pictures – unfortunately I didn’t get a hold of the limited preorder bonus anymore, as usually, click to enlarge:

Nekopara Nyankos

The chibi figures of our kitties, f.l.t.r.: Azuki, Cinnamon, Coconut, Vanilla, Chocola, Maple, Shigure (who’s actually human) and Miruku/Milk. Cute as fuck!

Ah yes, sorry for the licensing stamps. I was too lazy to create the usual two versions, one for here and one for off-site use. If somebody really wants a clean version of some of the photos, let me know in the comments – the license won’t change though.  Let’s look at the individual kittens, pair-wise:

 

Those two are the main catgirls, featuring a touching backstory of having been abandoned when very young and then saved by the main character whose role the player takes in the visual novel. Said backstory will be featured in a mini VN as well as a mini Anime OVA thanks to the Anime crowdfunding campaigns breaking through the $800.000 and $1.000.000 stretch goal milestones. They’re the youngest amongst the cats, and of unknown breed. Also, they’re twins, despite not looking the part.

Next:

 

Here we have the local tsundere loli Azuki (I do like her attitude!), a Munchkin cat, who is the oldest of the pack and the pretty perverted Cinnamon, whom we will see more of in the final VN that’ll come out some time this year – I forgot the actual release date. She’s supposed to be a Scottish Fold.

More:

 

Besides being the largest of the pack (in several ways), Coconuts’ most distinct feature are probably her dichromatic eyes. She’s the second-youngest and pretty naïve as well. The other one here is Maple, an American Curl who tends to get into fights with Coconut on a regular basis. Maple will also be shown more of in the final VN. And two more to go:

 

As said, Shigure is no actual catgirl, but the main characters sister, and a huge brocon as well (although there is no corresponding siscon going on here). She raised all of the first six catgirls shown above, and she supposedly did a great job with that. To the right we see Milk (Hepburn pronunciation “Mi-ru-ku”), who is a side character, not a member of the main cast. She’s a very young kitten helping her owner at a [Takoyaki] stand, and she’s cute as hell.

Yay for genetically engineered catgirls! (We seriously need a Kickstarter campaign for that!)

[1] The games feature explicit content only if you buy the original versions. If you get the games on Steam, they will be censored. There are content restoration patches for those versions as well, but in essence, you can get both uncensored as well as a slightly cheaper, censored versions. The Anime OVAs will not contain any explicit content.

2.) Illyasviel von Einzbern

I had actually sworn to myself that I would never ever watch Magical Girl stuff again (I did it once, in the late 90s’). But then, there came the Yuri, and when there is Girls Love, there’s just no way I’m gonna hold back, no matter the setting. So yes, I did watch all of Fate/Kaleid Liner Prisma☆Illya, and it’s… pretty questionable stuff, at least in season 2 & 3 if I remember correctly. This is Loli Yuri, you have been warned! In any case, let’s go all pink, sigh…

 

Yeah, I know. Just don’t say anything. I feel conflicted and embarrassed enough posting her here! Let’s take a closer look:

Illya, angle shot

Angle shot – I mean, yeah, she’s beautifully crafted, the detail level is high, and it’s a very dynamic pose. The problem with this clearly lies elsewhere :roll:

Let’s take a look at her face:

Illyas' face

Yep, Phat Companys’ sculptors and color producers did a pretty much perfect job here…

It gets much worse though…

Cardholder Analysis 1

This is wrong for too many reasons. Please do not watch the Anime, for I fear you might learn all of them.

You know, I’d love to tell you that my one complaint with this figure is the lack of detail on her card holder, and that I took this photo to show you said lack of detail… And of course, the card holder being totally out of focus here has nothing to do with anything! And that there is an actually correctly taken shot of this doesn’t mean anything either! Ah yeah, here:

Cardholder Analysis 2

The card holder *does* indeed lack some detail however… The FuRyu figure of her has the detailed pattern on it for instance.

Now if we can agree that none of this ever happened, we’re all good! But then again, I could’ve just not posted her in the first place, guess I feel like a criminal who does want to be caught after all.

Well, whatever, so what’s left? There will be two more scale figures in the future, of which one is particularly beautiful – look forward to a nice Kimono for once! And there are also some Nendoroids who’re missing for now, and depending on when a certain Madoka★Magica 2-figure set is really coming out, there will be another two to add to the list.

Guess I still have a lot of pictures to take. Could be that this blog will once more shift from technical stuff to weird Anime merchandise in next few weeks. :roll:

How to make HTML5 H.264/AVC video work on Windows XP/XP x64 (using Firefox)

 Media, t3h intarweb, Windows  14 Responses »
Feb 242017
 

Firefox + HTML5 + XP logo1.) Introduction & Explanation

This is one thing that has brought to me by two users ([SK1] on [Voodooalert]German flag and [liquidLD] who talked to me about this on IRC), and because I got a bit pissed off by it myself, I decided to look into the matter. Basically, HTML5 video on Windows XP / XP x64. But not just with webm (VP8/VP9), but also with H.264/AVC. Let’s face it, a lot of videos on the web rely on H.264 and sometimes you simply can’t watch certain videos or you won’t get all the available resolutions. Of course you could just rely on Adobe Flash, but since Google basically took over with their Pepperflash plugin and their Chrome browser no longer supports XP, it’s not the best move either. Especially when you think about Adobes’ history with critical security loopholes in Flash. HTML5 is just much, much safer, and free as well, and Firefox still supports XP.

Note that this guide is thus based on Firefox exclusively. Anything starting with version 47 should work, official support came in 49, and I’ll be using the current version, 51.0.1 at the time of writing.

So, why doesn’t it “just work” in the first place? It did a few years back, right? Because H.264 playback relies on a DRM plugin, on Linux it would be the Google Widevine plugin, on Windows it’s the Adobe Primetime plugin. So yes, Firefox does support DRM out of the box. But even if content isn’t signed and encrypted, the browser still relies on those plugins to play H.264. And the problem is, that Adobe found some problems with that plugin on XP, so they disabled support on the platform. Their version 17 plugin is still being rolled out with the browser however, and it is binary-compatible with XP, so let’s show you how to re-enable it!

2.) Making it work

On Windows XP and XP x64, the plugin should reside in the folder:

%USERPROFILE%\Application Data\Mozilla\Firefox\Profiles\<your profile folder>\gmp-eme-adobe\17\

That folder should contain the files eme-adobe.dll, eme-adobe.info and eme-adobe.voucher. If it doesn’t (maybe because you have a DRM-free version of Firefox), just create the folder structure yourself, get the necessary files from [here] and place them in that folder.

Having the files present won’t enable Adobe Primetime for you however as you can see on about:plugins (Note: The Cisco stuff you can see there is just for WebRTC, so it’s unusable for HTML5 <video>), we still need to tweak a few things on the about:config page of Firefox. Look for the following properties and set them to the values shown below. If a property doesn’t exist yet – media.gmp-eme-adobe.forceSupported most likely won’t – just create them yourself, all of them are boolean properties and all of them need to be set to true:

media.gmp-eme-adobe.enabled		true
media.gmp-eme-adobe.forceSupported	true
media.gmp-eme-adobe.visible		true
media.gmp.decoder.enabled		true
media.eme.enabled			true
media.mediasource.mp4.enabled		true
media.mp4.enabled			true

After making those changes, you’ll need to restart Firefox. Now you might already be good to go, but on some configurations, about:plugins might show something like this:

HTML5+H.264 on Firefox not yet working

Adobe Primetime seems enabled, but there is no file information? So it’s not actually loading the eme-adobe.dll yet (click to enlarge)

If that happens, open your preferences menu on the top right, click on “Add-ons”, then “Plugins” or just go to about:addons. What you should be seeing is this:

 

However, if Adobe Primetime shows a notice saying that it’s going to be “installed shortly”, forget it. Just do it manually on the plugins’ options page you can see on the right image. To do so, click “Check for Updates”. The warning should be gone momentarily. After that, re-check about:plugins, and you should be getting this:

Adobe Primetime fully enabled

Adobe Primetime fully enabled (click to enlarge)

3.) Testing

Now you can do a quick check on the [Youtube HTML5 page], and it should confirm that everything’s working:

Youtube confirming full HTML5 video support

Youtube confirming full HTML5 video support including H.264 and Media Source Extensions (click to enlarge)

With MSE, even Javascript players (like the Flowworks player) bytestreaming H.264 to Firefox should work! Of course, that’s not very thorough. What you’d want is a real playback test, since you can never be sure what you’re getting on Youtube without a bit of extra work. Decent playback tests are currently available on [Quirksmode], and it should look like this:

Firefox playing HTML5 H.264/AVC video in Firefox on Windows XP x64

Firefox playing HTML5 H.264/AVC video on Windows XP x64 (click to enlarge)

With this, even stuff like Netflix works, because you’re getting not just H.264 playback, but also DRM support. Now, whether DRM support is a good thing or not… You’ll have to decide that for yourself. I’m not supportive of DRM content on the web, but if you want to view or listen to such content, you can!

Just one last word of warning though: Adobe has ended their support for XP with a reason, as the Primetime content decryption plugin has shown problems and instabilities on XP! I’ve been using this for about a week now, and I’ve had one case of a video getting stuck, which is a typical symptom of Primetime throwing up on you. Don’t worry though, Firefox won’t crash. Just move the video slider a bit or restart the video, and it’ll work again! You don’t even need to restart the browser, and such occurrences seem to be quite rare, so I’m fine with it.

There you go!

4.) Thanks

Big thanks fly out to [the guys at MSFN] who came up with all of this. I basically got 100% of my information from them, so thank you! You rock! :)

Update: If you update your version of Firefox to the latest and final 52.0 ESR (extended support release), the last version which will be officially supported until 09-2017 for XP, you might notice that Adobe Primetime just disappeared after the update. That’s because the installer may delete the property media.gmp-eme-adobe.visible from your prefs.js. To reenable it, you’ll have to manually recreate the boolean property and set it to true:

media.gmp-eme-adobe.visible		true

Restart Firefox after the change, and the plugin should reappear on about:plugins and about:addons!

Compiling and installing GCC on CygWin

 Windows  No Responses »
Feb 152017
 

GCC on CygWin logoStill using my aging XP x64, I recently tried to compile a newer version of the GNU compiler collection (GCC) on my equally aging installation of [CygWin], v1.7.35(0.287/5/3). Reason is that I can no longer update CygWin itself, because the project did away with NT5.x compatibility, so now you need Windows Vista or 7 for the latest version. Given that CygWin uses a rolling release model, you can’t get any “in between” versions later on. Also, despite my best efforts to make use of the great work of Peter Castros’ [CygWin Timemachine], I still haven’t managed to get a later version of CygWin that still supports XP. The later versions all have some kind of massive problem with the bash/sh permanently crashing and coredumping. No idea what the reason is.

And even if it would work, I’d still be stuck with GCC 5.3.0 or 5.4.0 or something. It’s not that I absolutely need a fresh C/C++ compiler right now, but it’s good to be prepared, especially when it comes to the adoption of modern C++ standards. Since I’m doing my own Windows builds of libav and ffmpeg (also: For a new x265-based benchmark project similar to my old [x264 benchmark]), I wanted to be able to use a current version.

On Linux and BSD UNIX, compiling and using a new version of GCC is surprisingly simple! On CygWin however, it bombed for me trying to build the JNI (Java Native Interface), and after disabling it, it stumbled over some mysteriously missing files while I was following [this guide].

Luckily, a commenter named Joaquin provided a [solution] for this: CygWin seems to be missing some prerequisites that need to be downloaded. A script for doing that is included in the ./contrib/ folder of the unpacked GCC source tree, ./contrib/download_prerequisites! Let’s have a look inside:

  1. # Download some prerequisites needed by gcc.
  2. # Run this from the top level of the gcc source tree and the gcc
  3. # build will do the right thing.

Sounds useful… and:

  1. # Necessary to build GCC.
  2. MPFR=mpfr-2.4.2
  3. GMP=gmp-4.3.2
  4. MPC=mpc-0.8.1

Aha! So we’re missing “mpfr”, “gmp” and “mpc”. [mpfr] is a floating-point math library, [gmp] is another arithmetic library, and [mpc]… well, a math library as well. I have no idea why my CygWin would be missing those, or maybe it just doesn’t have the required versions? Uhm, and the following:

  1. # Necessary to build GCC with the Graphite loop optimizations.
  2. if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
  3.   ISL=isl-0.15

[ISL] is optional, but I guess it’s useful? I’m not actually sure what it really does though. Whatever it is, just call that helper script before the configuration stage, and everything should be fine. While sitting inside the root of the unpacked source tree, for GCC version 6.3.0 in my case (make SURE to choose a --program-suffix, or installation might effectively annihilate your platform compiler!), do something like this on your CygWin terminal:

./contrib/download_prerequisites
./configure --program-suffix=-6.3.0 --enable-languages=c,c++ --disable-shared 
make -j12
make install

I’m limiting myself to C/C++ here. I don’t need Fortran (I think) and the JNI component of the Java stuff breaks on CygWin anyway, so we’ll leave Java out. Also, we’ll have no link-time optimization (lto), but the important stuff will be there. The C++ shared library is disabled and I built the thing with -j12 to spawn 12 threads (or is it processes?) for speeding up the build, since I have 12 logical CPUs.

And that’s it!

To test things, I recompiled ffmpeg-3.2.4 with the new GCC 6.3.0 + yasm 1.3.0, and everything turned out just fine after rolling out the resulting ffmpeg.exe including some necessary CygWin libraries (cygwin1.dll and cygiconv-2.dll):

.\ffmpeg.exe -version | find /V "configuration"
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 6.3.0 (GCC)
libavutil      55. 34.101 / 55. 34.101
libavcodec     57. 64.101 / 57. 64.101
libavformat    57. 56.101 / 57. 56.101
libavdevice    57.  1.100 / 57.  1.100
libavfilter     6. 65.100 /  6. 65.100
libswscale      4.  2.100 /  4.  2.100
libswresample   2.  3.100 /  2.  3.100
libpostproc    54.  1.100 / 54.  1.100

A quick test showed the ffmpeg binary can cleanly decode H.265/HEVC video and also other stuff like FLAC, so it’s looking good! :)

x264 vs. x265: 1080p ultra low bitrate comparison

 Anime & Manga, Media Encoding  6 Responses »
Jan 252017
 

H.265/HEVC logo1.) Introduction

After doing a [somewhat proper comparison between x264 and x265] a while back, I thought I’d do another one at extremely low bitrates. It reminded me of the time I’ve been using ISDN at 64kbit/s (my provider didn’t let me use CAPI channel aggregation for 128kbit/s), which was the first true flat rate in my country. ‘Cause I’ve been thinking this:

“Can H.265/HEVC enable an ISDN user to stream 1080p content in any useful form?” and “What would H.264/AVC look like in that case?”

Let me say this first: It reaaally depends on how you define “useful”. :roll:

Pretty much nobody uses ISDN these days, and V.9x 56kbaud modems are dying out in the 1st world as well, so this article doesn’t make a lot of sense. To be fair, I didn’t even pick encoding settings fit for low-latency streaming either, nor are my settings fit for live encoding. So it’s just for the lulz, but still! I wanted to see whether it could be done at all, in theory.

To make it happen, I had to choose extremely low bitrates not just for video, but audio as well. There are even subtitles included in my example, which are present in Matroska-style zlib-compressed [.ass] format, so compressed text essentially.

For the audio part, I chose the Fraunhofer FhG-AAC encoder to encode to the lowest possible constant bitrate, which is 8kbit/s HEv2-AAC. That’s a VoIP-focused version of the codec targeted at preserving human speech as well as possible at conditions as bad as they get. And yes, it sounds terrible. But it still gets across just enough to be able to understand what people are saying and what type of sounds are occurring in a scene. Music and most environmental sounds are terrible in quality, but they are still discernible.

For video, I picked a 2-pass ABR mode with a 50kbit/s target bitrate, which is insanely low even for the Anime content I picked (my apologies, Mr. “[Anime is not what everyone watches]”, but yes, I picked Anime again). Note that 2D animated content is pretty easy on the encoders in this case, so the results would’ve likely been a lot worse with 1080p live action content. As for the encoder settings, you can find those [down below] and as for how I’m taking the screenshots, I’ll spare you those details, they’re pretty similar to the stuff shown in the link at the top.

Before we start with the actual quality comparison, I should mention that my test results actually overshot their target, so they’re really unsuitable for live streaming even in the ISDN case. I just didn’t care enough for trying to push the bitrate down any further. Regular streaming would still be possible with my result files, but not without prebuffering. See here:

$ ls -hs *.mkv
2.6M Piaceː Watashi no Italian - Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv
2.0M Piaceː Watashi no Italian - Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv
 76M Piaceː Watashi no Italian - Episode 02.mkv
$ for i in {'Piaceː Watashi no Italian - Episode 02.mkv','Piaceː Watashi no Italian - \
Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv','Piaceː Watashi no Italian - \
Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv'}; do mediainfo "$i" | grep -i "overall bit rate"; done
Overall bit rate                         : 2 640 kb/s
Overall bit rate                         : 88.6 kb/s
Overall bit rate                         : 69.2 kb/s

The first one is the source (note: From [Crunchyroll], legal to watch and record in my country at this time), the second my x264 and the third my x265 versions. Let’s show you the bitrate overshoot of just the video streams in my versions:

$ for i in {'Piaceː Watashi no Italian - Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv','Piaceː \
Watashi no Italian - Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv'}; do mediainfo \
--Output="Video;%BitRate/String%" "$i"; done
71.1 kb/s
51.6 kb/s

So as you can see, x264 messed up pretty big, overshooting by 21.1kbit/s (42.2%), whereas x265 almost landed on target, overshooting by a mere 1.6kbit/s (3.2%) overall. And still… well… Let’s give you an overview first (as usual, click to enlarge images):

2.) Quality comparisons

Note that the color shown in those thumbnails is not representative of the real images, this has been transformed to 256 color .png to make it easier to download (again, if your browser supports it, .webp will be loaded instead transparently). This is just to show you some basic differences between what x264 and x265 are able to preserve, and what they are not. Also, keep in mind, that “~50kbit/s nominal bitrate” means 71.1kbit/s for x264 and 51.6kbit/s for x265!

Overall, x264 fucks up big time. There are frames with partial macroblock drops and completely blank frames even! Also, a lot of frames lose their color either partially or completely as well, making them B/W. And that’s given that x264 even invested 42.2% more bitrate than what I aimed for!

x265 has no such severe issues, all frames are completely there in full color, and that at a bitrate reasonable close to the target. Let’s look at a few interesting cases side by side:

Scene 1 (left: x264, middle: x265, right: source file):

There are some indications of use of larger CTUs (coding tree unit, H.265s’ replacement for macroblocks) in x265s’ case, which is supposed to be one of its strong points, especially for very large resolution encoding (think: 4K/UHD, 8K). While larger blocks can mean loss of detail in that area, it’s ok for larger areas of uniform color, which this Anime has a ton of. H.264/AVC can’t do that so well, because the upper limit for a macroblocks’ size is rather low with 16×16 pixels. You can see the macroblock size pretty clearly in the blocky frame to the left. You need to look a bit more carefully in x265s’ case, but there are a few spots where I believe it can be seen as well. In my case the CTU size for x265 was 32×32px.

Hm, maybe --ctu 64 would’ve been better for this specific case, but whatever.

Lets look at two more mostly color-related comparisons:

Scenes 2 & 3 (left: x264, middle: x265, right: source file):

 

In the first case it seems as if x264 is trying to preserve shades of green more than anything, but in the second case, something terrible happens. There is a lot of red in the scene before this one, and there is quite some red on those can labels as well. It seems x264 doesn’t know where to put the color anymore, and the reds bleed almost all over the frame. And it stays like that for the entire scene as well, which means for several seconds. The greens and browns are lost. Block artifacts are excessive as well, but at least x264 managed to give us whole frames here, with some color even.

Well, the color kinda went everywhere, but uhm, yeah…

Two more:

Scenes 4 & 5 (left: x264, middle: x265, right: source file):

 

I really don’t know what’s with x264 and the reds. Shouldn’t green have priority? I mean, not just in the chroma subsampling, but in encoding as well? But red seems what x264 drops last, and it happens more than once. Given the detail and movements in that last part, even x265 fails though. Yes, it does preserve more color, but it doesn’t come remotely close to the source at this bitrate.

And that other frame with the cuteness overload? There are a lot like those, where x264 just kinda panics, drops everything it has and then frantically tries to (re?)construct the current frame, sometimes only partially until the next I-frame arrives or so.

So that’s it for my quick & dirty “ultra low bitrate” comparison between x264 and x265, at pretty taxing encoding settings once again.

3.) Additional information

x264 encoding settings:

$ mediainfo Piaceː\ Watashi\ no\ Italian\ -\ Episode\ 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv | grep -i \
"encoding settings"
Encoding settings                        : cabac=1 / ref=16 / deblock=1:-2:0 / analyse=0x3:0x133 / \
me=umh / subme=10 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 \
/ 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=18 / \
lookahead_threads=4 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / \
constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / \
open_gop=1 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / \
rc_lookahead=60 / rc=2pass / mbtree=1 / bitrate=50 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=81 \
/ qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:0.60

x265 encoding settings (note: 10 bits per color channel were chosen, same as for x264):

$ mediainfo Piaceː\ Watashi\ no\ Italian\ -\ Episode\ 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv | grep -i \
"encoding settings"
Encoding settings                        : cpuid=1049087 / frame-threads=3 / wpp / pmode / pme / \
no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=0 \
/ level-idc=0 / high-tier=1 / uhd-bd=0 / ref=6 / no-allow-non-conformance / no-repeat-headers / \
annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / \
keyint=250 / bframes=16 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=40 / \
lookahead-slices=0 / scenecut=40 / no-intra-refresh / ctu=32 / min-cu-size=8 / rect / amp / \
max-tu-size=32 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / rdoq-level=1 / signhide / \
no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=5 \
/ limit-refs=1 / limit-modes / me=3 / subme=4 / merange=57 / temporal-mvp / weightp / weightb / \
no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=6 / no-early-skip / rskip / \
no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / rdpenalty=0 / psy-rd=1.60 / \
psy-rdoq=5.00 / no-rd-refine / analysis-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=abr / \
bitrate=50 / qcomp=0.75 / qpstep=4 / stats-write=0 / stats-read=2 / stats-file=265/v.stats / \
cplxblur=20.0 / qblur=0.5 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=1.00 / cutree / \
zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / sar=1 / overscan=0 / \
videoformat=5 / range=1 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 \
/ max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / \
slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05

x264 version:

$ x264 --version
x264 0.148.x
(libswscale 3.0.0)
(libavformat 56.1.0)
built on Sep  6 2016, gcc: 6.2.0
x264 configuration: --bit-depth=10 --chroma-format=all
libx264 configuration: --bit-depth=10 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat license: nonfree and unredistributable
WARNING: This binary is unredistributable!

x265 version:

$ x265 --version
x265 [info]: HEVC encoder version 2.2+23-58dddcf01b7d
x265 [info]: build info [Linux][GCC 6.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

Encoding & testing platform:

$ uname -sr
Linux 2.6.32-573.8.1.el6.x86_64
$ cat /etc/redhat-release 
CentOS release 6.8 (Final)
$ cat /proc/cpuinfo | grep "model name" | uniq
model name	: Intel(R) Core(TM) i7 CPU       X 980  @ 3.33GHz

4.) Answers

Q: “Can H.265/HEVC enable an ISDN user to stream 1080p content in any useful form?“

A: It can probably stream something that at least resembles the original source in a recognizable fashion, but… whether you can call that “useful” or not is another thing entirely…

Q: “What would H.264/AVC look like in that case?”

A: Like shit! :roll:

Monitoring & restarting an unstable Windows service automatically

 Scripting, Server, Windows  No Responses »
Nov 242016
 

Broken Windows logo[1] I know what I should do if a system service on Microsoft Windows starts crashing of course; Fixing it is the way to go! But sometimes you simply can’t, because the component causing a certain instability can’t be swapped out or updated. Now Windows services do have a mechanism for monitoring and restarting a service upon failure, but it seems that only works if the system gets an actual error code back from the service upon termination. But it doesn’t seem to work (at least for me) if the service just dies abnormally. Windows recognizes the service has stopped somehow of course, but the restart procedure just doesn’t kick in.

So I thought I’d do it myself, programmatically. And it’s actually pretty easy. I solved this with VBScript, Windows Batch and Mark Russinovichs’ pslist plus grep. So the prerequisites are:

  • Microsoft Windows (well, huh..)
  • MS Windows Script(ing) Host / VBScript, Windows should come with this preinstalled since Windows 2000.
  • [pslist]
  • [grep][src] (grep is optional, I used GNU grep 2.5.4 in this case, licensed under the [GPLv3+])

Make sure the pstools and grep are within your %PATH%, so Windows can find those .exe files. If you don’t want to use grep, you can also use Microsofts’ own find command, if your version of Windows has it.

I divided this into two small scripts. Since the main part is Batch, it might be problematic if you run it at very short intervals, checking for the services’ status, because you get a command window popping up on the desktop. Since most users wouldn’t want that, another script acts as a launcher, hiding the cmd.exe window so it’s run fully in the background without disturbing any potential users or administrators. The launcher looks like this, in my case it’s meant to watch over an Apache web server:

  1. Set WshShell = CreateObject("WScript.Shell")
  2. WshShell.Run chr(34) & "C:\Server\Scripts\monitor-httpd.bat" & Chr(34), 0
  3. Set WshShell = Nothing

And that script C:\Server\Scripts\monitor-httpd.bat we’re launching looks like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| grep httpd`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

A version relying on Microsoft find instead of GNU grep could look like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| find /I "httpd"`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

To get a services’ exact name, just launch services.msc from Start \ Run or run the command net start on a cmd terminal.

As you can see, this greps “httpd” from the process list and pushes its output into %%I and finally into %HTTPDSTATUS%. We have to use a FOR /F for that, as Windows has no way of pushing command outputs from subshells into shell variables like UNIX has (like e.g. var=`command` or var=$(command)). Then we check for the status of that variable. If it’s not defined, then the process http.exe was nowhere to be found! In that case we restart the associated system service (needs proper permissions!). If the variable is defined, we do nothing but unsetting it, since we can assume the service is operating normally. Or at the very least it’s running. ;)

You can automate that by using the Windows task scheduler:

Scheduling an Apache web server "watchdog"

Scheduling an Apache web server “watchdog” (German Windows)

Create a Schedule to your liking and you’re done! If you can afford the affected service to be down for 5 minutes and no longer, just run it every 4 minutes or so.

The solution shown above can easily be adapted to monitor and restart any Windows service you have, as long as the service isn’t fundamentally broken so that it wouldn’t even start up anymore. Also, you can do a lot more, like sending notification eMails with a command line mailer like [blat] when crashes do occur. Of course, this is only useful for services that crash rarely. If it dies every few minutes, you should reaaally fix it instead of just pushing the restart button all the time… ;)

And that’s that!

[1] © Mar.0007. Original Version for desktopwallpapers4.me.

How to run the IBM SCSI ServeRAID manager on FreeBSD UNIX

 Security, Server, Storage, Unix  No Responses »
Nov 222016
 

FreeBSD IBM ServeRAID Manager logoAnd yet another FreeBSD-related post: After [updating] the IBM ServeRAID manager on my old Windows 2000 server I wanted to run the management software on any possible client. Given it’s Java stuff, that shouldn’t be too hard, right? Turned out not to be too easy either. Just copying the .jar file over to Linux and UNIX and running it like $ java -jar RaidMan.jar wouldn’t do the trick. Got nothing but some exception I didn’t understand. I wanted to have it work on XP x64 (easy, just use the installer) and Linux (also easy) as well as FreeBSD. But there is no version for FreeBSD?!

The ServeRAID v9.30.21 manager only supports the following operating systems:

  • SCO OpenServer 5 & 6
  • SCO Unixware 7.1.3 & 7.1.4
  • Oracle Solaris 10
  • Novell NetWare 6.5
  • Linux (only certain older distributions)
  • Windows (2000 or newer)

I started by installing the Linux version on my CentOS 6.8 machine. It does come with some platform-specific libraries as well, but those are for running the actual RAID controller management agent for interfacing with the driver on the machine running the ServeRAID controller. But I only needed the user space client program, which is 100% Java stuff. All I needed was the proper invocation to run it! By studying IBMs RaidMan.sh, I came up with a very simple way of launching the manager on FreeBSD by using this script I called serveraid.sh (Java is required naturally):

  1. #!/bin/sh
  2.  
  3. # ServeRAID Manager launcher script for FreeBSD UNIX
  4. # written by GAT. http://www.xin.at/archives/3967
  5. # Requirements: An X11 environment and java/openjdk8-jre
  6.  
  7. curDir="$(pwd)"
  8. baseDir="$(dirname $0)/"
  9.  
  10. mkdir ~/.serveraid 2>/dev/null
  11. cd ~/.serveraid/
  12.  
  13. java -Xms64m -Xmx128m -cp "$baseDir"RaidMan.jar com.ibm.sysmgt.raidmgr.mgtGUI.Launch \
  14. -jar "$baseDir"RaidMan.jar $* < /dev/null >> RaidMan_StartUp.log 2>&1
  15.  
  16. mv ~/RaidAgnt.pps ~/RaidGUI.pps ~/.serveraid/
  17. cd "$curDir"

Now with that you probably still can’t run everything locally (=in a FreeBSD machine with ServeRAID SCSI controller) because of the Linux libraries. I haven’t tried running those components on linuxulator, nor do I care for that. But what I can do is to launch the ServeRAID manager and connect to a remote agent running on Linux or Windows or whatever is supported.

Now since this server/client stuff probably isn’t secure at all (no SSL/TLS I think), I’m running this through an SSH tunnel. However, the Manager refuses to connect to a local port because “localhost” and “127.0.0.1” make it think you want to connect to an actual local RAID controller. It would refuse to add such a host, because an undeleteable “local machine” is always already set up to begin with, and that one won’t work with an SSH tunnel as it’s probably not running over TCP/IP. This can be circumvented easily though!

Open /etc/hosts as root and enter an additional fantasy host name for 127.0.0.1. I did it like that with “xin”:

::1			localhost localhost.my.domain xin
127.0.0.1		localhost localhost.my.domain xin

Now I had a new host “xin” that the ServeRAID manager wouldn’t complain about. Now set up the SSH tunnel to the target machine, I put that part into a script /usr/local/sbin/serveraidtunnel.sh. Here’s an example, 34571 is the ServeRAID agents’ default TCP listen port, 10.20.15.1 shall be the LAN IP of our remote machine hosting the ServeRAID array:

#!/bin/bash
ssh -fN -p22 -L34571:10.20.15.1:34571 mysshuser@www.myserver.com

You’d also need to replace “mysshuser” with your user name on the remote machine, and “www.myserver.com” with the Internet host name of the server via which you can access the ServeRAID machine. Might be the same machine or a port forward to some box within the remote LAN.

Now you can open the ServeRAID manager and connect to the made-up host “xin” (or whichever name you chose), piping traffic to and from the ServeRAID manager through a strongly encrypted SSH tunnel:

IBM ServeRAID Manager on FreeBSD

It even detects the local systems’ operating system “FreeBSD” correctly!

And:

IBM ServeRAID Manager on FreeBSD

Accessing a remote Windows 2000 server with a ServeRAID II controller through an SSH tunnel, coming from FreeBSD 11.0 UNIX

IBM should’ve just given people the RaidMan.jar file with a few launcher scripts to be able to run it on any operating system with a Java runtime environment, whether Windows, or some obscure UNIX flavor or something else entirely, just for the client side. Well, as it stands, it ain’t as straight-forward as it may be on Linux or Windows, but this FreeBSD solution should work similarly on other systems as well, like e.g. Apple MacOS X or HP-UX and others. I tested this with the Sun JRE 1.6.0_32, Oracle JRE 1.8.0_112 and OpenJDK 1.8.0_102 for now, and even though it was originally built for Java 1.4.2, it still works just fine.

Actually, it works even better than with the original JRE bundled with RaidMan.jar, at least on MS Windows (no more GUI glitches).

And for the easy way, here’s the [package]! Unpack it wherever you like, maybe in /usr/local/. On FreeBSD, you need [archivers/p7zip] to unpack it and a preferably modern Java version, like [java/openjdk8-jre], as well as X11 to run the GUI. For easy binary installation: # pkg install p7zip openjdk8-jre. To run the manager, you don’t need any root privileges, you can execute it as a normal user, maybe like this:

$ /usr/local/RaidMan/serveraid.sh

Please note that my script will create your ServeRAID configuration in ~/.serveraid/, so if you want to run it as a different user or on a different machine later on, you should recursively copy that directory to the new user/machine. That’ll retain the local client configuration.

That should do it! :)

Updating the IBM ServeRAID manager on my ancient server

 Server, Storage  No Responses »
Nov 212016
 

IBM ServeRAID Manager logoBelieve it or not, the server hosting the very web site you’re reading right now has all of its data stored on an ancient IBM ServeRAID II array made in the year 1995. That makes the SCSI RAID-5 controller 21 years old, and the 9.1GB SCA drives attached to it via hot-plug bays are from 1999, so 17 years old. Recently, I found out that IBMs’ latest SCSI ServeRAID manager from 2011 still supports that ancient controller as well as the almost equally ancient Windows 2000 Server I’m running on the machine. In hope for better management functionality, I chose to give the new software a try. So additionally to my antiquated NT4 ServeRAID manager v2.23.3 I’d also run v9.30.21 side-by-side! This is also in preparation for a potential upgrade to a much newer ServeRAID-4H and larger SCSI drives.

Just so you know how the old v2.23.3 looks, here it is:

IBM ServeRAID Manager v2.23.3

IBM ServeRAID Manager v2.23.3

It really looks like 1996-1997 software? It can do the most important tasks, but there are two major drawbacks:

  1. It can’t notify me of any problems via eMail
  2. It’s a purely standalone software, meaning no server/client architecture => I have to log in via KVM-over-IP or SSH+VNC to manage it

So my hope was that the new software would have a server part and a detachable client component as well as the ability to send eMails whenever shit happens. However, when first launching the new ServeRAID manager, I was greeted with this:

ServeRAID Manager v9.30.21 GUI failure

Now this doesn’t look right… (click to enlarge)

Note that this was my attempt to run the software on Windows XP x64. On Windows 2000, it looked a bit better, but still somewhat messed up. Certain GUI elements would pop up upon mouseover, but overall, the program just wasn’t usable. After finding out that this is Java software being executed by a bundled and ancient version of Sun Java (v1.4.2_12), i just tried to run the RaidMan.jar file with my platform Java. On XP x64 that’s the latest and greatest Java 1.8u112 (even though the installer says it needs a newer operating system this seems to work just fine) and on Windows 2000 it’s the latest supported on that OS: Java 1.6u31. To make RaidMan.jar run on a different JRE on Windows, you can just alter the shortcut the installer creates for you:

Changing the JRE that ServeRAID Manager should be executed by

Changing the JRE that ServeRAID Manager should be executed by

Here it’s run by the javaw.exe command that an old JDK 1.7.0 installer created in %WINDIR%\system32\. It was only later that I changed it to 1.8u112. After changing the JRE to a more modern one, everything magically works:

ServeRAID Manager v9.30.21, logged in

ServeRAID Manager v9.30.21, remotely logged in to my server (click to enlarge)

And this is already me having launched the Manager component on a different machine on my LAN, connecting to the ServeRAID agent service running on my server. So that part works. Since this software also runs on Linux and FreeBSD UNIX, I can set up a proper SSH tunnel script to access it remotely and securely from the outside world as well. Yay! Clicking on the controller gave me this:

ServeRAID Manager v9.30.21 array overview

Array overview (click to enlarge)

Ok, this reminds me of Adaptecs’/ICPs’ StorMan, and since there is some Adaptec license included on the IBM Application CD that this version came from, it might very well be practically the same software. It does show warnings on all drives, while the array and volume are “ok”. The warnings are pretty negligible though, as you can already see above, let’s have a more detailed look:

ServeRAID Manager v9.30.21 disk warranty warnings

So I have possible non-warranted drives? No shit, sherlock! Most of them are older than the majority of todays’ Internet users… I still don’t get how 12 of these drives are still running, seriously… (click to enlarge)

So that’s not really an issue. But what about eMail notifications? Well, take a look:

ServeRAID Manager v9.30.21 notification options

It’s there! (click to enlarge)

Yes! It can notify to the desktop, to the system log and to various email recipients. Also, you can choose who gets which mails by selecting different log levels for different recipients. The only downside is, that the ServeRAID manager doesn’t allow for SSL/TLS connections to mail servers and it can’t even provide any login data. As such, you need your own eMail server on your local network, that allows for unauthenticated and unencrypted SMTP access from the IP of your ServeRAID machine. In my case, no problem, so I can now get eMail notifications to my home and work addresses, as well as an SMS by using my 3G providers’ eMail-2-SMS gateway!

On top of that, you can of course check out disk and controller status as well:

ServeRAID Manager v9.30.21 disk status

Disk status – not much to see here at all (on none of the tabs), probably because the old ServeRAID II can’t do S.M.A.R.T. Maybe good that it can’t, I don’t really want to see 17 year old hard drives’ S.M.A.R.T. logs anyway. ;)

 

ServeRAID Manager v9.30.21 controller status

Status of my ServeRAID II controller, no battery backup unit attached for the 4MB EDO-DRAM write cache and no temperature sensors present, so not much to see here either.

Now there is only one problem with this and that is that the new ServeRAID agent service consumes quite a lot of CPU power in the background, showing as 100% peaks on a single CPU core every few seconds. This is clearly visible in my web-based monitoring setup:

ServeRAID Manager v9.30.21 agent CPU load

The background service is a bit too CPU hungry for my taste (Pentium Pro™ 200MHz). The part left of the “hole” is before installation, the part right of it after installation.

And in case you’re wondering what that hole is right between about 20:30 and 22:00, that’s the ServeRAID Managers’ SNMP components which killed my Microsoft SNMP services upon installation. My network and CPU monitoring solution is based on SNMP though, so that was not good. Luckily, just restarting the SNMP services fixed it. However, as you can see, one of the slow 200MHz cores is now under much higher load. I don’t like that because I’m short on CPU power all the time anyway, but I’ll leave it alone for now, let’s see how it goes.

ServeRAID Manager v9.30.21 splash screen

“Fast configuration”, but a pretty slow background service… :roll:

Now all I need to get is a large pack of large SCA SCSI drives, since I still have that much faster [ServeRAID 4H] with 128MB SDRAM cache and BBU lying around for 3 years anyway! Ah, and as always, the motivation to actually upgrade the server. ;)

Edit: It turns out I found the main culprit for the high CPU load. It seems to be IBMs’ [SNMP sub-agent component] after all, the one that also caused my SNMP service to shut down upon installation. Uninstalling the ServeRAID Manager v9.30.21 and reinstalling it with the SNMP component deselected resulted in a different load profile. See the following graph, the vertical red line separates the state before (with SNMP sub-agent) from the state after (without SNMP sub-agent). Take a look at the magenta line depicting the CPU core that the RAID service was bound to:

ServeRAID Manager v9.30.21 with reduced CPU load

Disabling the ServeRAID managers’ SNMP sub-agent lowers the CPU load significantly!

Thanks fly out to [these guys at Ars Technica] for giving me the right idea!

 Older Entries