Feb 242017
 

Firefox + HTML5 + XP logo1.) Introduction & Explanation

This is one thing that has brought to me by two users ([SK1] on [Voodooalert]German flag and [liquidLD] who talked to me about this on IRC), and because I got a bit pissed off by it myself, I decided to look into the matter. Basically, HTML5 video on Windows XP / XP x64. But not just with webm (VP8/VP9), but also with H.264/AVC. Let’s face it, a lot of videos on the web rely on H.264 and sometimes you simply can’t watch certain videos or you won’t get all the available resolutions. Of course you could just rely on Adobe Flash, but since Google basically took over with their Pepperflash plugin and their Chrome browser no longer supports XP, it’s not the best move either. Especially when you think about Adobes’ history with critical security loopholes in Flash. HTML5 is just much, much safer, and free as well, and Firefox still supports XP.

Note that this guide is thus based on Firefox exclusively. Anything starting with version 47 should work, official support came in 49, and I’ll be using the current version, 51.0.1 at the time of writing.

So, why doesn’t it “just work” in the first place? It did a few years back, right? Because H.264 playback relies on a DRM plugin, on Linux it would be the Google Widevine plugin, on Windows it’s the Adobe Primetime plugin. So yes, Firefox does support DRM out of the box. But even if content isn’t signed and encrypted, the browser still relies on those plugins to play H.264. And the problem is, that Adobe found some problems with that plugin on XP, so they disabled support on the platform. Their version 17 plugin is still being rolled out with the browser however, and it is binary-compatible with XP, so let’s show you how to re-enable it!

2.) Making it work

On Windows XP and XP x64, the plugin should reside in the folder:

%USERPROFILE%\Application Data\Mozilla\Firefox\Profiles\<your profile folder>\gmp-eme-adobe\17\

That folder should contain the files eme-adobe.dll, eme-adobe.info and eme-adobe.voucher. If it doesn’t (maybe because you have a DRM-free version of Firefox), just create the folder structure yourself, get the necessary files from [here] and place them in that folder.

Having the files present won’t enable Adobe Primetime for you however as you can see on about:plugins (Note: The Cisco stuff you can see there is just for WebRTC, so it’s unusable for HTML5 <video>), we still need to tweak a few things on the about:config page of Firefox. Look for the following properties and set them to the values shown below. If a property doesn’t exist yet – media.gmp-eme-adobe.forceSupported most likely won’t – just create them yourself, all of them are boolean properties and all of them need to be set to true:

media.gmp-eme-adobe.enabled		true
media.gmp-eme-adobe.forceSupported	true
media.gmp-eme-adobe.visible		true
media.gmp.decoder.enabled		true
media.eme.enabled			true
media.mediasource.mp4.enabled		true
media.mp4.enabled			true

After making those changes, you’ll need to restart Firefox. Now you might already be good to go, but on some configurations, about:plugins might show something like this:

HTML5+H.264 on Firefox not yet working

Adobe Primetime seems enabled, but there is no file information? So it’s not actually loading the eme-adobe.dll yet (click to enlarge)

If that happens, open your preferences menu on the top right, click on “Add-ons”, then “Plugins” or just go to about:addons. What you should be seeing is this:

 

However, if Adobe Primetime shows a notice saying that it’s going to be “installed shortly”, forget it. Just do it manually on the plugins’ options page you can see on the right image. To do so, click “Check for Updates”. The warning should be gone momentarily. After that, re-check about:plugins, and you should be getting this:

Adobe Primetime fully enabled

Adobe Primetime fully enabled (click to enlarge)

3.) Testing

Now you can do a quick check on the [Youtube HTML5 page], and it should confirm that everything’s working:

Youtube confirming full HTML5 video support

Youtube confirming full HTML5 video support including H.264 and Media Source Extensions (click to enlarge)

With MSE, even Javascript players (like the Flowworks player) bytestreaming H.264 to Firefox should work! Of course, that’s not very thorough. What you’d want is a real playback test, since you can never be sure what you’re getting on Youtube without a bit of extra work. Decent playback tests are currently available on [Quirksmode], and it should look like this:

Firefox playing HTML5 H.264/AVC video in Firefox on Windows XP x64

Firefox playing HTML5 H.264/AVC video on Windows XP x64 (click to enlarge)

With this, even stuff like Netflix works, because you’re getting not just H.264 playback, but also DRM support. Now, whether DRM support is a good thing or not… You’ll have to decide that for yourself. I’m not supportive of DRM content on the web, but if you want to view or listen to such content, you can!

Just one last word of warning though: Adobe has ended their support for XP with a reason, as the Primetime content decryption plugin has shown problems and instabilities on XP! I’ve been using this for about a week now, and I’ve had one case of a video getting stuck, which is a typical symptom of Primetime throwing up on you. Don’t worry though, Firefox won’t crash. Just move the video slider a bit or restart the video, and it’ll work again! You don’t even need to restart the browser, and such occurrences seem to be quite rare, so I’m fine with it.

There you go!

4.) Thanks

Big thanks fly out to [the guys at MSFN] who came up with all of this. I basically got 100% of my information from them, so thank you! You rock! :)

Feb 152017
 

GCC on CygWin logoStill using my aging XP x64, I recently tried to compile a newer version of the GNU compiler collection (GCC) on my equally aging installation of [CygWin], v1.7.35(0.287/5/3). Reason is that I can no longer update CygWin itself, because the project did away with NT5.x compatibility, so now you need Windows Vista or 7 for the latest version. Given that CygWin uses a rolling release model, you can’t get any “in between” versions later on. Also, despite my best efforts to make use of the great work of Peter Castros’ [CygWin Timemachine], I still haven’t managed to get a later version of CygWin that still supports XP. The later versions all have some kind of massive problem with the bash/sh permanently crashing and coredumping. No idea what the reason is.

And even if it would work, I’d still be stuck with GCC 5.3.0 or 5.4.0 or something. It’s not that I absolutely need a fresh C/C++ compiler right now, but it’s good to be prepared, especially when it comes to the adoption of modern C++ standards. Since I’m doing my own Windows builds of libav and ffmpeg (also: For a new x265-based benchmark project similar to my old [x264 benchmark]), I wanted to be able to use a current version.

On Linux and BSD UNIX, compiling and using a new version of GCC is surprisingly simple! On CygWin however, it bombed for me trying to build the JNI (Java Native Interface), and after disabling it, it stumbled over some mysteriously missing files while I was following [this guide].

Luckily, a commenter named Joaquin provided a [solution] for this: CygWin seems to be missing some prerequisites that need to be downloaded. A script for doing that is included in the ./contrib/ folder of the unpacked GCC source tree, ./contrib/download_prerequisites! Let’s have a look inside:

  1. # Download some prerequisites needed by gcc.
  2. # Run this from the top level of the gcc source tree and the gcc
  3. # build will do the right thing.

Sounds useful… and:

  1. # Necessary to build GCC.
  2. MPFR=mpfr-2.4.2
  3. GMP=gmp-4.3.2
  4. MPC=mpc-0.8.1

Aha! So we’re missing “mpfr”, “gmp” and “mpc”. [mpfr] is a floating-point math library, [gmp] is another arithmetic library, and [mpc]… well, a math library as well. I have no idea why my CygWin would be missing those, or maybe it just doesn’t have the required versions? Uhm, and the following:

  1. # Necessary to build GCC with the Graphite loop optimizations.
  2. if [ "$GRAPHITE_LOOP_OPT" = "yes" ] ; then
  3.   ISL=isl-0.15

[ISL] is optional, but I guess it’s useful? I’m not actually sure what it really does though. Whatever it is, just call that helper script before the configuration stage, and everything should be fine. While sitting inside the root of the unpacked source tree, for GCC version 6.3.0 in my case (make SURE to choose a --program-suffix, or installation might effectively annihilate your platform compiler!), do something like this on your CygWin terminal:

./contrib/download_prerequisites
./configure --program-suffix=-6.3.0 --enable-languages=c,c++ --disable-shared 
make -j12
make install

I’m limiting myself to C/C++ here. I don’t need Fortran (I think) and the JNI component of the Java stuff breaks on CygWin anyway, so we’ll leave Java out. Also, we’ll have no link-time optimization (lto), but the important stuff will be there. The C++ shared library is disabled and I built the thing with -j12 to spawn 12 threads (or is it processes?) for speeding up the build, since I have 12 logical CPUs.

And that’s it!

To test things, I recompiled ffmpeg-3.2.4 with the new GCC 6.3.0 + yasm 1.3.0, and everything turned out just fine after rolling out the resulting ffmpeg.exe including some necessary CygWin libraries (cygwin1.dll and cygiconv-2.dll):

.\ffmpeg.exe -version | find /V "configuration"
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 6.3.0 (GCC)
libavutil      55. 34.101 / 55. 34.101
libavcodec     57. 64.101 / 57. 64.101
libavformat    57. 56.101 / 57. 56.101
libavdevice    57.  1.100 / 57.  1.100
libavfilter     6. 65.100 /  6. 65.100
libswscale      4.  2.100 /  4.  2.100
libswresample   2.  3.100 /  2.  3.100
libpostproc    54.  1.100 / 54.  1.100

A quick test showed the ffmpeg binary can cleanly decode H.265/HEVC video and also other stuff like FLAC, so it’s looking good! :)

Jan 252017
 

H.265/HEVC logo1.) Introduction

After doing a [somewhat proper comparison between x264 and x265] a while back, I thought I’d do another one at extremely low bitrates. It reminded me of the time I’ve been using ISDN at 64kbit/s (my provider didn’t let me use CAPI channel aggregation for 128kbit/s), which was the first true flat rate in my country. ‘Cause I’ve been thinking this:

“Can H.265/HEVC enable an ISDN user to stream 1080p content in any useful form?” and “What would H.264/AVC look like in that case?”

Let me say this first: It reaaally depends on how you define “useful”. :roll:

Pretty much nobody uses ISDN these days, and V.9x 56kbaud modems are dying out in the 1st world as well, so this article doesn’t make a lot of sense. To be fair, I didn’t even pick encoding settings fit for low-latency streaming either, nor are my settings fit for live encoding. So it’s just for the lulz, but still! I wanted to see whether it could be done at all, in theory.

To make it happen, I had to choose extremely low bitrates not just for video, but audio as well. There are even subtitles included in my example, which are present in Matroska-style zlib-compressed [.ass] format, so compressed text essentially.

For the audio part, I chose the Fraunhofer FhG-AAC encoder to encode to the lowest possible constant bitrate, which is 8kbit/s HEv2-AAC. That’s a VoIP-focused version of the codec targeted at preserving human speech as well as possible at conditions as bad as they get. And yes, it sounds terrible. But it still gets across just enough to be able to understand what people are saying and what type of sounds are occurring in a scene. Music and most environmental sounds are terrible in quality, but they are still discernible.

For video, I picked a 2-pass ABR mode with a 50kbit/s target bitrate, which is insanely low even for the Anime content I picked (my apologies, Mr. “[Anime is not what everyone watches]”, but yes, I picked Anime again). Note that 2D animated content is pretty easy on the encoders in this case, so the results would’ve likely been a lot worse with 1080p live action content. As for the encoder settings, you can find those [down below] and as for how I’m taking the screenshots, I’ll spare you those details, they’re pretty similar to the stuff shown in the link at the top.

Before we start with the actual quality comparison, I should mention that my test results actually overshot their target, so they’re really unsuitable for live streaming even in the ISDN case. I just didn’t care enough for trying to push the bitrate down any further. Regular streaming would still be possible with my result files, but not without prebuffering. See here:

$ ls -hs *.mkv
2.6M Piaceː Watashi no Italian - Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv
2.0M Piaceː Watashi no Italian - Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv
 76M Piaceː Watashi no Italian - Episode 02.mkv
$ for i in {'Piaceː Watashi no Italian - Episode 02.mkv','Piaceː Watashi no Italian - \
Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv','Piaceː Watashi no Italian - \
Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv'}; do mediainfo "$i" | grep -i "overall bit rate"; done
Overall bit rate                         : 2 640 kb/s
Overall bit rate                         : 88.6 kb/s
Overall bit rate                         : 69.2 kb/s

The first one is the source (note: From [Crunchyroll], legal to watch and record in my country at this time), the second my x264 and the third my x265 versions. Let’s show you the bitrate overshoot of just the video streams in my versions:

$ for i in {'Piaceː Watashi no Italian - Episode 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv','Piaceː \
Watashi no Italian - Episode 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv'}; do mediainfo \
--Output="Video;%BitRate/String%" "$i"; done
71.1 kb/s
51.6 kb/s

So as you can see, x264 messed up pretty big, overshooting by 21.1kbit/s (42.2%), whereas x265 almost landed on target, overshooting by a mere 1.6kbit/s (3.2%) overall. And still… well… Let’s give you an overview first (as usual, click to enlarge images):

2.) Quality comparisons

Note that the color shown in those thumbnails is not representative of the real images, this has been transformed to 256 color .png to make it easier to download (again, if your browser supports it, .webp will be loaded instead transparently). This is just to show you some basic differences between what x264 and x265 are able to preserve, and what they are not. Also, keep in mind, that “~50kbit/s nominal bitrate” means 71.1kbit/s for x264 and 51.6kbit/s for x265!

Overall, x264 fucks up big time. There are frames with partial macroblock drops and completely blank frames even! Also, a lot of frames lose their color either partially or completely as well, making them B/W. And that’s given that x264 even invested 42.2% more bitrate than what I aimed for!

x265 has no such severe issues, all frames are completely there in full color, and that at a bitrate reasonable close to the target. Let’s look at a few interesting cases side by side:

Scene 1 (left: x264, middle: x265, right: source file):

There are some indications of use of larger CTUs (coding tree unit, H.265s’ replacement for macroblocks) in x265s’ case, which is supposed to be one of its strong points, especially for very large resolution encoding (think: 4K/UHD, 8K). While larger blocks can mean loss of detail in that area, it’s ok for larger areas of uniform color, which this Anime has a ton of. H.264/AVC can’t do that so well, because the upper limit for a macroblocks’ size is rather low with 16×16 pixels. You can see the macroblock size pretty clearly in the blocky frame to the left. You need to look a bit more carefully in x265s’ case, but there are a few spots where I believe it can be seen as well. In my case the CTU size for x265 was 32×32px.

Hm, maybe --ctu 64 would’ve been better for this specific case, but whatever.

Lets look at two more mostly color-related comparisons:

Scenes 2 & 3 (left: x264, middle: x265, right: source file):

 

In the first case it seems as if x264 is trying to preserve shades of green more than anything, but in the second case, something terrible happens. There is a lot of red in the scene before this one, and there is quite some red on those can labels as well. It seems x264 doesn’t know where to put the color anymore, and the reds bleed almost all over the frame. And it stays like that for the entire scene as well, which means for several seconds. The greens and browns are lost. Block artifacts are excessive as well, but at least x264 managed to give us whole frames here, with some color even.

Well, the color kinda went everywhere, but uhm, yeah…

Two more:

Scenes 4 & 5 (left: x264, middle: x265, right: source file):

 

I really don’t know what’s with x264 and the reds. Shouldn’t green have priority? I mean, not just in the chroma subsampling, but in encoding as well? But red seems what x264 drops last, and it happens more than once. Given the detail and movements in that last part, even x265 fails though. Yes, it does preserve more color, but it doesn’t come remotely close to the source at this bitrate.

And that other frame with the cuteness overload? There are a lot like those, where x264 just kinda panics, drops everything it has and then frantically tries to (re?)construct the current frame, sometimes only partially until the next I-frame arrives or so.

So that’s it for my quick & dirty “ultra low bitrate” comparison between x264 and x265, at pretty taxing encoding settings once again.

3.) Additional information

x264 encoding settings:

$ mediainfo Piaceː\ Watashi\ no\ Italian\ -\ Episode\ 02-H.264+HEv2AAC-V50kbit-A8kbit.mkv | grep -i \
"encoding settings"
Encoding settings                        : cabac=1 / ref=16 / deblock=1:-2:0 / analyse=0x3:0x133 / \
me=umh / subme=10 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 \
/ 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=18 / \
lookahead_threads=4 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / \
constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / \
open_gop=1 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / \
rc_lookahead=60 / rc=2pass / mbtree=1 / bitrate=50 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=81 \
/ qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:0.60

x265 encoding settings (note: 10 bits per color channel were chosen, same as for x264):

$ mediainfo Piaceː\ Watashi\ no\ Italian\ -\ Episode\ 02-H.265+HEv2AAC-V50kbit-A8kbit.mkv | grep -i \
"encoding settings"
Encoding settings                        : cpuid=1049087 / frame-threads=3 / wpp / pmode / pme / \
no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=1920x1080 / interlace=0 / total-frames=0 \
/ level-idc=0 / high-tier=1 / uhd-bd=0 / ref=6 / no-allow-non-conformance / no-repeat-headers / \
annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / open-gop / min-keyint=23 / \
keyint=250 / bframes=16 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=40 / \
lookahead-slices=0 / scenecut=40 / no-intra-refresh / ctu=32 / min-cu-size=8 / rect / amp / \
max-tu-size=32 / tu-inter-depth=4 / tu-intra-depth=4 / limit-tu=0 / rdoq-level=1 / signhide / \
no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=5 \
/ limit-refs=1 / limit-modes / me=3 / subme=4 / merange=57 / temporal-mvp / weightp / weightb / \
no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=6 / no-early-skip / rskip / \
no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / rdpenalty=0 / psy-rd=1.60 / \
psy-rdoq=5.00 / no-rd-refine / analysis-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=abr / \
bitrate=50 / qcomp=0.75 / qpstep=4 / stats-write=0 / stats-read=2 / stats-file=265/v.stats / \
cplxblur=20.0 / qblur=0.5 / ipratio=1.40 / pbratio=1.30 / aq-mode=3 / aq-strength=1.00 / cutree / \
zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / sar=1 / overscan=0 / \
videoformat=5 / range=1 / colorprim=2 / transfer=2 / colormatrix=2 / chromaloc=0 / display-window=0 \
/ max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / \
slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05

x264 version:

$ x264 --version
x264 0.148.x
(libswscale 3.0.0)
(libavformat 56.1.0)
built on Sep  6 2016, gcc: 6.2.0
x264 configuration: --bit-depth=10 --chroma-format=all
libx264 configuration: --bit-depth=10 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat license: nonfree and unredistributable
WARNING: This binary is unredistributable!

x265 version:

$ x265 --version
x265 [info]: HEVC encoder version 2.2+23-58dddcf01b7d
x265 [info]: build info [Linux][GCC 6.2.0][64 bit] 8bit+10bit+12bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2

Encoding & testing platform:

$ uname -sr
Linux 2.6.32-573.8.1.el6.x86_64
$ cat /etc/redhat-release 
CentOS release 6.8 (Final)
$ cat /proc/cpuinfo | grep "model name" | uniq
model name	: Intel(R) Core(TM) i7 CPU       X 980  @ 3.33GHz

4.) Answers

Q: “Can H.265/HEVC enable an ISDN user to stream 1080p content in any useful form?

A: It can probably stream something that at least resembles the original source in a recognizable fashion, but… whether you can call that “useful” or not is another thing entirely…

Q: “What would H.264/AVC look like in that case?”

A: Like shit! :roll:

Nov 242016
 

Broken Windows logo[1] I know what I should do if a system service on Microsoft Windows starts crashing of course; Fixing it is the way to go! But sometimes you simply can’t, because the component causing a certain instability can’t be swapped out or updated. Now Windows services do have a mechanism for monitoring and restarting a service upon failure, but it seems that only works if the system gets an actual error code back from the service upon termination. But it doesn’t seem to work (at least for me) if the service just dies abnormally. Windows recognizes the service has stopped somehow of course, but the restart procedure just doesn’t kick in.

So I thought I’d do it myself, programmatically. And it’s actually pretty easy. I solved this with VBScript, Windows Batch and Mark Russinovichs’ pslist plus grep. So the prerequisites are:

  • Microsoft Windows (well, huh..)
  • MS Windows Script(ing) Host / VBScript, Windows should come with this preinstalled since Windows 2000.
  • [pslist]
  • [grep][src] (grep is optional, I used GNU grep 2.5.4 in this case, licensed under the [GPLv3+])

Make sure the pstools and grep are within your %PATH%, so Windows can find those .exe files. If you don’t want to use grep, you can also use Microsofts’ own find command, if your version of Windows has it.

I divided this into two small scripts. Since the main part is Batch, it might be problematic if you run it at very short intervals, checking for the services’ status, because you get a command window popping up on the desktop. Since most users wouldn’t want that, another script acts as a launcher, hiding the cmd.exe window so it’s run fully in the background without disturbing any potential users or administrators. The launcher looks like this, in my case it’s meant to watch over an Apache web server:

  1. Set WshShell = CreateObject("WScript.Shell")
  2. WshShell.Run chr(34) & "C:\Server\Scripts\monitor-httpd.bat" & Chr(34), 0
  3. Set WshShell = Nothing

And that script C:\Server\Scripts\monitor-httpd.bat we’re launching looks like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| grep httpd`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

A version relying on Microsoft find instead of GNU grep could look like this:

  1. @ECHO OFF
  2. FOR /F "tokens=* delims= usebackq" %%I IN (`pslist ^| find /I "httpd"`) DO SET HTTPDSTATUS=%%I
  3. IF NOT DEFINED HTTPDSTATUS (net start "Apache2.2") ELSE (SET HTTPDSTATUS=)

To get a services’ exact name, just launch services.msc from Start \ Run or run the command net start on a cmd terminal.

As you can see, this greps “httpd” from the process list and pushes its output into %%I and finally into %HTTPDSTATUS%. We have to use a FOR /F for that, as Windows has no way of pushing command outputs from subshells into shell variables like UNIX has (like e.g. var=`command` or var=$(command)). Then we check for the status of that variable. If it’s not defined, then the process http.exe was nowhere to be found! In that case we restart the associated system service (needs proper permissions!). If the variable is defined, we do nothing but unsetting it, since we can assume the service is operating normally. Or at the very least it’s running. ;)

You can automate that by using the Windows task scheduler:

Scheduling an Apache web server "watchdog"

Scheduling an Apache web server “watchdog” (German Windows)

Create a Schedule to your liking and you’re done! If you can afford the affected service to be down for 5 minutes and no longer, just run it every 4 minutes or so.

The solution shown above can easily be adapted to monitor and restart any Windows service you have, as long as the service isn’t fundamentally broken so that it wouldn’t even start up anymore. Also, you can do a lot more, like sending notification eMails with a command line mailer like [blat] when crashes do occur. Of course, this is only useful for services that crash rarely. If it dies every few minutes, you should reaaally fix it instead of just pushing the restart button all the time… ;)

And that’s that!

[1] © Mar.0007. Original Version for desktopwallpapers4.me.

Nov 222016
 

FreeBSD IBM ServeRAID Manager logoAnd yet another FreeBSD-related post: After [updating] the IBM ServeRAID manager on my old Windows 2000 server I wanted to run the management software on any possible client. Given it’s Java stuff, that shouldn’t be too hard, right? Turned out not to be too easy either. Just copying the .jar file over to Linux and UNIX and running it like $ java -jar RaidMan.jar wouldn’t do the trick. Got nothing but some exception I didn’t understand. I wanted to have it work on XP x64 (easy, just use the installer) and Linux (also easy) as well as FreeBSD. But there is no version for FreeBSD?!

The ServeRAID v9.30.21 manager only supports the following operating systems:

  • SCO OpenServer 5 & 6
  • SCO Unixware 7.1.3 & 7.1.4
  • Oracle Solaris 10
  • Novell NetWare 6.5
  • Linux (only certain older distributions)
  • Windows (2000 or newer)

I started by installing the Linux version on my CentOS 6.8 machine. It does come with some platform-specific libraries as well, but those are for running the actual RAID controller management agent for interfacing with the driver on the machine running the ServeRAID controller. But I only needed the user space client program, which is 100% Java stuff. All I needed was the proper invocation to run it! By studying IBMs RaidMan.sh, I came up with a very simple way of launching the manager on FreeBSD by using this script I called serveraid.sh (Java is required naturally):

  1. #!/bin/sh
  2.  
  3. # ServeRAID Manager launcher script for FreeBSD UNIX
  4. # written by GAT. http://www.xin.at/archives/3967
  5. # Requirements: An X11 environment and java/openjdk8-jre
  6.  
  7. curDir="$(pwd)"
  8. baseDir="$(dirname $0)/"
  9.  
  10. mkdir ~/.serveraid 2>/dev/null
  11. cd ~/.serveraid/
  12.  
  13. java -Xms64m -Xmx128m -cp "$baseDir"RaidMan.jar com.ibm.sysmgt.raidmgr.mgtGUI.Launch \
  14. -jar "$baseDir"RaidMan.jar $* < /dev/null >> RaidMan_StartUp.log 2>&1
  15.  
  16. mv ~/RaidAgnt.pps ~/RaidGUI.pps ~/.serveraid/
  17. cd "$curDir"

Now with that you probably still can’t run everything locally (=in a FreeBSD machine with ServeRAID SCSI controller) because of the Linux libraries. I haven’t tried running those components on linuxulator, nor do I care for that. But what I can do is to launch the ServeRAID manager and connect to a remote agent running on Linux or Windows or whatever is supported.

Now since this server/client stuff probably isn’t secure at all (no SSL/TLS I think), I’m running this through an SSH tunnel. However, the Manager refuses to connect to a local port because “localhost” and “127.0.0.1” make it think you want to connect to an actual local RAID controller. It would refuse to add such a host, because an undeleteable “local machine” is always already set up to begin with, and that one won’t work with an SSH tunnel as it’s probably not running over TCP/IP. This can be circumvented easily though!

Open /etc/hosts as root and enter an additional fantasy host name for 127.0.0.1. I did it like that with “xin”:

::1			localhost localhost.my.domain xin
127.0.0.1		localhost localhost.my.domain xin

Now I had a new host “xin” that the ServeRAID manager wouldn’t complain about. Now set up the SSH tunnel to the target machine, I put that part into a script /usr/local/sbin/serveraidtunnel.sh. Here’s an example, 34571 is the ServeRAID agents’ default TCP listen port, 10.20.15.1 shall be the LAN IP of our remote machine hosting the ServeRAID array:

#!/bin/bash
ssh -fN -p22 -L34571:10.20.15.1:34571 mysshuser@www.myserver.com

You’d also need to replace “mysshuser” with your user name on the remote machine, and “www.myserver.com” with the Internet host name of the server via which you can access the ServeRAID machine. Might be the same machine or a port forward to some box within the remote LAN.

Now you can open the ServeRAID manager and connect to the made-up host “xin” (or whichever name you chose), piping traffic to and from the ServeRAID manager through a strongly encrypted SSH tunnel:

IBM ServeRAID Manager on FreeBSD

It even detects the local systems’ operating system “FreeBSD” correctly!

And:

IBM ServeRAID Manager on FreeBSD

Accessing a remote Windows 2000 server with a ServeRAID II controller through an SSH tunnel, coming from FreeBSD 11.0 UNIX

IBM should’ve just given people the RaidMan.jar file with a few launcher scripts to be able to run it on any operating system with a Java runtime environment, whether Windows, or some obscure UNIX flavor or something else entirely, just for the client side. Well, as it stands, it ain’t as straight-forward as it may be on Linux or Windows, but this FreeBSD solution should work similarly on other systems as well, like e.g. Apple MacOS X or HP-UX and others. I tested this with the Sun JRE 1.6.0_32, Oracle JRE 1.8.0_112 and OpenJDK 1.8.0_102 for now, and even though it was originally built for Java 1.4.2, it still works just fine.

Actually, it works even better than with the original JRE bundled with RaidMan.jar, at least on MS Windows (no more GUI glitches).

And for the easy way, here’s the [package]! Unpack it wherever you like, maybe in /usr/local/. On FreeBSD, you need [archivers/p7zip] to unpack it and a preferably modern Java version, like [java/openjdk8-jre], as well as X11 to run the GUI. For easy binary installation: # pkg install p7zip openjdk8-jre. To run the manager, you don’t need any root privileges, you can execute it as a normal user, maybe like this:

$ /usr/local/RaidMan/serveraid.sh

Please note that my script will create your ServeRAID configuration in ~/.serveraid/, so if you want to run it as a different user or on a different machine later on, you should recursively copy that directory to the new user/machine. That’ll retain the local client configuration.

That should do it! :)

Nov 212016
 

IBM ServeRAID Manager logoBelieve it or not, the server hosting the very web site you’re reading right now has all of its data stored on an ancient IBM ServeRAID II array made in the year 1995. That makes the SCSI RAID-5 controller 21 years old, and the 9.1GB SCA drives attached to it via hot-plug bays are from 1999, so 17 years old. Recently, I found out that IBMs’ latest SCSI ServeRAID manager from 2011 still supports that ancient controller as well as the almost equally ancient Windows 2000 Server I’m running on the machine. In hope for better management functionality, I chose to give the new software a try. So additionally to my antiquated NT4 ServeRAID manager v2.23.3 I’d also run v9.30.21 side-by-side! This is also in preparation for a potential upgrade to a much newer ServeRAID-4H and larger SCSI drives.

Just so you know how the old v2.23.3 looks, here it is:

IBM ServeRAID Manager v2.23.3

IBM ServeRAID Manager v2.23.3

It really looks like 1996-1997 software? It can do the most important tasks, but there are two major drawbacks:

  1. It can’t notify me of any problems via eMail
  2. It’s a purely standalone software, meaning no server/client architecture => I have to log in via KVM-over-IP or SSH+VNC to manage it

So my hope was that the new software would have a server part and a detachable client component as well as the ability to send eMails whenever shit happens. However, when first launching the new ServeRAID manager, I was greeted with this:

ServeRAID Manager v9.30.21 GUI failure

Now this doesn’t look right… (click to enlarge)

Note that this was my attempt to run the software on Windows XP x64. On Windows 2000, it looked a bit better, but still somewhat messed up. Certain GUI elements would pop up upon mouseover, but overall, the program just wasn’t usable. After finding out that this is Java software being executed by a bundled and ancient version of Sun Java (v1.4.2_12), i just tried to run the RaidMan.jar file with my platform Java. On XP x64 that’s the latest and greatest Java 1.8u112 (even though the installer says it needs a newer operating system this seems to work just fine) and on Windows 2000 it’s the latest supported on that OS: Java 1.6u31. To make RaidMan.jar run on a different JRE on Windows, you can just alter the shortcut the installer creates for you:

Changing the JRE that ServeRAID Manager should be executed by

Changing the JRE that ServeRAID Manager should be executed by

Here it’s run by the javaw.exe command that an old JDK 1.7.0 installer created in %WINDIR%\system32\. It was only later that I changed it to 1.8u112. After changing the JRE to a more modern one, everything magically works:

ServeRAID Manager v9.30.21, logged in

ServeRAID Manager v9.30.21, remotely logged in to my server (click to enlarge)

And this is already me having launched the Manager component on a different machine on my LAN, connecting to the ServeRAID agent service running on my server. So that part works. Since this software also runs on Linux and FreeBSD UNIX, I can set up a proper SSH tunnel script to access it remotely and securely from the outside world as well. Yay! Clicking on the controller gave me this:

ServeRAID Manager v9.30.21 array overview

Array overview (click to enlarge)

Ok, this reminds me of Adaptecs’/ICPs’ StorMan, and since there is some Adaptec license included on the IBM Application CD that this version came from, it might very well be practically the same software. It does show warnings on all drives, while the array and volume are “ok”. The warnings are pretty negligible though, as you can already see above, let’s have a more detailed look:

ServeRAID Manager v9.30.21 disk warranty warnings

So I have possible non-warranted drives? No shit, sherlock! Most of them are older than the majority of todays’ Internet users… I still don’t get how 12 of these drives are still running, seriously… (click to enlarge)

So that’s not really an issue. But what about eMail notifications? Well, take a look:

ServeRAID Manager v9.30.21 notification options

It’s there! (click to enlarge)

Yes! It can notify to the desktop, to the system log and to various email recipients. Also, you can choose who gets which mails by selecting different log levels for different recipients. The only downside is, that the ServeRAID manager doesn’t allow for SSL/TLS connections to mail servers and it can’t even provide any login data. As such, you need your own eMail server on your local network, that allows for unauthenticated and unencrypted SMTP access from the IP of your ServeRAID machine. In my case, no problem, so I can now get eMail notifications to my home and work addresses, as well as an SMS by using my 3G providers’ eMail-2-SMS gateway!

On top of that, you can of course check out disk and controller status as well:

ServeRAID Manager v9.30.21 disk status

Disk status – not much to see here at all (on none of the tabs), probably because the old ServeRAID II can’t do S.M.A.R.T. Maybe good that it can’t, I don’t really want to see 17 year old hard drives’ S.M.A.R.T. logs anyway. ;)

 

ServeRAID Manager v9.30.21 controller status

Status of my ServeRAID II controller, no battery backup unit attached for the 4MB EDO-DRAM write cache and no temperature sensors present, so not much to see here either.

Now there is only one problem with this and that is that the new ServeRAID agent service consumes quite a lot of CPU power in the background, showing as 100% peaks on a single CPU core every few seconds. This is clearly visible in my web-based monitoring setup:

ServeRAID Manager v9.30.21 agent CPU load

The background service is a bit too CPU hungry for my taste (Pentium Pro™ 200MHz). The part left of the “hole” is before installation, the part right of it after installation.

And in case you’re wondering what that hole is right between about 20:30 and 22:00, that’s the ServeRAID Managers’ SNMP components which killed my Microsoft SNMP services upon installation. My network and CPU monitoring solution is based on SNMP though, so that was not good. Luckily, just restarting the SNMP services fixed it. However, as you can see, one of the slow 200MHz cores is now under much higher load. I don’t like that because I’m short on CPU power all the time anyway, but I’ll leave it alone for now, let’s see how it goes.

ServeRAID Manager v9.30.21 splash screen

“Fast configuration”, but a pretty slow background service… :roll:

Now all I need to get is a large pack of large SCA SCSI drives, since I still have that much faster [ServeRAID 4H] with 128MB SDRAM cache and BBU lying around for 3 years anyway! Ah, and as always, the motivation to actually upgrade the server. ;)

Edit: It turns out I found the main culprit for the high CPU load. It seems to be IBMs’ [SNMP sub-agent component] after all, the one that also caused my SNMP service to shut down upon installation. Uninstalling the ServeRAID Manager v9.30.21 and reinstalling it with the SNMP component deselected resulted in a different load profile. See the following graph, the vertical red line separates the state before (with SNMP sub-agent) from the state after (without SNMP sub-agent). Take a look at the magenta line depicting the CPU core that the RAID service was bound to:

ServeRAID Manager v9.30.21 with reduced CPU load

Disabling the ServeRAID managers’ SNMP sub-agent lowers the CPU load significantly!

Thanks fly out to [these guys at Ars Technica] for giving me the right idea!

Nov 192016
 

FreeBSD GMABoost logoRecently, after finding out that the old Intel GMA950 profits greatly from added memory bandwidth (see [here]), I wondered if the overclocking mechanism applied by the Windows tool [here] had leaked into the public after all this time. The developer of said tool refused to open source the software even after it turning into abandonware – announced support for GMA X3100 and X4500 as well as MacOS X and Linux never came to be. Also, he did not say how he managed to overclock the GMA950 in the first place.

Some hackers disassembled the code of the GMABooster however, and found out that all that’s needed is a simple PCI register modification that you could probably apply by yourself on Microsoft Windows by using H.Oda!s’ [WPCREdit].

Tools for PCI register modification do exist on Linux and UNIX as well of course, so I wondered whether I could apply this knowledge on FreeBSD UNIX too. Of course, I’m a few years late to the party, because people have already solved this back in 2011! But just in case the scripts and commands disappear from the web, I wanted this to be documented here as well. First, let’s see whether we even have a GMA950 (of course I do, but still). It should be PCI device 0:0:2:0, you can use FreeBSDs’ own pciconf utility or the lspci command from Linux:

# lspci | grep "00:02.0"
00:02.0 VGA compatible controller: Intel Corporation Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller (rev 03)
 
# pciconf -lv pci0:0:2:0
vgapci0@pci0:0:2:0:    class=0x030000 card=0x30aa103c chip=0x27a28086 rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Mobile 945GM/GMS, 943/940GML Express Integrated Graphics Controller'
    class      = display
    subclass   = VGA

Ok, to alter the GMA950s’ render clock speed (we are not going to touch it’s 2D “desktop” speed), we have to write certain values into some PCI registers of that chip at 0xF0hex and 0xF1hex. There are three different values regulating clockspeed. Since we’re going to use setpci, you’ll need to install the sysutils/pciutils package on your machine via # pkg install pciutils. I tried to do it with FreeBSDs’ native pciconf tool, but all I managed was to crash the machine a lot! Couldn’t get it solved that way (just me being too stupid I guess), so we’ll rely on a Linux tool for this. Here is my version of the script, which I call gmaboost.sh. I placed that in /usr/local/sbin/ for global execution:

  1. #!/bin/sh
  2.  
  3. case "$1" in
  4.   200) clockStep=34 ;;
  5.   250) clockStep=31 ;;
  6.   400) clockStep=33 ;;
  7.   *)
  8.     echo "Wrong or no argument specified! You need to specify a GMA clock speed!" >&2
  9.     echo "Usage: $0 [200|250|400]" >&2
  10.     exit 1
  11.   ;;
  12. esac
  13.  
  14. setpci -s 02.0 F0.B=00,60
  15. setpci -s 02.0 F0.B=$clockStep,05
  16.  
  17. echo "Clockspeed set to "$1"MHz"

Now you can do something like this: # gmaboost.sh 200 or # gmaboost.sh 400, etc. Interestingly, FreeBSDs’ i915_kms graphics driver seems to have set the 3D render clock speed of my GMA950 to 400MHz already, so there was nothing to be gained for me in terms of performance. I can still clock it down to conserve energy though. A quick performance comparison using a crappy custom-recorded ioquake3 demo shows the following results:

  • 200MHz: 30.6fps
  • 250MHz: 35.8fps
  • 400MHz: 42.6fps

Hardware was a Core 2 Duo T7600 and the GPU was making use of two DDR-II/667 4-4-4 memory modules in dual channel configuration. Resolution was 1400×1050 with quite a few changes in the Quake III configuration to achieve more performance, so your results won’t be comparable, even when running ioquake3 on identical hardware. I’d post my ~/.ioquake3/baseq3/q3config.cfg here, but in my stupidity I just managed to freaking wipe the file out. Now I have to redo all the tuning, pfh.

But in any case, this really works!

Unfortunately, it only applies to the GMA950. And I still wonder what it was that was so wrong with # pciconf -w -h pci0:0:2:0 0xF0 0060 && pciconf -w -h pci0:0:2:0 0xF0 3405 and the like. I tried a few combinations just in case my byte order was messed up or in case I really had to write single bytes instead of half-words, but either the change wouldn’t apply at all, or the machine would just lock up. Would be nice to do this with only BSD tools on actual FreeBSD UNIX, but I guess I’m just too stupid for pciconf

Nov 142016
 

HP/Compaq nx6310/nc6320 logoA good while back, I got a free notebook from [The_Plague]German flag, a HP/Compaq nx6310[1][2] which he kinda pulled out of the trash at his company. It’s not exactly “Thinkpad T23” material, but it’s a pretty solid, well-built machine with a good keyboard. I’ve been using the thing as an operating system testbed for a while (Linux, ReactOS, Haiku OS, OpenBSD, Dragonfly BSD, and finally: FreeBSD UNIX). After settling for FreeBSD the machine clearly showed its limitations though, the most problematic being imposed by the very low-end i940GML chipset. That one has limited the machine to a single processor core and a 533MHz data rate FSB.

I did give the machine a Core Duo T2450, but switching dual core on in the BIOS results in a lockup at POST time. Also, the chipset cannot use dual-channel DDR-II and limits the user to 2GiB of memory, making the use of a 64-bit processor rather pointless. Which turned out to be bad, because some code doesn’t even provide full functionality for 32-bit anymore, like x265, which dropped deep color support on 32-bit architectures.

But now, The_Plague pulled another one out of the trash, it’s basically the exact same machine, but a higher-end model, the nc6320. This one has an i945GM chipset, which means dual core support, FSB667 and 4GiB dual-channel RAM capability! It came with a Core 2 Duo T5600 @ 1.83GHz with 2MiB L2 cache. I ordered the largest possible chip for this box from ebay Hong Kong, so now it has a Core 2 Duo T7600 @ 2.33GHz with 4MiB L2 cache. Also, 2×2=4GiB of DDR-II/667 CL4 are on their way already, together with a 12-cell secondary monster battery!

And of course, FreeBSD UNIX again, in its brand new version 11.0-RELEASE:

HP/Compaq nc6320 running FreeBSD 11.0 UNIX

HP/Compaq nc6320 running FreeBSD 11.0 UNIX (click to enlarge)

The CPU upgrade is actually even noticeable when browsing the web, lots of resource-hungry Javascript and CSS3, you know. Luckily, Chromium supports hardware acceleration on the Intel GMA950 GPU on FreeBSD, as the OS comes with a kernel modesetting compliant driver for almost all integrated Intel graphics chips. It’s too slow to do the rasterization stage on the GPU, but it still helps.

Once again, it shall serve mostly as a meeting and sysadmin machine, with a little bit of private-use-fun added on top. Let’s have a look at the software! Oh and by the way, I decided to make the screenshots 8-bit .png images, so some of them will look a bit bad. But still better+smaller than JPEG for that purpose:

Running screenfetch on the nc6320

Running screenfetch on the nc6320 (click to enlarge)

$ screenfetch is showing us some details about the machine, which also makes it clear that everything is “Tokisaki Kurumi”-themed. Since there’s a lot of red color on that girls’ garments it seems at least somewhat fitting for a FreeBSD machine.

Chromium with FVD Speed Dial

Chromium with FVD Speed Dial (click to enlarge)

I’m a [Vivaldi] fan personally, but that browser isn’t available on any BSD yet, so I installed a few extensions to make Chromium work somewhat like Vivaldi; The most important part being the static FVD speed dial you can see above. What you can’t see here are the other extensions that followed it: AdBlockPlus and Ghostery. I hear there are better/faster solutions than ABP for ad blocking these days however, so maybe I’ll revise that.

IBM Lotus Notes via wine 1.8

IBM Lotus Notes 6.5.1 via 32-bit wine 1.8.4 (click to enlarge)

Also, for work I would sometimes need IBM Lotus Notes, as it’s our Universities’ groupware solution (think of that what you will). While I couldn’t get the Linux version to run, our Domino servers still accept connections from older clients, so it’s Lotus Notes 6.5.1 running under a 32-bit [wine], which is a solution IBM officially recommended for running the software on Linux/UNIX a few years ago. And yeah, it still works. And if you have Windows software wine can’t cope with?

XP x64 via VirtualBox on FreeBSD

XP x64 via VirtualBox on FreeBSD (click to enlarge)

For anything that wine can’t handle, the VirtualBox port kicks in, as we can see here. Together with the CPUs VT-x extension and the guest tools, virtualizing Windows on FreeBSD UNIX works relatively well. Not all features are there (like USB passthrough), but it works ok for me. Will need a Windows 7 VM as well I think.

More stuff:

Communicating on FreeBSD

Communicating on FreeBSD (parts are censored, click to enlarge)

One important part is communication! Luckily, there is a version of licq in the ports tree now. It builds well together with its Qt4 UI, so no complaints there. Hexchat for IRC access is also available, but the tricky part was Skype; Not that I really need it, but I wanted to have the linuxulator up and running as well! For those of you who don’t know what the “linuxulator” is: It’s a series of kernel modules that extend FreeBSDs kernel with parts of the Linux kernel interface. On top of that, you can pull parts of Fedora 10 or CentOS 6.8 or some CentOS 7 Linux userspace components from the package servers. Together with the kernel modules those form a kind of runtime environment for executing Linux programs – Skype 4.3 in this case! So I have both wine and linuxulator ready for action, and with it access to ICQ, Jabber, MSN, IRC and Skype. Now, what about multimedia?

Multimedia on FreeBSD

smplayer and xmms on FreeBSD, unfortunately the 8-bit color is a bit too noticeable for this screenshot, my apologies (click to enlarge)

This is a part where the upgraded processor also helps. Here we can see (s)mplayer play the last episode of the Anime Hanayamata in taxing 2.5Mbit H.265/HEVC encoding, paired with AAC-LC audio. The Core 2 Duo T5600 had some issues with this, but the faster T7600 shows now problems. Additionally, xmms is playing a Commodore 64 SID tune using libsidplay2 and the reSID engine. xmms comes with a lot of funny plugins from the FreeBSD ports tree for Gameboy tunes or NES tunes, but the C64 one you need to compile for yourself. Not too hard though, you can fetch libsidplay2 and reSID from packages beforehand to make things easier! What else?

ioquake3

ioquake3, a cleaned up version of the Quake III Arena source code, here in its 64-bit FreeBSD build (click to enlarge)

A pretty fun part: Playing the native Quake3 port [ioquake3] in 64-bit, for whenever you just need to shoot something to blow off some steam. ;) I have to say, I had to tweak it quite a bit to run fluently on the WVA 1400×1050 display of this book given the weak GMA950 GPU, but it runs “rather ok” now. ioquake3 is also available for Windows, OSX and Linux by the way, including a more advanced OpenGL 2 renderer, which gives users access to some advanced graphical effects. And if I get bored by that…

HakuNeko Manga ripper and qComicbook

HakuNeko Manga ripper and qComicbook showing some sweet girls love! (click to enlarge)

Once again, fixing up HakuNekos’ build system and C++ code to work with FreeBSD properly took some time. Unfortunately there is no port for it yet (and I’m too stupid/lazy to create one), so you have to fix it by hand. Lots of replacing sed invocations with gsed, find with gfind etc. and the OS #ifdef parts, which need to be changed in several .cpp files, here’s an example from MangaConnector.cpp:

  1. #ifdef __LINUX__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

Something like that needs to turn into this to compile on FreeBSD, otherwise you’ll end up with a HakuNeko that can’t do shit (it’ll still compile and run, but like I said, it’d be devoid of function):

  1. #if defined __LINUX__ || __FreeBSD__
  2. wxString MCEntry::invalidFileCharacters = wxT("/\r\n\t");
  3. endif

This is true for the latest version 1.4.1 as well. I guess the modifications should also apply to other operating systems by adding things like __OpenBSD__ or similar.

Now all that’s left is to wait for that massive 12C battery, the RAM capacity+speed upgrade and some FreeBSD case sticker that I ordered from [unixstickers.com] (hint: That’s a referral URL, it’s supposed to give you some $5 coupon upon ordering, I hope it works). Upon my order, a small part was donated to the LLVM project – very fitting, given that I’ve used clang/llvm a lot to compile stuff on FreeBSD as of late. :)

FreeBSD case sticker (preview)

This is what it’s supposed to look like, and it’s going to replace the current Windows XP+Vista sticker

I hope it’ll look as good in real life! :) Ah, I think I’m gonna have a lot of fun with that old piece of junk. ;)

Ah, and thanks fly out to The_Plague, who saved this laptop from the trash bin and gave it to me for free! Prost!

Edit: And the memory is here, two G.Skill “performance” modules doing 4-4-4 latencies at 667MHz data rate, replacing a single Samsung module running 5-5-5. Now I was interested in how much going from single channel CL5 to dual channel CL4 would really affect performance. Let’s just say, it didn’t do too much for CPU processes. However, the effect on the integrated GMA950 GPU (using shared system memory!) was amazing. It seems the graphics chip was held back a lot by the memory interface! Let’s have a quick look at Quake III Arena performance using a quickly recorded demo just for this purpose (ioquake3 can’t play old Quake III Arena demos like the “001” demo):

  • ioquake3 1.36, single channel DDR-II/667 CL5:
  • 30.6fps
  • ioquake3 1.36, dual channel DDR-II/667 CL4:
  • 41.2fps

Roughly +35%!!

Tests were run three times, then three more times after a reboot. After that, an average was taken. For ioquake3 this wouldn’t even have been necessary though, as the results were extremely consistent. It’s amazing how much the added memory speed really affects the game engine! I rebooted and re-ran the tests several times because I couldn’t believe in that massive boost in performance, but it’s actually true and fully reproducible! This reminds me of how well modern AMD APU graphics chips scale with main memory speed and it explains why people were asking for quad-channel DDR4 on those Kaveri APU chips. Its built-in Radeons would’ve probably loved the added bandwidth!

I also kinda felt that browsing web sites got a lot more smooth using Chromium with most of its GPU acceleration turned on. So I tried the graphics-centric browser test [Motionmark] to put that to the test. Parts of the results were inconclusive, but let’s have a look first:

  • Motionmark 1.0 (medium screen profile), single channel DDR-II/667 CL5:
  • Overall result: 13.85 ±22.24%
  • Multiply: 119.26 ±2.95%
  • Canvas Arcs: 19.04 ±68.48%
  • Leaves: 3.00 ±133.33%
  • Paths: 85.30 ±6.57%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 1.76 ±5.22%
  • Images: 40.58 ±2.56%
  • Design: 18.89 ±8.00%
  • Suits: 24.00 ±37.50%
  • Motionmark 1.0 (medium screen profile), dual channel DDR-II/667 CL4:
  • Overall result: 22.47 ±15.93%
  • Multiply: 124.55 ±1.60%
  • Canvas Arcs: 26.00 ±138.46%
  • Leaves: 65.90 ±16.93%
  • Paths: 37.00 ±16.89%
  • Canvas Lines: 1.00 ±0.00%
  • Focus: 2.00 ±50.00%
  • Images: 41.58 ±3.59%
  • Design: 24.49 ±2.35%
  • Suits: 90.65 ±13.55%

Now first things first: This was just my first pick for any kind of graphics-heavy browser benchmark. I thought I needed something that would make the browser do a lot of stuff on the GPU, given that hardware acceleration was almost fully enabled on FreeBSD UNIX + Chromium + GMA950. However, after repeated runs it showed that the variance was just far too high on the following tests: Leaves, Paths, Suits. Those would also mess up the overall score. The ones that showed consistent performance were: Multiply, Canvas Arcs, Canvas Lines, Focus, Images, Design, so we should focus on those. Well, not all of those tests show promising results (Multiply, Canvas Lines), but some clearly do. It seems my feeling that parts of CSS3 etc. had gotten faster after the memory upgrade was spot-on!

Not bad, not bad at all! And tomorrow morning, the [x264 benchmark] will also have finished, showing how much a classic CPU-heavy task would profit from that upgrade (probably not much, but we’ll see tomorrow).

Edit 2: And here is the rest. Like I thought, the memory upgrade had only minimal impact on CPU performance:

  • x264 benchmark, single channel DDR-II/667 CL5:
  • Runtime: 04:40:08.621
  • x264 benchmark, dual channel DDR-II/667 CL4:
  • Runtime: 04:38:23.851

So yeah it’s faster. But only by a meager +0.62%. Completely negligible. But it’s still a good upgrade given the GPU performance boost and the fact that I can now use more memory for virtual machines. :)

Ah, and here’s the 12-cell ultra capacity battery, which gives me a total of 18 cells in conjunction with the 6-cell primary battery:

Nice hardware actually, you can check it’s charge (roughly) with a button and a 4-LED display, and it has it’s own charging plug. What surprised me most though was this:

$ hwstat | grep -i -e "serial number" -i -e battery
[ACPI Battery (sysctl)]
[battery0]
        Serial number:                  00411 2006/10/12
[battery1]
        Serial number:                  00001 2016/07/29

That probably explains how a still sealed battery could come with a ~25% pre-charge. Manufactured in July 2016, wow. And that for a notebook that’s 10 years old? Ok, it’s an aftermarket battery by [GRS], but that’s just damn fine still! With that I’ll surely have enough battery runtime to make it through longer meetings as well! :)

Edit 3: And today I used the notebook for a sysadmin task, helping our lead developer in debugging a weird problem in a Java-based student exam submission and evaluation system of ours at work. I suspected that the new CuPPIX (=KNOPPIX derivative) distribution I built for this was to blame, but it turned out to be a faulty Java library handling MySQL database access, hence crashing our server software under high parallel loads. In any case, I had the nc6320 with me during the entire morning up until 12:30 or so, walking away with a total charge of 49% left after the developer had fixed the problem. Not stellar given a total of 18 cells, but definitely good enough for me! :)

Edit 4: And my FreeBSD sticker from unixstickers is finally here! They even gave me a bunch of random free stickers to go with it! I gave those to some colleagues for their kids. ;) And here it is:

FreeBSD sticker from unixstickers.com

There was a Windows Vista/XP sticker before, now it shows some UNIX love! (click to enlarge)

The sticker shows some pretty good quality as well, nice stuff! :)

Nov 082016
 

G.SHDSL extender failure (logo)…and it wasn’t even my fault! Can you believe it?! Probably not if you know me, but it’s true nonetheless… Almost 4 days of downtime and we’re back up since just about 2½ hours or so. Given that I already had to do maintenance on the server once this year (replacing a bad hard drive and doing a thorough cleaning as well as dust filter installation), this has crushed the yearly 99%+ availability that I was so proud of. So for the first time since 2006, XIN.at failed to satisfy my personal requirement in that regard. Including the maintenance done on the server and several regular ISP maintenances on the G.SHDSL line, the full downtime should now amount to roughly 90 hours in 2016. If we assume a sum of 8760 hours per year, I’m now down to an availability of ~98.97%.

That value might get a bit worse though if my ISP decides to do another few rounds of maintenance on the DSLAMs in the automatic exchange hub.

So, how did this happen?

It all began when my RAID-6 started acting up, the one in my workstation though, not in the server. Ok, I know, that’s entirely unrelated, but still. It died no pretty death right there last Friday. And once again (this happened before!) it was not the disks to blame, neither the controller, nor the FBM, not even the hotplug bay that I suspected because all disk failures where happening in the same bay. It was the power cable extensions. Again. Even though they’re brand new! I mean, what the hell. At least I know now, that an Areca controller can force RAID-6 arrays to come back to life even if already completely failed with 3+ disks down. Nice one, Areca, I’ll have a cold one in your honor!

And when that RAID was back up, I wanted to pull up my rolling shutters a bit, just because. Which is when the belt ripped in half and the shutters went crashing down, damning me to darkness. Ok, after that I had a beer and just went to bed. Not my day. Next day I did some makeshift repairs on the shutters so they would at least be rolled all the way up and stay there. Having 0% daylight at 09:00am is pretty depressing after all. Ok, after that was done (it was Saturday now), I sat back down in my chair and thought: “Ok, let’s just read my emails…”.

And then my G.SHDSL extender burned up, sending me, my email client, my server and the rest of my digital existence offline…

And that’s when I just knew I had to get up, drive to the supermarket and get a TON of beer!

Seriously… There is bad luck and then there is…

Bad luck never comes alone!

When it rains, it pours, they say

So, the thing just went dark from one moment to the next! No fan, no LEDs, no nothing. At first I thought it might be its external power supply, some standard 12V DC unit. But I measured the voltage and it was perfectly fine. So the extender itself was obviously dead. Never seen such a thing happen with Paradyne/Zhone hardware, but what can you do. So here’s the new one (or maybe it’s refurbished, you never know with this stuff):

Paradyne/Zhone SNE2040G G.SHDSL network extender

Paradyne/Zhone SNE2040G G.SHDSL network extender (click to enlarge)

Now all that’s left is to send the defective unit back and that’s that. I hope I won’t see anything like that happen again… :( At least I got them on the phone on Saturday (business level support), but I only have the small service level agreement with my current contract, so I couldn’t get a technician on weekends. And I wasn’t available “on-site” (at home) on Monday, so the replacement unit had to be shipped via parcel service.

Oh, and neither the 3G fallback solution nor the large SLA (full 24/7 on-site support) will ever be agreed upon for XIN.at – too expensive at ~40€ a month. :( There is just so much money I can pour into a free server after all.

At least everything is back up now, so cheers! Prost!

Sep 072016
 

TeamViewer on Linux logoI’m not exactly a big fan of TeamViewer, since you’ll never know what’s going to happen with that traffic of yours, so I prefer VNC over SSH instead. A few weeks ago I got TeamViewer access to a remote workstation machine for the purpose of processing A/V files however. Basically, it was about video and audio transcoding on said machine.

Since the stream meta data (like the language of an audio stream) wasn’t always there, I wanted to check it by playing back the files remotely in foobar2000 or MPC-HC. TeamViewer does offer a feature to relay the audio from a remote machine to your local box, as long as the remote server has some kind of soundcard / sound chip installed. I was using TeamViewer 11 – the newest version at the time of writing – to connect from CentOS 6.8 Linux to a Windows 7 Professional machine. Playing back audio yielded nothing but silence though.

Now, TeamViewer is actually not native Linux software. Both its Linux and MacOS X versions come with a bundled Wine 1.6 distribution preconfigured to run the 32-bit TeamViewer Windows binary. It was thus logical to assume that the configuration of TeamViewers’ built-in Wine was broken. This may happen in cases where you upgrade TeamViewer from previous releases (which is what I had done, 7 -> 8 -> 9 -> 11).

There are a multitude of proposed solutions to fix this, and since none of them worked for me as-is, I’d like to add my own to the mix. The first useful hint came from [here]. You absolutely need a working system-wide Wine setup for this. I already had one that I needed for work anyway, namely Wine 1.8.6 from the [EPEL] repository, configured using [winetricks]. We’re going to take some files from that installation and essentially replace TeamViewers’ own Wine with the one distributed by EPEL.

So I had TeamViewer 11 installed in /opt/teamviewer/ and some important configuration files for it in ~/.local/share/teamviewer11/ and ~/.config/teamviewer/. First, we backup the wine files of TeamViewer and replace them with the platform ones (the paths may vary depending on your Linux distribution, but the file names should not):

# mv /opt/teamviewer/tv_bin/wine/bin/wine /opt/teamviewer/tv_bin/wine/bin/wine.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wineserver /opt/teamviewer/tv_bin/wine/bin/wineserver.BACKUP
# mv /opt/teamviewer/tv_bin/wine/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/wine-preloader.BACKUP
# mv /opt/teamviewer/tv_bin/wine/lib/libwine.so.1.0 /opt/teamviewer/tv_bin/wine/lib/libwine.so.1.0.BACKUP
# mv /opt/teamviewer/tv_bin/wine/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/wine.BACKUP/
# cp /usr/bin/wine /usr/bin/wineserver /usr/bin/wine-preloader /opt/teamviewer/tv_bin/wine/bin/
# cp /usr/lib/libwine.so.1.0 /opt/teamviewer/tv_bin/wine/lib/
# cp -r /usr/lib/wine/ /opt/teamviewer/tv_bin/wine/lib/

This will replace all the binaries and libraries, in my case shoving Wine 1.8.6 underneath TeamViewer. This isn’t all that’s needed however. We’ll also need the system registry hive of your working Wine installation (with sound). That should be stored in ~/.wine/system.reg! Let’s replace TeamViewers’ own hive with this one:

$ mv ~/.local/share/teamviewer11/system.reg ~/.local/share/teamviewer11/system.reg.BACKUP
$ cp ~/.wine/system.reg ~/.local/share/teamviewer11/

Ok, and the final part is adding the proper Linux audio backend to this Wines’ configuration. That part is stored in ~/.wine/user.reg. Replacing the whole file didn’t work for me though, as TeamViewer would crash upon launch, probably missing some keys from its own user.reg. So, let’s just edit its file instead, open ~/.local/share/teamviewer11/system.reg with your favorite text editor and add the following line in a proper location (it’s sorted alphabetically):

[Software\\Wine\\Drivers\\winepulse.drv] 1473239241

The corresponding file should be found within TeamViewers’ replaced Wine distribution now by the way, in my case it’s /opt/teamviewer/tv_bin/wine/lib/wine/fakedlls/winepulse.drv.

Now, run the TeamViewer profile updater (Some people say it’s required to make this work, it wasn’t for me, but it didn’t hurt either): $ /opt/teamviewer/tv_bin/TeamViewer --update-profile and then its’ Wine configuration: $ /opt/teamviewer/tv_bin/TeamViewer --winecfg. After that, you should be greeted with this:

TeamViewer 11 running its own winecfg

TeamViewer 11 running its own copy of winecfg.

Before the modifications, the configuration window would show “None” as the driver, without any way to change it. So no audio, whereas we have Pulseaudio now. Press “Test Sound” if you want to check whether it truly works. I haven’t tested the ALSA backend by the way. In my case, as soon as the registry was fixed, Wine just autoselected Pulseaudio, which is fine for me.

Now launch TeamViewer and check out the audio options in this submenu:

TeamViewer preferences

The TeamViewer 11 preferences can be found here.

It should look like this:

TeamViewers audio options

Make sure “Play computer sounds and music” is checked! (click to enlarge)

Now, after having connected and logged in, you may also wish to verify the conference audio settings in TeamViewers’ top menu:

TeamViewer 11 conference audio settings

TeamViewer 11 conference audio settings, make sure “Computer sound” is checked!

When you play a sound file on the remote computer, you should hear it on your local one as well. With that, I can finally test the audio files I’m supposed to use on that remote machine for their actual language (which is a rather important detail) where meta data isn’t available.

This seems to be a problem of TeamViewers installation / update procedure which hasn’t been addressed for several major released now. I presume just removing all traces of TeamViewer and installing it from scratch might also do the trick, but I didn’t try it for myself.

Ah, and one more thing: If you can’t launch TeamViewer on CentOS 6.x because you’re getting the following error…

teamviewerd error

TeamViewer Daemon not running…

…forget about the solutions on the web on top of what this message is telling you. TeamViewer 11 uses a systemd-style script for launching its daemon on Linux now, and that won’t do on SysV init systems. Just become root and launch the crap manually: # /opt/teamviewer/tv_bin/teamviewerd &, then press <CTRL>+<d> and it works!

Let’s hope that daemon isn’t doing anything evil while running as root. :roll: