Jun 022016
 

KERNEL_STACK_INPAGE_ERROR logoRecently, this server (just to remind you: an ancient quad Pentium Pro machine with SCSI storage and FPM DRAM) experienced a 1½ hour downtime due to a KERNEL_STACK_INPAGE_ERROR bluescreen, stop code 0x00000077. Yeah yeah, I’m dreaming about running OpenBSD on XIN.at, but it’s still the same old Windows server system. Bites, but hard to give up and/or migrate certain pieces of software. In any case, what exactly does that mean? In essence, it means that the operating systems’ paged pool memory got corrupted. So, heh?

More clearly, either a DRAM error or a disk error, as not the entire paged pool needs to actually be paged to disk. The paged pool is swappable to disk, but not necessarily swapped to disk. So we need to dig a bit deeper. Since this server has 2-error correction and 3-error reporting capability for its memory due to IBM combining the parity FPM-DRAM with additional ECC chips, we can look for ECC/parity error reports in the servers’ system log. Also, disk errors should be pretty apparent in the log. And look what we’ve got here (The actual error messages are German even though the log is being displayed on a remote, English system – well, the server itself is running a German OS):

Actually, when grouping the error log by disk events, I get this:

54 disk errors in total - 8 of which were dead sectors

54 disk errors in total – 8 of which were medium errors – dead sectors

8 unrecoverable dead sectors and 46 controller errors, starting from march 2015 and nothing before that date. Now the actual meaning of a “controller error” isn’t quite clear. In case of SCSI hardware like here, it could be many things. Starting from firmware issues over cabling problems all the way to wrong SCSI bus terminations. Judging from the sporadic nature and the limited time window of the error I guess it’s really failing electronics in the drive however. The problems started roughly 10 years after that drive was manufactured, and it’s an 68-pin 10.000rpm Seagate Cheetah drive with 36GB capacity by the way.

So yeah, march 2015. Now you’re gonna say “you fuck, you saw it coming a long time ago!!”, and yeah, what can I say, it’s true. I did. But you know, looking away while whistling some happy tune is just too damn easy sometimes. :roll:

So, what happened exactly? There are no system memory errors at all, and the last error that has been reported before the BSOD was a disk event id 11, controller error. Whether there was another URE (unrecoverable read error / dead sector) as well, I can’t say. But this happened exactly before the machine went down, so I guess it’s pretty clear: The NT kernel tried to read swapped kernel paged pool memory back from disk, and when the disk error corrupted that critical read operation (whether controller error or URE), the kernel space memory got corrupted in the process, in which case any kernel has to halt the operating system as safe operation can no longer be guaranteed.

So, in the next few weeks, I will have to shut the machine down again to replace the drive and restore from a system image to a known good disk. In the meantime I’ll get some properly tested drives and I’m also gonna test the few drives I have in stock myself to find a proper replacement in due time.

Thank god I have that remote KVM and power cycle capabilities, so that even a non-ACPI compliant machine like the XIN.at server can recover from severe errors like this one, no matter where in the world I am. :) Good thing I spent some cash on an expensive UPS unit with management capabilities and that KVM box…

Mar 152016
 

H.265/HEVC logoJust recently, I’ve tested the computational cost of decoding 10-bit H.265/HEVC on older PCs as well as Android devices – with some external help. See [here]. The result was, that a reasonable Core 2 Quad can do 1080p @ 23.976fps @ 3MBit/s in software without issues, while a Core 2 Duo at 1.6GHz will fail. Also, it has been shown that Android devices – even when using seriously fast quad- and octa-core CPUs can’t do it fluently without a hardware decoder capable of accelerating 10-bit H.265. To my knowledge there is a hack for Tegra K1- and X1-based devices used by MX Player, utilizing the CUDA cores to do the decoding, but all others are being left behind for at least a few more months until Snapdragon 820 comes out.

Today, I’m going to show the results of my tests on Intel Skylake hardware to see whether Intels’ claims are true, for Intel has said that some of their most modern integrated GPUs can indeed accelerate 10-bit video, at least when it comes to the expensive H.265/HEVC. They didn’t claim this for all of their hardware however, so I’d like to look at some lower-end integrated GPUs today, the Intel HD Graphics 520 and the Intel HD Graphics 515. Here are the test systems, both running the latest Windows 10 Pro x64:

  • HP Elitebook 820 G3 (tiny)
  • HP Elitebook 820 G3
  • CPU: Intel [Core i5-6200U]
  • GPU: Intel HD Graphics 520
  • RAM: 8GB DDR4/2133 9-9-9-28-1T
  • Cooling: Active
  • HP Elite X2 1012 G1 (tiny)
  • HP Elite X2 1012 G1 Convertible
  • CPU: Intel [Core m5-6Y54]
  • GPU: Intel HD Graphics 515
  • RAM: 8GB LPDDR3/1866 14-17-17-40-1T
  • Cooling: Passive

Let’s look at the more powerful machine first, which would clearly be the actively cooled Elitebook 820 G3. First, let’s inspect the basic H.265/HEVC capabilities of the GPU with [DXVAChecker]:

DXVAChecker on an Intel HD Graphics 520

DXVAChecker looks good with the latest Intel drivers provided by HP (version 4331): 10-Bit H.264/HEVC is being supported all the way up to 8K!

And this is the ultra-low-voltage CPU housing the graphics core:

Intel Core i5-6200U

Intel Core i5-6200U

So let’s launch the Windows media player of my choice, [MPC-HC], and look at the video decoder options we have:

In any case, both HEVC and UHD decoding have to be enabled manually. On top of that, it seems that either Intels’ proprietary QuickSync can’t handle H.265/HEVC yet, or MPC-HC simply can’t make use of it. The standard Microsoft DXVA2 API however supports it just fine.

Once again, I’m testing with the Anime “Garden of Words” in 1920×1080 at ~23.976fps, but this time with a smaller slice at a higher bitrate of 5Mbit. The encoding options were as follows for pass 1 and pass 2:

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 5000 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1
--slow-firstpass --stats v.stats --sar 1 --range full

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 5000 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2
--stats v.stats --sar 1 --range full

Let’s look at the performance during some intense scenes with lots of rain at the beginning and some less taxing indoor scenes later:

There is clearly some difference, but it doesn’t appear to be overly dramatic. Let’s do a combined graph, putting the CPU loads for GPU-assisted decoding over the regular one as an overlay:

CPU load with software decoding in blue and DXVA2 GPU-accelerated hardware decoding in red

Blue = software decoding, magenta (cause I messed up with the red color) = GPU-assisted hardware decoding

Well, using DXVA2 does improve the situation here, even if it’s not by too much. It’s just that I would’ve expected a bit more here, but I guess that we’d still need to rely on proprietary APIs like nVidia CUVID or Intel QuickSync to get some really drastic results.

Let’s take a look at the Elite X2 1012 G1 convertible/tablet with its slightly lower CPU and GPU clock rates next:

Its processor:

Core m5-6Y54

Core m5-6Y54

And this is, what DXVAChecker has to say about its integrated GPU:

DXVAChecker on an Intel HD Graphics 515

Whoops… Something important seems to be missing here…

Now what do we have here?! Both HD Graphics 520 and 515 should be [architecturally identical]. Both are GT2 cores with 192 shader cores distributed over 24 clusters, 24 texture mapping units as well as 3 rasterizers. Both support the same QuickSync generation. The only marginal difference seems to be the maximum boost clock of 1.05GHz vs. 1GHz, and yet HD Graphics 515 shows no sign of supporting the Main10 profile for H.264/HEVC (“HEVC_VLD_Main10”), so no GPU-assisted 10-bit decoding! Why? Who knows. At the very least they could just scratch 8K support, and implement it for SD, HD, FHD and UHD 4K resolutions. But nope… Only 8-bit is supported here.

I even tried the latest beta driver version 4380 to see whether anything has changed in the meantime, but no; It behaves in the same way.

Let’s look at what that means for CPU load on the slower platform:

CPU load with software decoding

The small Core m5-6Y54 has to do all the work!

We can see that we get close to hitting the ceiling with the CPUs’ boost clock going up all the way. This is problematic for thermally constrained systems like this one. During a >4 hour [x264 benchmark run], the Elite X2 1012 G1 has shown that its 4.5W CPU can’t hold boost clocks this high for a long time, given the passive cooling solution. Instead, it sat somehwere in between 1.7-2.0GHz, mostly in the 1.8-1.9GHz area. This might still be enough with bigger decoding buffers, but DXVA2 would help a bit here in making this slightly less taxing on the CPU, especially considering higher bitrates or even 4K content. Also, when upping the ambient temperature, the runtime could be pushed back by almost an hour, pushing the CPU clock rate further down by 100-200MHz. So it might just not play that movie on the beach in summer at 36°C. ;)

So, what can we learn from that? If you’re going for an Intel/PC-based tablet, convertible or Ultrabook, you need to pick your Intel CPU+graphics solution wisely, and optimally not without testing it for yourself first! Who knows what other GPUs might be missing certain GPU video decoding features like HD Graphics 515 does. Given that there is no actual compatibility matrix for this as of yet (I have asked Intel to publish one, but they said they can’t promise anything), you need to be extra careful!

For stuff like my 10-bit H.265/HEVC videos at reasonably “low” bitrates, it’s likely ok even with the smallest Core m3-6Y30 + HD Graphics 515 that you can find in devices like Microsofts’ own Surface Pro 4. But considering modern tablets’ WiDi (Wireless Display) tech with UHD/4K resolutions, you might want to be careful when choosing that Windows (or Linux) playback device for your big screens!

Feb 252016
 

SlySoft logoIt’s over – after 13 years of being almost constantly under pressure by US-based companies, SlySoft finally had to close its doors. Most notably known for software such as CloneCD or AnyDVD, the Antiguan-based company has provided people all over the world with ways to quickly and easily circumvent disc-based copy protection mechanisms such as Sony ArcCos, CSS, ACSS or BD+ and many others for years.

The companys’ founder, a certain Mr. Giancarla Bettini had already been sued – and successfully so – before an Antiguan court. While it was strictly up to Antiguan Authorities to actually sue SlySoft (because the AACS-LA could not do so themselves due to some legal constraints), this did finally happen in 2012, fining Mr. Bettini for a sum of USD $30.000. That didn’t result in SlySoft closing down however.

What it was that happened exactly a few days ago is unclear, as SlySoft seems to be under NDA or maybe legal pressure as to not release any statement regarding the reasons for the shutdown, quote, “We were not allowed to respond to any request nor to post any statement”. The only thing that we have besides a forum thread with next to zero information is the statement on the official website, which is rather concise as well:

“Due to recent regulatory requirements we have had to cease all activities relating to SlySoft Inc.
We wish to thank our loyal customers/clients for their patronage over the years.”

It should be relatively clear however, that this has to have something to do with the AACS-LA and several movie studios as well as software and hardware companies “reminding” the United States government of SlySofts illicit activities just recently. This would’ve resulted in Antigua being put onto the US priority [watchlist] of countries violating US/international copyright laws. Ultimately, being put onto that list can result in trade barriers being put up within a short time, hurting a countrys’ economy, thus escalating the whole SlySoft thing to an international incident. More information [here].

AnyDVD HD

This little program and its little brothers made it all the way to the top and became an international incident! Quite the career…

It seems – and here is where my pure speculations start – that there was some kind of agreement found between SlySofts’ founder and the AACS-LA and/or the Antiguan and US governments resulting in the immediate shutdown of SlySoft without further consequences for either its founder or other members of the company. If true then SlySoft will surely also have to break their promise of releasing a “final” version of AnyDVD HD including all the decryption keys from the online database in case they have to close their doors forever. This is, what “[…] we have had to cease all activities relating to SlySoft Inc. […]” means after all.

So what are the consequences, technically?

I can only say for AnyDVD HD as according to the forums over at SlySoft, but the latest version 7.6.9.0 supposedly includes some 130.000 AACS keys and should still be able to decrypt a lot of Blu-Rays, even if not all of them.

In the end however, the situation can only deteriorate as time passes and new versions of AACS keys and BD+ certificates are being released, even if you bypass the removed DNS A-Records of key.slysoft.com and access one of the key servers by resolving the IP address locally (via your hosts file). Thing is, nobody can tell when SlySoft will be forced to implement more effective methods of making their services inaccessible, like by just switching off the machines themselves.

But even if they stay online for years to come, no new keys or certificates are going to be added, so it’s probably safe to say that the red fox is truly dead.

AddendumJust to be clear for those of you who are scared of even accessing any SlySoft machines with their real IPs any longer; According to a SlySoft employee (you can read it in their forums), all of the servers are still 100% under SlySofts physical control, and their storage backends are encrypted. They were not raided or anything. So it seems you do not have to fear “somebody else listening” on SlySofts key servers.

PS.: A sad day if you ask me, a victorious one if you ask the movie industry. Maybe somebody should just walk over and tell them, that cheaper, DRM-free media actually work a lot better on the market, when compared to jailing users into some “trusted” (by them) black boxes with forced software updates and closed software. Yeah, I actually want to play my BD movies on the PC (legally!!), and on systems based on free software like Linux and BSD UNIX as well, not on some blackboxed HW player, so go suck it down, Hollywood. I mean, I’m even BUYING your shit, for Christs’ sake…

Oh, by the way, China is actually sitting on that copyright watchlist (I mean, obviously), and they gave us DVDFab. Also, there are MakeMKV and [others as well]. We’ll see whether the AACS-LA can hunt them all down… And even if they can… Will it really make them more money? Debatable at best…

Red Fox logoUpdate: Those guys work fast! While SlySoft is gone, several of the developers have grabbed the software and moved the servers to Belize, the discussion forums have already been migrated and a new version 7.6.9.1 of AnyDVD HD has been released, including new keys and reconfigured to access the new key servers as well. The company is now called “Red Fox” and the forums can be accessed via [forum.redfox.bz].

By now, AnyDVD HD respects the old licenses as well, and this will stay this way for the transition period. Ultimately however, according to posts on the forums, people will have to buy new licenses, even if they had a lifetime license before. They also said they’ll cook up “something nice” for people who bought licenses just recently. Probably some kind of discount I presume.

Still, if I may quote one of the developers: “SlySoft is dead, long live RedFox!”

Feb 202016
 

H.265/HEVC logoRecently, after [successfully compiling] the next generation x265 H.265/HEVC video encoder on Windows, Linux and FreeBSD, I decided to ask for guidance when it comes to compressing Anime (live action will follow at a later time) in the Doom9 forums, [see here]. Thing is, I didn’t understand all of the knobs x265 has to offer, and some of the convenient presets of x264 didn’t exist here (like --tune film and --tune animation). So for a newbie it can be quite hard to make x265 perform well without sacrificing far too much CPU power, as x265 is significantly more taxing on the processor than x264.

Thanks to [Asmodian] and [MeteorRain]/[LittlePox] I got rid of x265s’ blurring issues, and I took their settings and turned them up to achieve more quality while staying within sane encoding times. My goal was to be able to encode 1080p ~24fps videos on an Intel Xeon X5690 hexcore @ 3.6GHz all-core boost clock at >=1fps for a target bitrate of 2.5Mbit.

In this post, I’d like to compare 7 scenes from the highly opulent Anime [The Garden of Words] (言の葉の庭) by [Makoto Shinkai] (新海 誠) at three different average bitrates, 1Mbit, 2.5Mbit (my current x264 default) and 5Mbit. The Blu-Ray source material is H.264/AVC at roughly 28Mbit on average. Also, both encoders are running in 10-bit color depth mode instead of the common 8-bit, meaning that the internal arithmetic precision is boosted from 8- to 16-bit integers as well. While somewhat “non-standard” for H.264/AVC, this is officially supported by H.265/HEVC for Blu-Ray 4K. The mode of operation is 2-pass to aim for comparable file sizes and bitrates. The encoding speed penalty for switching from x264 to x265 at the given settings is around a factor of 8. Somewhat.

The screenshots below are losslessly compressed 1920×1080 images. Since this is all about compression, I chose to serve the large versions of the images in WebP format to all browsers which support it (Opera 11+, Chromium-based Browsers like Chrome, Iron, Vivaldi, the Android Browser or Pale Moon as the only Gecko browser). This is done, because at maximum level, WebP does lossless compression much more efficiently, so the pictures are smaller. This helps, because my server only has 8Mbit/s upstream. If your browser doesn’t support WebP (like Firefox, IE, Edge, Safari), it’ll be fed lossless PNG instead. All of this happens automatically, you don’t need to do anything!

Now, let’s start with the specs.

Specifications:

Here are the source material encoding settings according to the video stream header:

cabac=1 / ref=4 / deblock=1:1:1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=0.40:0.00 /
mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 /
fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=1 / sliced_threads=0 / slices=4 /
nr=0 / decimate=1 / interlaced=0 / bluray_compat=1 / constrained_intra=0 / bframes=3 / b_pyramid=1 /
b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=1 / weightp=1 / keyint=24 / keyint_min=1 /
scenecut=40 / intra_refresh=0 / rc_lookahead=24 / rc=2pass / mbtree=1 / bitrate=28229 /
ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / cplxblur=20.0 / qblur=0.5 /
vbv_maxrate=31600 / vbv_bufsize=30000 / nal_hrd=vbr / filler=0 / ip_ratio=1.40 / aq=1:0.60

x264 10-bit encoding settings (pass 1 & pass 2), 2.5Mbit example:

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 2500 --aq-mode 1 -p 1 --slow-firstpass --stats v.stats -t 2 --no-fast-pskip --cqm flat
--non-deterministic

--fps 24000/1001 --preset veryslow --tune animation --open-gop --b-adapt 2 --b-pyramid normal -f -2:0
--bitrate 2500 --aq-mode 1 -p 2 --stats v.stats -t 2 --no-fast-pskip --cqm flat --non-deterministic

x265 10-bit encoding settings (pass 1 & pass 2), 2.5Mbit example:

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 2500 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1
--slow-firstpass --stats v.stats --sar 1 --range full

--y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --bframes 16 --b-pyramid --bitrate 2500 --rect
--amp --aq-mode 3 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0
--rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2
--stats v.stats --sar 1 --range full

Since x265 can only read raw YUV and Y4M, the source video is being fed to it via [libavs’] avconv tool, piping it into x265. The avconv commandline for that looks as follows:

$ avconv -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>/dev/null

If you want to do something similar, but you don’t like avconv, you can use [ffmpeg] as a replacement, the options are completely the same. Note that you should always specify the correct frame rates (-r) for input and output, or the bitrate setting of the encoder will be applied wrongly!

x264 on the other hand was linked against libav directly, using its decoding capabilities without any workarounds.

Let’s compare:

“The Garden of Words” has a lot of rain. This is a central story element of the 46 minute movie, and it’s hard on any encoder, because a lot of stuff is moving on screen all the time. Let’s take a look at such a scene for our first side-by-side comparison. Each comparison is done in two rows: H.264/AVC first (including the source material), and below that H.265/HEVC, also including the source.

Let’s go:

Scene 1, H.264/AVC encoded by x264 0.148.x:

Scene 1, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

It has been said that x265 performs specifically well at two things: Very high resolutions (which we don’t have here) and low bitrates. And yep, it shows. When comparing the 1Mbit shots, it becomes clear pretty quickly that x265 manages to preserve more detail for the parts with lots of motion. x264 on the other hand starts to wash out the scene pretty severely, smearing out some raindrops, spray water and parts of the foliage. Also, it’s pretty bad around the outlines as well, but that’s true for both encoders. You can spot that easily with all the aliasing artifacts on the raindrops.

Moving up a notch, it becomes very hard to distinguish between the two. When zooming in you can still spot a few minor differences (note the kids umbrella, even if it’s not marked), but it’s quite negligible. Here I’m already starting to think x265 might not be worth it in all cases. There are still differences between the two 2.5Mbit shots and the original however, see the red areas of the umbrella and the most low-contrast, dark parts of the foliage.

At 5Mbit, I really can’t see any difference anymore. Maybe the colors are a little off or something, but when seen in motion, distinguishing between the two and the original becomes virtually impossible. Given that we just threw a really difficult scene at x264 and x265, this should be a trend to continue throughout the whole test.

Now, even more rain:

Scene 2, H.264/AVC encoded by x264 0.148.x:

Scene 2, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Now this is extreme at 1Mbit! Looking at H.264, the spray water on top of the cable can’t even be told apart from the cloud in the background anymore. Detail loss all over the scene is catastrophic in comparison to the original. Tons of raindrops are simply gone entirely, and the texture details on the tower and the angled brick wall of the house to the left? Almost completely washed out and smeared.

Now, let’s look at H.265 @ 1Mbit. The spray water is also pretty bad, but it’s amazing how much more detail was preserved overall. Sure, there are still parts of the raindrops missing, but it’s much, much closer to the original. We can now see details on the walls as well, even the steep angle one on the left. The only serious issue is the red light bleeding at the tower. There is very little red there in the original, so I’m not sure what happened there. x264 does this as well, but x265 is a bit worse.

At the next level, the differences are less pronounced again, but there is still a significant enough improvement when going from x264 to x265 at 2.5Mbit: The spray water on the cable becomes more well-defined, and more rain is being preserved. Also, the textures on the walls are a tiny little bit more detailed and crisp. Once again though, x265 is bleeding too much red light at the tower.

Since it’s noticeably not fully on the level of the source still, let’s look at 5Mbit briefly. x265 is able to preserve a tiny little bit more rain detail here, coming extremely close to the original. In motion, you can’t really see the difference however.

Now, let’s get steamy:

Scene 3, H.264/AVC encoded by x264 0.148.x:

Scene 3, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

1Mbit first again: Let me just say: It’s ugly. x264 pretty much messes up the steam coming from the iron. We get lots of block artifacts now. Some of the low-contrast patterns on the ironing board are being smeared out at a pretty terrible level. Also, the bokeh background partly shows block artifacts and banding. x265 produces quite a lot of banding here itself, but no blocks. Also, outlines and sharp contrasts are more well-defined, and the low contrast part is done noticeably better.

At 2.5Mbit, the patterns repeat themselves now. The steam is only slightly better with x265, outlines are slightly more well-defined, and the low-contrast patterns are slightly more visible. For some of the blurred parts, x265 seems to be a tiny little bit to prone to banding though, in a very few spots, x264 might be just that 1% better. Overall however, x265 wins this, and even if it’s just for the better outlines.

At 5Mbit, you really need to zoom in and analyze very small details, e.g. around the outer border of the steam. Yes, x265 does better again. But you’d not really be able to notice this when watching.

How about we go cry a little bit:

Scene 4, H.264/AVC encoded by x264 0.148.x:

Scene 4, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Cutting onions would be a classic fun part in a slice-of-life anime. Here, it’s just kitchen work. And quite the bad looking one for H.264 at 1Mbit. The letters on the knife are partly lost completely, becoming unreadable. The onion parts that fly off are visibly worse than when encoded with x265 at the same bitrate. Also, x264 produced block artifacts in the blurred bokeh areas again, that simply aren’t there with x265.

On the next level, the two come much closer to each other. However, x265 simply does the outlines better. Less artifacts and sharper, just like with the writing on the knifes’ blade as well. The issues with the bokeh are nearly gone. What’s left is a negligible amount of blocking for x264 and banding for x265. Not really noticeable however.

Finally, at 5Mbit, x265 shows those ever so slightly more well-done outlines. But that’s about it, the rest looks nice for both, and pretty much identical to the source.

Now, please, dig in:

Scene 5, H.264/AVC encoded by x264 0.148.x:

Scene 5, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Let’s keep this short: x264 does blocking, and bad transitions/outlines. x265 does it better. Plain and simple.

At 2.5Mbit, x265 nearly reaches quality transparency when compared to the original, something x264 falls short of, just a bit. While x265 does the outlines and the steam part quite like in the original frame, x264 rips the outlines apart a bit too much, and slight block artifacts can again be seen for the steam part.

At 5Mbit, x264 still shows some blocking artifacts in a part that most lossy image/video compression algorithms traditionally suck at: The reds. While not true for all human beings, most eyes perceive much finer gradients for greens, then blues, and do worst with reds. Meaning, our eyes have an unequal sensitivity distribution when it comes to colors. So image and video codecs try to save bitrate in the reds first, because supposedly it’d be less noticeable. To me subjectively, x265 achieves transparency here, meaning it looks just like the original. x264 doesn’t manage entirely. Close, but not not entirely.

Next:

Scene 6, H.264/AVC encoded by x264 0.148.x:

Scene 6, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

This is a highly static scene, with only few moving parts, so there is some rain again, and some shadow cast by raindrops as well. Now, for the static parts, incremental B frames really work wonders here. Most detail is being preserved by both encoders. What’s supposed to be happening is happening: The encoders save bit rate where the human eye can’t easily tell: In the parts where stuff is moving around very quickly. That’s how we lose a lot of raindrop shadows and some drops as well. x264 seems to have trouble separating the scene into even smaller macro blocks though? Not sure if that’s the reason, but a lot of mesh detail for the basket on the balcony on the top right is lost – x265 does better there! This is maybe because x264 couldn’t distinct the moving drops from the static background so well anymore?

At 2.5Mbit, the scenes become almost indistinguishable. The more static content we have, the easier it gets of course, so the transparency threshold becomes lower. And if you ask me, both of them reach perfect quality at 5Mbit.

Let’s throw another hard one at them for the last round:

Scene 7, H.264/AVC encoded by x264 0.148.x:

Scene 7, H.265/HEVC encoded by x265 1.9+15-425b583f25db:

Enough rain already? Pffh! Here we have a lot of foliage and low contrast added to the mix. And it gets smeared a lot by x264, rain detail lost, fine details of the bushes turning into green mud, that’s how it goes. x265 also loses too much detail here (I mean, 1Mbit is really NOT much), but again, it fares quite a bit better.

At 2.5Mbit, both encoders do very well. Somehow, this scene doesn’t seem to penalize x264 that much at the medium level. You’d really need your magnifying glass to find the spots where x265 still does better, which surprises me a bit for this scene. And finally, at 5Mbit – if you ask me – visual transparency is reached for both x264 and x265.

Final thoughts:

Clearly it’s true what a lot of people have been saying. x265 rocks at low bitrates, if configured correctly. But that isn’t gonna give me perfect quality or anything. Yeah, it sucks less – much less – than x264 in that department, but at a higher 2.5Mbit, where both start looking quite decent, x265 having just a slight edge… it becomes hardly justifiable to use it, simply because it’s that much slower to run it at decent settings.

Also, you need to take device compatibility into account. Sure, a powerful PC can always play the stuff. No matter if it’s some UNIX, Linux, MacOS X or Windows. But how about video game consoles? Older TVs? That kind of thing. Most of those can only play H.264/AVC. Or course, if you’re only using your PC and you have a lot of time and electricity to burn, then hey – why not?

But I’ll have to think really long and really hard about whether I want to replace x264 with x265 at this given point in time. Overall, it might not be practical enough on my current hardware yet. Maybe I’d need an AVX/AVX2-capable processor, as x265 has tons of optimizations for those instruction set extensions. But I’m gonna stay on my Xeon X5690 for quite a while, so SSE 4.2 is the latest I have.

I’d say, if you can stand some quality degradation, then x265 might be the way to go, as it can give you much smaller file sizes at lower bitrates with slight degradation.

If you’re aiming for high bitrates and quality, it might not be worth it right now, at least for 1080p. It’s been said the tables are turning once more when going up to 4K and UHD, but I haven’t tested that yet, as all my content – both Anime and live action movies – are “low resolution” *cough* 1080p or 720p.

Addendum:

Screenshots were taken using the latest stable mplayer 1.3.0 on Linux. Thank god they’re bundling it with ffmpeg now, making things much easier. This choice was made because mplayer can easily grab screenshots from specific spots in a video in an automated fashion. I used framesteps for this, like this:

$ mplayer ./TEST-H.265/HEVC-1mbit.mkv -nosound -vf framestep=24 \
 -vo png:z=9:outdir=./screenshots/1mbit/H.265/HEVC/:prefix=H.265/HEVC-1mbit-

This will decode every 24th frame of the video file TEST-H.265/HEVC-1mbit.mkv, and grab it into a .png file with maximum lossless compression as supported by mplayer. The .png files will be prefixed with a user-defined string and numbered, like H.265/HEVC-1mbit-00000001.pngH.265/HEVC-1mbit-00000002.png and so on, until the end of file is reached.

To encode the full size screenshots to WebP, the most recent [libwebp-0.5.0], or rather one of its companion tools – cwebp – was used as follows:

$ cwebp -z 9 -lossless -q 100 -noalpha -mt ./input.png -o ./output.webp

Now… somebody wanna grant me remote access to some quad socket Haswell-EX Xeon box for free?

No?

Meh… :roll:

Feb 122016
 

H.265/HEVC logo1.) Giving you the binaries:

Just recently I tried to give the x265 H.265/HEVC video encoder another chance to prove itself, because so far I’ve been using x264, so H.264/AVC. x264 does a really good job, but given that the marketing guys are talking about colossal efficiency/quality gains with H.265, I thought I’d put that to the test once again. It was quite easy to compile the software on my CentOS 6.7 Linux, but my old XP x64 machine proved to be a bit tricky.

But, after a bit of trial and error and getting used to the build toolchain, I managed to compile a seemingly stable version from the latest snapshot:

x265 cli, showing the version info

x265 cli, showing its version info.

Update 2: And here comes my first attempt to build an x86_64 multilib binary, that can encode H.265 at 8-bit, 10-bit and 12-bit per pixel color depths. You may wish to use this if you need more flexibility (like using 12-bit for PC only and 8- or 10-bit for a broader array of target systems like TVs, cellphones etc.). It’s currently still being tested for a short encoding run. You can specify the desired color depth with the parameter -D, like -D 8, -D 10 or -D 12:

Update: And here are the newer 1.9 versions, built from source code directly from the [MulticoreWare] (=the developers) servers. I haven’t tested these yet, but given that I configured & compiled them in the same way as before, they “should work™”:

And the old 1.7 versions from the Videolan servers:

So this has been built with MSVC10 and yasm 1.3.0 on Windows XP Pro x64 SP2, meaning this needs the Microsoft Visual C++ 2010 runtime. You can get it from Microsoft if you don’t have it yet: [32-bit version], [64-bit version]. v1.7 tested for basic functionality on XP 32-bit and tested for encoding on XP x64. The 32-bit version only supports 8-bit per pixel, which is default for x264 as well. The 64-bit versions support either 8-, 10- or 12 bits per pixel. Typically, higher internal precision per pixel results in finer gradients and less banding. 8-/10-bit is default for H.265/HEVC. 12-bit will likely not be supported by any hardware players, just as it was with 10-bit H.264/AVC before.

You may or may not know it, but as of now, x265 cannot be linked against either ffmpeg or libav, so it can only read raw input. To “feed” it properly you need either a frame server like [AviSynth] in combination with the pipe tool [Avs4x265], or a decoder that can pipe raw YUV to x265. I went for the latter version, because I already have libav+fdkaac compiled for Windows to get the avconv.exe binary. It’s quite similar to ffmpeg.

This I can only provide as a 64-bit binary, as I’m not going to build it for 32-bit Windows anytime soon I guess, so here you go:

This was compiled with GCC 4.9.2 and yasm 1.3.0 on CygWin x64. To use the two together, add the locations of your EXE files (avconv.exe and x265.exe) to your search path. Then, you can feed arbitrary video (VC1, H.264/AVC, MPEG-2, whatever) to x265. An example for a raw H.264/AVC input stream using the 64-bit versions of the software:

avconv -r 24000/1001 -i video-input.h264 -f yuv4mpegpipe -pix_fmt yuv420p - 2>NUL |^
 x265 - --wpp --y4m -D 12 -p slower --bframes 16 --bitrate 2000 --crf 18 --sar 1^
 --range full -o video-output.h265

Or another, reading some video stream from an MKV container, disabling audio and subtitles:

avconv -r 24000/1001 -i video-input.mkv -f yuv4mpegpipe -pix_fmt yuv420p -an -sn^
 - 2>NUL | x265 - --wpp --y4m -D 12 -p slower --bframes 16 --bitrate 2000 --crf 18^
 --sar 1 --range full -o video-output.h265

Just remove the carets and line breaks to make single-line commands out of those if preferred. To understand all the options, make yourself some readmes like this: avconv -h full > avconv-readme.txt and x265 --log-level full --help > x265-readme.txt or read the documentation online.

2.) How to compile by yourself:

2a.) Prerequisites:

I won’t describe how to build libav here, but just the x265 part. First of all, you need some software to be able to do this, some of it free, other not so much (unless you can swap MSVC with MSYS, I didn’t try that):

  • [cmake] (I used version 2.8.12 because that’s roughly what I have on Linux.)
  • [Mercurial] (Needed to fetch the latest version from x265′ versioning system. I used the latest Inno Setup installer.)
  • [yasm] (Put yasm.exe in your search path and you’re fine. This is optional, but you really want this for speed reasons.)
  • [Microsoft Visual Studio] (Use 2010 if you’re on Windows XP. Supported versions: 2008/VC9, 2010/VC10, 2012/VC11 & 2013/VC12)
  • [x265 source code] (Enter a target download path and use Mercurials hg.exe like hg clone https://bitbucket.org/multicoreware/x265 to fetch it)

2b.) Preparation of the solution:

Usually, you would use cmake to have it compile your entire project, but in this case it’ll build Visual Studio project files and a solution file for us. To do this, enter the proper build path. In my case I’m using Visual Studio 2010, so VC10, and I’d like to build the 64-bit version, so with the unpacked x265 source, I’d enter its subdirectory build\vc10-x86_64\ and then run the generation script: .\make-solutions.bat:

make-solutions.bat preparing cmake for us

make-solutions.bat is preparing cmake for us.

There are several things you need to make sure here: First, if you’re on Windows XP or Windows Vista, you need to toggle the WINXP_SUPPORT flag. Also, if you’re compiling for a 64-bit target, you may wish to enable HIGH_BIT_DEPTH as well to get to either 10-bit or even 12-bit per pixel other than just 8. The 32-bit code doesn’t seem to support high bit dephts right now.

Then there is one more important thing; With CMAKE_CONFIGURATION_TYPES set to Debug;Release;MinSizeRel;RelWithDebInfo, my build was unstable, throwing errors during encoding, like x265 [error]: scaleChromaDist wrap detected dist: -2029735662 lambda: 256. Setting CMAKE_CONFIGURATION_TYPES to just Release solved that problem! So you may wish to do the same.

Make sure ENABLE_CLI and ENABLE_ASSEMBLY are checked as well, then click Configure. If you’re building with high bit depth support, you’ll be presented with another option, MAIN12. You should enable this to get Main12 profile support in case you’re planning to build a 12-bit encoder. If you don’t pick it, you’ll get a 10-bit version instead, staying within Blu-Ray 4K specifications. After that, click Configure again. Generally, you need to click Configure unless no more red stuff pops up, then click Generate.

2c.) Compiling and installing the solution:

Load the resulting solution file x265.sln into Microsoft Visual Studio, then right click ALL_BUILD and pick Build. This will compile x265. If you want to install it from the IDE as well, right click INSTALL and select Build. This will install x265 in %PROGRAMFILES%\x265\ with the binary sitting in %PROGRAMFILES%\x265\bin\:

Microsoft Visual Studio 2010, ready to compile the x265 solution generated by cmake

Microsoft Visual Studio 2010 with the x265 solution generated by cmake loaded, compiled and installed.

3.) Running it:

Now we can feed x265 some raw YUV files like this, after adding x265.exe to the search path:

x265 encoding a raw YUV file to H.265/HEVC

x265 encoding a raw YUV 4:2:0 file to H.265/HEVC (The options given to x265 may actually suck, I’m still in the learning process).

Or we can use a decoder to feed it arbitrary video formats, even from MKV containers, like shown in the beginning. ffmpeg or avconv can decode pretty much anything, and then pipe it into x265 as raw YUV 4:2:0:

x265 being fed a H.264/AVC bitstream by avconv

x265 being fed a H.264/AVC bitstream by avconv.

And that’s it! Now all I need is some 18-core beast processor to handle the extreme slowness of the encoder at such crazy settings. When going almost all-out, it’s easily 10 times as slow as x264 (at equally insane settings)! Or maybe I can get access to some rack server with tons of cores or something… :roll:

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.

Jan 152016
 

qWebIRC logoWhen I had set XINs web chat up back in 2014, I thought I’d found the holy grail of free IRC web frontends, but that wasn’t quite the case. While it worked, it wasn’t overly stable, and its GUI was a pretty crappy high-load HTML5/JavaScript part that didn’t work in a lot of browsers. It was based on the “kind of pre-alpha” [webchat2], a project which was dropped somewhere in the middle of the development process.

The biggest issue however was, that when a user was idle for like 5-10 minutes, webchat2 would drop his IRC connection in the backend without telling the user. So while the user kept thinking “oh, nobody is saying anything”, people might have continued to talk without him seeing it. The error became apparent only if the affected user started to write something again, which is when the “connection lost”-or-something message appeared.

Webchat, joined a channel

webchat2 – It looks nice, but it doesn’t really work that well.

It seems that software was bad at maintaining persistent connections for extended periods of time.

Back then I had tried several other alternatives, but most are based on [node.js], which my ancient Windows 2000 server (yeah yeah, I know) cannot run. I did stumble over the Python-based [qWebIRC] back then, but for some reason I had probably failed to install it properly. That piece was developed by the [QuakeNet] guys, who’re running it on their own site as well.

Yesterday I decided to give it another shot, and well…

qWebIRC login

The minimalistic qWebIRC login screen. “LunaticNet” isn’t really an IRC network though, it’s just the XIN.at IRC server by itself…

I wanted it perfect as well, so I aimed at fulfilling all the dependencies, which are:

  • Some IRC server (Duh! I won’t cover that part in detail here, but I’m running UnrealIRCd).
  • Python 2.5.x, 2.6.x or 2.7.x (obviously, and keep in mind that it won’t work with any Python 3.x).
  • zope.interface (a contract-based programming interface required by Twisted).
  • Twisted (for event-driven networking, something IRC needs to push stuff  happening on the IRC server to the web frontend).
  • pyWin32 (to enable Python to interface with the Win32 APIs).
  • simplejson (optional; preferably a version including its C extensions, provides a performance boost).
  • pyOpenSSL (optional; required if you wish to connect to IRC+SSL servers and/or to host the web chat via HTTPS instead of HTTP).
  • Java (optional; used for JavaScript minify during compile time. Makes the JS much smaller to save bandwidth).
  • Mercurial (optional; fast versioning system, provides a qWebIRC performance boost for some reason I don’t quite get yet).
  • instsrv & srvany (optional; Used to create a Windows system service for qWebIRC).

Now that’s quite something, and given that I’m doing this on Windows 2000, there have to be compromises. While the latest Python 2.7.11 can work on Win2k, the installer will fail. 2.7.3 is the last which works “cleanly”. You can still install 2.7.11 on a modern Windows box and then just copy it over, but then you won’t have it registered in the OS. In any case, I decided to go with the much older Python 2.5.4, also because some of the modules listed above including machine code were nowhere to be found for Python 2.7.x in a pre-compiled state.

So, some software is brand-new (from 2016 even), and other parts not so much. I tried to use the newest possible software without having to compile any machine code myself (like the C extensions of simplejson), because that would’ve been a lot of work.

I packaged everything I picked for this into one archive for you to use, here it is:

What you get are the following versions:

  • qWebIRC #516de557ddc7
  • Python v2.5.4
  • zope.interface v3.8.0
  • Twisted v12.1.0
  • pyWin32 v220
  • simplejson v2.1.1 with C extensions
  • pyOpenSSL v0.13.12 built by egenix
  • Sun Java Runtime Environment v1.6u31
  • Mercurial v3.4.2

And that’s what it looks like when it’s up and running:

qWebIRC chat

What qWebIRC looks like for a user logged into the XIN.at IRC server.

Now how do you install this? Simply follow these step-by-step instructions:

  1. Install Python 2.5.4. Make sure python.exe is in your systems search path. If it isn’t, add it.
  2. Copy the zope\ folder from the zope.interface 3.8.0 to the Lib\ subdirectory of your Python 2.5 installation, so that it looks like: C:\Program Files\Python25\Lib\zope\. Make sure the user who will run qWebIRC has sufficient permissions on the folder.
  3. Install Twisted 12.1.0.
  4. Install pyWin32 220
  5. Install simplejson 2.1.1
  6. Install egenix’ pyOpenSSL 0.13.12.
  7. Install Java 1.6u31. Make sure to disable auto-updates in the system control panel and disable the browser plugins for security reasons. Java is only needed for JavaScript code compression when compiling qWebIRC and for nothing else!
  8. Install Mercurial 3.4.2.
  9. Copy qWebIRC to a target directory, copy config.py.example to config.py and configure qWebIRC to your liking by editing config.py.
  10. When done, open a cmd.exe shell, cd to your qWebIRC installation directory and run python .\compile.py (This will take a few seconds). To test it, run python .\run.py, which will launch qWebIRC on the default port 9090. You can terminate it cleanly by pressing CTRL+C twice in a row.
  11. Optional, if you want qWebIRC as a system service: Copy instsrv.exe and srvany.exe to %WINDIR%\system32\. Then run instsrv qWebIRC %WINDIR%\system32\srvany.exe. Actual service configuration is discussed below.
  12. Optional, if you want SSL, create a certificate and a private key in PEM format using OpenSSL. If you don’t know how to do that, get OpenSSL [from here] and [read this] for a quick and simple solution. Create a subfolder SSL\ in your qWebIRC installation directory and put the certificate and key files in there. When ran as a background service, the passphrase has to be removed from the key! Make sure to keep your key file safe from theft!

After that, you’ll have compiled Python byte code and compressed JavaScript code for the static part of the web frontend. If you chose to create the service stub as well, you’ll need to configure the service first, otherwise it won’t really do anything. Find the service in your registry by running regedit. It should be in HKLM\SYSTEM\CurrentControlSet\Services\, called qWebIRC.

Here:

qWebIRC service

A qWebIRC service, configured to run the XIN.at chat with SSL on port 8080.

My Windows 2000 Server is German, but I guess it’s still understandable. The values are all REG_SZ / strings. Set the following three:

  1. AppDirectory (the working directory, should be the installation dir of qWebIRC).
  2. Application (the application to be launched by the service, so python.exe).
  3. AppParameters (the parameters to be passed to Python for launching qWebIRCs’ run.py. Here, I’m specifying a port to run on, as well as SSL certificate and key files to load, so qWebIRC can automatically switch to HTTPS).

Now, go to your system control panel, create a simple, restricted user to run qWebIRC as (if you don’t have a suitable one already) and make sure that user has permissions to read & execute the qWebIRC and Python 2.5 installations. For the qWebIRC\ directory the user also needs write access. Then, go to the Administrative Tools in the system control panel and configure the service qWebIRC to run as that restricted user.

Start the service and you should be done.

Of course, you can always just run a shell and launch it interactively from the command prompt as well, which is very useful for debugging by the way.

If you click on the web chat on the top right on this page, you can try it out for yourself! :) It may not look as fancy as webchat2, but it works a lot faster and is far more stable!

Ah, you’d have to accept the self-signed certificate of course, your web browser will likely warn you about it.

And that’s that. Now visitors not only have easy access to my IRC chat server, but also one that works properly and doesn’t consume a ton of resources. ;)

Oct 312015
 

Arial Unicode MSThe last thing I wrote about font anti-aliasing was probably a little less Microsoft-related than this one, handling [Font AA in Wine on FreeBSD/Linux], but let’s still call this “round 2”. Recently, having to display symbols and asian characters in Windows Explorer a lot (yeah yeah, because of Anime), I found the font rendering quite lacking when compared to what I’m seeing on my CentOS Linux box and my FreeBSD 10 machine. While Windows’ NTFS file system supports a wide enough array of unicode code points with its UCS2-le / UTF-16 character set including [surrogates and supplementary characters], the standard font used for displaying these characters — “Tahoma” — is partly lacking.

Reason here is, that higher Unicode symbols and characters of Tahoma are not covered by Windows’ sub-pixel anti-aliasing, also known as “ClearType”. While zoomed-up screenshots don’t do sub-pixel AA any justice as the actual sub-pixel gradients can’t be visualized anymore, you’ll still see clearly what I mean. But that’s not the only problem. For certain characters the sizing and spacing are just awkward as well, worsening the situation considerably.

Now I’m not sure about Windows 8.x and 10 as I’ve removed my virtual machines, but at least Windows 7, XP and older definitely do it in an ugly way. See here:

Symbols as typically seen in Anime series' and episodes' titles as rendered by Microsoft Tahoma

Symbols as typically seen in Anime series’ and episodes’ titles as rendered by Microsoft Tahoma

And so you can see what I was complaining about when it comes to ClearType, look at this version:

Symbols as rendered by Microsoft Arial Unicode MS

Symbols as rendered by Microsoft Arial Unicode MS

Now that’s a lot better. Just what font may Arial Unicode MS be, you may ask. But let’s get to that later. First, how about some asian character comparison? Have some Katakana and some Kanji mixed in:

Katakana and Kanji characters as rendered by Microsoft Tahoma

Katakana and Kanji characters as rendered by Microsoft Tahoma

Katakana and Kanji characters as rendered by Microsoft Arial Unicode MS

Katakana and Kanji characters as rendered by Microsoft Arial Unicode MS

As said, when zooming up, ClearType or any other sub-pixel based font anti-aliasing can’t be represented properly anymore, because the brightness gradients are lost. If you’re on a CRT that might affect the native screenshots as well, to a certain degree. But still, let’s compare top-down again:

Zoomed-up symbols as rendered by Microsoft Tahoma

Zoomed-up symbols as rendered by Microsoft Tahoma

Zoomed-up symbols as rendered by Microsoft Arial Unicode MS

Zoomed-up symbols as rendered by Microsoft Arial Unicode MS

While character symmetry is still ok with Tahoma in this case (not true for all chars), you may notice that the spacing is just terrible for the musical notes, and the sizing of the notes and also the stars doesn’t quite look right either. What’s most noticeable though is the clear lack of any smoothing effect on the symbols. Let’s look at some Hiragana, Katakana and Kanji:

Hiragana, Katakana and Kanji as rendered by Microsoft Tahoma

Zoomed-up Hiragana, Katakana and Kanji as rendered by Microsoft Tahoma

Hiragana, Katakana and Kanji as rendered by Microsoft Arial Unicode MS

Zoomed-up Hiragana, Katakana and Kanji as rendered by Microsoft Arial Unicode MS

Despite the color-shifting that comes naturally with sub-pixel AA, it’s pretty clear what looks better here in my opinion. Ah yes, please note that Tahoma was actually set to 8px and Arial Unicode MS to 9px here. Despite that, we can now see that there is some loss of detail happening for the most complex Kanji characters with Arial Unicode MS. They’d still be readable, but we can’t deny that either. To do away with that you’d need bigger font sizes, preferably on a higher-dpi monitor, like say, maybe UHD instead of 1080p-1600p. Or in other words, 150-200dpi instead of 90-100dpi. Still, despite that problem on low-dpi screens, the font just looks much neater.

The main issue here is, that Tahoma is ancient. It was one of the early fonts which could cover a reasonable amount of code points from the UCS2-le character set that Windows NTs NTFS came with. TFTs and ClearType didn’t even really exist back then. And Windows doesn’t really have a lot of fonts that can render enough exotic characters to really be sufficiently multi-lingual for certain use cases, not even talking about symbols yet. If you pick a font that can’t cover the characters you’re using, all you’ll get is just a ton of empty boxes where those chars should be.

The font I used as a replacement — Microsoft Arial Unicode MS — was specifically released to tackle these issues. It’s a more modern font that covers a wide array of characters and symbols without having to surrender font anti-aliasing on XP+ either.

That font was included in certain Office versions (2000, XP, 2003) in varying versions and has also been available as a free download from Microsoft. Unfortunately, the free downloads were withdrawn, and I can’t just post the font here due to licensing issues. However, if you search for it on the web, you may find enough download sources for it. Just keep in mind, that it’s not free. If you’re on a newer version of Windows, you may have other options when it comes to choosing your font. I’m just not sure which ones… In any case, the latest version of Arial Unicode MS is 1.01, if you wanna look for it.

If you do have Arial Unicode MS, you can switch it on in your advanced appearance menu of your display properties after right-clicking on the desktop, at least on XP. The font you’ll need to change is the “Icon” one. It’s called that, because it’ll also change the font of your Desktop icons together with the file/folder pane of Windows Explorer:

Changing the icon font

Changing the icon font

After that, pick your desired Unicode-capable font and change the size to your liking, the changes will take effect immediately after confirming the change:

Our new "Icon" font is Arial Unicode MS, changed to a font size of 9px for enhanced readability

Our new “Icon” font is Arial Unicode MS, changed to a font size of 9px for enhanced readability

Of course you could also look for other, free TrueType fonts on the web which can cover enough code points to be sufficient for your needs. Maybe you can also just copy one over from some Linux or UNIX system too. It seems OpenType fonts won’t do though, the menu only allows TrueType fonts.

And that’s — amongst other things — how you make childish Anime episode titles look neat in Microsoft Windows Explorer. ;)

(It’d really be interesting if Windows 8 & 10 give you any better options out of the box…)

Sep 022015
 

colorecho logoRecently, I am doing a lot of audio/video transcoding, and for that I’m using tools like mkvextract, mkvmerge, x264, avconv and fghaacenc on the command line. Mostly this is to get rid of old files that are either very inefficiently encoded – like DivX3 or MPEG-2 – or are just too large/high-bitrate for what the content’s worth. Since I do have large batches of relatively similar files, I wrote myself a few nice loops that would walk through an entire stack of videos and process all of it automatically. Because I want to see at first glance how far through a given video stack a script is, I wanted to output some colored notifications on the shell after each major processing step, telling me the current video number. That way, I could easily see how far the job had progressed. We’re talking processes here that take multiple days, so it makes sense.

Turns out that this was harder than expected. In the old DOS days, you would just load ANSI.SYS for that, and get ANSI color codes to work with. The same codes still work in modern UNIX terminals. But not within the Windows cmd for whatever reason. Now there is an interface for that kind of stuff, but seemingly no tools to use it. Like e.g. the echo command simply can’t make use of it.

Now there is project porting ANSI codes back to the cmd, and it’s called [ANSICON]. Basically just a DLL hooked into the cmd. Turns out I can’t use that, because I’m already using [clink] to make my cmd more UNIX-like and actually usable, which also is additional code hooked into the cmd, and the two confict with each other. More information about that [here].

So how the hell can I do this then? Turns out there is a batch script that can do it, but it’s a very outlandish hack that doesn’t seem to want to play nice with my shells if I do @echo off instead of echo off as well. I guess it’s clinks work breaking stuff here, but I’m not sure. Still, here is the code, as found by [Umlüx]:

expand/collapse source code
  1. ECHO OFF
  2. SETLOCAL EnableDelayedExpansion
  3.  
  4. FOR /F "tokens=1,2 delims=#" %%A IN ('"PROMPT #$H#$E# & ECHO ON & FOR %%b IN (1) DO REM"') DO (
  5.   SET "DEL=%%A"
  6. )
  7.  
  8. ECHO say the name of the colors, dont read
  9.  
  10. CALL :ColorText 0a "blue"
  11. CALL :ColorText 0C "green"
  12. CALL :ColorText 0b "red"
  13. ECHO.
  14. CALL :ColorText 19 "yellow"
  15. CALL :ColorText 2F "black"
  16. CALL :ColorText 4e "white"
  17.  
  18. GOTO :eof
  19.  
  20. :ColorText
  21.  
  22. ECHO OFF
  23.  
  24.  "%~2"
  25. findstr /v /a:%1 /R "^$" "%~2" NUL
  26. DEL "%~2" > NUL 2>&1
  27.  
  28. GOTO :eof

So since those two solutions didn’t work for me, what else could I try?

Then I had that crazy idea: Since the interface is there, why not write a little C program (actually ended up being C++ instead) that uses it? Since I was on Linux at the time, I tried to write it there and attempt my first cross-compile for Windows using a pre-built [MinGW-w64] compiler, that luckily just played along nicely on my CentOS 6.6 Linux. You can get pre-built binaries for 32-Bit Windows targets [here], and for 64-bit Windows targets [here]. Thing is, I know zero C++, so that took some time and it’s quite crappy, but here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include "windows.h"
  3.  
  4. // Set NT commandline color:
  5. void SetColor(int value) {
  6.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  7. }
  8.  
  9. int main(int argc, char *argv[]) {
  10.   SetColor(atoi(argv[2]));
  11.   std::cout << "\r\n" << argv[1] << "\r\n";
  12.   SetColor(7);
  13.   return 0;
  14. }

Update 2016-06-22: Since I wanted to work with wide character strings in recent days (so, non-ANSI Unicode characters), my colorecho command failed, because it couldn’t handle them at all. There was no wide character support. I decided to update the program to also work with that using Windows-typical UCS-2le/UTF-16, here’s the code:

expand/collapse source code
  1. #include <iostream>
  2. #include <io.h>
  3. #include <fcntl.h>
  4. #include "windows.h"
  5.  
  6. // Set NT commandline color:
  7. void SetColor(int value) {
  8.     SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), value);
  9. }
  10.  
  11. // Unicode main
  12. int wmain(int argc, wchar_t *argv[]) {
  13.   SetColor(_wtoi(argv[2]));
  14.   _setmode(_fileno(stdout), _O_U16TEXT);      // Set stdout to UCS-2le/UTF-16
  15.   std::wcout << "\r\n" << argv[1] << "\r\n";  // Write unicode
  16.   SetColor(7);                                // Set color back to default
  17.   return 0;
  18. }

I also updated the download section and the screenshots further below. End of update.

This builds nicely using MinGW on Linux for 64-bit Windows targets. Like so: While sitting in the MinGW directory (I put my code there as echo.cpp as well, for simplicities’ sake), run: ./bin/x86_64-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe. If you want to target 32-bit Windows instead, you’d need to use the proper MinGW for that (it comes in a separate archive): ./bin/i686-w64-mingw32-g++ echo.cpp -o colorecho-x86_64.exe

When building like that, you’d also need to deploy libstdc++-6.so and libgcc_s_seh-1.dll along with the EXE files though. The DLLs can be obtained from your local MinGW installation directory, subfolder ./x86_64-w64-mingw32/lib/ for the 64-bit or ./i686-w64-mingw32/lib/ for the 32-bit compiler. If you don’t want to do that and rather rely on Microsofts own C++ redistributables, you can also compile it with Microsoft VisualStudio Express, using its pre-prepared command line. You can find that here, if you have Microsofts VisualStudio installed – version 2010 in my case:

VC2010 command lines

The command lines of Visual Studio (click to enlarge)

Here, the “Visual Studio Command Prompt” is for a 32-bit build target, and “Visual Studio x64 Win64 Command Prompt” is for building 64-bit command line programs. Choose the appropriate one, then change into a directory where you have echo.cpp and run the following command: cl /EHsc /W4 /nologo /Fecolorecho.exe echo.cpp, giving you the executable colorecho.exe.

Alternatively, you can just download my pre-compiled versions here:

Update 2016-06-22: And here are the Unicode versions:

End of update.

Note that this tool does exactly what I need it to do, but it’ll likely not do exactly what you’d need it to do. Like e.g. the line breaks it adds before and after its output. That’s actually a job for the shells echo command to do, not some command line tool. But I just don’t care. So that’s why it’s basically a piece of crap for general use. The syntax is as follows, as shown for the 64-bit VC10 build:

colorecho-x86_64-vc10.exe "I am yellow!" 14

When run, it looks like this:

Colorecho running

colorecho invoked on the Windows cmd

So the first parameter is the string to be echoed, the second one is the color number. That number is 2-digit and can affect both the foreground and the background. Here the first 16 of them, which are foreground only:

  • 0: Black
  • 1: Dark blue
  • 2: Dark green
  • 3: Dark cyan
  • 4: Dark red
  • 5: Dark purple
  • 6: Dark yellow
  • 7: Light grey
  • 8: Dark grey
  • 9: Light blue
  • 10: Light green
  • 11: Light cyan
  • 12: Light red
  • 13: Light purple
  • 14: Light yellow
  • 15: White

If you go higher than that, you’ll also start changing the background colors and you’ll get different combinations of foreground and background colorization.

The background colors actually follow the same order, black, dark blue, dark green, etc. Numbers from 0..15 are on black background, numbers from 16..31 are on a dark blue background and so on. This makes up pretty much the same list:

  • 0..15: 16 foreground colors as listed above on the default black background, and
  • 16..31: on a dark blue background
  • 32..47: on a dark green background
  • 48..63: on a dark cyan background
  • 64..79: on a dark red background
  • 80..95: ona dark purple background
  • 96..111: on a dark yellow background
  • 112..127: on a light grey background
  • 128..143: on a dark grey background
  • 144..159: on a light blue background
  • 160..175: on a light green background
  • 176..191: on a light cyan background
  • 192..207: on a light red background
  • 208..223: on a light purple background
  • 224..239: on a light yellow background
  • 240..255: on a white background

Going over 255 will simply result in an overflow causing the system to start from the beginning, so 256 is equal to 0, 257 to 1, 260 to 4 and so on.

Update 2016-06-22: With the Unicode version, you can now even do stupid stuff like this:

colorecho unicode with DejaVu Sans Mono font on Windows cmd

It could be useful for some people? Maybe?

Note that this has been rendered using the DejaVu Sans Mono font, you can get it from [here]. Also, you need to tell the Windows cmd which fonts to allow (only monospaced TTFs work), you can read how to do that [here], it’s a relatively simple registry hack. I have yet to find a font that would work and also render CJK characters though, but with the fonts I myself have available at the moment, asian stuff won’t work, you’ll just get some boxes instead:

colorecho unicode CJK fail

Only characters which have proper glyphs for all desired codepoints in the font in use can be rendered, at least on Windows XP.

Note that this is not a limitation of colorecho, it does output the characters correctly, “ですね” in this case. It’s just that DejaVu Sans Mono has no font glyphs to actually draw them on screen. If you ever come across a neat cmd-terminal-compatible font that can render most symbols and CJK, please tell me in the comments, I’d be quite happy about that! End of update.

If somebody really wants to use this (which I doubt, but hey), but wishes it to do things in a more sane way, just request it in the comments. Or, if you can, just take the code, change and recompile it by yourself. You can get Microsofts VisualStudio Express – in the meantime called [VisualStudio Community] – free of charge anyway, and MinGW is free software to begin with.

Jul 152015
 

XP x64 is dead logo[1] Update 2: It’s not looking good. At all. I (and some other users) have been playing around with Windows Storage Server 2003 systems (see the first update below), and while the OEMs are supposed to uphold support for the system, it seems they just don’t care. Two distributions have been inspected, one from Dell and one from Hewlett Packard. The latter actually does feature an updating software, that will however ignore all newer Microsoft security fixes. It just doesn’t do anything useful at all it seems.

For now, I am not even sure whether Microsoft truly ships any updates to the OEMs for rollout to customers, or whether they’ve just abandoned WSS03 completely, making their lifecycle statements regarding the operating system an empty promise, as no OEM could patch the OS itself like Microsoft can. It looks like we’re not getting anywhere with this. If you are a Windows Storage Server 2003 operator and you know more about this, or if you actually do have access to WSS03 Windows updates from after 2015-07-14, please do let me know in the comments! Thank you.

Update: While I’m not completely sure about it, it might be the case that we can disregard the text below this paragraph. A user that goes by the name of [tagg] pointed me towards a comment in the RyanVM forums, where 5eraph is releasing his XP x64 update packs. It seems that a special version of Windows Server 2003 called the “Windows Storage Server 2003” is actually supported until [October of 2016], which could mean that XP x64 could get yet another extension to its lifetime. I’m currently inspecting the operating system and methods to extract full update installers out of it to see whether it can truly be done. If yes, there might be life in the old dog yet!


The dark ages have finally arrived – for XP x64 users, that is. Yesterday, Microsofts extended support for Windows Server 2003 has ended, and with it also my [unofficial XP x64 update] project. There won’t be any more updates for Windows XP Professional x64 Edition SP2 and its on-board components from this day on. And while the regular 32-bit version with its older NT 5.1 kernel will live on until 2019 due to the [POSReady2009 hack], XP x64 will not, as there is no 64-bit embedded POSReady version of XP naturally, given that point-of-service systems like ticketing machines or airport terminals don’t need 64-bit address spaces.

So, as said on the update page, if you’re going to keep using XP x64, you’re doing so at your own risk. And you should really be knowing what you’re doing too, or you may not only put yourself at risk, but also other users on the web, should your machine become compromised. But given that XP x64 users I’ve encountered recently are also often freaks and avid Linux/UNIX users as well, I think the danger is much lower than with 32-bit XP.

Well that’s it. From the Microsoft side of things, this is pretty much where XP x64 ends.

It’s a bit sad that there is no free kernel API extension to make newer software run on the old system, like there is for Win9x with [KernelEx]. Software built with modern Microsoft development environments (Visual Studio 2013+) and in certain cases older ones will likely work on XP less and less from now on. People will no longer care so much for linking against the old platform SDKs and compiling for the v110_xp platform target.

Several free software projects (like FileZilla to name just one of many) have already ceased to support NT5.x even before the end of Server 2003. It’s even worse for commercial applications of course. Then there are others which still care to keep the platform supported, like the x265/HEVC encoder or closed software like AnyDVD HD.

But one thing’s for sure.

From this day on, it’ll only get darker and colder for those who decide to stay on this ship!

[1] Original image is © 2015 Windows 8 Wallpapers

Jun 272015
 

Corsair Logo #2This is just a minor update after [part 2], but anyway. My old workstation (the one I’m migrating away from) just broke down a few days ago, so I had to do something, and quickly. Since I still don’t have any disks for my new RAID-6, I had to pull the existing RAID array from my old box and attach it to my new workstation in a hurry. It does look quite ugly too, with the RAID lying around on the table beside an open Lian Li PC-A79B. This is not how it was supposed to be, but well… In the meantime I found out that it was my Tagan Piperock 1300W power supply which broke down (Built by Topower by the way). Sad, because I liked it for its sturdy metal screw-on modular plugs, but well. So the machine now sits in its final location, it just doesn’t look too good at the moment:

"Helios" RAID-6 array emergency migration

Now the new machine has to host the old “Helios” RAID-6 array. Not quite as planned (click to enlarge).

In any case, I wanted to play around with that new Corsair “Professional Series Platinum AX1200i” of mine, which is a fully digital power supply unit featuring an I²C port. With that, you can hook it up to Corsairs Link [Commander], or you can use the dongle provided with the unit and hook it up to an internal USB header on your mainboard. Here’s a crop of a photo previously shown, this is the dongle:

The Corsair Link dongle

The Corsair Link dongle.

Now what this actually is, is a [Silicon Labs] – or Silabs in short – I²C to [USBXPress] bridge chip. So it’s not using the same USB HID device class of the Link Commander, but a completely different protocol, which is also why we’re tied to using the Corsair Link software. The free software project [CorsairLinkPlusPlus] won’t work with it at all as it supports only the Link Commander itself.

Having said that, I can’t use the Corsair Link software – which uses .Net 4.5 – on XP x64, it just won’t work on the old OS. The drivers that come with the device though are from Silabs and DO support XP and XP x64. The USB vendor ID was changed from Silabs to Corsair though, so it’s not 10CE:1C00, but 1B1C:1C00, making it impossible to install original Silabs drivers. But that’s ok, what Corsair’s shipping with the power supply works just as well.

You may not wish to install the whole Corsair Link software on XP just to get the drivers though. So I have isolated the drivers from the package for you to install them separately. The Hydro water cooler driver is also provided, but you don’t need it if it’s just for a power supply like in my case:

But, while you can install the dongle, you can’t talk to it, lacking the userland software for that. Now when I said “how to run Corsair Link on XP x64” in the title, I have to admit I was lying a bit. Because what I did was to virtualize the dongle using Oracle VirtualBox 4.2.26 and then run the Corsair Link software on a Windows 7 x64 virtual machine. Now, before launching that, the XP x64 host systems device manager will show this:

Corsair/Silabs dongle installed on XP x64

Corsair/Silabs dongle installed on XP x64.

Just so it’s handled automatically for every boot of my Windows 7 VM, I created a USB device filter in the virtual machines’ settings:

VirtualBox Corsair Link Filter

VirtualBox USB filter for the Corsair Link dongle.

Now when you start up the VM, VirtualBox will grab the device and replace it with a device called “VirtualBox USB”, thus making it unavailable on the host machine. Just install Corsair Link in the VM, and everything will work nicely:

Corsair Link, virtualized

With the USBXPress dongle virtualized properly, we can run Corsair Link on a Windows 7 VM, controlling the host machines’ power supply (Click to enlarge).

Many have described the software as buggy and crappy, but for me it gets the job done. All I wanted was to change the behavior of the unit, disabling its passive mode at low loads. While a nice feature, the internal thermal probe reports temperatures of up to 60°C at 300W load with the fan sitting still, and I don’t quite like that. I don’t see why it is needed to artificially accelerate the aging process of the PSUs electrolytic capacitors like that, so I set the fan speed to 40%, resulting in slightly short of 800rpm. Very silent, and good enough even for high loads. I now get down to 28-35°C depending on ambient temperatures without perceiving any additional noise. It may reach 40°C on really hot days I guess, but that’s a lot better than 60°C.

Just sad that we can’t define a complete custom fan curve for this unit, based on load or temperature readings. It’s possible with system fans when working with the Link Commander, but not for this one.

Naturally, virtualization also works if you’re on Linux or BSD UNIX or Solaris or whatever. It’s cumbersome, yes, but if you need it only to tell the PSUs firmware to keep the fan spinning, it’s ok. You don’t need to keep the software running, that’s the sweet part. The settings will be stored in the power supplys’ firmware directly.

Only downside is: You need a Windows Vista/7 or newer license for that of course. But maybe we’ll see some free software in the future, people are working on it, that much’s for sure!

Now let’s hope part 4 of this log will be my new hard disks, because I’m really starting to run low on disk space already…

Edit: Part 4 should now be ready, because my new 6TB SAS drives are here. However, instead it turned out to be quite the disaster, which is why [it’s part 3½ instead]. There are some preliminary benchmarks for you to see however, at least something. ;)