Aug 232016
 

UnrealIRCd logoOne of the services I’ve been running on xin.at for years now has been the IRC server UnrealIRCd. It’s available for Linux, UNIX and also Windows, so it’s a pretty neat choice I think. A few days ago however, a user had notified me, that his client couldn’t connect when using SSL/TLS encryption after an update of the software. I’m pretty sure this was due to the OpenSSL developers disabling the SSL v3 protocol by default. So his client only had TLS and my old UnrealIRCd 3.x only had SSL v3 => handshake failure:

error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

So what now? Just shoving a newer SSL library under my IRC server wouldn’t work in a stable fashion. So far, the only software I have ever seen which can be “magically” upgraded to modern protocols and ciphers this way was the Gene6 FTP server. All the way from OpenSSL 0.9.6 to 1.0.2. No idea how they did it.

Two options: Have users recompile their libraries and clients to enable SSL v3 (yeah, as if…), or try and backport a current (=2016-07-28) UnrealIRCd 4 to my server. One that supports both modern TLS v2 with modern ciphers as well as good old SSL v3, so legacy clients may connect in an encrypted fashion as well.

Why backport? Because it’s freaking Windows 2000 (and no, newer versions do *not* work), and UnrealIRCd dropped support for that, so I absolutely needed to recompile the server and several libraries it depends on. Now that was one wild ride for a user like me, I’m telling you.

Ah yes, this isn’t exactly a good step-by-step guide or anything, so in case you just wanna grab the files, scroll all the way down! If you want to know a few of the details… I don’t even remember all the things I did, but let’s see…

Requirements:

Here’s what you need:

  1. The Microsoft [Visual C++ 2008 runtime SP1 redistributable package] (only on the system where the server is supposed to run, not on the build system)
  2. Microsoft VisualStudio 2008 (I guess 2010 also works, as long as you have the v90 toolset available)
  3. Perl. I used [Strawberry Perl 5.24].
  4. The latest UnrealIRCd [dev package]. It’s for UnrealIRCd v3.4, but that doesn’t matter.
  5. The UnrealIRCd [source code]. I used the current/bugfixed version 4.0.5 for this build.
  6. A precompiled version of pcre2 supporting Windows 2000, I only found one eligible one [here]. (I failed to recompile/relink pcre2 properly, even with the version from the dev package :( )
  7. The stock [tre 0.8.0 library] source code, because it supports VS2008. The version shipped with the dev package doesn’t.
  8. The latest [OpenSSL library] source code, it’ll serve as a replacement for the older one shipped with the dev package.

If you cannot obtain Visual Studio 2008 via any (legal!) means, that’d probably mean you’re out of luck though. Luckily, I got all versions from Microsofts MSDNAA / DreamSpark program, but if you’re stuck on something like VS2012, 2013 or 2015, I cannot help you. Maybe this can still work out, but you’ll still need the 2008 version to get the v90 toolset (I guess, not an expert here…)

Modifications:

There are quite a few, but here are the ones that I still remember:

1.) Additional headers are required to link some of the software, there are free ones available. You can grab them [here]. Put them into the VC\include\ subdirectory of your Visual Studio 2008 installation folder. On top of those two, inttypes.h and stdint.h you’ll also need unistd.h, but that one’s easy: Just make a copy of io.h in that same folder and rename that copy to unistd.h and you’re done.

2.) First, cURL-SSL was built with the nmake options ENABLE_IPV6=no and ENABLE_IDN=no set. IPv6 support on Windows 2000 does exist by using an [experimental update], but it’s function calls are different than with Microsofts’ final version, so it’s unusable by most software. Also, IDN support is only available [for Windows XP and later], so internationalized domain names using non-ASCII characters don’t work. UnrealIRCd is to be linked against this version.

3.) tre replaced with latest stock tre 0.8.0 and recompiled, UnrealIRCd is to be linked against this build.

4.) Before building OpenSSL, it may need modifications to its makefile ms\ntdll.mak, which is generated by the ms\do_nasm step described in OpenSSLs INSTALL.W32, depending on your requirements. It is here where you can enable older, weaker ciphers and the older SSL v3/v2 protocols. Enable these deprecated version only if you absolutely need them!

Look for line 21 (Note, that the ^ line breaks aren’t in the file originally, it’s all in one line. I just added them here for readability purposes):

  1. CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^
  2.  -DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE ^
  3.  -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT ^
  4.  -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM ^
  5.  -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DOPENSSL_USE_APPLINK -I. ^
  6.  -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_SSL2 -DOPENSSL_NO_KRB5 -DOPENSSL_NO_JPAKE ^
  7.  -DOPENSSL_NO_WEAK_SSL_CIPHERS -DOPENSSL_NO_STATIC_ENGINE

You could replace this with the following, allowing weak ciphers and SSL v3, but not SSL v2 for example:

  1. #CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^
  2. # -DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE ^
  3. # -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT ^
  4. # -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM ^
  5. # -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DOPENSSL_USE_APPLINK -I. ^
  6. # -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_SSL2 -DOPENSSL_NO_KRB5 -DOPENSSL_NO_JPAKE ^
  7. # -DOPENSSL_NO_WEAK_SSL_CIPHERS -DOPENSSL_NO_STATIC_ENGINE
  8. CFLAG= /MD /Ox /O2 /Ob2 -DOPENSSL_THREADS  -DDSO_WIN32 -W3 -Gs0 -GF -Gy -nologo ^
  9.  -DOPENSSL_SYSNAME_WIN32 -DWIN32_LEAN_AND_MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE ^
  10.  -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT ^
  11.  -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM ^
  12.  -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DOPENSSL_USE_APPLINK -I. ^
  13.  -DOPENSSL_NO_RC5 -DOPENSSL_NO_MD2 -DOPENSSL_NO_SSL2 -DOPENSSL_NO_KRB5 -DOPENSSL_NO_JPAKE ^
  14.  -DOPENSSL_NO_STATIC_ENGINE

Compile as shown in the documentation, and install somewhere.

5.) Before UnrealIRCd can use the new version of OpenSSL it may need modifications to match the ones patched into the OpenSSL makefile. By default, it will also block stuff like SSL v3. Enter its source tree and open ssl\ssl.c, then locate lines 245 and 321, which will look like this:

  1. SSL_CTX_set_options(ctx_server, SSL_OP_NO_SSLv3);

Just comment that out:

  1. /** SSL_CTX_set_options(ctx_server, SSL_OP_NO_SSLv3); **/

If you enabled SSLv2 as well and want the IRC server to be able to use it, do the same for lines 244 and 320, look for this…

  1. SSL_CTX_set_options(ctx_client, SSL_OP_NO_SSLv2);

…and comment it out again:

  1. /** SSL_CTX_set_options(ctx_client, SSL_OP_NO_SSLv2); **/

Now compile and link as shown in the UnrealIRCd documentation. Like the developers I’d recomment assembling a proper command line for this, as editing the makefile all the time can be cumbersome, especially if you’re running into trouble along the way.

What else?

Some of the VS project files may be preconfigured for platform toolsets you don’t have (like v100, v110, etc.) or may be set to produce a Debug build by default. Make sure you’re using only the v90 toolset and produce only Release builds. To learn how, check out the Visual Studio documentation online. It’s not that hard for the stuff you need to build with the GUI.

And here is the file:

Note that I may have done something horribly wrong along the way with this, because it really works only on Windows 2000. This is not how it should be. But launching it on a newer operating system yields something like this:

UnrealIRCd runtime error on anything greater than or equal to Windows XP

Yeah… umm… riiight…

And after pressing OK, this:

UnrealIRCd runtime error on anything greater than or equal to Windows XP #2

Whatever…

I searched for those errors on the web for a little, but couldn’t find anything that would’ve told me why it breaks like this on “modern” operating systems, yet still works on Windows 2000. Oh, the build system was XP x64 by the way. Well, it doesn’t really matter, the standard build of the developers works on XP+ anyway, and this works only on Windows 2000. Mission accomplished in any case.

In this incarnation, the server can support SSL v3 as well as TLS v1.2 protocols and supports the following ciphers:

ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA38
4:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:SRP-DSS-
AES-256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:DH-DSS-AES256-GCM-SH
A384:DHE-DSS-AES256-GCM-SHA384:DH-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA38
4:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DH-RSA-AES256-SHA256:DH-DSS-AES256
-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DH-RSA-AES256-SHA:DH-DSS-AES256-SH
A:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:DH-RSA-CAMELLIA256-SHA:DH-DSS-
CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA
-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SH
A:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA25
6:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:SRP-DSS-
AES-128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:DH-DSS-AES128-GCM-SH
A256:DHE-DSS-AES128-GCM-SHA256:DH-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA25
6:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DH-RSA-AES128-SHA256:DH-DSS-AES128
-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH-RSA-AES128-SHA:DH-DSS-AES128-SH
A:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DH-RSA-SEED-SHA:DH-DSS-SEED-SHA:DHE-RSA-CAME
LLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:DH-RSA-CAMELLIA128-SHA:DH-DSS-CAMELLIA128-SH
A:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256
:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-S
HA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:PSK-AES128-
CBC-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SH
A:RC4-SHA:RC4-MD5:PSK-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:SR
P-DSS-3DES-EDE-CBC-SHA:SRP-RSA-3DES-EDE-CBC-SHA:SRP-3DES-EDE-CBC-SHA:EDH-RSA-DES
-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DH-RSA-DES-CBC3-SHA:DH-DSS-DES-CBC3-SHA:ECDH-RSA-
DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA

The necessary tools for creating an SSL/TLS certificate and for installing a Windows service for the server are also included (openssl.exe, unrealsvc.exe).

Licensing:

UnrealIRCd and the software it was linked against in this case is released under the following licenses:

Any modifications to any of the software packages above as posted on this page are hereby licensed under the same license as the original software before modifications were applied. When downloading any unmodified source code, you’ll have to patch it yourself before building for a Windows 2000 platform target.

And what now?

Well, I guess my server supports IRC+TLS for all modern clients now, so yay! ;) URLs are the same as before: [irc+ssl://www.xin.at:6697] with SSL v3/TLS v1.2 or [irc://www.xin.at:6666] if you wish to connect without any encryption enabled, all plain text.

Jun 032016
 

H.265/HEVC logoAnd here’s another x265 build for Windows XP and Windows XP x64, following [1.9+141]. As usual, these work on modern versions of Windows just as well. Again, built with Microsoft Visual Studio 2010 SP1 and tested for correct encodes for 8-bit, 10-bit and 12-bit color depth. The 8-bit test has been done using the x86_32 version, the 10- & 12-bit tests has been done with the x86_64 version. I’m not running complicated test suits on this, just a simple encode with manual output checking.

Here is the software for 32-bit and 64-bit systems:

As usual, the builds depend on the Microsoft Visual C++ 2010 runtime which you can download from Microsoft [for 32-bit systems] and [for 64-bit systems] if you don’t have it already.

This time around, it’s a pure binary release, giving you the x265.exe and libx265.dll. I think I’m gonna keep it that way. It’s meant for users, not developers anyway.

I’m thinking I might create a project page for this, so that all releases get consolidated on a single spot, that’d probably better than creating a new post for each and every build I’m pushing out. If I’m gonna do that, links to it will be added to each post regarding information about how to build x265 for WinXP+, and also to all binary release posts.

Such a page could also give you an avconv release on the spot, so you can work with all kinds of video input to your liking, given that x265 can only accept raw YUV video by itself. Just need to build a 32-bit version of libav as well then.

Oh well, have fun! :)

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.

Apr 192016
 

H.265/HEVC logoLike I said, I’ll keep doing these. Following version [1.9+108], here comes another build of the x265 encoder for Windows XP+ and Windows XP x64/Server 2003 x64+, this time it’s version 1.9+141. I’m not sure for how long the developers at Multicoreware are going to keep up support for NT 5.1/5.2 based operating systems, but for as long as they do, I’ll keep releasing builds for the old MS operating systems. Just keep in mind that I’m not running automated build & test systems, so I’m going to release selected binaries every 1-2 months or so. If you need a specific version, please just request it (or try and build it yourself, see previous posts).

Whenever Multicoreware does drop support I’ll still continue as long as it’s easily patchable. We can’t be sure of anything though, they’ve dropped deep color support (10-bit/12-bit per color channel) on 32-bit x86 platforms before, so…

Well, here is 1.9+141:

Once more, this has been built with Microsofts’ VisualStudio 2010 SP1 + yasm 1.3.0, and tested doing a 2-pass encode & quick output video verification for all color depths. Requires the MS VC++ 2010 runtime, you can get it here: [32-bit version], [64-bit version].

Apr 082016
 

H.265/HEVC logoPreviously, I have shown you [how to compile x265 on Windows] using Microsoft Visual Studio 2010 in a way that results in binaries compatible with Windows NT 5.1/5.2, or in other words: Windows XP, XP x64 and Windows Server 2003. And while that works for most purposes, today I’d like to show you how to build an actual multilib binary, that can handle all three color bit depths supported by x265, the standardized 8- and 10-bit (MAIN and MAIN10 profiles) as well as 12-bit (MAIN12 profile). With that, it’s all in one exe instead of three. As before though, multilib x265 is only supported on 64-Bit Windows. But first, once again…

1.) Giving you the binaries

There were a lot of improvements since the last version I published back in February of course, also performance-wise. So here’s the current version from Multicoreware for both 32-bit and 64-bit Windows, compiled with MSVC 2010 SP1 and yasm 1.3.0. This requires the Microsoft Visual C++ 2010 runtime to work, see previous article:

This time around, the binaries have been tested as well! On regular 32-bit Windows XP, only fundamental binary compatibility was tested. However, all versions, so the 32-Bit one and the 64-Bit multilib ones have been ran through a 2-pass ABR encoding test with output verification for 8-bit color depth (32- & 64-bit) as well as 10- and 12-bit color depths (64-bit only) on Windows XP Professional x64 Edition using the following command line (see previous post for details):

avconv -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL^ 
 | .\x265.exe - --y4m -D 10 --fps 24000/1001 -p veryslow^
 --open-gop --ref 6 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3^
 --no-sao --qcomp 0.75 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0^
 --rdoq-level 1 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 1^
 --slow-firstpass --stats v.stats --sar 1 --range full -o pass1.h265 & avconv^
 -r 24000/1001 -i input.h264 -f yuv4mpegpipe -pix_fmt yuv420p -r 24000/1001 - 2>NUL^
 | .\x265.exe - --y4m -D 10 --fps 24000/1001 -p veryslow --open-gop --ref 6^
 --bframes 16 --b-pyramid --bitrate 2500 --rect --amp --aq-mode 3 --no-sao --qcomp 0.75^
 --no-strong-intra-smoothing --psy-rd 1.6 --psy-rdoq 5.0 --rdoq-level 1^
 --tu-inter-depth 4 --tu-intra-depth 4 --ctu 32 --max-tu-size 16 --pass 2^
 --stats v.stats --sar 1 --range full -o pass2.h265

Needless to say, they should work fine on Windows Vista/7/8/8.1/10/Server 2008/Server 2012/HS 2007/HS 2011 as well.

From time to time, I’ll release new binaries, so you might wanna check back every few months or so, if you’re interested. You can also request a build in the comments if you’re growing impatient and need a specific version more quickly because of some bugfix / feature improvement in x265.

2.) Compiling an XP/2003-compatible x265 multilib binary yourself

First, please look at the previous article I linked to in the beginning, point 2. You need the software prerequisites listed in 2a and you might still wish to read through 2b to understand some of the stuff better. You don’t need to actually run any of the commands shown there though.

Now, the multilib build is done a bit differently from the rest, as everything is scripted, so this is 100% command line work, no graphical cmake, no running the full Visual Studio IDE. Usually, with all software in place, sitting in the root directory of the x265 source tree, all you need to do is to go to build\vc10-x86_64\ and run ./multilib.bat. This won’t give us an XP/2003-compatible binary however, and the reason lies within the build script multilib.bat, here is the stock version:

expand/collapse source code (multilib.bat)
  1. @echo off
  2. if "%VS100COMNTOOLS%" == "" (
  3.   msg "%username%" "Visual Studio 10 not detected"
  4.   exit 1
  5. )
  6.  
  7. call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat"
  8.  
  9. @mkdir 12bit
  10. @mkdir 10bit
  11. @mkdir 8bit
  12.  
  13. @cd 12bit
  14. cmake -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON
  15. if exist x265.sln (
  16.   MSBuild /property:Configuration="Release" x265.sln
  17.   copy/y Release\x265-static.lib ..\8bit\x265-static-main12.lib
  18. )
  19.  
  20. @cd ..\10bit
  21. cmake -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DENABLE_SHARED=OFF -DENABLE_CLI=OFF
  22. if exist x265.sln (
  23.   MSBuild /property:Configuration="Release" x265.sln
  24.   copy/y Release\x265-static.lib ..\8bit\x265-static-main10.lib
  25. )
  26.  
  27. @cd ..\8bit
  28. if not exist x265-static-main10.lib (
  29.   msg "%username%" "10bit build failed"
  30.   exit 1
  31. )
  32. if not exist x265-static-main12.lib (
  33.   msg "%username%" "12bit build failed"
  34.   exit 1
  35. )
  36. cmake -G "Visual Studio 10 Win64" ../../../source -DEXTRA_LIB="x265-static-main10.lib;x265-static-main12.lib" -DLINKED_10BIT=ON -DLINKED_12BIT=ON
  37. if exist x265.sln (
  38.   MSBuild /property:Configuration="Release" x265.sln
  39.   :: combine static libraries (ignore warnings caused by winxp.cpp hacks)
  40.   move Release\x265-static.lib x265-static-main.lib
  41.   LIB.EXE /ignore:4006 /ignore:4221 /OUT:Release\x265-static.lib x265-static-main.lib x265-static-main10.lib x265-static-main12.lib
  42. )
  43.  
  44. pause

So I took all the options from the files generated by the original cmake when doing the normal build, and added them to the script to ensure our output binaries would be XP-compatible. This is the fixed build script:

expand/collapse source code (multilib.bat, patched for XP/2003)
  1. @echo off
  2. if "%VS100COMNTOOLS%" == "" (
  3.   msg "%username%" "Visual Studio 10 not detected"
  4.   exit 1
  5. )
  6.  
  7. call "%VS100COMNTOOLS%\..\..\VC\vcvarsall.bat"
  8.  
  9. @mkdir 12bit
  10. @mkdir 10bit
  11. @mkdir 8bit
  12.  
  13. @cd 12bit
  14. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DWINXP_SUPPORT=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF -DMAIN12=ON
  15. if exist x265.sln (
  16.   MSBuild /property:Configuration="Release" x265.sln
  17.   copy/y Release\x265-static.lib ..\8bit\x265-static-main12.lib
  18. )
  19.  
  20. @cd ..\10bit
  21. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DHIGH_BIT_DEPTH=ON -DEXPORT_C_API=OFF -DWINXP_SUPPORT=ON -DENABLE_SHARED=OFF -DENABLE_CLI=OFF
  22. if exist x265.sln (
  23.   MSBuild /property:Configuration="Release" x265.sln
  24.   copy/y Release\x265-static.lib ..\8bit\x265-static-main10.lib
  25. )
  26.  
  27. @cd ..\8bit
  28. if not exist x265-static-main10.lib (
  29.   msg "%username%" "10bit build failed"
  30.   exit 1
  31. )
  32. if not exist x265-static-main12.lib (
  33.   msg "%username%" "12bit build failed"
  34.   exit 1
  35. )
  36. cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_CONFIGURATION_TYPES="Release" -G "Visual Studio 10 Win64" ../../../source -DWINXP_SUPPORT=ON -DEXTRA_LIB="x265-static-main10.lib;x265-static-main12.lib" -DLINKED_10BIT=ON -DLINKED_12BIT=ON
  37. if exist x265.sln (
  38.   MSBuild /property:Configuration="Release" x265.sln
  39.   :: combine static libraries (ignore warnings caused by winxp.cpp hacks)
  40.   move Release\x265-static.lib x265-static-main.lib
  41.   LIB.EXE /ignore:4006 /ignore:4221 /OUT:Release\x265-static.lib x265-static-main.lib x265-static-main10.lib x265-static-main12.lib
  42. )
  43.  
  44. pause

You can just rename your original script for backup and put the fixed code in its place, build\vc10-x86_64\multilib.bat, then run it on the command line. If all the required tools are present, it will compile a 12-bit library, then a 10-bit library (both static) and finally an 8-bit binary that will have the other two libraries statically linked in. The final x265.exe can then be found in build\vc10-x86_64\8bit\Release\. To check whether it’s the real thing, look for the bitness by running .\x265.exe --version while sitting in that folder on the command line. You should see something like this:

x265 multilib binary

A x265 multilib binary shows that it’s “8-bit+10-bit+12-bit”

Per-color-channel bitness can be defined with x265s’ command line option -D. So that’d be -D 8, -D 10 or -D 12. Note that only 8- and 10-bit are part of the official Blu-Ray UHD/4k specification however.

3.) A side note

In case you’re new to this, you might not get why “8-bit” and “10-bit” etc. Aren’t color spaces supposed to be 16-bit, 24-bit, 32-bit etc.? Well, it seems that in the world of video processing, people don’t refer to whole color space bitness, but rather individual color channel bitness. So with three channels (red, green, blue for instance), you’d have 8/10/12 bits per channel, so that’s 24-, 30- and 36-bit total, or 16.7 million, 1 billion and 64 billion colors.

The more important part – and the reason why nobody encodes to 12-bit – is the internal arithmetic precision of x265 though (same applies to x264). At 8-bit color depth, arithmetic precision is also at 8-bits. When you hop over to 10-bit, you can’t use 8-bit operations and data types any longer, so everything is done at 16-bit precision. This makes the code slower, but also more efficient in preserving color gradients. Since 10-bit H.265/HEVC is officially a part of Blu-Ray UHD/4k, this would be the sweet spot, unless you’re dealing with devices too slow to play it.

Going to 12-bit won’t boost the precision further, it just gives you more colors, that most of today’s displays won’t be able to show anyway. Not much benefit.

So that’s that.

Have fun! :)

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.

Feb 122016
 

H.265/HEVC logo1.) Giving you the binaries:

Just recently I tried to give the x265 H.265/HEVC video encoder another chance to prove itself, because so far I’ve been using x264, so H.264/AVC. x264 does a really good job, but given that the marketing guys are talking about colossal efficiency/quality gains with H.265, I thought I’d put that to the test once again. It was quite easy to compile the software on my CentOS 6.7 Linux, but my old XP x64 machine proved to be a bit tricky.

But, after a bit of trial and error and getting used to the build toolchain, I managed to compile a seemingly stable version from the latest snapshot:

x265 cli, showing the version info

x265 cli, showing its version info.

Update 2: And here comes my first attempt to build an x86_64 multilib binary, that can encode H.265 at 8-bit, 10-bit and 12-bit per pixel color depths. You may wish to use this if you need more flexibility (like using 12-bit for PC only and 8- or 10-bit for a broader array of target systems like TVs, cellphones etc.). It’s currently still being tested for a short encoding run. You can specify the desired color depth with the parameter -D, like -D 8, -D 10 or -D 12:

Update: And here are the newer 1.9 versions, built from source code directly from the [MulticoreWare] (=the developers) servers. I haven’t tested these yet, but given that I configured & compiled them in the same way as before, they “should work™”:

And the old 1.7 versions from the Videolan servers:

So this has been built with MSVC10 and yasm 1.3.0 on Windows XP Pro x64 SP2, meaning this needs the Microsoft Visual C++ 2010 runtime. You can get it from Microsoft if you don’t have it yet: [32-bit version], [64-bit version]. v1.7 tested for basic functionality on XP 32-bit and tested for encoding on XP x64. The 32-bit version only supports 8-bit per pixel, which is default for x264 as well. The 64-bit versions support either 8-, 10- or 12 bits per pixel. Typically, higher internal precision per pixel results in finer gradients and less banding. 8-/10-bit is default for H.265/HEVC. 12-bit will likely not be supported by any hardware players, just as it was with 10-bit H.264/AVC before.

You may or may not know it, but as of now, x265 cannot be linked against either ffmpeg or libav, so it can only read raw input. To “feed” it properly you need either a frame server like [AviSynth] in combination with the pipe tool [Avs4x265], or a decoder that can pipe raw YUV to x265. I went for the latter version, because I already have libav+fdkaac compiled for Windows to get the avconv.exe binary. It’s quite similar to ffmpeg.

This I can only provide as a 64-bit binary, as I’m not going to build it for 32-bit Windows anytime soon I guess, so here you go:

This was compiled with GCC 4.9.2 and yasm 1.3.0 on CygWin x64. To use the two together, add the locations of your EXE files (avconv.exe and x265.exe) to your search path. Then, you can feed arbitrary video (VC1, H.264/AVC, MPEG-2, whatever) to x265. An example for a raw H.264/AVC input stream using the 64-bit versions of the software:

avconv -r 24000/1001 -i video-input.h264 -f yuv4mpegpipe -pix_fmt yuv420p - 2>NUL |^
 x265 - --wpp --y4m -D 12 -p slower --bframes 16 --bitrate 2000 --crf 18 --sar 1^
 --range full -o video-output.h265

Or another, reading some video stream from an MKV container, disabling audio and subtitles:

avconv -r 24000/1001 -i video-input.mkv -f yuv4mpegpipe -pix_fmt yuv420p -an -sn^
 - 2>NUL | x265 - --wpp --y4m -D 12 -p slower --bframes 16 --bitrate 2000 --crf 18^
 --sar 1 --range full -o video-output.h265

Just remove the carets and line breaks to make single-line commands out of those if preferred. To understand all the options, make yourself some readmes like this: avconv -h full > avconv-readme.txt and x265 --log-level full --help > x265-readme.txt or read the documentation online.

2.) How to compile by yourself:

2a.) Prerequisites:

I won’t describe how to build libav here, but just the x265 part. First of all, you need some software to be able to do this, some of it free, other not so much (unless you can swap MSVC with MSYS, I didn’t try that):

  • [cmake] (I used version 2.8.12 because that’s roughly what I have on Linux.)
  • [Mercurial] (Needed to fetch the latest version from x265′ versioning system. I used the latest Inno Setup installer.)
  • [yasm] (Put yasm.exe in your search path and you’re fine. This is optional, but you really want this for speed reasons.)
  • [Microsoft Visual Studio] (Use 2010 if you’re on Windows XP. Supported versions: 2008/VC9, 2010/VC10, 2012/VC11 & 2013/VC12)
  • [x265 source code] (Enter a target download path and use Mercurials hg.exe like hg clone https://bitbucket.org/multicoreware/x265 to fetch it)

2b.) Preparation of the solution:

Usually, you would use cmake to have it compile your entire project, but in this case it’ll build Visual Studio project files and a solution file for us. To do this, enter the proper build path. In my case I’m using Visual Studio 2010, so VC10, and I’d like to build the 64-bit version, so with the unpacked x265 source, I’d enter its subdirectory build\vc10-x86_64\ and then run the generation script: .\make-solutions.bat:

make-solutions.bat preparing cmake for us

make-solutions.bat is preparing cmake for us.

There are several things you need to make sure here: First, if you’re on Windows XP or Windows Vista, you need to toggle the WINXP_SUPPORT flag. Also, if you’re compiling for a 64-bit target, you may wish to enable HIGH_BIT_DEPTH as well to get to either 10-bit or even 12-bit per pixel other than just 8. The 32-bit code doesn’t seem to support high bit dephts right now.

Then there is one more important thing; With CMAKE_CONFIGURATION_TYPES set to Debug;Release;MinSizeRel;RelWithDebInfo, my build was unstable, throwing errors during encoding, like x265 [error]: scaleChromaDist wrap detected dist: -2029735662 lambda: 256. Setting CMAKE_CONFIGURATION_TYPES to just Release solved that problem! So you may wish to do the same.

Make sure ENABLE_CLI and ENABLE_ASSEMBLY are checked as well, then click Configure. If you’re building with high bit depth support, you’ll be presented with another option, MAIN12. You should enable this to get Main12 profile support in case you’re planning to build a 12-bit encoder. If you don’t pick it, you’ll get a 10-bit version instead, staying within Blu-Ray 4K specifications. After that, click Configure again. Generally, you need to click Configure unless no more red stuff pops up, then click Generate.

2c.) Compiling and installing the solution:

Load the resulting solution file x265.sln into Microsoft Visual Studio, then right click ALL_BUILD and pick Build. This will compile x265. If you want to install it from the IDE as well, right click INSTALL and select Build. This will install x265 in %PROGRAMFILES%\x265\ with the binary sitting in %PROGRAMFILES%\x265\bin\:

Microsoft Visual Studio 2010, ready to compile the x265 solution generated by cmake

Microsoft Visual Studio 2010 with the x265 solution generated by cmake loaded, compiled and installed.

3.) Running it:

Now we can feed x265 some raw YUV files like this, after adding x265.exe to the search path:

x265 encoding a raw YUV file to H.265/HEVC

x265 encoding a raw YUV 4:2:0 file to H.265/HEVC (The options given to x265 may actually suck, I’m still in the learning process).

Or we can use a decoder to feed it arbitrary video formats, even from MKV containers, like shown in the beginning. ffmpeg or avconv can decode pretty much anything, and then pipe it into x265 as raw YUV 4:2:0:

x265 being fed a H.264/AVC bitstream by avconv

x265 being fed a H.264/AVC bitstream by avconv.

And that’s it! Now all I need is some 18-core beast processor to handle the extreme slowness of the encoder at such crazy settings. When going almost all-out, it’s easily 10 times as slow as x264 (at equally insane settings)! Or maybe I can get access to some rack server with tons of cores or something… :roll:

Update: All x265 releases have now been consolidated on [this page]! All future XP- and XP x64-compatible releases of x265 plus a relatively recent version of avconv to act as a decoder for feeding x265 with any kind of input video streams will be posted there as well.

Jan 272016
 

HakuNeko logoJust yesterday I’ve showed you how to modify and compile the [HakuNeko] Manga Ripper so it can work on FreeBSD 10 UNIX, [see here] for some background info. I also mentioned that I couldn’t get it to build on CentOS 6 Linux, something I chose to investigate today. After flying blind for a while trying to fix include paths and other things, I finally got to the core of the problem, which lies in HakuNekos’ src/CurlRequest.cpp, where the code calls CURLOPT_ACCEPT_ENCODING from cURLs typecheck-gcc.h. This is critical stuff, as [cURL] is the software library needed for actually connecting to websites and downloading the image files of Manga/Comics.

It turns out that CURLOPT_ACCEPT_ENCODING wasn’t always called that. With cURL version 7.21.6, it was renamed to that from the former CURLOPT_ENCODING as you can read [here]. And well, CentOS 6 ships with cURL 7.19.7…

When running $ ./configure && make from the unpacked HakuNeko source tree without fixing anything, you’ll run into this problem:

g++ -c -Wall -O2 -I/usr/lib64/wx/include/gtk2-unicode-release-2.8 -I/usr/include/wx-2.8 -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES -D__WXGTK__ -pthread -o obj/CurlRequest.cpp.o src/CurlRequest.cpp
src/CurlRequest.cpp: In member function ‘void CurlRequest::SetCompression(wxString)’:
src/CurlRequest.cpp:122: error: ‘CURLOPT_ACCEPT_ENCODING’ was not declared in this scope
make: *** [obj/CurlRequest.cpp.o] Error 1

So you’ll have to fix the call in src/CurlRequest.cpp! Look for this part:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Change CURLOPT_ACCEPT_ENCODING to CURLOPT_ENCODING. The rest can stay the same, as the name is all that has really changed here. It’s functionally identical as far as I can tell. So it should look like this:

  1. void CurlRequest::SetCompression(wxString Compression)
  2. {
  3.     if(curl)
  4.     {
  5.         curl_easy_setopt(curl, CURLOPT_ENCODING, (const char*)Compression.mb_str(wxConvUTF8));
  6.         //curl_easy_setopt(curl, CURLOPT_ACCEPT_ENCODING, (const char*)memcpy(new wxByte[Compression.Len()], Compression.mb_str(wxConvUTF8).data(), Compression.Len()));
  7.     }
  8. }

Save the file, go back to the main source tree and you can do:

  • $ ./configure && make
  • # make install

And done! Works like a charm:

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux

HakuNeko fetching Haiyore! Nyaruko-san on CentOS 6.7 Linux!

And now, for your convenience I fixed up the Makefile and rpm/SPECS/specfile.spec a little bit to build proper rpm packages as well. I can provide them for CentOS 6.x Linux in both 32-bit as well as 64-bit x86 flavors:

You need to unzip these first, because I was too lazy to allow the rpm file type in my blogging software.

The naked rpms have also been submitted to the HakuNeko developers as a comment to their [More Linux Packages] support ticket which you’re supposed to use for that purpose, so you can get them from there as well. Not sure if the developers will add the files to the projects’ official downloads.

This build of HakuNeko has been linked against the wxWidgets 2.8.12 GUI libraries, which come from the official CentOS 6.7 package repositories. So you’ll need to install wxGTK to be able to use the white kitty:

  • # yum install wxGTK

After that you can install the .rpm package of your choice. For a 64-bit system for instance, enter the folder where the hakuneko_1.3.12_el6_x86_64.rpm file is, run # yum localinstall ./hakuneko_1.3.12_el6_x86_64.rpm and confirm the installation.

Now it’s time to have fun using HakoNeko on your Enterprise Linux system! Totally what RedHat intended you to use it for! ;) :roll: