Aug 282016
 

KERNEL_DATA_INPAGE_ERROR logoHere is how a responsible system administrator should handle downtimes and replacements of faulty hardware: Give advance notice to all users and make sure to give everybody enough time to prepare for services going offline, if possible. Specify a precise time window which is as convenient as possible for most users. Also, explain the exact technical reasons in words as simple as possible.

How I handled the replacement of XINs’ system hard disk? See that nice blue logo on the top left side? KERNEL_DATA_INPAGE_ERROR, bugcheck code 0x0000007a. And [it isn’t the first of its kind either], last one was a KERNEL_STACK_INPAGE_ERROR, clearly disk related given that the disk had logged controller errors as well as unrecoverable dead sectors. And NO, that one wasn’t the first one too. :roll: So yeah, I rebooted the [monster], and decided that it’s too much of a pain in the ass to fix it and hoped (=told myself while in denial) that it would just live on happily ever after! Clearly in ignorance of the obvious problem, just so I could walk over to my workstation and continue to watch some Anime and have a few cold ones in peace…

So, my apologies for being lazy in a slightly dangerous way this time. Well, it’s not like there aren’t any system backups or anything, but still. In the end, it caused an unannounced and unplanned downtime 3½ hours long. This still shouldn’t hurt XINs’ >=99% yearly availability, but it clearly wasn’t the right way to deal with it either…

Well, it’s fixed now, because this time I got a bit nervous and pissed off as well. Thanks to [Umlüx], the XIN server is now running a factory-new HP/Compaq 15000rpm 68p LVD/SE SCSI drive, essentially a Seagate Cheetah 15k.3. As I am writing this the drive has only 2.9h of power on time accumulated. Pretty nice to find such pristine hardware!

Thanks do however also fly out to [Grindhavoc]German flag and [lommodore]German flag from [Voodooalert]German flag, who also kindly provided a few drives, of which some were quite usable. They’re in store now, for when the current HP drive starts behaving badly.

Now, let’s hope it was just the disk and no Controller / cabling problem on top of that, but it looks like this should be it for now. One less thing to worry about as well. ;)

May 302013
 

Hack logoTonight at around 10:30PM, XIN.at came under attack by what I assume to be a larger bot network. As a result, all CPU-intensive web applications on my server, including this weblog, forums, CMS’ etc. went offline because of the attacks causing excessive PHP CPU cycle usage and timeouts of all scripts. It seems the primary target of the attacks was this weblogs login page, but the largely inactive forum was also a target obviously. Log file analysis suggests, that this was a distributed brute force attack to break into XIN.at web services (and probably a few thousand other web services on other servers). The attacking hosts were many machines spread across multiple IP pools, so I have to assume that this is some major attack to break into multiple web services at once, for purposes currently not known to me.

Amusingly, the attack failed on this server while remaining in its infancy. As a security measure, runtime of PHP scripts is limited on this server. The attack was so massively parallelized, that it simply overloaded everything, causing all scripts to return HTTP 503 errors to almost all attacking hosts. In laymans terms: This server was simply too SLOW to allow the attackers scripts to ever succeed. I can only laugh about that simple fact. This server was just too old and crappy for this shit. ;)

Well, still, for that reason, this weblog and most of XIN.at’s web services went offline for around 2½ hours. For now, it is back online, and hopefully the brute force attacks have ceased by now, as my monitoring would suggest. We’ll see.

May 292012
 

The deal has been made: 8/8Mbit/s are coming for XIN.at! That is, if there are still enough telephone lines available in the hub of this building. There might or might not be enough. Thing is, for symmetric G.SHDSL I need one line per 2Mbits/s, so that means I need another two for a total of four to reach 8/8. On top of that, the provider may instruct the local Telekom to add a fifth line in case bandwith scaling is not good enough. It will definitely take another few weeks for everything to be done though. First, a technician of the Telekom (or A1, as they’re now called here) will come to see if there are enough lines. If there are, the guy will patch them through into my flat.

When that’s done, another appointment will be made with the company KAPSCH for them to send yet another technician. Those guys are doing the actual on-site work for my provider, so they will connect my new four-port G.SHDSL extender and take the old one with them. Not that I couldn’t do that myself, but the on-site service was included or so the salesperson told me, so I HAD to take it. Meh, but ok. I suppose if everything goes as planned, I might have 8Mbit/s available for all XIN.at users including myself within the next three weeks. I’ll keep you updated on the progress.

Update #1: I have just been called by the A1 guys and the date for the line installation has been fixed to 2012-06-08. I will have to move that to another date in case I go to the Novarock festival (which is not sure yet), but basically that should be it. The guy also told me, that the line is supposed to be put online on 2012-06-13, which doesn’t leave too much time for the UPC technician to come and do the on-site installation, but I guess it should be ok. Maybe they do it exactly on the 13th? We’ll see. I suppose UPC will call me in the coming days, or send me an eMail with information about their exact procedure.

Update #2:  Another call, this time from Kapsch. Those guys are doing the on-site installations for UPC, so they will bring me the new G.SHDSL extender/router. I have fixed an appointment on monday, 09:00am, which is absolutely optimal, since I have taken both monday and tuesday off, additionally to friday. Novarock has now fallen through since my collegue can’t come. No fun alone. So I’ll focus on the installation now, which means all the appointments are finally fixed.

Update #3:  A1 has sent a young (though surprisingly competent) girl to install the two new lines. She allowed me to peek into the local distribution hub, revealing 10 free line connections. Since a dead one from my flat has been re-used, only one of those has been used up, meaning there are 9 free, which is quite a lot. That’s relieving, as it means that even a future upgrade to 8 lines is totally possible. But for now, a total of 4 lines / plugs has been successfully installed, now I have to wait for monday for the final installation of the new G.SHDSL extender. Could be a Cisco instead of Paradyne this time. We’ll see.

Update #4:  The Kapsch technician has now been here to bring up the new line. He told me that the link speed of the new Zhone 2040 extender (Zhone = former Paradyne, so it’s the same manufacturer as before) has to be set to 100mbit full duplex. That made some sense, als the older Paradyne 2020 had to be set to 10mbit full duplex to avoid weird connection problems. Also, the network card hooked up to the extender had to be set to that speed. Only that this time the technician was WRONG. The correct network interface card link speed setting for a Zhone 2040 in factory configuration is auto-negotiated. If I set it statically, the entire line becomes unstable. Dammit. Now that that’s done, there is another problem. Seemingly, the two old lines were not hooked up to the same DSLAM in the central hub. For the entire line to synchronize, all four links have to be hooked up to the same DSLAM in the central hub. Perfect. Now I have to wait for another week or so before i really get those 8/8mbits. Fuck it, how much do I love waiting games, eh? At least one good thing: The new 2040 extender box gives me considerably better upstream speeds for some strange reason. We’ll see how that goes when the missing two links are being brought online. In the meantime, here are a few pictures for you to look at:

Update #5:  Well guess what, today at 02:00pm – 04:00pm, A1 alias the Austrian Telekom was supposed to hook up the missing two lines in the central hub. Guess what they DIDN’T manage to do? Yeah, right. Plugging two fucking cables vom A to B, seems too hard for them! Best thing is, UPC never even received any feedback about what happened or why something didn’t work out as planned. I guess they never even WENT there. Such a simple task, yet they fail. God dammit. UPC support will try to find out what happened tomorrow morning. I’m slowly feeling the anger piling up here. Let’s see what they have to say tomorrow…

Update #6:  Wow. I don’t know what they’re thinking, but it seems that A1 suspects the local cabling to be the culprit. Let’s just say I don’t believe that, but they want to re-do the entire cabling now, replacing the cable conduits with screened ones. I doubt that’ll do anything, but I won’t let them slip away before the line is working this time, that much is for sure. The next appointment for another round of cable installations is next monday. Had to take another day off for that. :( Damn you, A1!

Update #7:

Zhone SNE2040G with all four lines upGuess what, A1 was wrong. The cabling is perfectly fine, so their assumption that the screening was to be blamed was incorrect. But the actual mistake wasn’t made by A1 still. It was UPC and/or the Kapsch technician who had the old two lines deactivated, but not re-activated after everything was hooked up. Beautiful. Now finally they did what was necessary and configured their DSLAM properly, and all of a sudden, all 4 bonded lines are up and running! Finally! :)

Feb 152012
 

Internet LogoMy Internet provider UPC just informed me, that there will be some undefined maintainance work going on on the 29th of February, that’s exactly in two weeks from now. According to the announcement, the downtime might range from midnight to 06:00 AM. That means, that all XIN.at services including this website will be down between 00:00 AM – 06:00 AM on 2012-02-29. I hope we can expect everything to just work as expected after the maintainance is done. But you never know with Internet providers, not even when running a business-class line, so better be prepared for anything.

Services that will be down include, but are not limited to the XIN.at web server, mail server, FTP server, IRC server and other minor services like time serving etc. Since the work is being done at night, chances are that only very few people will even notice this. Let’s hope that it goes that way!