if it's owən » Blog

Building a NAS: Stage 2 (Software and Maintenance)

Owen — Sat, 24 Apr 2010 23:42:58 +0000

With the system up and running, it made sense for me to make an effort to see it stayed that way. I needed to be able to have the following things happening without any intervention on my behalf:

Users’ Windows PCs backing up automatically to their respective folder on the network
These backups being automatically rsync’d with an external disk, for a true backup
Reports on the status of the machine being made available to me at regular intervals

The first is turning out to be the most difficult. Initially I wanted to write a batch script that handled this for me and then configure it to run once a week using Task Scheduler in Windows. However, there are three different versions of Windows for me to contend with out of my five users, (XP, Vista, 7), so it wasn’t as straightforward as I thought it would be, with Vista and 7 including “Robocopy” out of the box, and XP not. However, we all know the most straightforward solution isn’t always the best one. Even with Robocopy, you cannot do incremental/differential file copies. You can coarsely say “all files changed after some date”, but it still copies everything in full.

Enter FTPSync, a nice little FTP tool which has a built-in differencing algorithm and allows full, incremental and differential backups. Its configuration is somewhat obscure, but nothing too difficult to understand with the help of their online documentation (which as it always happens, was actually down the first time I wanted to access it). After some initial tests on a small dataset, I think it will be most suitable for my purposes, for two reasons: (a) incremental/differential backups, saving time, and (b) it does this over FTP, which is the most CPU-efficient protocol for my underpowered file server.

Setting up the user home directories was pretty easy: just enable the [homes] share in your Samba configuration, and you’re away. Mine are actually redirected via symlinks to another disk, so you also have to add “symlinks = yes” to the [global] section of your config, or your home directories won’t appear as intended. The rsync configuration was also pretty straightforward, but I’ll include my backup script here for your reference:

#!/bin/sh
# 200GB is 209715200 bytes
# rsync.pwd file must be owned by root as script is executed from roots crontab
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/home/owen/ssh-key/owen-rsync-key
RUSER=owen
RHOST=192.168.1.50
RPATH=/home/owen/backup/fileserver-backups/owen/
LPATH=/var/data/owen
# $RSYNC -az -e "$SSH -p 768 -i $KEY" $RUSER@$RHOST:$RPATH $LPATH
# rsync -az /var/data/owen/ owen@boxenmkiv::owen –password-file /home/owen/rsync.pwd
homeSize=`du -sD $LPATH | cut -f 1`
emailMSG=""
MAILSUBJ="Backup summary for $LPATH on `date`"
RECIPIENT="owen"
if [ $homeSize -gt 209715200 ]; then
emailMSG="User $RUSER rsync backup failed.\nReason: $LPATH size is ~ $(($homeSize/1024/1024)) Gb"
else
emailMSG="$LPATH is ~ $(($homeSize/1024/1024)) Gb, proceeding with backup at `date`.\n"
# $RSYNC -az -e "$SSH -p 768 -i $KEY" $RUSER@$RHOST:$RPATH $LPATH
rsync -az /var/data/owen/ owen@boxenmkiv::owen –password-file /home/owen/rsync.pwd
emailMSG=$emailMSG"$LPATH backup completed at `date`."
fi
echo $emailMSG | nail -r "my@email.net" -s "Owen’s Backup Result for `date`" my@email.net

There’s actually a lot of junk at the start of the script which I haven’t cleaned out yet (since changing from an rsync over SSH approach), but the important part, the part that does all the work, is the rsync call on line 28. The simple if construction checks the size of the data to be backed up, and then proceeds if below 200Gb, otherwise does nothing. The result of the script is e-mailed to me when it finishes. Also, I have created a little script which runs every now and then to collect the temperature of the hard disks, a df output or two, and some other info, which is also e-mailed to me periodically, just to keep an eye on things.

There is a version of the above script for each of the five users on my system, and they are all executed using roots cron daemon. As noted in the script, the rsync.pwd file must be owned by root for this to work.

Building a NAS: Stage 1 (Rationale and Hardware)

Owen — Sun, 18 Apr 2010 04:33:36 +0000

Currently I live in a share house with four other people. It’s a very internet-connected sharehouse; three of us are heavily into gaming, all of us are heavily into internet, three of us are students, and all of us have precious data which is important that we’d rather not lose.

Enter me, and my extensive ability to play with neat toys. Surely there’s something I can do, or build, or make, that can benefit everyone? What do you know: there is! I picked up an old file server system from my parents’ place which was serving as a back up system for my dads business. It had 2x 500Gb Western Digital hard disks running in a RAID1 (mirror) array. The system ran fine for about three years and was based on the following, which I had aquired from eBay and the local computer store some years prior:

Intel Celeron 1000Mhz (Tualatin)
2x256Mb PC100 SDRAM
Intel i815-based mainboard
Sil3114 4-port PCI SATA150 card
Realtek-based PCI network card

It ran Arch Linux (a mistake, but once it was setup, it never needed administering, so not too bad I guess), and worked without issue for several years. Come a few months ago, and it was no longer needed, so I reclaimed it. Aside from one of the disks having ~1700 bad sectors (but thankfully still in warranty), the rest of the system was fine, if a little dusty.

Of course my operating system of choice was Linux. Two main reasons: (1) it doesn’t cost anything, and (2) (and this is the real reason) should my hardware die, (which is more probable than it not dying, on account of the fact that it is coming up on ten years old) with a Linux-based system I can literally plug the hard disks into a totally different CPU, motherboard & RAM and have it back up and running in no time at all. In the event of a mainboard, CPU or memory failure, the system would not need to be rebuilt. Linux’s ability to cope with massive hardware changes is perfect for my untrustworthy hardware.

In the meantime though I thought to myself that the old Celeron could be used once more: the PCI SATA card still had two free ports, and PCI gigabit network cards are cheap. So, I bought two Western Digital “Green Power” 1Tb drives and a Realtek-based PCI gigabit NIC and away I went with a Ubuntu 9.10 Server install. After dealing with a small issue of the operating system not booting because the version of GRUB it ships with had some strange bug whereby it couldn’t find PATA hard disks, I had it up and running.

Some figures: the maximum write speed I can put to the system over network is a fairly pedestrian 25Mb/sec; that’s over FTP on a “Gigabit” network (I say gigabit in quotes because I went cheap on the switch; using iperf to benchmark it yields a maximum throughput of about 370Mbit). The CPU usage is 100%. scp-ing stuff over the same link maxes out at about 9Mb/sec, and rsyncing maxes out at about 2.5Mb/sec, all three figures because the CPU is maxed out. The only good figure I see out of the system is its power draw: using an inline measurement tool I picked up from Jaycar, the system pulls a max. of about 44 watts. That’s with:

Intel Celeron 1000Mhz (Tualatin)
2x256Mb PC100 SDRAM
Intel i815-based mainboard
Sil3114 4-port SATA150 card
Realtek RTL-8169 gigabit NIC
Western Digital 200Gb WDC WD2000JB-00GVC0 (boot disk)
Western Digital “Green Power” 1Tb WDC WD10EADS-65M2B0
Western Digital “Green Power” 1Tb WDC WD10EADS-00P8B0
Western Digital “Blue” 640Gb WDC WD6400AAKS-00A7B0
Western Digital 500Gb WDC WD5000AAKS-00YGA0
2x 120mm fans, 1x80mm CPU fan, 1x80mm PSU fan

For general file server usage it is fine. I can stream 4x 720p movies from the system without any trouble. Copying large datasets to it over Samba or FTP isn’t too bad. However, the main purpose of the system is to be a backup of important documents etc with each person in the household having up to 200Gb available. At present, the rsync of a ~180Gb dataset has taken approximately 15 hours; and it is still running on account of that heavily CPU-limited 2.5Mb/sec that rsync manages. This is however on the first pass, so I wonder if the performance will be so terrible on the second pass?

The puzzle I have is thus: with all the traffic going over the PCI bus, which maxes out at 133Mb/sec, is that 25Mb/sec write speed I see over FTP on account of the CPU being maxed out from calculating the data flow (ie, the data itself), or is it maxing out calculating the PCI subsystem (ie, co-ordinating the data)? Theoretically I should see at least 66Mb/sec (PCI bus total speed divided by two); but I have measured the average write performance of one of the “Green Power” disks to be around 38Mb/sec. So, it won’t reach the theoretical maximum of the PCI bus when writing, however that 25Mb/sec still falls short. At least I have some headroom to maximise the performance of my disk subsystem with faster buses.

There are fortunately a lot of options becoming available which I can use to my advantage; specifically, a lot of Atom-based mini-ITX boards with low power computing in mind. Unfortunately many of them are not really designed for low-power NAS duties with most only having two SATA ports and no expension options. There are a few however that are going ‘all-out’ on the features department, but many are not available in Australia yet. The Zotac boards, Supermicro boards, and the Gigabyte boards have 4+ SATA ports, with the Zotac being my favourite as it also has a single PCIe x16 slot – with the addition of a simple graphics card, the system could go from NAS to full-fledged HD-capable HTPC. While I don’t currently have that purpose in mind, there’s no reason why it couldn’t happen in the future.

But, would this really improve the performance of my server? I am going to assume that it would. The NM10 chipset has modern data buses to support it: DDR memory, PCIe, and USB2.0 to name a few. What of the CPU, though? The D510 is a dual-core, low power solution running at 1.6Ghz. But, the Atom achieves its low power through some trickery: it’s actually a RISC CPU with a CISC wrapper around it. There was some worry when they were first released that their performance with unoptimized software would be terrible.

PassMark CPU chart for the Atom D510, and PassMark CPU chart for an Intel Celeron 1000Mhz (scroll waaay down) – no competition, right? Let’s hope so. With this, and anecdotal evidence collected from forums stating that an Atom-based NAS can achieve 1Gbps+ (with dual NICs in a team), I shouldn’t expect to see a drop in performance.

At present, my favourite local retailer has “preorder” status on the Gigabyte GA-D510UD for around $110. I can’t find Zotac solutions being offered for sale anywhere in Australia. I will have to keep an eye on the performance of the system to see if upgrading would be worthwhile.

Apple sucks.

Owen — Fri, 09 Apr 2010 14:34:59 +0000

Haha. My favourite:

Thing 2.0

The tech world is on the verge of frenzy at the release on Tuesday of Thing 2.0 by market leader StuffCorp.

“It completely revolutionises the nature of stuff. Thing 2.0 is basically like nothing we have ever seen before. The game has literally changed.” said StuffCorp CEO Chris Warner.

And it seems like the public agrees, with over 2,000 people lining up outside of StuffCorp’s Silicon Valley superstore in order to be one of the first to get their hands on Thing 2.0.

IT aficionado Sam Doyle, who we spoke to after having lined up for 14 hours to get his hands on Thing 2.0, was adamant that it was worth the wait.

“Thing 2.0 is just in a league of it’s own. It does stuff twice as fast as its closest competitor, up to 12 trillion things per second, and that’s just unrivalled in the current market. I love it.”

But, behind all the hype, what does Thing 2.0 really offer? Well, the main change to Thing 2.0 that differentiates it from the original Thing is that it is 33% smaller, 20% lighter and can do literally ten times more shit than the original thing could do.

The improved performance of the Thing 2.0 essentially comes down to the BuzzWordDrive, which features 3 blue LED lights and an aerodynamic design, allowing it to perform the vast array of functions that the original Thing could only dream of.

170 million units of the Thing 2.0 have already been preordered, merely confirming how, in today’s fast paced world, it is simply irresponsible to get left behind.

Thing 2.0’s tagline “Do more shit per second, today.”, has clearly resonated with a public that has more to do, and less free time, than ever before.

But will the public be satisfied? This reporter thinks so, but with rumours of Thing 3.0 already in the works, public expectation is on the rise.

I can’t buy the netbook I want.

Owen — Tue, 16 Mar 2010 03:43:40 +0000

I want a netbook that has:

1366×768 resolution or more (~11.6″ screen)
Competent graphics (no Intel GMA500; it isn’t Linux compatible)
Upgradeable memory (3Gb at least, please)
Good battery life (3 hours minimum with WiFi)
Preferrably with a built-in 3G modem
As already mentioned, Linux friendly
Sub $1000 AUD

It seems as this kind of a netbook simply doesn’t exist in Australia at the moment. A prime candidate would be the HP Mini 311; however, it’s not available in Australia. This laptop has received great reviews and meets all my requirements.

There are similar models available in Australia, but they all seem to miss one or another of my requirements. Why is it so hard to buy a high-end Atom netbook in this country? Christ, the platform has been out for at least 18 months, if not longer!

Why do I want all this? Well, some use-case scenarios:

At home on the couch: I can fire it up and be browsing the ‘net in seconds. By being ultra low-power, I rarely have to worry about where the power cord is. A dual-core Atom and nVidia ION graphics will ensure I can handle online 720p/1080p YouTube streaming. While working on documents or doing whatever.
At parties: connected to a home theatre system via HDMI, it’s a portable media centre for use at parties. I have a certain fondness for ProjectM which may be slightly unhealthy.
At work: in a data centre, despite being surrounded by computers and networks, you don’t always have easy access to power or internet. With at least 3Gb of RAM, I could create a RAM disk to store my working files on temporarily and shut down the hard disk completely, putting the netbook into a kind of ultra frugal battery mode. The built-in 3G modem means I don’t have to worry about dongles hanging off it while I work in confined spaces.

I can’t buy from the US because I’ll have nil support from HP warranty. That sucks.

My laptop is dying

Owen — Sat, 13 Mar 2010 07:33:52 +0000

Unfortunately it seems as though it’s nearly time for my laptop to die. Earlier today, while processing some photos, I saw the temperature of the graphics chip exceed 80 degrees celcius. It stayed at 83 degrees for some time, and then the screen began to artefact. I have subsequently removed the bottom panel which normally covers the cooling fan and heatpipe arrangement, and placed the laptop, open and on its side on my desk, with cables sticking out vertically, using an external monitor.

I thought maybe it was just the temperature at that point in time causing the issue. At present, the graphics chip sits at 54 degrees celcius, but similar graphics artefacts are occasionally occuring. This makes me a sad panda. Although I have had this laptop for just short of two years, I don’t really want to buy another. What really annoys me about this is that I know it’s not my fault the thing is dying.

I look after my things. This is the reason it’s dying. nVidia and Dell fucked up a few years ago; nVidia made some dodgy graphics chips and then those dodgy graphics chips made it into several models of Dell laptop, mine included. Although Dell admirably extended the warranty on all affected laptops for 12 months for this particular issue, I still feel as though I’ve been cheated somewhat. I still have a desktop system that I built late in 2004. It’s still running perfectly fine because I thought out the system before I purchased it, built it properly, and looked after it. Despite the fact that I use my computers more than the average person (that system in particular was operated nearly 24/7, literally for about 4 years), they stand the test of time. Not so with my laptop, because nVidia and Dell fucked up.

So, although I really like Dell laptops, and I have since bought other nVidia products, I might have to look elsewhere for a future laptop, if indeed I buy one at all. My needs have changed and I rarely make use of a full laptop system anymore. I could probably do with a basic netbook, although even for simple tasks the abysmal screen resolution on those things would annoy me. I think I’ll have to have a ponder.

Fixing things…

Owen — Sun, 07 Mar 2010 07:41:51 +0000

So, a while back I set up my MythTV system to use PulseAudio because I thought that’s what I needed to get mulitple audio outputs happening on my HTPC. Turns out I was wrong. You see, ALSA is perfectly fine for software mixing. Even though I’m still not sure whether I actually need to use software mixing, it’s what’s working for me at the moment.

The troubles I had with PulseAudio were numerous, and I should really have figured this out sooner, though I think I was too lazy to really think about it properly. I had setup PulseAudio with mpd; that was easy. With MythTV it wasn’t so easy. MythTV doesn’t support PA, and rightly so: compared to ALSA, for what MythTV wants to get done, PA plainly sucks. PA does not support AC3/Dolby Digital/DTS passthrough. PA also adds a lot of latency to the audio – depending on your hardware of course. On my system it was somewhere in the range of 100ms, or a little more. Which was too much for my A/V receivers built-in A/V sync function to deal with, anyway. The result of this? Any audio I got out of MythTV through PA to my A/V receiver worked normally ~80% of the time (though only stereo, so no DVD audio or 5.1 from digital television). The other 20% of the time, you’d get no audio at all, or you’d get extremely loud static hissing. It wasn’t very pleasant. All my Linux-bashing friends made sure to note this problem each and every time it occured while they were trying to watch something on my HTPC.

No more! The solution I have found is simple. After doing a little bit of reading and copying and pasting, ALSA is now correctly configured to handle AC3/DTS passthrough AND all other audio. The only drawback that I can see, which really isn’t that bad at all, is that MythTV cannot be in playback mode while mpd is trying to output, and vice-versa. I can’t think of a usecase of wanting to have music playing while a television show is on, so that’s OK by me.

I’ll dump my various config files below for future reference.

/etc/asound.conf

pcm.dmixer {
type dmix
ipc_key 1024
ipc_key_add_uid false
ipc_perm 0666 # mixing for all users
slave {
pcm “hw:0,1″
period_time 0
period_size 1024
buffer_size 8192
rate 44100
}
bindings {
0 0
1 1
}
}

pcm.dsp0 {
type plug
slave.pcm “dmixer”
}

pcm.!default {
type plug
slave.pcm “dmixer”
}

pcm.default {
type plug
slave.pcm “dmixer”
}

ctl.mixer0 {
type hw
card 0
}

/etc/mpd.conf is simple:

audio_output {
type “alsa”
name “My ALSA Device”
}

And my MythTV settings:

Audio output device: ALSA:default
Speakers configuration: 5.1
Upconvert stereo to 5.1 surround (unticked; this is at your preference however)
Digital output device: ALSA:iec958:{ AES0 0×02 } (autodetected/populated by MythTV, yours should be too, otherwise use ‘aplay -l’ on commandline to get your device name/number)
Dolby Digital and DTS boxes both ticked
Advanced Audio Configuration ticked; Overried SRC quality ticked, Sample Rate Conversion: best
Aggressive Sound card buffering ticked

The other issue I had with MythTV which I fixed today was that the menu button on my remote never seemed to work. I was puzzled and hunted through my MythTV lircrc. This is another thing which I’ve been wanting to fix for ages but never really put any effort into. The fix, again, was simple: the autogenerated lircrc file for MythTV had two identical entries for what MythTV was meant to do when the menu button was pushed. Erasing the duplicate entry in the file made it work. Wha-la!

And all this greatness achieved while being severely hungover. It’s amazing what you can do when you put your mind to it.

Censorship on music can GET FUCKEDY FUCK FUCKED IN THE FUCKING ARSE WITH A FUCKING FUCK STICK! FUCK!!!!!!!!!

Owen — Sat, 27 Feb 2010 11:38:31 +0000

Ugh. I bought a three-pack of Korn albums in a music store a while ago, then I ripped them and forgot about them. Tonight, I think to myself, “ah, a bit of Korn for something different” – I’m listening to ‘Here to Stay’ and I’m hearing strange audio artefacts, I’m thinking maybe it was a bad rip. Closer listen. It’s swear words only. WHAT THE FUCK! I’ve managed to buy some fucking retarded censored version where the swear words are jumbled up so to the casual listener, it still sounds like lyrics, but really it’s just garble. WHAT THE FUCK. ARG!

Words like “fuck” (OMG!) and “shit” (OMG!) are now garbled messes. Thank-you whoever fuckhead decided this was a good idea. You are a dickhead. A fuckbrain. A no-hoping moron. A fucking lunatic. A spastic deranged moron with a fucking shoe for a brain. Eat a bag of dicks.

*deep breath*

OK, I think I’m over it now. But censored music is still fucking stupid.

Fish tank: Stage 3

Owen — Wed, 03 Feb 2010 00:21:44 +0000

After doing a little more reading about planting fish tanks, I realised that the state of my plants needed to be changed. I had planted them as they came from the shop – in bundles of 5-6 stems, and touching or protruding from the surface of the water.

You can see in the above picture the three main groupings of plants.

What I did was this:

Untie the lead (?) wire keeping the plant stems together at the bottom;
Sort the plant stems according to height, and then halve the tallest ones, taking care not to cut away new roots. These are generally easy to identify as pale white strands which protrude from joints in the plant stems;
I then removed any leaves which appeared to be dead or dying (easily identified by their lack of colour), and removed any leaves (whether dead or new) near the bottom of the plant; any leaves buried would only burden the plant as they contribute nothing toward photosynthesis;
Re-group the new plant cuttings with 2-3 stems per group, and replant, taking care to ensure the roots were covered by the substrate

So, this is what my tank looks like now:

As you can see, there are now plants all across the tank horizontally. Until they grow and thicken the coverage, there is still not a lot of places to hide for my rasboras, so they will continue to hide at the back under the filter until I can provide them some more places to hide. I have read that the more places fish have to hide, the more likely they are to come out, as they know a safe place is not far away.

I will hopefully soon aquire some drift wood and river rocks to put in the tank. I’ll have to do some ‘research’ (looking at other peoples’ established tanks) to get a feel for tank layouts that not only look nice but also provide plenty of areas for the fish to enjoy.

The hardest part of setting up this fish tank has so far been my lack of patience. Most other personal projects that I work on only happen at the speed I can complete them or at the speed of my computer – this fish tank, however, requires I wait for mother nature (helped along in some respects) to do her thing. Hopefully I will learn to be patient! I will have to, or I risk wasting my time, and money, and worst of all, I risk killing my fish!

So far so good, however.

Fish tank: Stage 2

Owen — Sun, 31 Jan 2010 11:56:33 +0000

I have introduced my first fish. Ashamedly, I will admit I can’t remember what they are. Something-tails; their tails are quite interesting.

Nothing much has changed; with the addition of fish, I have added a heater, and a bit of a blingy air stone.

The fish are doing fine. I can’t wait to get more! Though, I’ll have to actually wait a week or so to add any more.

One thing I’m quite annoyed about is that the tank looked clean before I set it up, but now that it’s full of water etc, I can see really very obvious previous high-water marks all through it, it’s gross.

Fish tank: Stage 1

Owen — Sat, 30 Jan 2010 22:41:46 +0000

Stage 1 of my fish tank has been completed.

Today I purchased:

Aqua One 2x24W “T8″ fluorescent lighting ($140, the most expensive part so far)
Seachem biological stabliser; to combat “new tank syndrome”
General purpose plant fertiliser/nutrients
Three plants, of which I forget the names

It's cloudy as the gravel settles. I did wash all the gravel before putting the water in, but apparently not well enough." class="g2image_centered" />

~~I’m going to leave the tank sit for a week~~. I’m going to go to the fish shop today and buy… something. Although I purchased a biological stabliser to reduce the effects of new tank syndrome (tl;dr version: new tank syndrome is the lack of “good” bacteria in the tank which helps to cycle waste products from plants and fish to reusable products by the plants); I really should wait a while before introducing any fish to the tank. Though, if I buy a test kit from the fish shop and everything comes up good, I may as well go ahead and buy myself some fish today. Provided the water has cleared up, anyway.

As I’m in the middle of moving house, the tank is actually at my new place, so I will have to check with my new housemates if they’re able to feed any fish I buy.