The machine is based on an Intel Atom 330 (dual-core 1.6Ghz) with 2Gb of DDR2 800Mhz RAM. It has one onboard Realtek 8168B PCIe gigabit ethernet controller (LAN side) and I have installed a VIA Rhine II 100Mbit PCI network card (PPPoE/WAN side). I have also installed a Marvell 88SE9123 PCIe SATA 6.0 Gb/s SATA controller and a Silicon Image SiI 3114 3Gb/s SATA controller for extra hard disks. The Marvell chip on the former card is the same one installed on many new motherboards which boast SATA3 support, and will be until Intel, AMD, VIA et. al. start incorporating SATA3 controllers into their chipsets, so it is already well supported and will be for some time. It is only a 2-port card but that is perfect for RAID0 or RAID1 configuration, not that I am using the fakeraid hardware controller on board.

So far the system has had several teething problems. I am using a standard Billion 7300 modem/router as a simple ADSL2+ bridge, with the ClearOS box performing PPPoE negotiation and all routing (NAT) duties. I have left my previous modem/router, a Billion 7404VGPX configured and connected to the LAN on standby duty in case I have problems with ClearOS, which I have. All I need to do is move the phone line from the Billion 7300 to the 7404 and re-enable the DHCP server on the 7404 to revert to the old setup.

The ClearOS installation procedure is fairly straightfoward although it is somewhat limited by the fact that you must install from USB or CDROM if you are setting up ClearOS in Gateway mode like I have done. You cannot do an install from the internet unless you are running standalone mode. This doesn’t really make sense to me, surely it wouldn’t be hard to specify temporary network settings during the installation procedure to gain access to the internet?

Some of the problems I have had so far with ClearOS:

• Hard locks (hardware problems): the Sil 3114 card has caused some hard system locks while accessing attached disks. Considering this card has been in use for the last 3-4 years in an older fileserver with four disks attached (it now only has two), this is either indicative of the card reaching its end of life, a problem with its physical installation or some incompatibility between this and the motherboard. The latter is highly unlikely, but the former are possible. That said, it has been working well for the entire time in a well-ventilated case and is essentially ‘good as new’, so who really knows? I have reseated the card in the PCI slot and so far the system has been operating for 25 hours straight. Still early days, but previously it’d only worked for less than that before problems arose.
• Soft locks (software problems): I have managed to lock the system two or three times by clicking around the web control panel. It seems as though running certain reports causes catastrophic failures somewhere, so much so that it ceases function as a router and does not even respond to SSH any longer, requiring a hard reset. This has not happened since I reseated the Sil 3114 card though, so maybe that was at play here.
• Random service failures: the web configuration sometimes dies and comes back. Usually after performing a CPU-intensive task, like listing the currently open connections or generating a report.
• PPPoE: doesn’t seem to receive DNS server address information from the ISP. Or if it does, it doesn’t redistribute this information via DHCP automatically. I’ve had to manually input the preferred DNS servers into the DHCP client.
• ClearOS doesn’t support transparent web proxying AND QoS bandwidth allocation. This is a well-discussed problem with ClearOS, apparently the developers think the problem is too hard to solve without more money. However, other freely-available Linux-based routers can do transparent web proxying and QoS bandwidth allocation. Guess it wasn’t too hard for them…
• Also doesn’t provide NFS services out of the box.

Aside from it crashing a few times (which is looking more and more like it was my fault), the system has been pretty good. It has most of the features anyone would need as an SMB or advanced home server.

Routing and WAN performance with routing and such is excellent. Pings to local gaming servers seem to be slightly reduced by 20-30 milliseconds. The web proxying cache seems to have sped things up a little bit, though with most of the content I access being hosted locally in Australia the speed is already there, it’s more to do with saving bandwidth on not re-downloading the same content all the time.

I still can’t figure out what the deal with network fileserving is. The CPU load while transferring files at gigabit speeds are around 80-90%; I guess this is owing to the fact that I am using an mdadm Linux software RAID0? Samba to Windows Vista only yeilds about 30Mb/sec; though FTP is 80-120 Mb/sec (given this high number is impossible, I am doubting the accuracy of these numbers). iperf between systems is about 850Mbit though so I know the network isn’t the problem. It will require further troubleshooting methinks.

• Users’ Windows PCs backing up automatically to their respective folder on the network
• These backups being automatically rsync’d with an external disk, for a true backup
• Reports on the status of the machine being made available to me at regular intervals

The first is turning out to be the most difficult. Initially I wanted to write a batch script that handled this for me and then configure it to run once a week using Task Scheduler in Windows. However, there are three different versions of Windows for me to contend with out of my five users, (XP, Vista, 7), so it wasn’t as straightforward as I thought it would be, with Vista and 7 including “Robocopy” out of the box, and XP not. However, we all know the most straightforward solution isn’t always the best one. Even with Robocopy, you cannot do incremental/differential file copies. You can coarsely say “all files changed after some date”, but it still copies everything in full.

Enter FTPSync, a nice little FTP tool which has a built-in differencing algorithm and allows full, incremental and differential backups. Its configuration is somewhat obscure, but nothing too difficult to understand with the help of their online documentation (which as it always happens, was actually down the first time I wanted to access it). After some initial tests on a small dataset, I think it will be most suitable for my purposes, for two reasons: (a) incremental/differential backups, saving time, and (b) it does this over FTP, which is the most CPU-efficient protocol for my underpowered file server.

Setting up the user home directories was pretty easy: just enable the [homes] share in your Samba configuration, and you’re away. Mine are actually redirected via symlinks to another disk, so you also have to add “symlinks = yes” to the [global] section of your config, or your home directories won’t appear as intended. The rsync configuration was also pretty straightforward, but I’ll include my backup script here for your reference:

There’s actually a lot of junk at the start of the script which I haven’t cleaned out yet (since changing from an rsync over SSH approach), but the important part, the part that does all the work, is the rsync call on line 28. The simple if construction checks the size of the data to be backed up, and then proceeds if below 200Gb, otherwise does nothing. The result of the script is e-mailed to me when it finishes. Also, I have created a little script which runs every now and then to collect the temperature of the hard disks, a df output or two, and some other info, which is also e-mailed to me periodically, just to keep an eye on things.

There is a version of the above script for each of the five users on my system, and they are all executed using roots cron daemon. As noted in the script, the rsync.pwd file must be owned by root for this to work.

1. Left-click the icon and select ‘Preferences’; click the checkbox next to ‘Start applet on session login’ and close the dialogue.
2. Next, left-click the icon and select ‘Configure local sound server’; click the ‘Network Server’ tab and check the boxes next to ‘Enable network access to local sound devices’ and then ‘Don’t require authentication’. This allows applications to stream audio data direct to PulseAudio instead of through PulseAudio’s implementation of the ALSA API. This is necessary for all applications that support talking directly to PulseAudio.

Next, you’ll want to open the Volume Control from the tray icon menu and select your output device on the Configuration tab. PulseAudio should have enumerated all of the available combinations on your system, so choose the option you want. If you’re using optical output to your A/V receiver or television, choose digital, SPDIF or IEC958.

Setting up mpd for use with PulseAudio is straightforward. Open your /etc/mpd.conf file and comment out all the configuration related to alsa, then scroll down a bit and you should see a commented-out section related to PulseAudio. Make your config look like the below, and restart mpd.

# An example of an ALSA output:
#
#audio_output {
# type "alsa"
# name "My ALSA Device"
# device "hw:0,1" # optional
# format "44100:16:2" # optional
# mixer_device "default" # optional
# mixer_control "PCM" # optional
# mixer_index "0" # optional
#}
audio_output {
type "pulse"
name "My Pulse Output"
# server "remote_server" # optional
# sink "remote_server_sink" # optional
}

Next, you’ll need to setup MythTV. This is pretty straightforward, all I did was go through the General settings to the audio options menu and set the following options:

Update! MythTV trunk supports PulseAudio, so all you need to do is select ‘PulseAudio:default’ as the audio output device.

• Output:ALSA:plughw:0,1 (0,1 being the card id, device id key, obtainable through the command ‘aplay -l’; you may have something different in your setup but you can freely edit this text field for the correct device)
• Passthrough output device: Default
• Max audio channels: Stereo
• Upmix: passive
• Enable AC3 to SPDIF passthrough: no (force AC3 decoding by MythTV if your receiver can’t do it)
• Enable DTS to SPDIF passthrough: yes (pass Dolby/DTS to your receiver, for DVDs)

And that’s it! All done. You should now have mpd and MythTV routable through PulseAudio which allows you to make use of the excellent projectM audio visualisation suite (basically an implementation of Milkdrop in OpenGL for Linux). Excellent for parties!

A note: This does actually output PCM 44.1Khz correctly. I was previously routing my audio through HDMI cable to the television, then from the television to the receiver. The TV was outputing a 48Khz signal; direct from the computer via optical, it’s a sweet 44.1Khz. Nice!

Now, I’m not specifically talking about any operating system or platform, but I will use Windows Vista and Ubuntu Linux 9.04 as examples, as they are the latest (and supposedly smartest) versions of their respective lines.

Over the weekend I backed up my Linux laptop onto my Vista desktop so I could install the latest Ubuntu (9.04). I did that and then started the process of copying all my old files back onto my laptop.

To make the whole process easier for me, I simply backed up my entire home directory. This meant that when I went to copy it into the new installation, some common files and folders would require a conflict resolution (eg /home/owen/Desktop). Of course, I hadn’t thought about this at the time and went away for a few hours, only to come back and find the transfer stalled, waiting for me to resolve a conflict.

My question is: why are file transfers not smarter than this? Surely my 95Gb of files could have been copied faster if only there was a more intelligent way. For example: why can the list of files to be copied be some kind of linked list, as opposed to a FIFO queue? In that case, if a file conflict arises, and no response is received within some sane time, the file causing the conflict is removed from its current position in the list and placed further down, so that the next file may be copied in the background while the conflict resolution for the previous file is on hold. With this action, the bulk of non-conflicting files could then be copied in the background, thus saving time later.

1. Media Centre Musings: Part 1 – The Theory
2. Media Centre Musings: Part 2 – The Plan
3. Media Centre Musings: Part 3 – Third Time’s the Charm
4. Media Centre Musings: Part 4 – Starting Implementation
5. Media Centre Musings: Part 5 – Putting it all Together

Introduction

As I am now living in my own place with people of my generation who understand my desire to have “everything now” (as most other late Generation X-ers and early Generation Y-ers will understand), I have decided that it is time to build myself a media centre for use in the lounge/common room.

The popularity of digital media management solutions has steadily risen over the past few years. Solutions based around commodity PC hardware and open-source operating systems have been around since 2002 at least, with commercial options such as TiVo being available for a few years beforehand.

Although TiVo has been available in America and surrounding countries for nearly 10 years now, it has only recently become avilable in countries such as Australia, Taiwan etc more recently. Thus contributing to the popularisation (amongst geeks at least) of free/open-source solutions such as the aforementioned MythTV.

Software

It is this software system, on top of my as always favourite Linux operating system that I will build my media centre solution. I have decided that using Arch will be ideal for a few reasons:

1. it’s fast. With my older hardware, this is important, especially when considering playback of HD content
2. I know how to use it.
3. I prefer some “under the hood” access – configuring everything from scratch, for me, is not a chore. I prefer it to having everything set to defaults like with Ubuntu/Mythbuntu. With either of those distributions, I’d end up changing all the defaults anyway.

Hardware

My hardware of choice is my ex-gaming PC, a veteran system by today’s standards which is over four years old now!

• AMD Athlon64 3400+ (2400Mhz)
• DFI LANParty UT nForce 250GB
• 2x512Mb Corsair Dual-Channel RAM
• Albatron nVidia GeForce 6800GT 256Mb
• Creative Sound Blaster Live! 5.1

Storage

Currently the system has about 480GB of storage spread across three seperate hard drives (the joys of incremental upgrades). However, due to the planned dual-role of this computer as media centre and secure backup/storage system, I will be buying some new drives and setting up a simple software RAID system of SATA disks. With the ridiculous cost of hard disk storage solutions these days, where off-the-shelf 1TB drives command an approximate cost of 16 cents per gigabyte, there will be no shortage of disk space.

For reasons of performance and reliability I will buy at least two identical drives (probably a pair of Western Digital 7200RPM ‘Green Power’ drives) and place them in a RAID1 configuration. Whether I will use the onboard RAID capabilities of the system board or an implementation in software (mdadm) is largely irrelevant in terms of performance as both rely on the CPU to perform RAID-related calculations. However, using the software solution will abstract the RAID functions from the hardware, which will eliminate the single point of failure. If the motherboard were to be replaced, I would effectively lose my RAID system, and all data with it. The underlying hardware is irrelevant in this scenario when using software RAID.

Video Capture

Finding the right video capture card isn’t going to be as easy as the rest of the hardware. While there is a lot of information (MythTV hardware wiki, OCAU MythTV Wiki) available about which are the best cards to buy, it is a matter of matching the exact cards which are known as fully working to those actually available for purchase.

For example, the Leadtek DTV-2000H is a well-rounded card which offers the features I require. However, there are two different revisions available, the ‘I’ revision and the ‘J’ revision. The former works flawlessly and the latter has intermittent problems and requires a lot more tweaking to get it to play nicely. Both cards use the same highly recommended chipset (Connextant 2388x) but apparently are different enough to cause problems (see the MythTV wiki entry for this card). This makes my choice a very difficult one as retailers generally aren’t too happy about having customers open boxes and inspect hardware version numbers prior to a purchase being made.

Remote Control

The only other hardware consideration I have is what remote control to use. Some of the available video capture cards come with their own, or I may buy another. It is an incidental cost in the grand scheme of things but adds so much functionality to the system. I do not want to have a keyboard and mouse hanging around in my loungeroom if I can avoid it, and indeed, I can. More to the point, though, a remote control with familiar buttons makes the system much more accessible to the non-technical people who will be using the system, and that increases the chances it’ll be used.

Conclusion

As this is only part one of my upcoming series of posts on the matter, you will have to wait a week or two until I’ve properly researched my hardware choices, made my purchases and begun the build. Rest assured though, I’ll do a full write-up!

Over that time period, I’ve come to appreciate the level of understanding that the people I’m developing for have of what I’m doing: very little. But, the little understanding they do have has enabled me to build a sturdy communique with them.

At uni, we’re taught one of the major problems that face software developers is properly understanding what it is they’re being asked to develop. To that end, we’ve been told a few horror stories, where the developers meet with the clients, the clients say “we want an X, and we want it to do Y, Z and B”. Then, the developers go away for a long time and they come back with what they think X looks like, doing what they think Y, Z and B should look like. Usually they get it completely wrong and the clients are furious.

So, we are taught that one of the most important parts of software development is having an almost always open dialogue between the developers and the clients in order to ensure that what the developers are developing matches up with what the clients think they’re developing. Initially, the guys at work didn’t really understand that. I don’t blame them, because I even asked, “have you ever had anyone develop software for you before?” to which they replied simply, “no.”

So, my task was to try to educate them on how to properly interact with me when it came to developing software. I explained that they couldn’t simply say “we want a PQR”, leave me alone for two weeks, and have me come back with exactly what they were after. As they’re quite receptive people, they understood.

But, how do you deal with people who aren’t receptive? More specifically, how can you convince your boss, a non-technical person, that you need to allow for certain timeframes and certain communications when you are trying to create something new for someone? So far I’ve not had much luck with my boss. I explain to him the same things I explained to the others, but I don’t know if he gets it or not. He usually responds with a nod, says “OK”, and changes the subject. He’s a solutions-oriented man; he talks about solutions, final products, and results like they’re simply a matter of moving from A to B. I need to find some way of leveraging this type of thought process to my advantage, so I may communicate with him more effectively.

An excerpt:

Daniel_K:

We are aware that you have been assisting owners of our Creative sound cards for some time now, by providing unofficial driver packages for Vista that deliver more of the original functionality that was found in the equivalent XP packages for those sound cards. In principle we don’t have a problem with you helping users in this way, so long as they understand that any driver packages you supply are not supported by Creative. Where we do have a problem is when technology and IP owned by Creative or other companies that Creative has licensed from, are made to run on other products for which they are not intended. We took action to remove your thread because, like you, Creative and its technology partners think it is only fair to be compensated for goods and services. The difference in this case is that we own the rights to the materials that you are distributing. By enabling our technology and IP to run on sound cards for which it was not originally offered or intended, you are in effect, stealing our goods. When you solicit donations for providing packages like this, you are profiting from something that you do not own. If we choose to develop and provide host-based processing features with certain sound cards and not others, that is a business decision that only we have the right to make.

Although you say you have discontinued your practice of distributing unauthorized software packages for Creative sound cards we have seen evidence of them elsewhere along with donation requests from you. We also note in a recent post of yours on these forums, that you appear to be contemplating the release of further packages. To be clear, we are asking you to respect our legal rights in this matter and cease all further unauthorized distribution of our technology and IP. In addition we request that you observe our forum rules and respect our right to enforce those rules. If you are in any doubt as to what we would consider unacceptable then please request clarification through one of our forum moderators before posting.

Phil O’Shaughnessy
VP Corporate Communications
Creative Labs Inc.

Forum Moderator
Creative Labs

Essentially, Creative has a bad history of customer support. Their driver packs typically provide less features than are advertised. Whenever a new operating system comes along (or in my case, even a new service pack for an existing operating system), Creative throw their hands in the air and shout “no!” — they refuse to support older products on new operating systems. At all.

Some of you might say, “fair enough” — but that’s not entirely the issue here. A forum member, Daniel_K, was packaging and releasing — for free — to the public updated driver packs that as I understand it were created using chopped-up existing driver packs. This was all fine and dandy until of course Creative stepped in and instructed Daniel_K to discontinue this activity, as apparently it was a breach of Creative’s copyrights.

It’s absolutely ridiculous; but you don’t need me to tell you that. You can read the thread here.

This post is password protected. To view it please enter your password below:

Password:

This post is password protected. To view it please enter your password below:

Password:

Instead of being restricted by proprietary licencing and arbitrarily enforced limitations, open-source software leaves developers and users alike to make free choices about the software they use. Unrestricted by vendor lock-in, users are free to switch software solutions as they please, allowing IT departments and home users alike to make quick and resolute decisions concerning software they use.

This of course is much to the chagrin of large software organisations that rely on said vendor lock-in to maintain market share. This has been one of the main sticking points for many governments, businesses and other organisations plans to transition to free and open-source software solutions. However, recently the US Department of Defence and the Croatian government have both made serious efforts toward evaluating and implementing open-source software solutions for use in their day-to-day tasks; the Croatian government has implemented an open-source software policy for all government departments, which encompasses the following guidelines:

• Government institutions will choose and/or develop open source solutions as much as possible, instead of using closed source alternatives.
• The government will support development of closed source solutions that use open standards for protocols and file formats, and which are developed in Croatia.
• The government will support the use of open source programs and open standards outside of its institutions.
• The government will support the use of open source solutions in educational institutions; both closed and open source solutions will be equally presented to students.

(provided by NewsForge, “Croatian Government adopts open-source software policy“)

Indeed, from the NewsForge article linked above, here are the words of Domagoj Juricic, deputy state secretary at the Central State Administrative Office for e-Croatia, who is the leader of Croatia’s Open Source Software Policy project:

“The use of information technology in government administration bodies is increasingly becoming important. So far, most of the software we use is proprietary software, so we cannot modify or complement it, or link software from different vendors. These software products impose rigid commercial conditions of use and limit our possibilities. In this way, government administration bodies may be led into a dependent position on the supplier of the software. This could lead to closed information systems, which make the success and efficiency of our eAdministration project more difficult.”

Similar sentiments are put forward by the US Deparment of Defence’s report, The Open Technology Road Map. The report identifies critical differences between the acqusition of physical goods vs the aquisition and use of digital media. While currently, many proprietary software programmes are treated the same way as a physical good, the report argues the arbitrary limitations of proprietary software, stating:

Digital goods (software code, music, movies, etc.) once created can be copied perfectly with relative ease: limiting distribution enforces scarcity, but that scarcity is arbitrary and negotiated, rather than an innate property of the product.

This arbitrary limitation of software availability also has an impact on inter-departmental software sharing:

Currently within DoD, there is no internal distribution policy or mechanism for DoD developed and paid for software code. By not enabling internal distribution, DoD creates an arbitrary scarcity of its own software code, which increases the development and maintenance costs of information technology across the Department. Other negative consequences include lock-in to obsolete proprietary technologies, the inability to extend existing capabilities in months vs. years, and snarls of interoperability that stem from the opacity and stove-piping of information systems.

Alongside vendor lock-in and the consequent reliance upon the software vendor, proprietary software suffers from lack of features, as well slow turnaround time for bugfixes and feature requests. Open software does not suffer the same limitations, as software developers can be contracted as needed to make additions to existing software, without the arbitrary licencing restrictions placed upon alternative proprietary software. Software developed for use in one department can be easily deployed in other departments, and collaboration of efforts between inter-departmental software development is made easier.

If the Croatian government and the US Department of Defence can adopt open-source ideals, technology and software, there is no reason why other governments and government departments cannot do the same. Open software allows for consistent interoperability (code-sharing, systems compatability, data management), offers system transparency (ease of auditing, bugtracking and fixing, feature implementations) and improves efficiency (reduction of code duplication through code re-use, ease of integration of new features and technology, streamlining of developer contracting) and overall, produces not only a better system of software, but at the same time reduces cost, eliminates vendor lock-in, and produces community-friendly code, which benefits everyone.