themonkeysdidit

Everyone should backup their data. We all know it, but how many of us actually do it? Well until today, I didn’t. But I knew I should so I began looking into various solutions.

First off , you need a lot of space for backups, and however much space you have now, won’t be enough a few years down the line. So all the free online services were out as space is pretty dismal.

Secondly, it had to be cheap. It is for home use only after all so fire proof data centres are hardly required.

Thirdly, it has to be automatic, otherwise you just don’t bother or forget to run your backup and that’s when disaster strikes.

So what did I decide? As of early this year I use a mac book pro as my primary home computer, which means it comes with the excellent Time Machine software, perfect for my needs. Next problem was where to tell Time Machine to store the data. I could use the server in our house but this doesn’t guard against fire, theft etc as presumably both laptop and server would suffer the same fate. So I bought a 1TB USB drive from amazon and back up to this. The clever(?) bit is the disk is actually kept in the office at work and I only bring it home once a week (ish) to run a back up before taking it back to work the next day. Thus I have 1TB of off site backup storage – awesome.

And how did I get a network speed of 1456MB/s? Well, it takes about 12 minutes to get from home to work on my bike, so with the disk in my bag, that’s 1TB of data transfer in 12 minutes, which equates to about 1456MB/s bandwidth (all be it with horrible latency )

Following on from the design issues in XMediaStream, the plan of implementing a queue to hold the SimpleMessageObjects received over multicast has been implemented.

This was done with three new classes, and a modification to MulticastShoutcastServer.java.

SimpleQueue.java

This class holds the queue implementation. In a bid for code re-use, the underlying data structure is a simple array of Objects. A read pointer and write pointer point to the next read and write position (wrapping back to zero when they reach array.length). When an object is read, that position in the array is set to null. Therefore whenever a write is taking place, the position in the array is checked to make sure it is null. If it isn’t null, the write pointer has caught up to the read pointer and we’re about to overwrite an object that hasn’t yet been read. If this happens, the queue will re-size and copy the unread objects into the new queue, setting the read and write pointers appropriately.

MulticastPacketListener.java

This is pretty simple, it continuously listens for new multicast packets and for each one receives, check’s it’s for this configured source and if so, add it to the SimpleQueue.

QueueShoutcastServerStreamer.java

This continuously reads from the queue and sends the mp3 bytes to the listening squeezebox.

The only problem encountered was if the squeezebox disconnected (or was started and stopped). Both MulticastPacketListener’s and QueueShoutcastServerStreamer’s are Threads, so if the squeezebox disconnected, the QueueShoutcastServerStreamer stopped reading from the queue but the MulticastPacketListener kept writing new packets to the queue. The queue therefore continuously re-sized until the JVM ran out of memory.

To get round this problem, the SimpleQueue keeps a timestamp every time an object is added to or read from the queue. The MulticastPacketListener then checks each time it writes a new object when the last read took place, if it was more than 30s ago, it deems the reading client is dead and ends.

This implementation has been running successfully for a couple of days now with no glitches in the music, no dropped multicast packets and only a few queue resizes.

I finally decided to take the plunge and test whether or not XMediaStream  is dropping packets or not. Every so often when listening to the shoutcast stream, there’s a glitch in the music so I was pretty confident something wasn’t quite right.

First thing was to prove whether or not packets were getting dropped. This was quite simple, every SimpleMessageObject sent as a multicast packet now has a sequence number added to it. The receiver of these packets then checks the sequence number to make sure it is what it is expecting and spits a warning out to the log if it misses one.

I connected a listener to the feed over night and checked the log file, low and behold, a packet was getting dropped about once every two and a half minutes. At this point I was faced with 4 options:

1. Do nothing, one dropped packet every 2 minutes 30 seconds isn’t so bad and results in only a slight audio glitch.
2. Implement a recovery mechanism to allow the recovery of missed packets.
3. Find out where in the network the packet was being dropped and fix it.
4. Do something else.

After a while, the audio glitches are a bit annoying and it’s not a particularly intelligent solution so that was 1 out of the window. A recovery mechanism would mean the receiver would have to request a particular packet, that’s OK, we now have a sequence number we can use. But the sender has no packet history so implementing a recovery mechanism would involve storing history in some sort of data structure. This sounds like a fair amount of work so I won’t be doing 2. Both sender and receiver run on the same server so there’s no routers/switches etc in the way that could be dropping packets so no point doing anything for 3. Interestingly, this also means that we must be filling a Solaris TCP/IP buffer somewhere…

This leaves 4 (did you see that coming?). The way the MulticastShoutcastServerStreamer works (that’s the object that the squeezebox connects to and receives the shoutcast stream from) is that it grabs a SimpleMessageObject off the multicast stream (blocking), extracts the raw mp3 data and then sends it to the squeezebox using the OutputStream from the Socket. Once this write() completes, it then grabs the next SimpleMessageObject from the multicast stream. Here’s where the problem lies. I think the socket writes can block if the send buffer is full, meaning the receiving multicast buffer is filling up while we’re waiting to write to the squeezebox.

As a quick side note, there is some flow control here. The multicast sender sends data pretty much at the bit rate of the mp3, so it won’t swamp the network with packets. However, this simple method isn’t quite enough.

My current way of thinking is to split the MulticastShoutcastServerStreamer further. We’ll grab a load of the multicast SimpleMessageObject off the network and write them to an internal queue. The shoutcast streaming part will then read these from the queue and send the data to the squeeze box. So as long as our queue is big enough, we shouldn’t drop any more packets…

There are plenty of instructions for installing bibblepro on Gentoo using ebuilds, but none that I could find for bibblelite. As such, this is a step by step guide to modifying the bibblepro ebuild to provide a bibblelite ebuild. (If you have no idea what an ebuild is, I’d read up on it a bit first…). Please note, all commands preceded with the ‘%’ character should be run as root, those preceded by the ‘$’ character should be run as a regular user.

Step 1 – Pull in the bibblepro ebuild

Firstly, you’ll need to install layman which provides a set of portage overlays (the bibblepro ebuild is available in one such overlay).

% emerge layman

Next, need to bring in the xwing overlay:

% layman -o http://www.gentoo.org/proj/en/overlays/layman-global.txt
% layman -a xwing

If this the first overlay you’ve pulled in with layman:

% echo "source /usr/portage/local/layman/make.conf" >> /etc/make.conf

Finally, check bibblepro is available to install.

emerge --search bibble
Searching...
[ Results for search key : bibble ]
[ Applications found : 1 ]

*  media-gfx/bibblepro-bin [ Masked ]
      Latest version available: 4.10.1
      Latest version installed: [ Not Installed ]
      Size of files: 25,141 kB
      Homepage:      http://www.bibblelabs.com
      Description:   Professional photo workflow and RAW conversion software
      License:       bibblepro

Now we have the layman overlay, we’ll need to create a user overlay and add a modified version of the bibblepro ebuild.

Step 2 – Create a manual overlay

Add the following line “/usr/portage/local/my_overlay” to /usr/portage/local/layman/make.conf so the file now looks like this:

PORTDIR_OVERLAY="
/usr/portage/local/layman/xwing
/usr/portage/local/my_overlay
$PORTDIR_OVERLAY
"

Now add the bibblepro files to our overlay, changing the names so they are now bibblelite:

N.B. The cp command should be typed all on one line

% mkdir -p /usr/portage/local/my_overlay/media-gfx/bibblelite-bin/
% cp -r /usr/portage/local/layman/xwing/media-gfx/bibblepro-bin/bibblepro-bin-4.10.1.ebuild
/usr/portage/local/my_overlay/media-gfx/bibblelite-bin/bibblelite-bin-4.10.1.ebuild

You’ll now have to make some changes to the ebuild file “bibblelite-bin-4.10.1.ebuild” so it builds the lite version, not the pro version. As this file is updated, these instructions will become out of date, but the basic idea is to replace all instances of bibblepro with bibblelite and all instances of BibblePro with BibbleLite.

Now that’s done, you can create the digest:

% ebuild bibblelite-bin-4.10.1.ebuild digest

With any luck it will go and grab the bibblelite binary package from the bibblelabs website and perform some mystical portage voodoo* on it. So long as the voodoo ended without error, we’re good to install!

Step 3 – Install

A quick check to make sure our new package is there:

% emerge --search bibble
Searching...
[ Results for search key : bibble ]
[ Applications found : 2 ]

*  media-gfx/bibblelite-bin [ Masked ]
      Latest version available: 4.10.1
      Latest version installed: [ Not Installed ]
      Size of files: 24,907 kB
      Homepage:      http://www.bibblelabs.com
      Description:   Professional photo workflow and RAW conversion software
      License:       bibblelite

*  media-gfx/bibblepro-bin [ Masked ]
      Latest version available: 4.10.1
      Latest version installed: [ Not Installed ]
      Size of files: 25,141 kB
      Homepage:      http://www.bibblelabs.com
      Description:   Professional photo workflow and RAW conversion software
      License:       bibblepro

Yippee! Let’s install (it’s masked by default so we’ll have to unmask it first)

% echo "media-gfx/bibblelite-bin" >> /etc/portage/package.keywords
% emerge -va bibblelite-bin

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ] media-gfx/bibblelite-bin-4.10.1  USE="-andypro -anselpro -cammy -color-popper
-contrast-control -dinkypro -foxypro -ginapro -gradpro -mattypro -ndgrad -percypro -roypro
-sadiepro -shadypro -sharpiepro -siggypro -spectrum -tonypro" 0 kB [1]

Total: 1 package (1 new), Size of downloads: 0 kB
Portage tree and overlays:
 [0] /usr/portage
 [1] /usr/portage/local/my_overlay

Would you like to merge these packages? [Yes/No]   Yes

Depending on your system setup and window manager, you might even have a handy icon added to your graphics menu to start bibble with. If not (like me), you should be able to simply start it from the command line (and be immediately presented with an error):

$ bibblelite
bibblelite: error while loading shared libraries: libkodakcms.so: cannot open shared object file:
No such file or directory

We should probably fix that

Step 4 – Troubleshoot

Add the missing library location to the env.d area, regenerate the ld cache and we should be good to go:

% LDPATH=/usr/lib/bibblelabs/bibblelite/libs
% env-update
$ bibblelite
AppPath: /usr/bin
SysHome: /usr/lib/bibblelabs/bibblelite/

Success!

*It’s not really mystical portage voodoo, just recording some checksums and the like.