Tuesday, January 28, 2014

How I learned to stop worrying (about power cuts) and love my Raspberry PI

As many other hobbyists I got myself a Raspberry PI a few months ago. It's very cool that such a functional piece of hardware can be so cheap. I'm using mine as a backup server for my photos, documents, videos etc, it is my Samba Server, FTP server and, most importantly an XBMC network server.

I'll outline some of the things that I did to make mine working the way I wanted to, but first I'd like to address the biggest issue that I experienced with my Raspberry PI: the fact that it doesn't deal well with power cuts. The PI itself isn't really to blame for that, its a general issue with Linux and other unix-type operating systems: sudden power losses can make their boot/os disk unreadable. With Linux running on a laptop you don't really have that issue, as your laptop has a built-in UPS, but my PI doesn't have that. Which means that I often happened to find my device in a state that it wouldn't boot up again. Apparently we had a power blip.
While some companies are starting to sell UPS solutions for the PI, one of the main issues with that is that the UPS often costs more than the PI itself. If you don't want to spend that extra money, I have used a different setup that gave me the capability to survive PI boot disk corruptions without too much work or cost.

Backup your PI boot SD card

Some people suggested that the best way to prevent your PI from corrupted boot disks is to make the disk readonly. While certainly a good idea, this doesn't really work for me as I often experiment with my PI and change its installed packages regularly. So I didn't want to restrict myself that much.
What I did instead was use Billw's cloning script to make a backup copy of the main SD card on a second one. I used an old SD card that I still had lying around and got a USB SD card adapter, which sells on ebay for about £1 including shipping. Using that script I make regular backups of my main SD card onto its clone. When experiencing one of those regular power blips and the PI doesn't want to start up, I simply swap the SD cards - the one from the USB goes into the main SD card port and vice-versa! That will allow me to start up the PI again with all my configuration as I had it. Then simply run Billw's script again to clone the SD card that's still working back on the corrupted one at which point I'll have two working SD cards again!
I've used this mechanism for a few months now and it works pretty well. Certainly much cheaper than a UPS and although a little bit of work (you need to ensure that the backup SD-card stays up-to-date) I found that it worked great. You can also automate the cloning so that it happens automatically every night or so...

In addition to the backup script there are a few other things that I tweaked for my PI. They might be a little different than what you're used to on Linux, so I'm outlining them here.

Get a Hard Disk with power-save

If you're planning to use your PI as a file server, like I do, get a USB harddrive that has a power-save feature. This means that you can have your system running 24/7 but that there won't be any spinning parts if nobody's using it, and very low power consumption too in that case. I got this Seagate 2TB one for about £60 which works great! The only thing you need to do is enable the power save feature once, which you have to do on a Windows PC using the program that comes on the drive. Once power-save enabled, you can connect it to the PI and it works perfectly. Note that you do need to sudo apt-get install ntfs-3g to use most pre-formatted USB drives.

Use UUIDs to mount your drives

Another issue that I was experiencing was that my USB drives didn't always get the same disk identifier after a PI reboot. One disk would sometimes be sdb and other times it would be sdc, for example. This is problematic if you use the standard 
  /dev/sda1 /disks/x1 auto noatime 0 0
in your /etc/fstab to mount your drives to a directory, as your mounted directory would sometimes be mounted to one disk and sometimes to the other...
A good solution to that is to mount your drives by UUID instead. If you look in /dev/disk/by-uuid/ you can see the UUIDs of all the available disks. Find the right one and use that to mount your disk. The following makes sure that always the same disk is mounted to /disks/x1:
  /dev/disk/by-uuid/5A22DB2D2DB0CBF /disks/x1 auto noatime 0 0

Rotate the logs

If you're doing a lot of file transfer to-from your PI, the Samba/FTP log files can get pretty big. If you, like me, use a crappy old 4GB SD card to run off, it might make sense to keep the size of the log files down. Logrotate allows you to do this, and the standard Raspbian distribution comes with this installed. I just tweaked it to keep the log files smaller, by only remembering 1 week of logs. To do this, you tweak /etc/logrotate.conf, I'm using these settings:
  # rotate log files weekly
  weekly

  # keep 1 weeks worth of backlogs
  rotate 1

  # create new (empty) log files after rotating old ones
  create

  # compress log files
  compress


Serve XBMC throughout the house

Many people use the PI to run XBMC directly. XBMC is a wonderful media playback application that gives you a really nice interface to your media collection. And a PI connected to an audio/video system can certainly be used for running it. However, I was looking for something different. I wanted to use my video library on any device in the house. Maybe a laptop, maybe our multimedia pc which is connected to the tv, maybe even on an Android smartphone. XBMC has features that allow you to do this. To get this working, you don't actually need XBMC on the Raspberry PI at all! You need the video files on it, a network file access protocol, such as Samba, and an installation of MySQL which is what the XBMC installations on the client computers communicate with to show your video library etc. The whole setup is really nice as it allows you to watch your videos on any device in the house and it even remembers where you left off if you stopped halfway, so that you can easily continue watching on another computer. Once you've installed Samba and MySQL for XBMC on the PI, you need to install XBMC on all the client devices that you want to use. There are builds available for most platforms. Next you need to configure XBMC to use MySQL. I actually use different profiles to organize my video files, for example you may have holiday videos in one and educational videos in another profile (or any other separation of your choice ;). You can do this by putting the advancedsettings.xml file in the .../userdata/profile/<Profile Name> directory and name the database in it to separate the content, like this:
  <advancedsettings>
    <videodatabase>
      <type>mysql</type>
      <host>my_pi_host</host>
      <port>3306</port>
      <user>mysql_username</user>
      <pass>mysql_password</pass>
      <name>my_holiday_movies</name>
    </videodatabase>
  </advancedsettings>
The name tag in the configuration is used to connect to separate MySQL databases running in the same database server. That allows you to serve different media depending on the XBMC profile you're in. In the Holidays profile it lists my holiday videos, in the Education profile it has my educational ones.

Finally, when you add videos to your database, make sure to select add the directory on the PI using a network protocol that works on all your devices. For example a Samba URL: 
  smb://my_pi_host/share/holiday/videos/
And voilĂ , you will get all your movies on all your XBMC devices.

Ok, so this was a random collection of bits and pieces that I did to get my Raspberry PI to do what I wanted. If you have a nugget of PI goodness, leave a comment to share :)

Monday, January 13, 2014

OSGi Subsytems on Apache Felix

Note: more up-to-date information on getting started with Apache Aries Subsystems can be found here: http://aries.apache.org/modules/subsystems.html

Last year I blogged about running the OSGi Subsystems implementation from Apache Aries. At the time Equinox was the only OSGi framework that had the full Core R5 implementation supported, which is needed in order to run OSGi Subsystems.

With some recent work on the Apache Felix container, it is now very close to supporting the full OSGi R5 specification. One of this first things that I tried with that is to run the Aries Subsystems implementation on it. And it works :)

OSGi Subsystems are a great way to package a number of OSGi bundles together, for example if you want to distribute an OSGi-based application. Subsystems provide a really convenient deployment model, without compromising the modularity of your application. For more background on Subsystems, see my earlier post about how to use OSGi Subsystems to deploy your applications.

What I'm doing here is take the example from that blog post and run it on Apache Felix with Aries Subsystems.

First of all you need the latest and greatest Felix Framework. I always build it as follows:

  svn co http://svn.apache.org/repos/asf/felix/trunk felix
  cd felix/framework
  mvn install 
  cd ../main
  mvn install 

This will give you a framework runtime in the felix/main folder.
Then add the following bundles to the bundle subdirectory:

All of the bundles above have links behind them where they can be downloaded, except for the Felix Coordinator implementation, as this one is very new. You can just build it from the coordinator directory of your Felix checkout. Or, if you prefer to use released components, you can also use the Equinox implementation org.eclipse.equinox.coordinator_1.1.0.v20120522-1841, but in order to run that on Felix you also need the Equinox supplement bundle org.eclipse.equinox.supplement_1.5.0.v20130812-2109.
Finally, I'm adding my little gogo command bundle as described in my older post, to add subsystem:list, subsystem:install, subsystem:uninstall, subsystem:start and subsystem:stop commands to Gogo. You can also download that as a bundle here: subsystem-gogo-command-1.0.0.jar

Ok, let's start up Felix and look what's inside:
.../felix/main $ java -jar bin/felix 
... some log messages appear ...
____________________________
Welcome to Apache Felix Gogo
g! lb
START LEVEL 1
   ID|State      |Level|Name
    0|Active     |    0|System Bundle (4.3.0.SNAPSHOT)
    1|Active     |    1|Apache Aries Application API (1.0.0)
    2|Active     |    1|Apache Aries Application Modelling (1.0.0)
    3|Active     |    1|Apache Aries Application Utils (1.0.0)
    4|Active     |    1|Apache Aries Blueprint Bundle (1.1.0)
    5|Active     |    1|Apache Aries Proxy Bundle (1.0.1)
    6|Active     |    1|Apache Aries Subsystem API (1.0.0)
    7|Active     |    1|Apache Aries Subsystem Core (1.0.0)
    8|Active     |    1|Apache Aries Util (1.1.0)
    9|Active     |    1|Apache Felix Bundle Repository (1.6.6)
   10|Active     |    1|Apache Felix Configuration Admin Service (1.8.0)
   11|Active     |    1|Apache Felix Coordinator Service (0.0.1.SNAPSHOT)
   12|Active     |    1|Apache Felix Gogo Command (0.12.0)
   13|Active     |    1|Apache Felix Gogo Runtime (0.10.0)
   14|Active     |    1|Apache Felix Gogo Shell (0.10.0)
   15|Active     |    1|Apache Felix Resolver (1.0.0)
   16|Active     |    1|Region Digraph (1.1.0.v20120522-1841)
   17|Active     |    1|slf4j-api (1.7.5)
   18|Resolved   |    1|slf4j-simple (1.7.5)
   19|Active     |    1|Subsystem Gogo Command (1.0.0)
   20|Active     |    1|org.osgi.service.subsystem.region.context.0 (1.0.0)

g! subsystem:list
0 ACTIVE org.osgi.service.subsystem.root

Looking at the bundles, you can see the Gogo command line ones that come with Felix, as well as its Bundle Repo. You can also see bundle 20, which is a synthesised bundle that the Subsystems implementation added and which represents the root subsystem. The subsystem:list command reports that there is one subsystem: the root one.

Now let's look at my example applications again. I have the following two subsystems:


Subsystems are a great way to distribute multi-bundle applications, and my two subsystems each contain 2 specific bundles as well as a shared bundle.

Let's install then and see what happens:

g! subsystem:install http://coderthoughts.googlecode.com/files/subsystem1.esa
Installing subsystem: http://coderthoughts.googlecode.com/files/subsystem1.esa
Subsystem successfully installed: subsystem1; id: 1
g! subsystem:start 1
g! lb
START LEVEL 1
   ID|State      |Level|Name
...
   21|Active     |    1|SharedBundle (1.0.0)
   22|Active     |    1|BundleA (1.0.0)
   23|Active     |    1|BundleB (1.0.0)
The three bundles needed by subsystem1 are installed and all started with the subsystem:start command.

And let's add the second subsystem...
g! subsystem:install http://coderthoughts.googlecode.com/files/subsystem2.esa
Installing subsystem: http://coderthoughts.googlecode.com/files/subsystem2.esa
Subsystem successfully installed: subsystem2; id: 2
g! subsystem:start 2
g! lb
START LEVEL 1
   ID|State      |Level|Name
...
   21|Active     |    1|SharedBundle (1.0.0)
   22|Active     |    1|BundleA (1.0.0)
   23|Active     |    1|BundleB (1.0.0)
   24|Active     |    1|BundleC (1.0.0)
   25|Active     |    1|BundleD (1.0.0)

As expected, the two new bundles for subsystem2 are now also installed. And, because we're talking about a feature subsystem here, where everything is shared, the SharedBundle (21) is not installed a second time, but rather reused from subsystem1. 

The topic of subsystems is much bigger. Subsystems can provide a certain level of isolation, they can work with Repositories for provisioning and you have a lot of options wrt to what you can put inside an .esa file: you can put all the bundles in there that your application needs, or you can just have a textual descriptor that declares your main bundles and let the OSGi Resolver and Repository find the dependencies and deploy these for you. Details about these various options can be found in the OSGi Subsystem spec (chapter 134 in the OSGi R5 Enterprise spec). 

All in all good news - Subsystems now works in Apache Felix as well. Right now you need to build the latest snapshot, but hopefully we'll have a Felix Framework release for this soon!