Friday, April 19, 2013

Using OSGi Subsystems to deploy your Applications

One of the major new specs in the OSGi R5 Enterprise release is the Subsystem specification. While this spec itself is quite large and covers a wide number of angles and use-cases, I find the simplest way to explain Subsystems really as something like Application deployment for OSGi where an application is comprised of a number of bundles.

OSGi always encourages modular development and during development this is indeed great, because you can focus on the module at hand and have clear visibility of the impact of any changes that you make. However, once you want to deploy your application of 150 bundles this may become a little bit complicated - you certainly don't want to hand the person performing the deployment of 150 different files to deploy. You want to put them together in some way or form. This has caused many projects to come up with their own solution around this. Karaf and Eclipse have features, Eclipse Virgo has plans and Apache Aries has applications. The OSGi Subsystem specification now provides a standard for combining a number of bundles in a single deployable (an .esa file) which means that an .esa file can be deployed in any compliant Subsystem implementation.
I am really happy that the good people at Apache Aries have recently released version 1.0 of the Aries Subsystem implementation, so you can now use this without having to build an implementation yourself.

I'm going to look at how to create and use Subsystems later in this post, but first let's get an OSGi framework set up with Subsystems. The Aries implementation with its dependencies consists of 15 bundles. In my example below I'm using Equinox as the OSGi framework, but it should obviously work just as well on any other OSGi R5 compliant framework.

Setting up the Subsystem infrastructure
I'm going to be using the following:
  • Equinox 3.8.2 (which comes with Eclipse 4.2.2 or 3.8.2)
  • The gogo shell (which comes with Equinox and also with Felix)
  • The Aries Subsystem 1.0 implementation
  • dependencies of the above...
Let's start getting Equinox up and running with the shell, add the following bundles to your Equinox runtime. These all come shipped with Eclipse so if you're working in Eclipse you can simply select them in the 'OSGi Framework' launch configuration:

org.eclipse.osgi_3.8.2.v20130124-134944
org.eclipse.osgi.services_3.3.100.v20120522-1822
org.apache.felix.gogo.runtime_0.8.0.v201108120515
org.apache.felix.gogo.shell_0.8.0.v201110170705
org.eclipse.equinox.console_1.0.0.v20120522-1841

Now add the following bundles to install the Aries Subsystem implementation (the links below can be used to download them from Maven Central):

I'll leave it to the reader to find a convenient way to install all these (see also the comments section with a note about how to install this on a framework other than Equinox; you need an extra bundle). You can do it with a script, using a repository, etc... There is also a subsystem-bundle artifact that may help. In any case, this alone validates one of the key points why Subsystems were designed in the first place. If you have an application that is formed by a number of bundles you'd really want a nice and convenient way to deploy them. Once we have a subsystem implementation in place we can do this and start deploying large applications that consist of many bundles by simply deploying a single Subsystem archive file.

With the above bundles started the Subsystem Service is registered and ready to deploy subsystems:
osgi> services (objectClass=*Subsystem)
  {org.osgi.service.subsystem.Subsystem}=
  {subsystem.id=0, subsystem.state=ACTIVE, subsystem.version=1.0.0,
   subsystem.type=osgi.subsystem.application,
   subsystem.symbolicName=org.osgi.service.subsystem.root, ...}
  "Registered by bundle:" org.apache.aries.subsystem.core_1.0.0 [6]

There is one issue though - we have no tool yet that utilizes the subsystem service so we can interact with it. It would be really nice if we could add a command to the OSGi console to do this. Using the extensible Gogo command shell this is childs play. So let's add a few subsystem commands.

Add some Subsystem commands to Gogo
Gogo is becoming the de-facto standard for shell commands in an OSGi framework. It's used by Equinox, Felix, Karaf and other OSGi distributions these days. Gogo is extensible and adding a few new commands to it is as simple as registering an OSGi service.

I created a bundle with only a single class, the activator which provides the following commands:
  subsystem:list
  subsystem:install <url>
  subsystem:uninstall <id>
  subsystem:start <id>
  subsystem:stop <id>

import java.io.IOException;
import java.net.URL;
import java.util.*;
import org.osgi.framework.*;
import org.osgi.service.subsystem.Subsystem;

public class Activator implements BundleActivator {
  private BundleContext bundleContext;

  public void start(BundleContext context) throws Exception {
    bundleContext = context;
    Dictionary<String, Object> props = new Hashtable<String, Object>();
    props.put("osgi.command.function",
      new String [] {"install", "uninstall", "start", "stop", "list"});
    props.put("osgi.command.scope", "subsystem");
    context.registerService(getClass().getName(), this, props);
  }

  public void install(String url) throws IOException {
    Subsystem rootSubsystem = getSubsystem(0);
    Subsystem s = rootSubsystem.install(url, new URL(url).openStream());
    System.out.println("Subsystem successfully installed: " +
      s.getSymbolicName() + "; id: " + s.getSubsystemId());
  }

  public void uninstall(long id) {
    getSubsystem(id).uninstall();
  }

  public void start(long id) {
    getSubsystem(id).start();
  }

  public void stop(long id) {
    getSubsystem(id).stop();
  }

  public void list() throws InvalidSyntaxException {
    for (ServiceReference<Subsystem> ref :
         bundleContext.getServiceReferences(Subsystem.class, null)) {
      Subsystem s = bundleContext.getService(ref);
      if (s != null) {
        System.out.printf("%d\t%s\t%s\n", s.getSubsystemId(), s.getState(), s.getSymbolicName());
      }
    }
  }

  private Subsystem getSubsystem(long id) {
    try {
      for (ServiceReference<Subsystem> ref :
           bundleContext.getServiceReferences(Subsystem.class, "(subsystem.id=" + id + ")")) {
        Subsystem svc = bundleContext.getService(ref);
        if (svc != null)
          return svc;
      }
    } catch (InvalidSyntaxException e) {
      throw new RuntimeException(e);
    }
    throw new RuntimeException("Unable to find subsystem " + id);
  }

  public void stop(BundleContext context) throws Exception {}
}

I shared a bundle that contains this command, you can get it from here: http://coderthoughts.googlecode.com/files/subsystem-gogo-command-1.0.0.jar

Once the above bundles are installed and everything is started I have the following bundles in my framework:
0  ACTIVE org.eclipse.osgi_3.8.2.v20130124-134944
1  ACTIVE org.eclipse.osgi.services_3.3.100.v20120522-1822
2  ACTIVE org.apache.felix.gogo.runtime_0.8.0.v201108120515
3  ACTIVE org.apache.felix.gogo.shell_0.8.0.v201110170705
4  ACTIVE org.eclipse.equinox.console_1.0.0.v20120522-1841
5  ACTIVE org.apache.aries.subsystem.api_1.0.0
6  ACTIVE org.apache.aries.subsystem.core_1.0.0
7  ACTIVE org.apache.aries.subsystem.obr_1.0.0
8  ACTIVE org.apache.aries.application.api_1.0.0
9  ACTIVE org.apache.aries.application.modeller_1.0.0
10 ACTIVE org.apache.aries.application.utils_1.0.0
11 ACTIVE org.apache.aries.blueprint_1.1.0
12 ACTIVE org.apache.aries.proxy_1.0.1
13 ACTIVE org.apache.aries.util_1.1.0
14 ACTIVE org.apache.felix.bundlerepository_1.6.6
15 ACTIVE org.apache.felix.resolver_1.0.0
16 ACTIVE org.eclipse.equinox.coordinator_1.1.0.v20120522-1841
17 ACTIVE org.eclipse.equinox.region_1.1.0.v20120522-1841
18 ACTIVE slf4j.api_1.7.5, Fragments=19
19 RESOLV slf4j.simple_1.7.5, Master=18
20 ACTIVE org.osgi.service.subsystem.region.context.0_1.0.0
21 ACTIVE subsystem-gogo-command_1.0.0

Note that bundle 20 is a synthesized bundle created automatically by the subsystem implementation. We can safely ignore it.

Now I can start doing something. Let's list the available subsystems using our new command from the subsystem-gogo-command listing/bundle:
osgi> subsystem:list
0 ACTIVE org.osgi.service.subsystem.root 

At this point there is only a single subsystem: the root one.

Working with Subsystems
Let's create some sample subsystems to look at what you can do.

I'm going to create two basic subsystems that should allow us to play with it. The subsystem specification defines a number of different subsystem types. In this post I will be looking at the feature subsystem type which deploys all the bundles from the subsystem in a shared space. As if you were just installing all the bundles in a plain framework. (note: other subsystem types provide isolation for the subsystems.)
Subsystem archives typically use the .esa file extension. Both my example subsystems contain 3 bundles. The subsystem1.esa file contains Bundle A, Bundle B and a bundle called Shared Bundle. subsystem2.esa contains Bundle C, Bundle D and also the same Shared Bundle. Both subsystems package the Shared Bundle as they both have a dependency on it. So in order to get a fully working system for either subsystem I need that Shared Bundle. However since these are feature subsystems, where everything is shared I only need the Shared Bundle deployed once.

Creating a subsystem file is pretty easy. The .esa file is really just a zip file that contains the embedded bundles in the root. Additionally it contains a subsystem manifest. I created mine simply using the jar command, but you can also use tools such as the esa-maven-plugin. Here's what you'll find inside:

$ jar tvf subsystem1.esa
    99 Fri Apr 19 08:34:08 IST 2013 OSGI-INF/SUBSYSTEM.MF
  1181 Fri Apr 19 08:33:06 IST 2013 BundleA_1.0.0.jar
  1058 Fri Apr 19 08:33:06 IST 2013 BundleB_1.0.0.jar
   906 Fri Apr 19 08:33:06 IST 2013 SharedBundle_1.0.0.jar


As you can see the zip file contains the relevant bundles in the root plus a subsystem manifest. Here's what the SUBSYSTEM.MF file in subsystem1.esa looks like:
  Subsystem-SymbolicName: subsystem1
  Subsystem-Version: 1.0.0
  Subsystem-Type: osgi.subsystem.feature
It looks a bit like a Bundle Manifest. Most of the information in there is optional...

The subsystem2.esa file is very similar. You can download the sample subsystem files from here: subsystem1.esa and subsystem2.esa.

Let's deploy a subsystem:
  osgi> subsystem:install http://coderthoughts.googlecode.com/files/subsystem1.esa
  Subsystem successfully installed: subsystem1; id: 1

If we list the bundles the three bundles that were in subsystem1.esa are now added.
  22 INSTALLED SharedBundle_1.0.0
  23 INSTALLED BundleA_1.0.0
  24 INSTALLED BundleB_1.0.0

Now let's start the subsystem:
  osgi> subsystem:start 1
...
  22 ACTIVE SharedBundle_1.0.0
  23 ACTIVE BundleA_1.0.0
  24 ACTIVE BundleB_1.0.0
This is pretty handy: starting the subsystem will start all of the bundles that it contains!

Let's add the other subsystem:
  osgi> subsystem:install http://coderthoughts.googlecode.com/files/subsystem2.esa
  Subsystem successfully installed: subsystem2; id: 2
  osgi> subsystem:start 2

Now both subsystems are active:
  22 ACTIVE SharedBundle_1.0.0
  23 ACTIVE BundleA_1.0.0
  24 ACTIVE BundleB_1.0.0
  25 ACTIVE BundleC_1.0.0
  26 ACTIVE BundleD_1.0.0
And we can see that the SharedBundle was only deployed once, because it could be shared across subsystems.

You can also query the subsystems known in the system:
  osgi> subsystem:list
  0 ACTIVE org.osgi.service.subsystem.root
  1 ACTIVE subsystem1
  2 ACTIVE subsystem2

Another interesting aspect is how stopping and un-installation works. Especially in relation to the SharedBundle. I'll leave is as an exercise for the reader but you can see that the Subsystems implementation keeps track of the bundle sharing. If you only stop subsystem1, the SharedBundle will remain ACTIVE. Only when both subsystems that use the bundle are stopped the bundle will move to the RESOLVED state. Uninstalling works similarly. When you uninstall subsystem1, BundleA and BundleB will be uninstalled, but the SharedBundle won't as it is still being used by subsystem2. Only when subsystem2 is uninstalled as well all of the bundles associated with subsystem1 and subsystem2 are uninstalled.

There is a lot more to talk about in relation to subsystems. For example, subsystems don't have to actually embed their dependencies. They can also download them from an OSGi Repository service. In that case your .esa file can be limited to only contain a SUBSYSTEM.MF which lists what your root application bundles should be. The subsystem implementation can also use the Repository Service to automatically find transitive dependencies.

In my little example, the subsystems only contain 3 bundles each, but using .esa files can become really handy when your application becomes large and contains tens or hundreds of bundles. You can even nest them, so subsystems can contain other subsystems - becoming building blocks of higher-level subsystems.

OSGi Subsystems should make the distribution and deployment of larger OSGi applications much easier. The .esa file provides a portable format which allows you to hand your users a single artifact to deploy, regardless of how many bundles your application is made up of.

For more information about OSGi Subsystems see chapter 134 of the OSGi R5 Enterprise specification: http://www.osgi.org/Download/Release5