Dynamic Provisioning OSGi Cloud Ecosystems

In my recent posts I looked at building a cloud ecosystem using OSGi and adding extra control to remote service invocations in such ecosystems.

In this post I'm starting to look at the provisioning of such ecosystems. OSGi frameworks are dynamic which means they can dynamically be assigned a function and when the need arises they can be dynamically repurposed. This repurposing is interesting in particular because it saves you from sending the cloud vm image over the network again. And those images can be big. You can reuse the image that you started with, changing its function and role over time.

In my previous post I was using pre-configured VM images, each representing a different deployment in the ecosystem. In this post I will start out with a number of identical images. In addition, I will deploy one other special image: the provisioner. However, since that one is also running in an OSGi framework, even that one can be repurposed or moved if the need arises.

You'll see that, as in the previous posts, none of the OSGi bundles are statically configured to know where the remote services they use are located. They are all part of the dynamic OSGi Cloud Ecosystem which binds all services together via the OSGi Remote Services compliant Discovery mechanism.
I wrote a very simple demo provisioner which keeps an eye on what frameworks are up, provisions them with OSGi bundles, and if one goes down it redeploys services to an alternative.

Altogether, the setup is as follows:

Provisioning an ecosystem of initially identical OSGi frameworks

The ecosystem contains 5 OSGi frameworks, 4 of which are identical: osgi1, osgi2, osgi3 and web. The only difference wrt to the last one is that it has a well-known hostname because it's hosting the web component that will be accessed from the outside world. The 5th framework has the same baseline as the first 4, but it acts as the provisioner/management agent, so this one has some extra bundles deployed that can provision OSGi bundles in the other frameworks in the ecosystem.

To be able to work with such a dynamic ecosystem of OSGi frameworks, the provisioner needs to monitor the frameworks available in it. Since these frameworks are represented as OSGi services this monitoring can simply be done through OSGi service mechanics, in my case I'm using a simple ServiceTracker, but you could do the same with Blueprint or Declarative Services or any other OSGi services-based framework.

Filter filter = bundleContext.createFilter(

  "(&(objectClass=org.coderthoughts...api.OSGiFramework)(service.imported=*))");
frameworkTracker = new ServiceTracker(bundleContext, filter, null) {
  @Override
  public Object addingService(ServiceReference reference) {
    handleTopologyChange();
    return super.addingService(reference);
  }

  @Override
  public void removedService(ServiceReference reference, Object service) {
    handleTopologyChange();
    super.removedService(reference, service);
  }

};

My ServiceTracker listens for services with OSGiFramework as objectClass which have the service.imported property set. This means that it listens to remote frameworks only. Whenever a framework appears or disappears the handleTopologyChange() will be called.

For this posting I wrote a really simple demo provisioner that has the following rules that are evaluated when the topology changes (frameworks arrive and disappear in the ecosystem):

• Deploy the web bundles to a framework with an ip address starting with 'web-'.
• Deploy one instance of the service provider bundles to the framework with the largest amount of memory available. 
• Ensure that if the vm hosting the service provider bundles disappears for some reason, they get redeployed elsewhere. Note that by virtue of using the OSGi Remote Services Discovery functionality the client will automatically find the moved service.

Obviously these aren't real production rules, but they give you an idea what you can do in a management/provisioning agent:

• Decide what to deploy based on image characteristics, I used image name for the web component, but you can also use other metadata such as image type (Red Hat OpenShift/Amazon/Azure/etc), location (e.g. country where the VM is hosted) or other custom metadata.
• Dynamically select a target image based on a runtime characteristic, in my case I'm using memory available.
• Keep an eye on the topology. When images disappear or new ones arrive the provisioner gets notified - simply through OSGi Service dynamics - and changes can be made accordingly. For example a redeploy action can be taken when a framework disappears.
• No hardcoded references to where the target services live. The service consumer use ordinary OSGi service APIs to look up and invoke the services in the cloud ecosystem. I'm not using a component system like Blueprint or DS here but you can also use those, of course. 

the demo provisioner code is here: DemoProvisioner.java

As before, I'm using Red Hat's free OpenShift cloud instances, so anyone can play with this stuff, just create a free account and go.

My basic provisioner knows the sets of bundles that need to be deployed in order to get a resolvable system. In the future I'll write some more about the OSGi Repository and Resolver services which can make this process more declarative and highly flexible.

However, the first problem I had was: what will I use to do the remote deployment to my OSGi frameworks? There are a number of OSGi specs that deal with remote management, most notably JMX. But remote JMX deployment doesn't really work that well in my cloud environment as it requires me to open an extra network port, which I don't have. Probably the best solution would be based on the REST API that is being worked on in the EEG (RFC 182) and hook that in with the OSGi HTTP service. For the moment, I decided to create a simple deployment service API that works well with OSGi Remote Services:

public interface RemoteDeployer {

    long getBundleID(String location);

    String getSymbolicName(long id);

    long installBundle(String location, byte [] base64Data);

    long [] listBundleIDs();

    void startBundle(long id);

    void stopBundle(long id);

    void uninstallBundle(long id);

}

This gives me very basic management control over remote frameworks and allows me to find out what they have installed. It's a simple API that's designed to work well with Remote Services. Note that the installBundle() method takes the actual bundle content as base-64 encoded bytes. This saves me from having to host the bundles as downloadable URLs from my provisioner framework, and makes it really easy to send over the bytes.

I'm registering my RemoteDeployer service in each framework in the ecosystem using the Remote Services based cloud ecosystem service distribution I described in my previous post. This way the provisioner can access the service in the remote frameworks and deploy bundles to each.

That's pretty much it! I've simplified the web servlet a little from the previous post to only show a single TestService invocation, let's try it out...

Try it out!
First thing I'm doing is start the Discovery Server in the ecosystem. This is done exactly as in my earlier post: Cloud Ecosystems with OSGi.

Having the discovery server running, I begin by launching my provisioner framework.  After that I'll be adding the empty OSGi frameworks one-by-one to see how it all behaves.
Finally I'll kill a VM to see redeployment at work.

The common OSGi Framework bits
The provisioner framework is created by first adding the baseline infrastructure to a new OpenShift DIY clone:

$ git clone ssh://blahblahblah@provisioner-coderthoughts.rhcloud.com/~/git/provisioner.git/

$ cd provisioner

$ git fetch https://github.com/bosschaert/osgi-cloud-infra.git

$ git merge -Xtheirs FETCH_HEAD

Don't forget to set up the Discovery SSH tunnel, which is also described in Cloud Ecosystems with OSGi.

The specific Provisioner bits
Specific to the provisioner vm are the provisioner bundles. The demo provisioner I wrote can be built and deployed from the osgi-cloud-prov-src github project:

$ git clone https://github.com/bosschaert/osgi-cloud-prov-src.git
$ cd osgi-cloud-prov-src

$ mvn install 

$ ./copy-to-provisioner.sh .../provisioner

finally add the changed configuration and provisioner bundle to git in the provisioner clone, then commit and push:

$ cd .../provisioner

$ git add osgi

$ git commit -m "Provisioner VM"
$ git push

The provisioner image will be uploaded to the cloud and if you have a console visible (which you'll get by default by doing a git push on OpenShift) you will eventually see the OSGi Framework coming up.

Adding the identical OSGi Frameworks
I'm launching 3 identical cloud images all containing an unprovisioned OSGi framework. As they become active my demo provisioner will exercise the RemoteDeployer API to provision them with bundles.

First add one called web-something. Mine is called web-coderthoughts. The demo provisioner looks for one with the 'web-' prefix and will deploy the servlet there, this will allow us to use a fixed URL in the browser to see what's going on. This is exactly the same as what is descibed above in the common OSGi Framework bits:

$ git clone ssh://blahblahblah@web-coderthoughts.rhcloud.com/~/git/web.git/

$ cd web

$ git fetch https://github.com/bosschaert/osgi-cloud-infra.git

$ git merge -Xtheirs FETCH_HEAD

And don't forget to set up the SSH tunnel.

Once the web framework is started, the provisioner starts reacting, if you have a console, you'll see messages appear such as:

remote: *** Found web framework
remote: *** Web framework bundles deployed

Great! Let's look at the servlet page:

So far so good. We are seeing 2 frameworks in the ecosystem: the web framework and the provisioner. No invocations where made on the TestService because it isn't deployed yet.

So let's deploy our two other frameworks. The steps are the same as for the web framework, except the vm names are osgi1 and osgi2 and reload the web page:

click to see full size

The two new frameworks are now visible in the list. Our TestService, which is invoked by the servlet/web vm, is apparently deployed to osgi1. You can see that by comparing the UUIDs.

Reacting to failure
Let's kill the framework hosting the TestService and see what happens :)
OpenShift has the following command for this, if you're using a different cloud provider I'm sure they provide a means to stop a Cloud VM image.

$ rhc app stop -a osgi1

Give the system a few moments to react, then refresh the web page:

click to see full size

The osgi1 framework has disappeared from the list and the TestService invocation reports that it's now running in the osgi2 framework. The provisioner has reacted to the disappearance of osgi1 and reprovisioned the TestService bundles to osgi2. The service client (the servlet) was automatically reconfigured to use osgi2 throught the Remote Services Discovery mechanism. No static configuration or binding file, simply OSGi service dynamics at work across VMs in the cloud.

Repurposing
Obviously the provisioner in this posting was very simplistic and only really an example of how you could write one. Additionally, things will become even more interesting when you start using an OSGi Repository Service to provide the content to the provisioner. That will bring a highly declarative model to the provisioner and allows you do create a much more generic management agent. I'll write about that in a future post.

Another element not worked out in this post is the re-purposing of OSGi frameworks. However it's easy to see that with the primitives available this can be done. Initially the frameworks in the cloud ecosystem are identical, and although provisioner will fill them with different content, the content can be removed (the bundles can be uninstalled) and the framework can be given a different role by the provisioner when the need arises.

Note that all the code in the various github projects used in this post has been tagged 0.3.

Controlling OSGi Services in Cloud Ecosystems

In my previous post I looked at the basics of setting up an OSGi Cloud Ecosystem. With that as a basis I'm going to look at an issue that was brought up during the second OSGi Cloud Workshop, held this year at EclipseCon in Washington, where this list of ideas was produced. One of topics on this list is about new ways in which services can fail in a cloud scenario. For example, a service invocation might fail because the service consumer's credit card has expired. Or the number of invocations allocated to a certain client has been used up, etc.

In this post I'll be looking at how to address this in the OSGi Cloud Ecosystem architecture that I've been writing about.

First lets look at the scenario where a cloud service grants a maximum number of invocations to a certain consumer.

An invocation policy for a remote OSGi service

From the consumer side, I'm simply invoking my remote demo TestService as usual:

  TestService ts = ... from the Service Registry ...

  String result = ts.doit();

On the Service Provider side we need to add additional control. As this control (normally) only applies to remote consumers from another framework in the cloud I've extended the OSGi Remote Services implementation to provide extra control points. I did this on a branch of the Apache CXF-DOSGi project, which is the Remote Services implementation I'm using. I came up with the idea of a RemoteServiceFactory. Conceptually a bit similar to a normal OSGi ServiceFactory. Where the normal OSGi ServiceFactory can provide a new service instance for each client bundle, a RemoteServiceFactory can provide a new service (and exert other control) for each remote client. I'm currently distinguishing each client by IP address, not completely sure whether this covers all the cases, maybe some sort of a security context would also make sense in here.
My initial version of the RemoteServiceFactory interface looks like this:
  public interface RemoteServiceFactory {
    public Object getService(String clientIP, ServiceReference reference);
    public void ungetService(String clientIP, ServiceReference reference, Object service);
  }

Now we can put the additional controls in place. I can limit service invocations to 3 per IP address:
public class TestServiceRSF implements RemoteServiceFactory, ... {

  ConcurrentMap<String, AtomicInteger> invocationCount = new ConcurrentHashMap<>();

  @Override
  public Object getService(String clientIP, ServiceReference reference) {
    AtomicInteger count = getCount(clientIP);
    int amount = count.incrementAndGet();
    if (amount > 3)
      throw new InvocationsExhaustedException("Maximum invocations reached for: " + clientIP);

    return new TestServiceImpl(); // or reuse an existing one
  } 

  private AtomicInteger getCount(String ipAddr) {
    AtomicInteger newCnt = new AtomicInteger();
    AtomicInteger oldCnt = invocationCount.putIfAbsent(ipAddr, newCnt);
    return oldCnt == null ? newCnt : oldCnt;
  }

A RemoteServiceFactory allows me to add other policies as well, for example it can prevent concurrent invocations from a single consumer, see here for an example, select provided functionality based on the client or even charge the customer per invocation.

To register the RemoteServiceFactory in the system I'm currently adding it as a service registration property:

public class Activator implements BundleActivator {

  // ...
  public void start(BundleContext context) throws Exception {

    TestService ts = new TestServiceImpl();
    Dictionary<String, Object> tsProps = new Hashtable<>();
    tsProps.put("service.exported.interfaces", "*");
    tsProps.put("service.exported.configs", "org.coderthoughts.configtype.cloud");
    RemoteServiceFactory tsControl = new TestServiceRSF(context);
    tsProps.put("org.coderthoughts.remote.service.factory", tsControl);
    tsReg = context.registerService(TestService.class.getName(), ts, tsProps);
   ...

More control to the client

Being able to control the service as described above is nice, but seeing an exception in the client when trying to invoke the service isn't great. It would be good if the client could prevent such a situation by asking the framework whether a service it's hosting will accept invocations. For this I added service variables to the OSGiFramework interface. You can ask frameworks in the ecosystem for metadata regarding the services it provides:
  public interface OSGiFramework {
    String getServiceVariable(long serviceID, String name);
  }

I implemented this idea using the OSGi Monitor Admin service (chapter 119 of the Compendium Specification)

Service Variables are accessed via OSGi Monitor Admin

From my client servlet I'm checking the service status before calling the service:

  fw.getServiceVariable(serviceID, OSGiFramework.SV_STATUS);

(note given a service reference, you can find out whether it's remote by checking for the service.imported property, you can find the hosting framework instance by matching their endpoint.framework.uuid properties and you can get the service ID of the service in that framework by looking up the endpoint.service.id of the remote service - see here for an example).

So I can ask the OSGiFramework whether I can invoke the service, it can respond with various return codes, as a starting point for possible return values, I took the following list, largely inspired by the HTTP response codes:

  SERVICE_STATUS_OK // HTTP 200

  SERVICE_STATUS_UNAUTHORIZED // HTTP 401

  SERVICE_STATUS_PAYMENT_NEEDED // HTTP 402

  SERVICE_STATUS_FORBIDDEN // HTTP 403

  SERVICE_STATUS_NOT_FOUND // HTTP 404

  SERVICE_STATUS_QUOTA_EXCEEDED // HTTP 413

  SERVICE_STATUS_SERVER_ERROR // HTTP 500

  SERVICE_STATUS_TEMPORARY_UNAVAILABLE // HTTP 503

It might also be worth adding some response codes around saying that things are still ok, but not for long, I was thinking of these responses:

  SERVICE_STATUS_OK_QUOTA_ALMOST_EXCEEDED

  SERVICE_STATUS_OK_PAYMENT_INFO_NEEDED_SOON

These need more thought but I think they can provide an interesting mechanism in preventing outages.

Under the hood I'm using the OSGi Monitor Admin Specification. I took an implementation from KnowHowLab (thanks guys!). It gives a nice Whiteboard pattern-based approach to providing metadata via a Monitorable service. As the RemoteServiceFactory is the place where I'm implementing the policies for my TestService, it provides me a natural place to publish the metadata too.

When the client calls OSGiFramework.getServiceVariable(id, SV_STATUS) the OSGiFramework service implementation in turn finds the matching Monitorable, which provides the status information. The Monitorable for my TestService is implemented by its RemoteServiceFactory:

public class TestServiceRSF implements ..., Monitorable {

  ConcurrentMap<String, AtomicInteger> invocationCount = new ConcurrentHashMap<>(); 

  // ...

  @Override

  public StatusVariable getStatusVariable(String var) throws IllegalArgumentException {

    String ip = getIPAddress(var);

    AtomicInteger count = invocationCount.get(ip);

    String status;

    if (count == null) {

      status = OSGiFramework.SERVICE_STATUS_NOT_FOUND;

    } else {

      if (count.get() < 3)

        status = OSGiFramework.SERVICE_STATUS_OK;

      else

        status = OSGiFramework.SERVICE_STATUS_QUOTA_EXCEEDED;

    }

    return new StatusVariable(var, StatusVariable.CM_SI, status);

  }

  private String getIPAddress(String var) {
    // The client IP is the suffix of the variable name
    if (!var.startsWith(OSGiFramework.SERVICE_STATUS_PREFIX))
      throw new IllegalArgumentException("Not a valid status variable: " + var);


    String ip = var.substring(OSGiFramework.SERVICE_STATUS_PREFIX.length());
    return ip;
  }

Using MonitorAdmin thought the OSGi ServiceRegistry gives me a nice, loosely coupled mechanism to provide remote service metadata. It fits nicely with the RemoteServiceFactory approach but can also be implemented otherwise.

When I use my web servlet again I can see it all in action. It invokes the test service 5 times:

In the Web UI you can see the 2 OSGi Frameworks in this cloud ecosystem. The local one (that also hosts the Web UI) and another one in a different cloud VM.
The Servlet hosting the webpage invokes the TestService 5 times. In this case there is only a remote instance available. After 3 invocations it reports that the invocation quota have been used up.

The Web UI servlet also invokes another remote service (the LongRunningService) twice concurrently. You can see the policy that prevents concurrent invocation in action where only one invocation succeds (it waits for a while then returns 42) and the other reports an error and does not invoke the actual service.

The demo simply displays the service status and the return value from the remote service, but given this information I can do some interesting things.

• I can make the OSGi service consumer aware and avoid services that are not OK to invoke. Standard OSGi service mechanics allow me to switch to another service without much ado. 
• I can go even further and add a mechanism in the OSGi framework that automatically hides services if they are not OK to invoke. I wrote a blog post a while ago on how that can be done: Altering OSGi Service Lookups with Service Registry Hooks - the standard OSGi Service Registry Hooks allow you to do things like this.

Do it yourself!

I have updated osgi-cloud-infra (and its source) with the changes to the OSGiFramework service. The bundles in this project are also updated to contain the Monitor Admin service implementation and the changes to CXF-DOSGi that I made on my branch to support the RemoteServiceFactory.

Additionally I updated the demo bundles in osgi-cloud-disco-demo to implement the RemoteServiceFactory, add Service Variables and update the webui as above.

There are no changes to the discovery server component.

Instructions to run it all are identical to what was described in the previous blog post - just follow the steps from 'try it out' in that post and you'll see it in action.
Note that it's possible that this code will be updated in the future. I've tagged this version as 0.2 in git.

Cloud ecosystems with OSGi

One of the areas where I think that the dynamic services architecture of OSGi can really shine is in the context of cloud. And what I have in mind here is a cloud ecosystem comprised of multiple nodes in a cloud, or possibly across clouds, where each node in this ecosystem potentially has a different role from the other. In such a system the various nodes need to be able to work together to perform some function, and hooking the pieces together is really where the fun starts because how do you know from inside one cloud vm where the other ones are? Various people are working on solutions for this which range from elastic IP addresses to plugging in variables when launching a VM and various other ones. While I agree that these solutions provide value I think that they should not necessarily bleed into the space of the developer or even the deployer. The deployer should simply be able to select a cloud, create a few instances and associate them together. At that point they should nicely work together. 


This is where OSGi Services come in. OSGi Services implement a Java interface (we might see OSGi services in other languages too in the not too distant future) and are registered in the OSGi Service Registry. Consumers of these services are not tied to the provider as they select the service on its interface or other properties. The provider could be any other bundle in the OSGi Framework, or when using OSGi Remote Services they could be in a different framework. The OSGi Remote Services specs also describe a mechanism for discovery which makes it possible to find remote OSGi services using the standard OSGi Service Registry mechanisms (or component frameworks such as Blueprint, DS, etc).


So I started prototyping such a cloud ecosystem using Red Hat's OpenShift cloud combined with OSGi Remote Services. However you'll see that my bundles are pure OSGi bundles that don't depend on any type of cloud - they simply use the OSGi Service Registry as normal...

In the diagram each OSGi Framework is potentially running in its own Cloud VM, although multiple frameworks could also share VMs (this would be a deployment choice and doesn't affect the architecture).

Before diving into the details, my setup allows me to:

• register OSGi Services to be shared with other OSGi frameworks within the ecosystem.
• see what other frameworks are running in this ecosystem. This would be useful information for a provisioning agent.

What's so special about this? Isn't this just OSGi Remote Services? Yes, I'm using those, but the interesting bit is the discovery component which binds the cloud ecosystem together. The user of the Remote Services doesn't need to know where they physically are. Similarly, the provider of the remoted service doesn't need to know how its distributed.

As with any of my blog articles, I'm sharing the details of all this below, so do try this at home :) Most of the work here relates to setting up the infrastructure. Hopefully we can see cloud vendors provide something like this in the not too distant future which would give you a nice and clean deployment infrastructure for creating dynamic OSGi-based cloud ecosystems (i.e. an OSGi PAAS).

The view from inside an OSGi bundle

Very simple. The provider of a service marks it as shared for use in the cloud ecosystem by adding 2 extra service registration properties. I'm using the standard OSGi Remote Service property service.exported.interfaces for this, however with a special config type: org.coderthoughts.configtype.cloud. This config type is picked up by the infrastructure to mean that it needs to be shared in the current cloud ecosystem.

I wrote a set of demo bundles to show the OSGi cloud ecosystem in action. One of the demo bundles registers a TestService, using the standard BundleContext API and adds these properties:

public class Activator implements BundleActivator {

  public void start(BundleContext context) throws Exception {

    TestService dr = new TestServiceImpl();

    Dictionary props = new Hashtable();

    props.put("service.exported.interfaces", "*");

    props.put("service.exported.configs",
              "org.coderthoughts.configtype.cloud");

    context.registerService(TestService.class.getName(), dr, props);

  }

  ....

}

You can see the full provider class here: Activator.java

Consuming the service is completely non-intrusive. My demo also contains a Servlet that provides a simple Web UI to test the service and makes invocations on it. It doesn't need to specify anything special to use an OSGi service that might be in another framework instance. It uses a standard OSGi ServiceTracker to look up the TestService:

ServiceTracker testServiceTracker = new ServiceTracker(context, TestService.class.getName(), null) {
  public Object addingService(ServiceReference reference) {
    testServicesRefs.add(reference);
    return super.addingService(reference);
  }

  public void removedService(ServiceReference reference, 
                             Object service) {
    testServicesRefs.remove(reference);
    super.removedService(reference, service);
  }
};
testServiceTracker.open();

For the whole thing, see here: MyServlet.java

I used plain OSGi Service APIs here, but you can also use Blueprint, DS or whatever OSGi Component technology to work with the services...

The main point is that we are doing purely Service Oriented Programming. As long as the services are available somewhere in the ecosystem their consumers will find them. If a cloud VM dies or another is added, the dynamic capabilities of OSGi Services will rebind the consumers to the changed service locations. The code that deals with the services doesn't deal with the physical cloud topology at all.

Try it out!

As always on this blog I'm providing detailed steps to try this out yourself. Note that I'm using Red Hat's OpenShift which gives you 3 cloud instances for development purposes for free. The rest is all opensource software so you can get going straight away.

Also note that you can set this up using other clouds too, or even across different clouds, the OSGi bundles aren't affected by this at all. So if you prefer another cloud the only thing you need to do there is setup the Discovery system for that cloud; the same OSGi bundles will work.

Cloud instances

For this example I'm using 3 cloud VMs to create my ecosystem. All of which are based on the OpenShift 'DIY' cartridge as I explained in my previous posting. They have the following names:

• discoserver - provides the Discovery functionality
• osgi1 and osgi2 - two logically identical OSGi frameworks

Discovery

The Discovery functionality is based on Apache ZooKeeper and actually runs in its own cloud vm. Everything you need is available from the github project osgi-cloud-discovery.

Here's how I get it into my cloud image (same as described in my previous post):

$ git clone ssh://blahblahblah@discoserver-coderthoughts.rhcloud.com/~/git/discoserver.git/ (this is the URL given to you when you created the OpenShift vm)

$ cd discoserver

$ git fetch https://github.com/bosschaert/osgi-cloud-discovery.git

$ git merge -Xtheirs FETCH_HEAD 

then launch the VM:

$ git push

... after a while you'll see:

Starting zookeeper ... STARTED

Done - I've got my discovery system started in the cloud.

I didn't replicate discovery (for fault tolerance) here for simplicity, that can be added later.

The OSGi Frameworks

For the OSGi Frameworks I'm starting off with 2 identical frameworks which contain the baseline infrastructure. I put this infrastructure in the osgi-cloud-infra github project. To get this into your VM clone as provided by the OpenShift 'DIY' cartridge do similar to the above:

$ git clone ssh://blahblahblah@osgi1-coderthoughts.rhcloud.com/~/git/osgi1.git/

$ cd osgi1

$ git fetch https://github.com/bosschaert/osgi-cloud-infra.git

$ git merge -Xtheirs FETCH_HEAD 

At this point it gets a little tricky as I'm setting up an SSH tunnel to the discovery instance to make this vm part of the discovery domain. To do this, I create an SSH key which I'm adding to my OpenShift account, then each instance that's part of my ecosystem uses that key to set up the SSH tunnel.

Create the key and upload it to OpenShift:

$ cd disco-tunnel

$ ssh-keygen -t rsa -f disco_id_rsa

$ rhc sshkey add -i disco -k disco_id_rsa.pub 

Create a script that sets up the tunnel. For this we also need to know the SSH URL of the discovery VM. This is the blahblahblah@discoserver-coderthoughts.rhcloud.com identifier (or whatever OpenShift provided to you). In the disco-tunnel directory is a template for this script, copy it and add the identifier to the DISCOVERY_VM variable in the script:

$ cp create-tunnel-template.sh create-tunnel.sh

$ vi create-tunnel.sh

... set the DISCOVERY_VM variable ...

finally add the new files in here to git:

$ git add create-tunnel.sh disco_id_rsa*

For any further OSGi framework instances, you can simply copy the files added to git here (create-tunnel.sh and disco_id_rsa*) and add to the git repo. 

As you can see, this bit is quite OpenShift specific. It's a once-off thing that needs setting up and it's not really ideal, I hope that cloud vendors will make something like this easier in the future :)

Add Demo bundles

At this point I have my cloud instances set up as far as the infrastructure goes. However, they don't do much yet given that I don't have any application bundles. I want to deploy my TestService as described above and I'm also going to deploy the little Servlet-based Web UI that invokes it so that we can see it happening. The demo bundles are hosted in a source project: osgi-cloud-disco-demo.

To deploy, clone and build the demo bundles:

$ git clone git://github.com/bosschaert/osgi-cloud-disco-demo.git

$ cd osgi-cloud-disco-demo

$ mvn install

Next thing we need to do is deploy the bundles. For now I'm using static deployment but I'm planning to expand to dynamic deployment in the future.

I'm deploying the Servlet-based Web UI bundle first. The osgi-cloud-demo-disco source tree contains a script that can do the copying and updates the configuration to deploy the bundles in the framework:

$ ./copy-to-webui-framework.sh ~/clones/osgi1

In the osgi1 clone I can now see that the bundles have been added and the configuration to deploy them updated:

$ git status

#    modified:   osgi/equinox/config/config.ini

# Untracked files:

#    osgi/equinox/bundles/cloud-disco-demo-api-1.0.0-SNAPSHOT.jar

#    osgi/equinox/bundles/cloud-disco-demo-web-ui-1.0.0-SNAPSHOT.jar

Add them all to git and commit and push the git repo:

$ git add osgi/equinox

$ git commit -m "An OSGi Framework Image"

$ git push

The cloud VM is started as part of the 'git push'.

Let's try the demo web ui, go to the /webui context of the domain that OpenShift provided to you and it will display the OSGi Frameworks known to the system and all the TestService instances:

There is 1 framework known (the one running the webui) and no TestService instances. So far so good.

Next we'll make the TestService available in another Cloud vm.
Create another cloud VM (e.g. osgi2) identical to osgi1, but without the demo bundles.

Then deploy the demo service provider bundles:

$ ./copy-to-provider-framework.sh ~/clones/osgi2

In the osgi2 clone I can now see that the bundles have been added and the configuration to deploy them updated:

$ git status

# On branch master

#    modified:   osgi/equinox/config/config.ini

#

# Untracked files:

#    osgi/equinox/bundles/cloud-disco-demo-api-1.0.0-SNAPSHOT.jar

#    osgi/equinox/bundles/cloud-disco-demo-provider-1.0.0-SNAPSHOT.jar

Add them all to git and commit and push the git repo:

$ git add osgi/equinox 

$ git commit -m "An OSGi Framework Image"

$ git push

Give the system a minute or so, then refresh the web UI:

You can now see that there are 2 OSGi frameworks available in the ecosystem. The web UI (running in osgi1) invokes the test service (running in osgi2) which, as a return value, reports its UUID to show that its running in the other instance.

Programming model

The nice thing here is that I stayed within the OSGi programming model. My bundles simply use an OSGi ServiceTracker to look up the framework instances (which are represented as services) and the TestService. I don't have any configuration code to wire up the remote services. This all goes through the OSGi Remote Services-based discovery mechanism.
Also, the TestService is invoked as a normal OSGi Service. The only 'special' thing I did here was to mark the TestService as exported in the cloud with some service properties.

Conclusion

This is just a start... I think it opens up some very interesting possibilities and I intend to write more posts in the near future about dynamic provisioning in this context, service monitoring and other cloud topics. The example I've been running here is on my employer's (Red Hat) OpenShift cloud - but it can work on any cloud or even across clouds and the bundles providing the functionality generally don't need to know at all what cloud they're in...

Some additional notes

Cloud instances typically have a limited set of ports they can open to the outside world. In the case of OpenShift you currently get only one: port 8080 which is mapped to external port 80. If you're running multiple applications in a single cloud VM this can sometimes be a problem as they each may want to have their own port. OSGi actually helps here too. It provides the OSGi HTTP Service where bundles can register any number of resources, servlets etc on different contexts of the same port. So in my example I'm running my Servlet on /webui but I'm also using Apache CXF-DOSGi as the Remote Services implementation which exposes the OSGi Services on the same port but different contexts. As many web-related technologies in OSGi are built on the HTTP Service they can all happily coexist on a single port.

OSGi on OpenShift (Free, OpenSource Cloud)

While JBoss OSGi support for OpenShift is being looked at, I had a go at getting Apache Felix to work on OpenShift. And I have to say, the results were good :)

OpenShift is Red Hat's cloud. You can create and use cloud instances for free, for development purposes - a free cloud playground!! 
OpenShift comes with a number of cartridges, including a somewhat reduced version of JBoss AS7 (no OSGi), Node.js and Jenkins. However OpenShift also has a DIY application type. With this you can deploy whatever you like :) So, I took the Felix OSGi framework and deployed that in OpenShift using the DIY type!

Here's what I did.
1. Get a free OpenShift account by signing up at http://openshift.redhat.com
2. Once you have your account you can create an application:

Select the Do-It-Yourself application type.

3. Create a URL for the application:

I went for http://felix-coderthoughts.rhcloud.com.
When you click create, the an empty holder application is created for you. We can fill the application in using git.

4. Now you can clone the application using git, OpenShift will tell you the URL to use, mine looks like this:

$ git clone ssh://blahblahblah@felix-coderthoughts.rhcloud.com/~/git/felix.git

Oh, yes, it only works once you have an SSH key added to your account. You can do that in the 'My Account' section of the web console, or from the command line, see here for more info.

5. At this point you can start putting your cloud content into git. Initially your clone contains a README, some mostly empty directories and some openshift hooks:

$ ls -a

.git        README        misc
.openshift  diy

So what next?
I started by adding a modified Apache Felix installation to a directory called osgi/felix-framework-4.0.2. I took Felix 4.0.2 and instead of the gogo terminal-based console bundles I used the Web Console with Config Admin and Pax-Web.

Next we need to modify the OpenShift action hooks to start the OSGi Framework. This is what my .openshift/action_hooks/start script does:

cd $OPENSHIFT_GEAR_DIR/repo/osgi/felix-framework-4.0.2
export felixcmd="java -Dorg.ops4j.pax.web.listening.addresses=$OPENSHIFT_INTERNAL_IP -jar bin/felix.jar"
nohup $felixcmd &

First the current directory is changed to where Felix is stored, and then Felix is launched with as extra option for Pax Web, the IP address that OpenShift wants us to use. The port number is configured as 8080, as required by OpenShift, this is specified in the Felix configuration file.

That's pretty much it - I also created a fairly blunt stop script that kills all the Java processed. I guess that could be refined ;)

Ok, so let's try it out.
As a convenience I added this setup as a github project called felix-openshift, you can get it into your freshly cloned OpenShift repo with:

$ git fetch git://github.com/bosschaert/felix-openshift.git
From git://github.com/bosschaert/felix-openshift
 * branch            HEAD       -> FETCH_HEAD

Then merge the content in with:
$ git merge -Xtheirs FETCH_HEAD
Note that -Xtheirs automatically takes the start and stop scripts from the fetched project and overwrites the local ones. If your git client doesn't support this option you'll probably have to merge these scripts by hand...

To see it all in action, run:

$ git push
Counting objects: 32, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (21/21), done.
Writing objects: 100% (27/27), 3.49 MiB | 86 KiB/s, done.
Total 27 (delta 2), reused 10 (delta 1)
remote: Starting application...
remote: Done
... you will see more logging messages, and then ...

remote: 2012-05-17 16:19:11.133:INFO:oejs.AbstractConnector:Started NIOSocketConnectorWrapper@127.2.146.129:8080 STARTING
remote: 2012-05-17 16:19:11.561:INFO:oejsh.ContextHandler:started HttpServiceContext{httpContext=org.apache.felix.webconsole.internal.servlet.OsgiManagerHttpContext@174d93a}
Aha - the webconsole has started!

I can now launch it at http://felix-coderthoughts.rhcloud.com/system/console/bundles (or whatever your OpenShift URL is). Default Felix Web Console credentials are admin/admin:

From here on I can use the Felix Web Console to deploy more bundles, control the framework and so on.

After pushing the git repo, you can also control the cloud instance from the commandline with rhc:

$ rhc app start -a felix -l user@domain.org

and

$ rhc app stop -a felix -l user@domain.org

There you go, OSGi running in the cloud. And you can develop and play with this stuff without the need to give anyone your credit card details :)

Running the Penrose (or OpenJDK 8) tests

As a follow-up on my previous post, Building project Penrose (or OpenJDK 8), I wanted to run the unit tests. And, as with many other things in OpenJDK, it's different from what I was used to. While other projects typically use JUnit for unit testing, OpenJDK uses a tool called jtreg, which you'll have to install manually before you can run any tests.

Once you have jtreg installed you need to tell the build where it is (the directory specified contains linux/bin/jtreg) and add it to your path - note that the JT_HOME variable is used by the OpenJDK build:

export JT_HOME=/home/david/apps/jtreg

export PATH=$JT_HOME/linux/bin:$PATH

Now you can run the whole test suite (in your clone of http://hg.openjdk.java.net/penrose/jigsaw or whatever your clone of OpenJDK 8 is):

$ make test

The above takes a very long time (edit: in fact it doesn't properly finish at all for me, it just hangs after a number of hours).

If you're developing it's better to run a portion of the tests, for instance a single test directory:

.../jdk/test$ jtreg -testjdk:/home/david/hg/pj_230212/build/linux-i586/jdk-module-image org/openjdk/jigsaw/cli
Test results: passed: 10
Report written to /home/david/hg/pj_230212/jdk/test/JTreport/html/report.html
Results written to /home/david/hg/pj_230212/jdk/test/JTwork

or a single test class:

.../jdk/test$ jtreg -testjdk:/home/david/hg/pj_230212/build/linux-i586/jdk-module-image org/openjdk/jigsaw/cli/ModuleFileTest.java
Test results: passed: 1
Report written to /home/david/hg/pj_230212/jdk/test/JTreport/html/report.html
Results written to /home/david/hg/pj_230212/jdk/test/JTwork

Note that in the above command lines it's important to pass in the full absolute path of the JDK you're running with as some of the tests rely on that to launch certain executables (like javac).

Building project Penrose (or OpenJDK 8)

As OpenJDK Project Penrose was approved last week, I started looking at getting set up to actually be able to contribute to OpenJDK and Penrose.
As I had never built OpenJDK before I started looking for the build page. While OpenJDK build page says that there is no OpenJDK 8 build readme yet, the link behind the 'coming soon' text actually contains some useful information but it relates to old versions of Linux and I want to get this working on Fedora 16 :)

Currently the Penrose codebase is still the same as the Jigsaw code base, so the steps below also apply to building Jigsaw, and they also work for plain OpenJDK 8.
At some point there will be changes in the Penrose codebase, but I don't expect that they will change the build process much, if at all.

While Julien Ponge nicely describes how to get this working on Ubuntu, the steps are a bit different on Fedora 16, so I'm documenting here how to get started from a fresh new Fedora 16 machine.

Get a few dependencies first:

# set up gcc and the static version of libstdc++

sudo yum install gcc gcc-c++ libstdc++-static

# install build deps on alsa, freetype and cups

sudo yum install alsa-lib-devel freetype-devel cups-devel

# install some X11 headers
sudo yum libXt-devel libXext-devel libXrender-devel libXtst-devel

# get mercurial

sudo yum install mercurial

# install Java 7 and ant

sudo yum install java-1.7.0-openjdk-devel ant

Now clone the whole forest of Mercurial trees (you can also use fclone with the Forest extension instead, which will save you from doing calling the bash script manually):
$ hg clone http://hg.openjdk.java.net/penrose/jigsaw pj
$ cd pj
$ bash get_source.sh

Set the following environment variables:
export LANG=C
export ALT_BOOTDIR=/usr/lib/jvm/java-1.7.0-openjdk
export ALLOW_DOWNLOADS=true

At this point 'make sanity', which does a rough check on the setup, should succeed:
$ make sanity
...
Sanity check passed.

Ok, we're ready to build:

$ make

... go for a coffee, walk the dog, book a holiday

... after quite a while you'll see that it's finished

-- Build times ----------
Target all_product_build
Start 2012-02-24 08:15:23
End   2012-02-24 08:52:47
00:00:58 corba
00:17:31 hotspot
00:00:19 jaxp
00:00:23 jaxws
00:17:44 jdk
00:00:29 langtools
00:37:24 TOTAL
-------------------------

There you go, you built Penrose and the result is a JDK 8 installation based on the Penrose codebase:
$ cd build/linux-i586/jdk-module-image/bin
$ ./java -version
openjdk version "1.8.0-internal"
OpenJDK Runtime Environment (build 1.8.0-internal-david_2012_02_24_08_15-b00)
OpenJDK Client VM (build 23.0-b11, mixed mode)

Looking forward to start hacking around in the code :)

Thanks to Andrew Haley and Alan Bateman for helping me with some issues I came across along the way.

Backing up through FTP on a Mac

It's been a little over half a year now that I made the transition to mac. I was more or less forced into it by Dell because they didn't make the 1920x1200 high-res laptops any more that I like working with.
But I have to say, the Apple hardware is second to none - I haven't experienced any mysterious crashes since.

However I had to find Mac alternatives for all the software that I was using, and I managed to find most of what I was looking for quite quickly...

• For viewing zip archives I currently use Zipeg, it only does extraction but that's good enough more most cases.
• I really like Breeze for keyboard-based window alignment, similar to what Windows 7 has.
• And I started using BetterTouchTool to map a keystroke the mousebuttons (saves me from having to press the touchpad all the time, which I don't really like).
• I'm still undecided about iTunes, I don't really like it (e.g. it's missing a feature that shows what you're playing in the dock or something like that) but it kinda does what it needs to do and there doesn't seem to be anything better on the Mac, strangely enough...
• Most other applications that I had been using before have Mac versions, except for...

... a backup solution! I was always using Areca in the past which works great and had both Windows and Linux versions available. I guess everybody on Mac simply buys a Time Machine but I wanted to use my existing network storage drive for my backups.

I found iBackup and used it for about half a year. It was seemingly able to do the work over a Samba mount to my network drive, although I did notice that I was getting many failure runs, in fact over half of the runs were failures. It might have been caused by the SMB implementation on my NAS drive, I don't really know but I noticed that it was having issues with relatively large files (over a couple of megs).

I started looking at using FTP to do the backups instead (my NAS drive supports FTP) but iBackup doesn't support that.

After looking around for a while I couldn't find any freely available backup option for Mac that uses FTP. I might have missed one or two, but in the end I started writing a little shell script to do the job. One of the things that I was worried about was the backup speed. My script logs in and out for every directory recursively so I was concerned that it would take much longer to complete than the SMB option. I was very surprised that for my data it took 9 minutes to complete, where the SMB-based option took several hours if it completed at all.

So here it is, a simple little bash script to backup any directory on your mac using FTP, I invoke it like this

example..

ftp-bku.sh ~/bin ~/docs ~/etc ~/utils

Note that I have my FTP credential in my ~/.netrc file so that I don't need to provide them in the script itself...

For those who are interested, you can find the script below (btw one thing that isn't completely reliable yet is the return code of the ftp program. It's 0 in some cases when there actually is a problem. Not sure if this can be fixed, although I could grep the output for error messages...)

FTPURL=ftp://User@192.168.1.24:2121
FTPBASEDIR=MyDir
TIMESTAMP=$(date +"%d%m%y-%H%M%S")
BACKUPDIR=backup_`hostname`_$TIMESTAMP
# Perform the actual FTP backup
function backup {
    PNAME="$1"
    DNAME="$2"
    ftp -i -V $FTPURL<<EOF
    bin
    cd "$FTPBASEDIR"
    mkdir "$BACKUPDIR"
    cd "$BACKUPDIR"
    lcd "$PNAME"
    mkdir "$DNAME"
    cd "$DNAME"
    mput *
    quit
EOF
    if [ "$?" == "0" ]; then
        echo Backup succeeded: "$BACKUPDIR"
    else
        echo Backup failed: "$BACKUPDIR"
    fi
}
# Backup the directory specified as $2 recursively where the base directory is $1
# So to backup /Users/david/etc/tmp as etc/tmp you pass in 
#  /Users/david/etc/tmp etc/tmp

function backup-dir {
    cd "$1"
    if [ -d "$2" ]; then
        BASE="$1"
        DIR="$2"
        BASELEN="${#BASE}"
        SUBDIR="${DIR:BASELEN}"
        TARGET="`basename "$1"`$SUBDIR"
        backup "$2" "$TARGET"
    fi
    cd "$2"
    for dir in *
    do
        if [ -d "$2/$dir" ]; then
            backup-dir "$1" "$2/$dir"
        fi
    done
}
# Staring point, backup all directories specified on the command line
for directory in "$@"
do
    backup-dir "$directory" "$directory"
done