Adding Aspects to OSGi Services with no Bytecode Weaving

In my previous post I looked at how easy it is to apply the 'branch by abstraction' pattern to OSGi services, where the Service API is the abstraction layer and you can branch at runtime without even taking down the service client.
Another part of the story here is that it can be useful to add aspects to an implementation. This can also be done using the branch by abstraction pattern and in OSGi you can add these aspects to existing services without the need to modify them, and also without the need for bytecode manipulation. In OSGi you can proxy services by hiding the original and placing a proxy service in the service registry that consumers will use instead. This makes it possible to add a chain of aspects to the service invocations without the need to either change the clients nor the services themselves.

To illustrate, I started a little project called service-spector that allows you to add service aspects to any kind of service. The aspects themselves are also implemented as services. Here's what a ServiceAspect looks like:

public interface ServiceAspect {
  String announce();
  void preServiceInvoke(ServiceReference service, Method method, Object[] args);
  void postServiceInvoke(ServiceReference service, Method method, Object[] args, Object result);
  String report();
}

Aspects are called before and after a service invocation is done via preServiceInvoke()/postServiceInvoke(). They are called with a reference to the service being invoked and with the parameters that are provided to the service. The post call is also provided with the result of the invocation. The aspect API allows you to do things like obtaining metrics for services, add logging to existing service calls or anything else really that an aspect could do. Additionally it contains annouce() and report() to present information to the user.

The service-spector is configured using Configuration Admin with a list of filters for services that need have the aspects applied in the service.filters property. There is an additional configuration item that states whether the original services need to be hidden. The proxies are automatically registered with a higher service ranking than the original service. If the consumer of the original only uses a single service the original does not need to be hidden as the one with the higher ranking will be selected. However, if a consumer uses all the services of a certain type you do want to hide the originals. In general it's safer to hide the original service, but if you want to run without the HidingHooks you can switch this off.

I am using the Felix File Install to provide the configuration and have it stored in a file in load/org.coderthoughts.service.spector.impl.ServiceSpector.config A little known feature of File Install is that is supports typed configuration data in .config files, so the following file contains a Boolean hide.services property and a String[] service.filters property.

hide.services=B"true"
service.filters=["(objectClass\=org.coderthoughts.primes.service.PrimeNumberService)",
    "(objectClass\=org.apache.felix.bundlerepository.RepositoryAdmin)"]

In the ServiceSpector main component I can easily consume this configuration using the new Declarative Services annotation based approach:

@Component(immediate=true)
public class ServiceSpector implements ServiceListener {
  // This API is populated by Config Admin with configuration information
  @interface Config {
    String [] service_filters();
    boolean hide_services() default false;
  }

...

  @Activate
  private synchronized void activate(BundleContext bc, Config cfg) {
    ...
    System.out.println("Service filters: " +
      Arrays.toString(cfg.service_filters()));
    System.out.println("Hide services: " + cfg.hide_services());
  }

The Config annotation is automatically populated with the configuration information by matching the property keys with the annotation methods, which I can easily read out using its typed API (the dots in configuration keys are mangled to underscores).

Service-spector has a Service Listener that looks for services appearing and checks if they need to have aspects applied. If so they will be proxied. Then, when the proxied service is invoked, all the ServiceAspect services are called to let them do their thing. For example the CountingServiceAspect simply counts how often a certain API is invoked: 

@Component(scope=ServiceScope.PROTOTYPE)
public class CountingServiceAspect implements ServiceAspect {
  Map<String, LongAdder> invocationCounts = new ConcurrentHashMap<>();

  @Override
  public String announce() {
    return "Invocation counting started.\n";
  }

  @Override
  public void preServiceInvoke(ServiceReference service, Method method, Object[] args) {
    Class declaringClass = method.getDeclaringClass();
    String key = declaringClass.getSimpleName() + "#" + method.getName();
    invocationCounts.computeIfAbsent(key, k -> new LongAdder()).increment();
  }

  @Override
  public void postServiceInvoke(ServiceReference service, Method method, Object[] args, Object result){
    // nothing to do
  }

  @Override
  public String report() {
    StringBuilder sb = new StringBuilder();
    sb.append("Invocation counts\n");
    sb.append("=================\n");
    for (Map.Entry<String, LongAdder> entry : invocationCounts.entrySet()) {
      sb.append(entry.getKey() + ": " + entry.getValue() + "\n");
    }
    return sb.toString();
  }
}

The CountingServiceAspect is included in the Service Spector bundle. Lets add this and the File Install and Configuration Admin bundles to the framework setup used in the previous post that computes primes. I now have the following bundles installed. I removed bundle 9, the incorrect prime number generator that returns only 1's for now.

g! lb
START LEVEL 1
ID|State     |Level|Name
 0|Active    | 0|System Bundle (5.4.0)|5.4.0
 1|Active    | 1|Apache Felix Bundle Repository (2.0.6)|2.0.6
 2|Active    | 1|Apache Felix Gogo Command (0.16.0)|0.16.0
 3|Active    | 1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
 4|Active    | 1|Apache Felix Gogo Shell (0.10.0)|0.10.0
 5|Active    | 1|Apache Felix Declarative Services (2.0.2)|2.0.2
 6|Active    | 1|service (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
 7|Active    | 1|impl (1.0.1.SNAPSHOT)|1.0.1.SNAPSHOT
 8|Installed | 1|client (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
10|Active    | 1|Apache Felix Configuration Admin Service (1.8.8)|1.8.8
11|Active    | 1|Apache Felix File Install (3.5.2)|3.5.2
12|Active    | 1|service-spector (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT

Additionally I have the configuration file from above in the ./load/org.coderthoughts.service.spector.impl.ServiceSpector.config location. This is the default location where the Felix File Install looks. When I start the service-spector bundle it reports the active aspects and its configuration:

g! start 12
Invocation counting started.

Service Spector
===============
Service filters: [(objectClass=org.coderthoughts.primes.service.PrimeNumberService), (objectClass=org.apache.felix.bundlerepository.RepositoryAdmin)]
Hide services: true

Ok so let's start our client:
g! start 8
Services now: $Proxy6(PrimeNumbers)
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! stop 8

The version of primes on master has a PrimeServiceReporter that reports the name and the classname of the service. Here we can see that the service calls itself "PrimeNumbers" but that the actual service instance is a Proxy.

As our filters state that we are also proxying the Repository Admin which comes with the Felix implementation we can also exercise that, for example via the shell commands:
 
g! obr:repos list
... some output ...
g! obr:list
... some output 

Ok so we've now used both of the services that are being proxied. Let's see what APIs have been utilized. The service spector calls report() on all the ServiceAspects when the bundle is being stopped:

g! stop 12
Invocation counts
=================
PrimeNumberService#getServiceName: 1
PrimeNumberService#nextPrime: 30
RepositoryAdmin#listRepositories: 1
RepositoryAdmin#discoverResources: 2

Contributing more aspects

Aspects are contributed as services themselves so they don't need to be part of the service-spector bundle itself. The service-spector-method-timer bundle is a separate bundle that contributes a ServiceAspect that provides average method invocations times.

Get the binaries

The code in this blog post can easily be built using maven, but if you just want the binaries, you can get them from here:
service-spector: https://github.com/coderthoughts/service-spector/releases
method-timer: https://github.com/coderthoughts/service-spector-method-timer/releases

Branch by Abstraction and OSGi

Inspired by my friend Philipp Suter who pointed me at this wired article http://www.wired.com/2016/02/rebuilding-modern-software-is-like-rebuilding-the-bay-bridge which relates to Martin Fowler's Branch by Abstraction I was thinking: how would this work in an OSGi context?

Leaving aside the remote nature of the problem for the moment, let's focus on the pure API aspect here. Whether remote or not really orthogonal... I'll work through this with example code that can be found here: https://github.com/coderthoughts/primes

Let's say you have an implementation to compute prime numbers:

public class PrimeNumbers {
  public int nextPrime(int n) {
    // computes next prime after n - see https://github.com/coderthoughts/primes details
    return p;
  }
}

And a client program that regularly uses the prime number generator. I have chosen a client that runs in a loop to reflect a long-running program, similar to a long-running process communicating with a microservice:

public class PrimeClient {
  private PrimeNumbers primeGenerator = new PrimeNumbers();
  private void start() {
    new Thread(() -> {
      while (true) {
        System.out.print("First 10 primes: ");
        for (int i=0, p=1; i<10; i++) {
          if (i > 0) System.out.print(", ");
          p = primeGenerator.nextPrime(p);
          System.out.print(p);
        }
        System.out.println();
        try { Thread.sleep(1000); } catch (InterruptedException ie) {}
      }
    }).start();
  } 

  public static void main(String[] args) {
    new PrimeClient().start();
  }
}

If you have the source code cloned or forked using git, you can run this example easily by checking out the stage1 branch and using Maven:

.../primes> git checkout stage1
.../primes> mvn clean install
... maven output
[INFO] ------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------

Then run it from the client submodule:

.../primes/client> mvn exec:java -Dexec.mainClass=\
org.coderthoughts.primes.client.PrimeClient
... maven output
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
... and so on ...

Ok so our system works. It keeps printing out prime numbers, but as you can see there is a bug in the output. We also want to replace it in the future with another implementation. This is what the Branch by Abstraction Pattern is about.

In this post I will look at how to do this with OSGi Services. OSGi Services are just POJOs registered in the OSGi Service Registry. OSGi Services are dynamic, they can come and go, and OSGi Service Consumers dynamically react to these changes, as well see. In the following few steps will change the implementation to an OSGi Service. Then we'll update the service at runtime to fix the bug above, without even stopping the service consumer. Finally we can replace the service implementation with a completely different implementation, also without even stopping the client.

Turn the application into OSGi bundles

We'll start by turning the program into an OSGi program that contains 2 bundles: the client bundle and the impl bundle. We'll use the Apache Felix OSGi Framework and also use OSGi Declarative Services which provides a nice dependency injection model to work with OSGi Services.

You can see all this on the git branch called stage2:

.../primes> git checkout stage2
.../primes> mvn clean install

The Client code is quite similar to the original client, except that it now contains some annotations to instruct DS to start and stop it. Also the PrimeNumbers class is now injected instead of directly constructed via the @Reference annotation. The greedy policyOption instructs the injector to re-inject if a better match becomes available:

@Component
public class PrimeClient {
  @Reference(policyOption=ReferencePolicyOption.GREEDY)
  private PrimeNumbers primeGenerator;
  private volatile boolean keepRunning = false; 

  @Activate
  private void start() {
    keepRunning = true;
    new Thread(() -> {
      while (keepRunning) {
        System.out.print("First 10 primes: ");
        for (int i=0, p=1; i<10; i++) {
          if (i > 0) System.out.print(", ");
          p = primeGenerator.nextPrime(p);
          System.out.print(p);
        }
        System.out.println();
        try { Thread.sleep(1000); } catch (InterruptedException ie) {}
      }
    }).start();
  } 

  @Deactivate
  private void stop() {
    keepRunning = false;
  }
}

The prime generator implementation code is the same except for an added annotation. We register the implementation class in the Service Registry so that it can be injected into the client:

@Component(service=PrimeNumbers.class)
public class PrimeNumbers {
  public int nextPrime(int n) {
    // computes next prime after n
    return p;
  }
}

As its now an OSGi application, we run it in an OSGi Framework. I'm using the Apache Felix Framework version 5.4.0, but any other OSGi R6 compliant framework will do.

> java -jar bin/felix.jar
g! start http://www.eu.apache.org/dist/felix/org.apache.felix.scr-2.0.2.jar
g! start file:/.../clones/primes/impl/target/impl-0.1.0-SNAPSHOT.jar
g! install file:/.../clones/primes/client/target/client-0.1.0-SNAPSHOT.jar

Now you should have everything installed that you need:

g! lb
START LEVEL 1
ID|State |Level|Name
0|Active | 0|System Bundle (5.4.0)|5.4.0
1|Active | 1|Apache Felix Bundle Repository (2.0.6)|2.0.6
2|Active | 1|Apache Felix Gogo Command (0.16.0)|0.16.0
3|Active | 1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
4|Active | 1|Apache Felix Gogo Shell (0.10.0)|0.10.0
5|Active | 1|Apache Felix Declarative Services (2.0.2)|2.0.2
6|Active | 1|impl (0.1.0.SNAPSHOT)|0.1.0.SNAPSHOT
7|Installed | 1|client (0.1.0.SNAPSHOT)|0.1.0.SNAPSHOT

We can start the client bundle:

g! start 7
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
... and so on ..

You can now also stop the client:

g! stop 7

Great - our OSGi bundles work :)
Now we'll do what Martin Fowler calls creating the abstraction layer.

Introduce the Abstraction Layer: the OSGi Service

Go to the branch stage3 for the code:

.../primes> git checkout stage3
.../primes> mvn clean install

The abstraction layer for the Branch by Abstraction pattern is provided by an interface that we'll use as a service interface. This interface is in a new maven module that creates the service OSGi bundle.

public interface PrimeNumberService {
    int nextPrime(int n);
}

Well turn our Prime Number generator into an OSGi Service. The only difference here is that our PrimeNumbers implementation now implements the PrimeNumberService interface. Also the @Component annotation does not need to declare the service in this case as the component implements an interface it will automatically be registered as a service under that interface:

@Component
public class PrimeNumbers implements PrimeNumberService {
    public int nextPrime(int n) {
      // computes next prime after n
      return p;
    }
}

Run everything in the OSGi framework. The result is still the same but now the client is using the OSGi Service:

g! lb
START LEVEL 1
   ID|State      |Level|Name
    0|Active     |    0|System Bundle (5.4.0)|5.4.0
    1|Active     |    1|Apache Felix Bundle Repository (2.0.6)|2.0.6
    2|Active     |    1|Apache Felix Gogo Command (0.16.0)|0.16.0
    3|Active     |    1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
    4|Active     |    1|Apache Felix Gogo Shell (0.10.0)|0.10.0
    5|Active     |    1|Apache Felix Declarative Services (2.0.2)|2.0.2
    6|Active     |    1|service (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
    7|Active     |    1|impl (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
    8|Resolved  |    1|client (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
g! start 8
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31

You can introspect your bundles too and see that the client is indeed wired to the service provided by the service implementation:

g! inspect cap * 7
org.coderthoughts.primes.impl [7] provides:
-------------------------------------------
...
service; org.coderthoughts.primes.service.PrimeNumberService with properties:
   component.id = 0
   component.name = org.coderthoughts.primes.impl.PrimeNumbers
   service.bundleid = 7
   service.id = 22
   service.scope = bundle
   Used by:
      org.coderthoughts.primes.client [8]

Great - now we can finally fix that annoying bug in the service implementation: that it missed 2 as a prime! While we're doing this we'll just keep the bundles in the framework running...

Fix the bug in the implementation whitout stopping the client

The prime number generator is fixed in the code in stage4:

.../primes> git checkout stage4
.../primes> mvn clean install

It's a small change to the impl bundle. The service interface and the client remain unchanged. Let's update our running application with the fixed bundle:

First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
g! update 7 file:/.../clones/primes/impl/target/impl-1.0.1-SNAPSHOT.jar
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29

Great - finally our service is fixed! And notice that the client did not need to be restarted! The  DS injection, via the @Reference annotation, handles all of the dynamics for us! The client code simply uses the service as a POJO.

The branch: change to an entirely different service implementation without client restart

Being able to fix a service without even restarting its users is already immensely useful, but we can go even further. I can write an entirely new and different service implementation and migrate the client to use that without restarting the client, using the same mechanism. 

This code is on the branch stage5 and contains a new bundle impl2 that provides an implementation of the PrimeNumberService that always returns 1. 

.../primes> git checkout stage5
.../primes> mvn clean install

While the impl2 implementation obviously does not produce correct prime numbers, it does show how you can completely change the implementation. In the real world a totally different implementation could be working with a different back-end, a new algorithm, a service migrated from a different department etc...

Or alternatively you could do a façade service implementation that round-robins across a number of back-end services or selects a backing service based on the features that the client should be getting.
In the end the solution will always end up being an alternative Service in the service registry that the client can dynamically switch to.

So let's start that new service implementation:

First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! start file:/.../clones/primes/impl2/target/impl2-1.0.0-SNAPSHOT.jar
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! stop 7
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
First 10 primes: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1

Above you can see that when you install and start the new bundle initially nothing will happen. At this point both services are installed at the same time. The client is still bound to the original service as its still there and there is no reason to rebind, the new service is no better match than the original. But when the bundle that provides the initial service is stopped (bundle 7) the client switches over to the implementation that always returns 1. This switchover could happen at any point, even halfway thought the production of the list, so you might even be lucky enough to see something like:

First 10 primes: 2, 3, 5, 7, 11, 13, 1, 1, 1, 1

I hope I have shown that OSGi services provide an excellent mechanism to implement the Branch by Abstraction pattern and even provide the possibility to do the switching between suppliers without stopping the client!

In the next post I'll show how we can add aspects to our services, still without modifying or even restarting the client. These can be useful for debugging, tracking or measuring how a service is used.

PS - Oh, and on the remote thing, this will work just as well locally or Remote. Use OSGi Remote Services to turn your local service into a remote one... For available Remote Services implementations see https://en.wikipedia.org/wiki/OSGi_Specification_Implementations#100:_Remote_Services

With thanks to Carsten Ziegeler for reviewing and providing additional ideas.