Monday, January 29, 2018

Blockchain Smart Contracts are the new Serverless!

Smart Contracts (thanks to Michael Bacina)
Over the recent past I've been experimenting with Smart Contracts for block chain implementations. Smart Contracts are essentially programs running on the block chain infrastructure. For example Ethereum supports Smart Contracts written in Solidity. EOS is another example of a blockchain that will support smart contracts. I've been looking at EOS an its smart contracts in more detail, here the smart contracts can be written in C/C++ so you don't have to learn a new language for it.


A CryptoKitty
So what can you do with a smart contract? Smart contracts are designed to provide some sort of computation and store the result (immutably) on the blockchain. The computation is a custom program that applies to your domain, so for example you could be renting out holiday accommodation and your website might store the holiday home rental contract including the price, optional extras, insurance etc for a given period of time, after having computed it, on the blockchain. In most cases smart contracts can also handle the payment, so let's say the rental home costs 500 Euro per week, then the equivalent in Ether (ETH) or whatever the current blockchain/crypto is, can be transferred to the owner as part of the smart contract execution.
Or, more creatively as has been done on the ethereum network, your contract could compute a unique CryptoKitty for you that is a cute looking creature created just for you to look at and store the result on the blockchain.

You can even take this a little bit further. As shown with the CryptoKitty the smart contract does not need to have anything to do with transferring money from a to b or writing some sort of financial contract. In theory you could use the smart contract to do anything you might be able to use asynchronous computing power for.

After playing briefly with Solidity, the smart contract language for Ethereum I moved to play a bit more with EOS smart contracts. Why EOS? EOS as a blockchain is still in it's early phase and under heavy development although they do have a test network up and running at this stage. However I find EOS interesting because it has a couple of interesting aspects:
  • First of all it's quite easy to run a test EOS node on your own machine which allows you as a developer to play with it and understand it in a sandbox type environment.
  • EOS aims to provide much higher transaction rates than the current major block chains can provide. It promises up to 50000 transactions per second which all of a sudden is big enough to handle amounts of payments similar to major credit card companies like Visa and Mastercard.
  • Smart contracts on EOS can be written in C/C++ which is really nice, as you don't need to learn a new programming language for it.
So let's take a look at how I got my example EOS contract deployed to my own test EOS node.

Build the EOS code

I tried this on Ubuntu 16.04. Compile the EOS code base:

First, clone the EOS code
  ~$ git clone https://github.com/eosio/eos ~/eos
  
~$ cd eos

Then build the whole lot:
  ~/eos$ ./build.sh ubuntu full
This takes a while but once it's finished you should be able to run your EOS node:
  ~/eos/build/programs/eosiod$ ./eosiod

It now exits with an error. You need to set up data-dir/config.ini as described in the EOS docs: https://github.com/EOSIO/eos

At this point your EOS node should be happily up and running. Which is really neat. You've got a EOS block chain node running for development purposes on your local machine!

~/eos/build/programs/eosiod $ ./eosiod 


Before you can use your EOS node you need to create a wallet and an account. Since our smart contract will be computing the Fibonacci sequence, I'm going to call the account fibonacci. The following commands do that for you. They use a demo account inita that is created in the config.ini file when its set up as above.

Here we use the eosioc program which is a client to the EOS network:
~/eos/build/programs/eosioc $ ./eosioc wallet create
~/eos/build/programs/eosioc $ ./eosioc wallet open

Import the inita demo key:
~/eos/build/programs/eosioc $ ./eosioc wallet import 5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3

Create two private keys for the fibonacci account:
~/eos/build/programs/eosioc $ ./eosioc create key
Private key:###
Public key: ###
~/eos/build/programs/eosioc $ ./eosioc create key
Private key:###
Public key: ###

Create the fibonacci account:
~/eos/build/programs/eosioc $ ./eosioc create account inita fibonacci {private key1} {private key2}

Create the smart contract

I created a little testproject called fibonacci which computes the fibonacci sequence to a certain iteration in the EOS smart contract and stores the result in the EOS database.

The code can be found in github here: https://github.com/coderthoughts/fibonacci

It exists of two components: an external contract, in the fibonacci.abi file which defines how the application communicates with the outside world. The actual communication typically happens in JSON:

{
  "structs": [{
      "name": "compute",
      "base": "",
      "fields": {
        "iterations": "uint64"
      }
  },{
    "name": "result",
    "base": "",
    "fields": {
      "id": "name",
      "value": "uint64"
    }
  }],
  "actions": [{
    "action_name": "compute",
    "type": "compute"
  }],
  "tables": [{
    "table_name": "results",
    "type": "result",
    "index_type": "i64",
    "key_names": ["id"],
    "key_types": ["name"]
  }]
}

And there is the fibonacci.cpp file that contains the source code in C++ of the contract. The main bit of the C++ contract is just the apply() method that gets invoked when the EOS Smart Contract receives a message:
    uint64_t fibonacci(uint64_t iterations) {
        uint64_t first = 0;
        uint64_t second = 1;

        if (iterations == 0)
            return 0;

        eosio::print("1 ");
        for (uint64_t i=1; i < iterations; i++) {
            uint64_t res = first + second;
            first = second;
            second = res;
            eosio::print(res, " ");
        }
        return second;
    }

    /// The apply method implements the dispatch of events to this contract
    void apply(uint64_t code, uint64_t action) {
        if (action == N(compute)) {
            auto message = eosio::current_message<compute>();
            eosio::print("Calling fibonacci\n");
            uint64_t num = fibonacci(message.iterations);

            result res(eosio::name(code), num);
            Results::store(res, res.id);
            eosio::print("Stored result in database\n");
        }
    }

There are also specific EOS libraries available, the documentation is here: https://eosio.github.io/eos/group__contractdev.html

Deploy your own smart contract

An interesting part of the EOS smart contract development lifecycle is that these contract don't get compiled in regular machine language, what C++ compilers normally do, but it gets compiles into a WebAssembly .wast file. This file is some sort of assembler language but then platform independent and this is what EOS uses at runtime.

Once deployed, you can execute your contract by sending a message to it. The contract executes and can write its output to a database location for later retrieval by a client.

The easiest way to compile the fibonacci source is to put the files with the other example smart contracts in the EOS codebase, in the contracts directory. Then you've got everything in the path as the compiler expects it:

Compile the project:
~/eos/contracts$ ../build/tools/eoscpp -o fibonacci/fibonacci.wast fibonacci/fibonacci.cpp
Upload the fibonacci smart contract to the EOS node:
~/eos/contracts$ eosioc set contract fibonacci fibonacci/fibonacci.wast fibonacci/fibonacci.abi
Now we can start executing it. Lets run the fibonacci compute for 8 iterations and store the result:
~/eos/contracts$ eosioc push message fibonacci compute '{"iterations":8}' -S fibonacci

On the EOS demon console you can see some debug output from the smart contract

> Calling fibonacci
> 1 1 2 3 5 8 13 21 Stored result

However a real user will obviously never be able to see this. So to obtain the result of the computation, we'll look it up in the EOS database:

/build/programs/eosioc$ ./eosioc get table fibonacci fibonacci results 


{
  "rows": [{
      "id": "fibonacci",
      "value": 21
    }
  ],
  "more": false
}


So the result is 21. We've executed our smart contract and obtained the result on the EOS blockchain!

Conclusion

I'm pretty excited by the possibilities of smart contracts with regard to the possibilities that these, once matured, can provide. It becomes similar to what today is labeled 'serverless' computing. Things that are at the moment possible through large providers such as Amazon Lambda and Microsoft Azure Functions will also be provided via block chain networks. One difference is that the computation is not done by a single cloud entity, but rather by a collection of nodes that are run by individuals who have mining machines for that crypto. In my eyes it's still early days and certain things in the contracts can certainly be improved, e.g. the available APIs usable from within the smart contracts are still fairly limited, but that will probably improve over time. The fun thing is: it's pretty easy to get started experimenting and writing smart contracts, even from a simple Linux box, so you can learn and develop your smart contracts while the blockchain teams are working on maturing the infrastructure.

Tuesday, March 1, 2016

Adding Aspects to OSGi Services with no Bytecode Weaving

In my previous post I looked at how easy it is to apply the 'branch by abstraction' pattern to OSGi services, where the Service API is the abstraction layer and you can branch at runtime without even taking down the service client.
Another part of the story here is that it can be useful to add aspects to an implementation. This can also be done using the branch by abstraction pattern and in OSGi you can add these aspects to existing services without the need to modify them, and also without the need for bytecode manipulation. In OSGi you can proxy services by hiding the original and placing a proxy service in the service registry that consumers will use instead. This makes it possible to add a chain of aspects to the service invocations without the need to either change the clients nor the services themselves.

To illustrate, I started a little project called service-spector that allows you to add service aspects to any kind of service. The aspects themselves are also implemented as services. Here's what a ServiceAspect looks like:

public interface ServiceAspect {
  String announce();
  void preServiceInvoke(ServiceReference service, Method method, Object[] args);
  void postServiceInvoke(ServiceReference service, Method method, Object[] args, Object result);
  String report();
}

Aspects are called before and after a service invocation is done via preServiceInvoke()/postServiceInvoke(). They are called with a reference to the service being invoked and with the parameters that are provided to the service. The post call is also provided with the result of the invocation. The aspect API allows you to do things like obtaining metrics for services, add logging to existing service calls or anything else really that an aspect could do. Additionally it contains annouce() and report() to present information to the user.

The service-spector is configured using Configuration Admin with a list of filters for services that need have the aspects applied in the service.filters property. There is an additional configuration item that states whether the original services need to be hidden. The proxies are automatically registered with a higher service ranking than the original service. If the consumer of the original only uses a single service the original does not need to be hidden as the one with the higher ranking will be selected. However, if a consumer uses all the services of a certain type you do want to hide the originals. In general it's safer to hide the original service, but if you want to run without the HidingHooks you can switch this off.

I am using the Felix File Install to provide the configuration and have it stored in a file in load/org.coderthoughts.service.spector.impl.ServiceSpector.config A little known feature of File Install is that is supports typed configuration data in .config files, so the following file contains a Boolean hide.services property and a String[] service.filters property.

hide.services=B"true"
service.filters=["(objectClass\=org.coderthoughts.primes.service.PrimeNumberService)",
    "(objectClass\=org.apache.felix.bundlerepository.RepositoryAdmin)"]

In the ServiceSpector main component I can easily consume this configuration using the new Declarative Services annotation based approach:

@Component(immediate=true)
public class ServiceSpector implements ServiceListener {
  // This API is populated by Config Admin with configuration information
  @interface Config {
    String [] service_filters();
    boolean hide_services() default false;
  }

...

  @Activate
  private synchronized void activate(BundleContext bc, Config cfg) {
    ...
    System.out.println("Service filters: " +
      Arrays.toString(cfg.service_filters()));
    System.out.println("Hide services: " + cfg.hide_services());
  }

The Config annotation is automatically populated with the configuration information by matching the property keys with the annotation methods, which I can easily read out using its typed API (the dots in configuration keys are mangled to underscores).

Service-spector has a Service Listener that looks for services appearing and checks if they need to have aspects applied. If so they will be proxied. Then, when the proxied service is invoked, all the ServiceAspect services are called to let them do their thing. For example the CountingServiceAspect simply counts how often a certain API is invoked: 

@Component(scope=ServiceScope.PROTOTYPE)
public class CountingServiceAspect implements ServiceAspect {
  Map<String, LongAdder> invocationCounts = new ConcurrentHashMap<>();

  @Override
  public String announce() {
    return "Invocation counting started.\n";
  }

  @Override
  public void preServiceInvoke(ServiceReference service, Method method, Object[] args) {
    Class declaringClass = method.getDeclaringClass();
    String key = declaringClass.getSimpleName() + "#" + method.getName();
    invocationCounts.computeIfAbsent(key, k -> new LongAdder()).increment();
  }

  @Override
  public void postServiceInvoke(ServiceReference service, Method method, Object[] args, Object result){
    // nothing to do
  }

  @Override
  public String report() {
    StringBuilder sb = new StringBuilder();
    sb.append("Invocation counts\n");
    sb.append("=================\n");
    for (Map.Entry<String, LongAdder> entry : invocationCounts.entrySet()) {
      sb.append(entry.getKey() + ": " + entry.getValue() + "\n");
    }
    return sb.toString();
  }
}


The CountingServiceAspect is included in the Service Spector bundle. Lets add this and the File Install and Configuration Admin bundles to the framework setup used in the previous post that computes primes. I now have the following bundles installed. I removed bundle 9, the incorrect prime number generator that returns only 1's for now.

g! lb
START LEVEL 1
ID|State     |Level|Name
 0|Active    | 0|System Bundle (5.4.0)|5.4.0
 1|Active    | 1|Apache Felix Bundle Repository (2.0.6)|2.0.6
 2|Active    | 1|Apache Felix Gogo Command (0.16.0)|0.16.0
 3|Active    | 1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
 4|Active    | 1|Apache Felix Gogo Shell (0.10.0)|0.10.0
 5|Active    | 1|Apache Felix Declarative Services (2.0.2)|2.0.2
 6|Active    | 1|service (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
 7|Active    | 1|impl (1.0.1.SNAPSHOT)|1.0.1.SNAPSHOT
 8|Installed | 1|client (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
10|Active    | 1|Apache Felix Configuration Admin Service (1.8.8)|1.8.8
11|Active    | 1|Apache Felix File Install (3.5.2)|3.5.2
12|Active    | 1|service-spector (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT


Additionally I have the configuration file from above in the ./load/org.coderthoughts.service.spector.impl.ServiceSpector.config location. This is the default location where the Felix File Install looks. When I start the service-spector bundle it reports the active aspects and its configuration:

g! start 12
Invocation counting started.

Service Spector
===============
Service filters: [(objectClass=org.coderthoughts.primes.service.PrimeNumberService), (objectClass=org.apache.felix.bundlerepository.RepositoryAdmin)]
Hide services: true


Ok so let's start our client:
g! start 8
Services now: $Proxy6(PrimeNumbers)
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! stop 8


The version of primes on master has a PrimeServiceReporter that reports the name and the classname of the service. Here we can see that the service calls itself "PrimeNumbers" but that the actual service instance is a Proxy.

As our filters state that we are also proxying the Repository Admin which comes with the Felix implementation we can also exercise that, for example via the shell commands:
 

g! obr:repos list
... some output ...
g! obr:list
... some output
 

Ok so we've now used both of the services that are being proxied. Let's see what APIs have been utilized. The service spector calls report() on all the ServiceAspects when the bundle is being stopped:

g! stop 12
Invocation counts
=================
PrimeNumberService#getServiceName: 1
PrimeNumberService#nextPrime: 30
RepositoryAdmin#listRepositories: 1
RepositoryAdmin#discoverResources: 2

Contributing more aspects

Aspects are contributed as services themselves so they don't need to be part of the service-spector bundle itself. The service-spector-method-timer bundle is a separate bundle that contributes a ServiceAspect that provides average method invocations times.

Get the binaries

The code in this blog post can easily be built using maven, but if you just want the binaries, you can get them from here:
service-spector: https://github.com/coderthoughts/service-spector/releases
method-timer: https://github.com/coderthoughts/service-spector-method-timer/releases

Monday, February 8, 2016

Branch by Abstraction and OSGi

Inspired by my friend Philipp Suter who pointed me at this wired article http://www.wired.com/2016/02/rebuilding-modern-software-is-like-rebuilding-the-bay-bridge which relates to Martin Fowler's Branch by Abstraction I was thinking: how would this work in an OSGi context?

Leaving aside the remote nature of the problem for the moment, let's focus on the pure API aspect here. Whether remote or not really orthogonal... I'll work through this with example code that can be found here: https://github.com/coderthoughts/primes

Let's say you have an implementation to compute prime numbers:
public class PrimeNumbers {
  public int nextPrime(int n) {
    // computes next prime after n - see
https://github.com/coderthoughts/primes details
    return p;
  }
}
And a client program that regularly uses the prime number generator. I have chosen a client that runs in a loop to reflect a long-running program, similar to a long-running process communicating with a microservice:
public class PrimeClient {
  private PrimeNumbers primeGenerator = new PrimeNumbers();
  private void start() {
    new Thread(() -> {
      while (true) {
        System.out.print("First 10 primes: ");
        for (int i=0, p=1; i<10; i++) {
          if (i > 0) System.out.print(", ");
          p = primeGenerator.nextPrime(p);
          System.out.print(p);
        }
        System.out.println();
        try { Thread.sleep(1000); } catch (InterruptedException ie) {}
      }
    }).start();
  }
 
  public static void main(String[] args) {
    new PrimeClient().start();
  }
}
If you have the source code cloned or forked using git, you can run this example easily by checking out the stage1 branch and using Maven:
.../primes> git checkout stage1
.../primes> mvn clean install
... maven output
[INFO] ------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------
Then run it from the client submodule:
.../primes/client> mvn exec:java -Dexec.mainClass=\
org.coderthoughts.primes.client.PrimeClient
... maven output
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
... and so on ...
Ok so our system works. It keeps printing out prime numbers, but as you can see there is a bug in the output. We also want to replace it in the future with another implementation. This is what the Branch by Abstraction Pattern is about.

In this post I will look at how to do this with OSGi Services. OSGi Services are just POJOs registered in the OSGi Service Registry. OSGi Services are dynamic, they can come and go, and OSGi Service Consumers dynamically react to these changes, as well see. In the following few steps will change the implementation to an OSGi Service. Then we'll update the service at runtime to fix the bug above, without even stopping the service consumer. Finally we can replace the service implementation with a completely different implementation, also without even stopping the client.

Turn the application into OSGi bundles

We'll start by turning the program into an OSGi program that contains 2 bundles: the client bundle and the impl bundle. We'll use the Apache Felix OSGi Framework and also use OSGi Declarative Services which provides a nice dependency injection model to work with OSGi Services.

You can see all this on the git branch called stage2:
.../primes> git checkout stage2
.../primes> mvn clean install
The Client code is quite similar to the original client, except that it now contains some annotations to instruct DS to start and stop it. Also the PrimeNumbers class is now injected instead of directly constructed via the @Reference annotation. The greedy policyOption instructs the injector to re-inject if a better match becomes available:
@Component
public class PrimeClient {
  @Reference(policyOption=ReferencePolicyOption.GREEDY)
  private PrimeNumbers primeGenerator;
  private volatile boolean keepRunning = false;
 
  @Activate
  private void start() {
    keepRunning = true;
    new Thread(() -> {
      while (keepRunning) {
        System.out.print("First 10 primes: ");
        for (int i=0, p=1; i<10; i++) {
          if (i > 0) System.out.print(", ");
          p = primeGenerator.nextPrime(p);
          System.out.print(p);
        }
        System.out.println();
        try { Thread.sleep(1000); } catch (InterruptedException ie) {}
      }
    }).start();
  }
 
  @Deactivate
  private void stop() {
    keepRunning = false;
  }
}
The prime generator implementation code is the same except for an added annotation. We register the implementation class in the Service Registry so that it can be injected into the client:
@Component(service=PrimeNumbers.class)
public class PrimeNumbers {
  public int nextPrime(int n) {
    // computes next prime after n
    return p;
  }
}
As its now an OSGi application, we run it in an OSGi Framework. I'm using the Apache Felix Framework version 5.4.0, but any other OSGi R6 compliant framework will do.
> java -jar bin/felix.jar
g! start http://www.eu.apache.org/dist/felix/org.apache.felix.scr-2.0.2.jar
g! start file:/.../clones/primes/impl/target/impl-0.1.0-SNAPSHOT.jar
g! install file:/.../clones/primes/client/target/client-0.1.0-SNAPSHOT.jar
Now you should have everything installed that you need:
g! lb
START LEVEL 1
ID|State |Level|Name
0|Active | 0|System Bundle (5.4.0)|5.4.0
1|Active | 1|Apache Felix Bundle Repository (2.0.6)|2.0.6
2|Active | 1|Apache Felix Gogo Command (0.16.0)|0.16.0
3|Active | 1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
4|Active | 1|Apache Felix Gogo Shell (0.10.0)|0.10.0
5|Active | 1|Apache Felix Declarative Services (2.0.2)|2.0.2
6|Active | 1|impl (0.1.0.SNAPSHOT)|0.1.0.SNAPSHOT
7|Installed | 1|client (0.1.0.SNAPSHOT)|0.1.0.SNAPSHOT
We can start the client bundle:
g! start 7
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
... and so on ..
You can now also stop the client:
g! stop 7
Great - our OSGi bundles work :)
Now we'll do what Martin Fowler calls creating the abstraction layer.

Introduce the Abstraction Layer: the OSGi Service

Go to the branch stage3 for the code:
.../primes> git checkout stage3
.../primes> mvn clean install
The abstraction layer for the Branch by Abstraction pattern is provided by an interface that we'll use as a service interface. This interface is in a new maven module that creates the service OSGi bundle.
public interface PrimeNumberService {
    int nextPrime(int n);
}
Well turn our Prime Number generator into an OSGi Service. The only difference here is that our PrimeNumbers implementation now implements the PrimeNumberService interface. Also the @Component annotation does not need to declare the service in this case as the component implements an interface it will automatically be registered as a service under that interface:
@Component
public class PrimeNumbers implements PrimeNumberService {
    public int nextPrime(int n) {
      // computes next prime after n
      return p;
    }
}
Run everything in the OSGi framework. The result is still the same but now the client is using the OSGi Service:
g! lb
START LEVEL 1
   ID|State      |Level|Name
    0|Active     |    0|System Bundle (5.4.0)|5.4.0
    1|Active     |    1|Apache Felix Bundle Repository (2.0.6)|2.0.6
    2|Active     |    1|Apache Felix Gogo Command (0.16.0)|0.16.0
    3|Active     |    1|Apache Felix Gogo Runtime (0.16.2)|0.16.2
    4|Active     |    1|Apache Felix Gogo Shell (0.10.0)|0.10.0
    5|Active     |    1|Apache Felix Declarative Services (2.0.2)|2.0.2
    6|Active     |    1|service (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
    7|Active     |    1|impl (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
    8|Resolved  |    1|client (1.0.0.SNAPSHOT)|1.0.0.SNAPSHOT
g! start 8
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
You can introspect your bundles too and see that the client is indeed wired to the service provided by the service implementation:
g! inspect cap * 7
org.coderthoughts.primes.impl [7] provides:
-------------------------------------------
...
service; org.coderthoughts.primes.service.PrimeNumberService with properties:
   component.id = 0
   component.name = org.coderthoughts.primes.impl.PrimeNumbers
   service.bundleid = 7
   service.id = 22
   service.scope = bundle
   Used by:
      org.coderthoughts.primes.client [8]
Great - now we can finally fix that annoying bug in the service implementation: that it missed 2 as a prime! While we're doing this we'll just keep the bundles in the framework running...

Fix the bug in the implementation whitout stopping the client

The prime number generator is fixed in the code in stage4:
.../primes> git checkout stage4
.../primes> mvn clean install
It's a small change to the impl bundle. The service interface and the client remain unchanged. Let's update our running application with the fixed bundle:
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
First 10 primes: 3, 5, 7, 11, 13, 17, 19, 23, 29, 31
g! update 7 file:/.../clones/primes/impl/target/impl-1.0.1-SNAPSHOT.jar
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
Great - finally our service is fixed! And notice that the client did not need to be restarted! The  DS injection, via the @Reference annotation, handles all of the dynamics for us! The client code simply uses the service as a POJO.

The branch: change to an entirely different service implementation without client restart

Being able to fix a service without even restarting its users is already immensely useful, but we can go even further. I can write an entirely new and different service implementation and migrate the client to use that without restarting the client, using the same mechanism. 

This code is on the branch stage5 and contains a new bundle impl2 that provides an implementation of the PrimeNumberService that always returns 1. 
.../primes> git checkout stage5
.../primes> mvn clean install
While the impl2 implementation obviously does not produce correct prime numbers, it does show how you can completely change the implementation. In the real world a totally different implementation could be working with a different back-end, a new algorithm, a service migrated from a different department etc...

Or alternatively you could do a façade service implementation that round-robins across a number of back-end services or selects a backing service based on the features that the client should be getting.
In the end the solution will always end up being an alternative Service in the service registry that the client can dynamically switch to.

So let's start that new service implementation:
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! start file:/.../clones/primes/impl2/target/impl2-1.0.0-SNAPSHOT.jar
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
g! stop 7
First 10 primes: 2, 3, 5, 7, 11, 13, 17, 19, 23, 29
First 10 primes: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
First 10 primes: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
Above you can see that when you install and start the new bundle initially nothing will happen. At this point both services are installed at the same time. The client is still bound to the original service as its still there and there is no reason to rebind, the new service is no better match than the original. But when the bundle that provides the initial service is stopped (bundle 7) the client switches over to the implementation that always returns 1. This switchover could happen at any point, even halfway thought the production of the list, so you might even be lucky enough to see something like:
First 10 primes: 2, 3, 5, 7, 11, 13, 1, 1, 1, 1
I hope I have shown that OSGi services provide an excellent mechanism to implement the Branch by Abstraction pattern and even provide the possibility to do the switching between suppliers without stopping the client!

In the next post I'll show how we can add aspects to our services, still without modifying or even restarting the client. These can be useful for debugging, tracking or measuring how a service is used.

PS - Oh, and on the remote thing, this will work just as well locally or Remote. Use OSGi Remote Services to turn your local service into a remote one... For available Remote Services implementations see https://en.wikipedia.org/wiki/OSGi_Specification_Implementations#100:_Remote_Services

With thanks to Carsten Ziegeler for reviewing and providing additional ideas.

Tuesday, May 19, 2015

My favourite free Mac OSX apps

I have been a Windows guy for many, many years, starting with Windows 2.x in the late 1980's all the way up to Windows 7. Then, about 4 years ago I gave up. Not because Windows was so bad (I actually quite liked Windows 7) but because at the time I could not find any decent laptop for it! So I got a Mac in 2011. And, over the years I got to like it, especially the hardware. I was always a big fan of opensource and freeware/shareware which was always widely available for Windows. Over the past years a collected some good legal free software for the Mac and I thought I'd share with everyone what I like. The interesting bit here is that most of this stuff is not actually distributed via the Apple Appstore...

Here's my list in no particular order, starting with a bundle of freely available utilities...

Handbrake



Great video converter tool. Convert any videos that you might have into any other format. BTW if you're looking for good settings that give you videos that play/stream on most devices and are relatively modest in size, I use the following settings (everything not mentioned is left at default):
  • Format: MP4, Web Optimized
  • Picture: Size: source = destination, Cropping: automatic, Anamorphic: none, Filters: all off
  • Video: Codec: H.264, FPS: Same as source-variable, Quality: RF 22, Use Advanced Options
  • Audio: AAC Stereo, Samplerate auto, Bitrate 128
  • Advanced: Reference Frames: 6, CABAC Entropy Encoding: yes, 8x8 transform: yes, Weighted P-Frames: yes, Subpixel ME & Mode: 2  SATD qpel, Trellis: off, No DCT Decimation: off.
Download from http://handbrake.fr.

VLC

The video player that plays everything. If you install the right drivers it can even play Blu Ray discs, something the native DVD player on OSX cannot do. Download from http://www.videolan.org/vlc.

Kodi


Previously called XBMC. Another media player. This one is really good for streaming over the network and has tons of plugins. You can also create a catalog that runs on a central server and stream that to all the devices in your house, for example. I wrote some details on how I use my Raspberry PI as such a media server here: http://coderthoughts.blogspot.com/2014/01/how-i-learned-to-stop-worrying-about.html. Download from http://kodi.tv.

BetterTouchTool

This tool has a lot of features for the touchpad, but the main reason why I use it is not on the touchpad. I don't like the click feature of the touchpad, the fact that you have to press the touchpad itself. I prefer pressing a button to act as a mouseclick. On my UK keyboard there is a key that I never use, just above the TAB key there is a key with the § sign. The above screenshot shows how I used the BetterTouchTool to remap that key to be the mouseclick. I still use the trackpad to position my mouse, but no need to press it down, I do that with my other hand by pressing that mapped key. Download from here: http://blog.boastr.net
BetterTouchTool also has Windows 7-like window snapping functionality, which is really handy too. Although I personally use Breeze for that, which has more features in this area. It's not a free tool so it doesn't belong in this list, but its very cheap so worth checking out :)

Flycut

This is a very handy little tool, that simply extends the clipboard with a history of up to 25 entries. Instead of cmd-V, use shift-cmd-V to cycle through all your past clipboard contents. Extremely useful. This one is free on the apple appstore: https://itunes.apple.com/ie/app/flycut-clipboard-manager/id442160987?mt=12.

Zipeg

This is a free WinZip-like tool for the Mac. It doesn't have as many features as WinZip, but I tend to find it very good for quickly peeking in a .zip or .jar file. And the program icon is just really cute! Download from: http://www.zipeg.com.

MacVim

I'm a VIM guy and this Mac version is really good! Download from https://code.google.com/p/macvim.

KeePassX

I'm relatively new to password management software but the recent vulnerabilities and attacks made me move to it. I'm a bit weary of storing my passwords in a cloud-based solution. I'd rather have a local file that's properly encrypted, so that it doesn't matter who actually sees that raw file. I also want something that automatically works with web browsers so that passwords are automatically entered. KeePass is a great Windows-based app that does all that. There's an Android version (KeePassDroid) that even works on my BlackBerry Z10. And there are Chrome (chromeIPass) and Firefox (PassIFox) integrations with the KeePass HTTP server that seamlessly enter your passwords in the appropriate websites if you approve this action.
There are a number of opensource projects to get this working on the Mac too. However, they seem to be in various stages of incompleteness. There is the KeePassX project, but it doesn't have support for the HTTP server (yet) that you need to integrate with the browser. There is a new project called MacPass which is another KeePass clone, but it also doesn't support the HTTP server (yet). Some people use Mono to run the Windows KeePass on OSX, but I found that not very satisfying and at some point it just started crashing continuously. In the end I found Keith Bennett's fork of KeePassX 2.0Alpha5 that has HTTP server support. The UI is a bit buggy, but it's the best I could find in the opensource arena for password management on OSX, so this is what I'm using right now. Looking forward to having a more stable version of KeePass on the Mac with HTTP server support.

PDF Split and Merge

I love PDFs :) (hey I work for Adobe). And there are tons of tools available for creating PDFs both from Adobe and from other vendors. PDFSam is a really nice little tool that helps you work with PDFs once they are created. Rearrange pages, split chapters off, merge multiple PDFs together, that kind of thing. I find myself using this tool pretty often. You can get it from here: http://www.pdfsam.org.

OSX Screen Saver runner

Ok, should maybe not be in this list, because it's not freeware or opensource, as it's shipped as part of OSX itself but it's free in that everyone who has OSX already has it, but you may not be aware of its existence. This is just a really simple way to lock your screen, if you need to walk away from your computer, for example. You simply find it on your local disk at /System/Library/Frameworks/ScreenSaver.framework/Versions/A/Resources/ScreenSaverEngine.app
and I normally have it locked to my dock, so I can just click it to lock my screen.

Applications

There are also a few larger applications where I think there are some great opensource products available that I use on a daily basis.

Openoffice / Libreoffice

Personally I prefer this office suite to the commercial ones available. Great features, great usability and it works on all the platforms that I use outside of OSX, including Linux. Download from https://www.openoffice.org.

Thunderbird and Firefox


These two need no further introduction. Just an excellent webbrowser and a great email client. Get them at https://www.mozilla.org.

Eclipse

Still the best IDE around for Java and other programming languages. Available from http://www.eclipse.org.

That's it - these are my favourite opensource / freeware / shareware apps that I use all the time. Enjoy :)

Wednesday, May 28, 2014

Running bndtools OSGi builds with Maven - bnd-maven-plugin now released!

The bndtools.org project provides some really great tooling for building OSGi bundles. This includes help for generating the bundle manifest, templates for working with Declarative Services, testing support and more. There's tooling for resolving deployments against repositories, and last but certainly not least - automatic semantic versioning. The tooling for semantic versioning will tell you whether the version of your exported packages is correct, by comparing the content of your APIs with the previous release of the bundle.
I won't go into the full feature set of bndtools here, have a look at the bndtools website for all the goodness. Another thing to say about bndtools is that it has a really active community, which is always a good thing to know when you're deciding on using an opensource project.

Example IDE screen, taken from bndtools.org
The bndtools project itself focuses on providing an Eclipse-based graphical IDE for developing OSGi bundles. However you normally also want to be able to build your bundles in a headless build. For example as part of a Jenkins/Hudson process or simply to build it from the command-line yourself.
The bndtools project provides headless build options using Ant and Gradle, but for many people being able to use Maven for builds is essential.

Most OSGi development in Maven is done these days using the maven-bundle-plugin or using Eclipse Tycho. Tycho focuses on building OSGi bundles using a manifest-first approach which is supported by the Eclipse PDE tooling. 
The maven-bundle-plugin uses the pom.xml file as the source of information for the bundle and while it uses bnd under the covers, just like the bndtools project, maven-bundle-plugin is a little different in that it uses the pom.xml as the source of information for the build. The bndtools IDE uses bnd.bnd files for this.

This is where the bnd-maven-plugin comes in. Toni Menzel started it a while ago, Peter Kriens worked on it, and I have been putting bits and pieces in recently. The bnd-maven-plugin basically builds a bndtools project exactly like the bndtools IDE would build it, but then from Maven. As of today the first version of the bnd-maven-plugin  (called 1.0.2 - mea culpa ;) has been released to Maven Central, so you can easily use it in your Maven builds.

Precise instructions on how to use it can be found in the project readme, but you can easily try it out by looking at the bnd-maven-plugin samples in github. They show what your pom should look like. The samples contain the following test projects:
TestBundle - this is a simple default-layout bndtools project that can be built with Maven.
TestBundle2 - a bndtools project that follows Maven conventions for directory locations of source and output files. It can be built using Maven and the bndtools IDE. This project also contains an example unit test.
TestBundle3 - this project contains an OSGi Framework integration test, which can be run both from Maven and Bndtools.
Additionally, the sample structure contains
ParentPom - a module that contains a parent pom used by all the other submodules.
MBPBundle - a module that builds a bundle using the maven-bundle-plugin, showing that it can be used alongside the bnd-maven-plugin, although within a single module you need to choose one or the other.

To run it with Maven you can simply go into the parent directory and execute mvn install:
$ .../samples/sample-projects $ mvn install
... lots of maven output ...
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ TestBundle2 ---
[INFO] Surefire report directory: /Users/David/clones/bnd/bnd-maven-plugin-parent/samples/sample-projects/TestBundle2/target/surefire-reports
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.foo.bar.UnitTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in org.foo.bar.UnitTest
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
...
[INFO] --- bnd-maven-plugin:1.0.2:integration-test (default-integration-test) @ TestBundle3 ---
[INFO] Running the Bnd OSGi integration tests
Tests run  : 1
Passed     : 1
Errors     : 0
Failures   : 0
No Errors
...
[INFO] Finished running the Bnd OSGi integration tests
[INFO] ----------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ParentPom ......................................... SUCCESS [0.185s]
[INFO] TestBundle ........................................ SUCCESS [4.134s]
[INFO] TestBundle2 ....................................... SUCCESS [0.361s]
[INFO] TestBundle3 ....................................... SUCCESS [1.041s]
[INFO] MBPBundle ......................................... SUCCESS [0.842s]
[INFO] sample-projects ................................... SUCCESS [0.041s]
[INFO] ----------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ----------------------------------------------------------------------
[INFO] Total time: 11.707s

The above has built all modules, ran the unit tests in TestBundle2 and the integration tests in TestBundle3.

Use them with the bndtools IDE!

As the TestBundle, TestBundle2 and TestBundle3 projects are bndtools Eclipse projects, they can be imported via File -> Import -> Existing Projects into Workspace (along with the bnd cnf project):

Then you can use the IDE for all your dev needs, e.g. to execute the OSGi integration tests:


This is only the first release, but should be good enough to start playing with it. See if it works for you. Give us feedback! Help developing it further! You can fork, file issues and patches in the bndtools/bnd project on github. Enjoy :)

Tuesday, March 25, 2014

Apache Felix Framework 4.4.0 provides full OSGi Core R5 support

Felix Framework 4.4.0 was released today (http://felix.apache.org/news.html) and now provides for OSGi Core R5 support. From a features point of view, it means that the following R5 features are now fully supported:
  • New org.osgi.framework.UnfilteredServiceListener interface
  • New org.osgi.framework.VersionRange class
  • The Resource API and it's use in the Wiring API
  • New osgi.identity namespace
  • New value and new default for org.osgi.framework.bsnversion framework property
  • support for static valueOf methods in the Filter
  • Enhanced Bundle.adapt() to support AccessControlContext
  • Updates to the Bundle Hook Specification (Bundle Collision Hook)
But the main benefit as I see it that we can now run OSGi Subsystems, such as the Apache Aries implementation, on Apache Felix! For more details on that see this blog post: http://coderthoughts.blogspot.com/2014/01/osgi-subsytems-on-apache-felix.html

Tuesday, January 28, 2014

How I learned to stop worrying (about power cuts) and love my Raspberry PI

As many other hobbyists I got myself a Raspberry PI a few months ago. It's very cool that such a functional piece of hardware can be so cheap. I'm using mine as a backup server for my photos, documents, videos etc, it is my Samba Server, FTP server and, most importantly an XBMC network server.

I'll outline some of the things that I did to make mine working the way I wanted to, but first I'd like to address the biggest issue that I experienced with my Raspberry PI: the fact that it doesn't deal well with power cuts. The PI itself isn't really to blame for that, its a general issue with Linux and other unix-type operating systems: sudden power losses can make their boot/os disk unreadable. With Linux running on a laptop you don't really have that issue, as your laptop has a built-in UPS, but my PI doesn't have that. Which means that I often happened to find my device in a state that it wouldn't boot up again. Apparently we had a power blip.
While some companies are starting to sell UPS solutions for the PI, one of the main issues with that is that the UPS often costs more than the PI itself. If you don't want to spend that extra money, I have used a different setup that gave me the capability to survive PI boot disk corruptions without too much work or cost.

Backup your PI boot SD card

Some people suggested that the best way to prevent your PI from corrupted boot disks is to make the disk readonly. While certainly a good idea, this doesn't really work for me as I often experiment with my PI and change its installed packages regularly. So I didn't want to restrict myself that much.
What I did instead was use Billw's cloning script to make a backup copy of the main SD card on a second one. I used an old SD card that I still had lying around and got a USB SD card adapter, which sells on ebay for about £1 including shipping. Using that script I make regular backups of my main SD card onto its clone. When experiencing one of those regular power blips and the PI doesn't want to start up, I simply swap the SD cards - the one from the USB goes into the main SD card port and vice-versa! That will allow me to start up the PI again with all my configuration as I had it. Then simply run Billw's script again to clone the SD card that's still working back on the corrupted one at which point I'll have two working SD cards again!
I've used this mechanism for a few months now and it works pretty well. Certainly much cheaper than a UPS and although a little bit of work (you need to ensure that the backup SD-card stays up-to-date) I found that it worked great. You can also automate the cloning so that it happens automatically every night or so...

In addition to the backup script there are a few other things that I tweaked for my PI. They might be a little different than what you're used to on Linux, so I'm outlining them here.

Get a Hard Disk with power-save

If you're planning to use your PI as a file server, like I do, get a USB harddrive that has a power-save feature. This means that you can have your system running 24/7 but that there won't be any spinning parts if nobody's using it, and very low power consumption too in that case. I got this Seagate 2TB one for about £60 which works great! The only thing you need to do is enable the power save feature once, which you have to do on a Windows PC using the program that comes on the drive. Once power-save enabled, you can connect it to the PI and it works perfectly. Note that you do need to sudo apt-get install ntfs-3g to use most pre-formatted USB drives.

Use UUIDs to mount your drives

Another issue that I was experiencing was that my USB drives didn't always get the same disk identifier after a PI reboot. One disk would sometimes be sdb and other times it would be sdc, for example. This is problematic if you use the standard 
  /dev/sda1 /disks/x1 auto noatime 0 0
in your /etc/fstab to mount your drives to a directory, as your mounted directory would sometimes be mounted to one disk and sometimes to the other...
A good solution to that is to mount your drives by UUID instead. If you look in /dev/disk/by-uuid/ you can see the UUIDs of all the available disks. Find the right one and use that to mount your disk. The following makes sure that always the same disk is mounted to /disks/x1:
  /dev/disk/by-uuid/5A22DB2D2DB0CBF /disks/x1 auto noatime 0 0

Rotate the logs

If you're doing a lot of file transfer to-from your PI, the Samba/FTP log files can get pretty big. If you, like me, use a crappy old 4GB SD card to run off, it might make sense to keep the size of the log files down. Logrotate allows you to do this, and the standard Raspbian distribution comes with this installed. I just tweaked it to keep the log files smaller, by only remembering 1 week of logs. To do this, you tweak /etc/logrotate.conf, I'm using these settings:
  # rotate log files weekly
  weekly

  # keep 1 weeks worth of backlogs
  rotate 1

  # create new (empty) log files after rotating old ones
  create

  # compress log files
  compress


Serve XBMC throughout the house

Many people use the PI to run XBMC directly. XBMC is a wonderful media playback application that gives you a really nice interface to your media collection. And a PI connected to an audio/video system can certainly be used for running it. However, I was looking for something different. I wanted to use my video library on any device in the house. Maybe a laptop, maybe our multimedia pc which is connected to the tv, maybe even on an Android smartphone. XBMC has features that allow you to do this. To get this working, you don't actually need XBMC on the Raspberry PI at all! You need the video files on it, a network file access protocol, such as Samba, and an installation of MySQL which is what the XBMC installations on the client computers communicate with to show your video library etc. The whole setup is really nice as it allows you to watch your videos on any device in the house and it even remembers where you left off if you stopped halfway, so that you can easily continue watching on another computer. Once you've installed Samba and MySQL for XBMC on the PI, you need to install XBMC on all the client devices that you want to use. There are builds available for most platforms. Next you need to configure XBMC to use MySQL. I actually use different profiles to organize my video files, for example you may have holiday videos in one and educational videos in another profile (or any other separation of your choice ;). You can do this by putting the advancedsettings.xml file in the .../userdata/profile/<Profile Name> directory and name the database in it to separate the content, like this:
  <advancedsettings>
    <videodatabase>
      <type>mysql</type>
      <host>my_pi_host</host>
      <port>3306</port>
      <user>mysql_username</user>
      <pass>mysql_password</pass>
      <name>my_holiday_movies</name>
    </videodatabase>
  </advancedsettings>
The name tag in the configuration is used to connect to separate MySQL databases running in the same database server. That allows you to serve different media depending on the XBMC profile you're in. In the Holidays profile it lists my holiday videos, in the Education profile it has my educational ones.

Finally, when you add videos to your database, make sure to select add the directory on the PI using a network protocol that works on all your devices. For example a Samba URL: 
  smb://my_pi_host/share/holiday/videos/
And voilĂ , you will get all your movies on all your XBMC devices.

Ok, so this was a random collection of bits and pieces that I did to get my Raspberry PI to do what I wanted. If you have a nugget of PI goodness, leave a comment to share :)