More on Constructors

Warning: This post contains fictional grammar, ideas off the cuff and high levels of programmer nerdery. Proceed at your own risk. The whole shebang was started off here: If I Would Design a Language.

Here’s some more you could do with implicit constructors. Remember this class?

class Greeter(String name = "stranger") {

    function greet() returns Void {
        echo("Hello #{$name}!");
    }
}

It turns out that the constructor argument “name” live on within the scope of the class. That’s interesting, because one of the main pains in Java is beans, and bean accessors. We’ve already seen that we can alleviate that with field access. But how about making it even simpler. Have a look at the following, mutable and completely awesome bean:

@Mutable
class Person(@Field String firstName, @Field String lastName) { }

Adding the “@Field” annotation generates automatic fields, including getters for the argument, and since he class is mutable, it also generates setters. The above would be equivalent to the following Java bean:

public class Person {

    private String firstName;
    private String lastname;

    public Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }
}

The amount of crud reduced is ridiculous. There is of course the problem that if you mix “@Field” annotations with ordinary arguments and an explicit constructor it might get too messy. But hey, it’s a neat idea.

If I Would Design a Language

Warning: This post contains fictional grammar, ideas off the cuff and high levels of programmer nerdery. Proceed at your own risk.

I’ve had a look at the influx of new languages for the JVM the last couple of months. In particular I’ve read up a bit on Kotlin, Fantom and Clojure. And naturally I started thinking about what I would do myself if I designed a new language.

Well… the world don’t need a new computer language. But nevertheless, here’s what I’m thinking right now.

Main Ideas

Here’s what I’d aim for primarily:

  • Readability: In real software projects in real life you need to understand other peoples code. This means you have to trade-off write-ability for read-ability. It also limits you in terms of variations: there should only ever be one way of expressing any particular statement. It also means I’d probably skip type inference, but more on that later.
  • Modularity: Ceylon and Fantom all have it right, modularity should be build in from the start. However, it seems to me they are missing a very obvious extension to the language level modularity, which is…
  • Dependency Injection: This is a kicker, as I don’t know of any language that have it built in. Think of Guice but as a core feature of the language itself. If this could be possible to combine with the language modularity, you could a potentially awesome productivity right from the start.
  • Concurrency: I’m a server kind of guy, this is important. I think Fantom is on the right track so I’d probably end up with something similar. Ie. an actor style pattern where the only allowed interaction between actors are immutable objects. In fact, I think immutability is going to be a main feature as well…
  • Immutability: Immutability enables cool stuff: Clojure has this down pat. And although I’m not sure at this point, I think all classes will be immutable by default, and you’d have to declare a class if you want it mutable. People would hate it though.

Hello World

Let’s get right down to it, shall we? We’ll do the standard “hello world” a bit more complicated immediately to highlight a bit more syntax:

/*
 * class is 'public' and immutable by default, the constructor
 * signature is immediately visible and has a default value
 */
class Greeter(String name = "stranger") {

    /*
     * member is 'public' and has an automatic accessor, it
     * is also initiated with the value from the constructor
     */
    String name = $name;

    /*
     * function is 'public' by default
     */
    function greet() returns Void {
        /*
         * 'echo' is a short hand for system out, and
         * strings are evaluated
         */
        echo("Hello #{name}!");
    }
}

So what do we have here? Let’s break it down.

Constructor

The constructor signature is declared immediately after the class name. It can contain default values for arguments. The actual implementation is done either implicitly, or in a “constructor” statement:

class Greeter(String name = "stranger") {

    String name;

    constructor {
        /*
         * 'this' denotes a raw field access and is only
         * allowed in constructors and setters
         */
        this.name = $name;
    }
}

Note that this constructor is equivalent with the previous example. Does this mean that a class can only have one constructor? Yes it does, but with default values for arguments, and named argument passing it should be enough.

And by the way, the constructor arguments are accessible through the entire class. So the greeter can be simplified:

class Greeter(String name = "stranger") {

    function greet() returns Void {
        echo("Hello #{$name}!");
    }
}

Pretty cool huh? This means there’s an implicit field for each constructor argument in each class, which is readable but not writable.

Class Immutability and Field Access

I’ll leave the collection classes for now, but ordinary classes are immutable by default, and all fields have a generated “getter”, and optionally a “setter” as well.  So, to start with, this would be illegal:

Greeter greeter = new Greeter();
greeter.name = "kalle"; // illegal!

If we truly want an object that can be mutated, we’d have to declare it as such:

@Mutable
class Greeter(String name = "stranger") {

    String name = $name;

}

And now, we can change it post-construction:

Greeter greeter = new Greeter();
greeter.name = "Adam"; // legal!

Access to object fields are done via accessors that are generated automatically. If you want to change their behavior you can, but only the “setter”, the access is always reading the field as it is:

@Mutable
class Greeter(String name = "stranger") {

    String name = $name with setter {
        /*
         * 'this' denotes a raw field access and is only
         * allowed in constructors and setters, and there's an
         * implicit "argument" which is what the setter was
         * called with
         */
        this.name = name;
    }
}

But all objects have another implicit function attached to fields, namely the clone operator. With this you can get a clone of the current object with one or more fields changed. If our Greeter was imutable, we could do this:

Greeter greeter = new Greeter();
greeter = greeter.name -> "Adam";

Under the hood this created a new Greeter object which is equivalent with the first, except for the new name. It would be nice if you could chain clone operations as well, but I’ll get back to you on that.

Functions and High Order Functions

We’ve seen a function already and there’s not much to add. The declaration is in the form:

function <name>(<arguments>) returns <type> { }

To this we’ll add a few short-hands. OK, I know I said we’d only have one way of articulating statements. But hey, I think these may be worth it. Consider the following example:

function greet() returns Void {
    echo("Hello ${name}!");
}

Mr Eagle-eye will spot two possible ways of putting this simpler: Why use parenthesis when there’s no arguments, it’s pretty obviously a method declaration anyway. And why declare a “return” when it’s void any way. So…

function greet {
    echo("Hello ${name}!");
}

In addition function should be possible to pass around as any other object. These high order functions comes in two flavors “strict” and “relaxed”.

  • “strict” functions only accepts immutable arguments and can only return immutable objects
  • “relaxed” functions accepts mutable arguments and may return mutable objects

By default all functions are relaxed as this is a great deal easier to work with. But by declaring strict functions we’ll enable functional programming out of the box a such functions will be guaranteed not to have side effects.

Here’s a bit of syntax for you. First, let’s imagine a “foreach” declaration on an interface:

function foreach(function(T item) visitor) returns Void;

And in a list of Greeter objects you’d call it like so:

list.foreach(new function(Greeter greeter) {
    greeter.name = "Adam";
});

“Primitive” Types and Literals

Everything is objects, and no excuse. Integers and Doubles are 64 bit. Strings are UTF-16. Characters are not integers. We have Bytes. So without further ado, here’s the list of literals:

true; // boolean
123; // integer
12.3; // double
1b // byte
§0010 // bits literal
0xAB // integer in hex
'n' // character
"n" // string
`http://www.google.com` // uri
2s // duration (2 seconds)

The duration comes in the following flavors:

1ps // picosecond
1ms // millisecond
1s // second
1m // minute
1h // hour
1d // day
1w // week
1M // month
1Y // year

In addition to the ones above, here’s some shorthand, taken straight from Fantom:

[1, 2] // list
[1:"Adam", 2:"David"] // map

Modularity

A “module” represent a unit of code which belongs to a particular namespace, and which is versioned and is packaged must like a Java “jar” file, ie. in a ZIP archive together with some meta-data. I haven’t thought through the declaration syntax yet, but here’s what we want in a declaration:

  • Namespace
  • Module name
  • Version
  • Dependencies
  • DI Configuration

We’ll steal the reference syntax from Maven. So this…

net.larsan.test.lang:Core:1.1.0

… would be the namespace “net.larsan.test.lang”, the module “Core” and the version “1.1.0″.  The format of the version is probably fixed, but mayinclude a “-SNAPSHOT” postfix option.

Here’s something though: the declaration will not include build information such as “source folder” or similar. That’s mixing apples and oranges, and unfortunately both Fantom and Kotlin are doing it.

Let’s do some imaginary code:

module {
    namespace = new Namespace("net.larsan.test.lang");
    moduleName = "Core";
    version = new Version("1.1.0");
    addDependency("com.whatever:Module:2");
}

Now, the module syntax is probably just a shorthand for this:

class <name> extends Module {

    constructor {
        namespace = new Namespace("net.larsan.test.lang");
        moduleName = "Core";
        version = new Version("1.1.0");
        addDependency("com.whatever:Module:2");
    }
}

The name of the module class would be anonymous and generated at runtime.

Dependency Injection

With modularity in place, DI is just a small step away. Let’s start with a type to use:

class Transformer {

    function transform(String str) returns String {
        return str.replace('k', 'y');
    }
}

Not very useful admitedly. But hang on:

class Greeter(String name = "stranger") {

    @Inject
    Transformer transformer;

    function greet {
        echo(transformer.transform("Hello #{$name}"));
    }
}

It get’s neater if the transformer was actually an interface:

interface Transformer {
    function transform(String str) returns String;
}

No we can play with polymorphism. I’ll skip ahead a bit here, suffice to say that I’m going to steal straight from Guice, including assisted inject, but this time with closures, so if you want to to imagine how it would look, go ahead. Some thing to note:

  • By default all types are injectable, given that the module knows how to 1) identify them; and 2) construct them.
  • Simple identification is made with a “name” attribute on the “@Inject” annotation, and a “@Name”  annotation on types.

The great thing here is that we have the perfect place to configure our type binding. Consider the following example:

module {
 namespace = new Namespace("net.larsan.test.lang");
 moduleName = "Core";
 version = new Version("1.1.0");
 addDependency("com.whatever:Module:2");
 bind(Transformer).to(MyTransfomerImpl);
}

Add to this high order provider functions, and make the module extendible for binding overrides (which would be neat for unit testing) and you have something awesome in the works.

Concurrency

We’ll do an actor model with a twist inspired by Clojure. We’ll need two concepts:

  • “Executor” – Basically your Java thread pool abstraction
  • “Actor” an actor is associated with a thread pool and has a state object and an inbox for functions

An actor is an immutable object with an immutable state. However, this state can be updated by visiting functions. So, let’s imagine our Greeter from before is immutable, we could have an actor with a Greeter state:

Actor<Greeter> actor = new Actor(new Greeter());
Future<Greeter> future = actor.act(new function(Greeter state) returns Greeter {
    return state.name -> "Adam";
});

That’s All?

No obviously not, only my eyes are starting to blur. But this should have given you some ideas, if nothing else about my personal taste. The concurrency model probably needs a bit of re-thinking. Should we allow Actors on mutable objects (provided they are cloned initially)? This would make the code one hell lot easier to write and should speed up execution for complex applications. Also actors needs to be able to talk to other actors.

There’s a lot of other stuff left that I can’t be bothered to cover tonight:

  • Tuples. Yes we should have free-from data structures =  immutable anonymous classes.
  • Type inference, yes or no? I’m leaning towards no for readability reasons in distributed projects.
  • Should unit testing be a part of the language?
  • The whole class / inheritance / mixing / interface stuff.
  • Immutable collections and operations on them. Think Clojure.
  • Generics.
  • Etc…

And now my brain is hurting.

Unit Tests with Guice and Warp Persist

Note: this post is cross-posted on Cubeia.com.

Yesterday I needed to do some unit testing for a project using Guice and Warp Persist. And one thing that seems to be lacking, or wasn’t to be found be me yesterday, is the kind of unit testing you can do with Spring, where each method is wrapped by a transaction which is rolled back when the method ends.

To simplify, it enables you to do this:

@Test
public void createUser() {
    User u = userService.createUserWithId(1);
}

@Test
public void createUserWithSameId() {
    User u = userService.createUserWithId(1);
}

If you assume the “userService” is transactional, we’re on a database that spans multiple methods, and that the user ID must be unique, the above pseudo-code should fail as we’re trying to create two users with the same ID. If, however, there was a transaction wrapping both methods, and that transaction was rolled back, we’d be fine.

So, can you do it with Guice and Warp Persist? Sure you can!

(I’m using TestNG and JPA by the way, but you’ll get the point).

We’ll create a base class that sets up the Guice context as well as handles the transactions for us. Let’s start with creating the methods we need:

public abstract class JpaTestBase {

    @BeforeClass
    public void setUpJpa() throws Exception {
        // setup guice + jpa here here
    }

    @AfterClass
    public void tearDownJpa() throws Exception {
        // stop jpa here
    }

    /*
     * Return the module needed for the test.
     */
    protected abstract List<? extends Module> testModules();

    @BeforeMethod
    public void setupTransaction() throws Exception {
        // create "session in view"
    }

    @AfterMethod
    public void cleanTransaction() throws Exception {
        // rollback transaction and close session
    }
}

We’re using the TestNG “before” and “after” class to setup and tear down JPA. If the test was in a bigger suite you could probably do it around all test classes, but this will do for now. I’m also including a “testModules” method that subclasses should use to make sure their own test classes are setup correctly.

Before we go on, I’ll add some dependencies which will be explained later:

@Inject
protected PersistenceService service;

@Inject
protected WorkManager workManager;

@Inject
protected Provider<EntityManagerFactory> emfProvider;

We’ll setup the Guice context and the JPA persistence service in the “setUpJpa” method:

@BeforeClass
public void setUpJpa() throws Exception {
    // create list with subclass modules
    List<Module> list = new LinkedList<Module>(testModules());
    // add persistence module
    list.add(PersistenceService.usingJpa()
        .across(UnitOfWork.REQUEST)
        .forAll(Matchers.annotatedWith(Transactional.class), Matchers.any())
        .buildModule());
    // modules to array and create 
    GuiceModule[] arr = list.toArray(new Module[list.size()]);
    injector = Guice.createInjector(arr);
    // make sure we get our dependencies
    injector.injectMembers(this);
    // NB: we need to start the service (injected)
    service.start();
}

The persistence service will work across UnitOfWork.REQUEST as that’s what we’re emulating here. I’m also matching the transaction for any classes, this enables me to mark an entire class as transactional, as opposed to single methods, which I find handy.

All we need to do to shut down is to close the service, like so:

@AfterClass
public void tearDownJpa() throws Exception {
    service.shutdown();
}

Now, let’s wrap the methods. Remember our dependency injections earlier? Well, here’s where they come in handy. The WorkManager is used by Warp Persist to manage a session, we can tell it to start and end “work” and this will correspond to an EntityManager bound to the current thread. Let’s start with that:

@BeforeMethod
public void setupTransaction() throws Exception {
    workManager.beginWork();
}

@AfterMethod
public void cleanTransaction() throws Exception {
    workManager.endWork();
}

So before each method we’ll open a new EntityManager which will be closed when the method ends. So far so good. Now, in order to enable a rollback when the method ends we first need to wrap the entire execution in a transaction, which means we need to get hold of the EntityManager. Luckily, Warp Persist has a class called ManagedContext which holds currently bound objects (a bit like a custom scope), in which we’ll find our EntityManager keyed to it’s persistence context, ie. the EntityManagerFactory that created it. Take a look again at the injected dependencies we injected above: As the test only handles one persistence context we can safely let Guice inject a Provider for an EntityManagerFactory for us and assume it is going to be the right one.

Still with me? OK, let’s do it then:

@BeforeMethod
public void setupTransaction() throws Exception {
    // begin work
    workManager.beginWork();
    // get the entity manager, using it's factory
    EntityManagerFactory emf = emfProvider.get();
    EntityManager man = ManagedContext.getBind(EntityManager.class, emf);
    // begin transactionman.getTransaction().begin();
}

@AfterMethod
public void cleanTransaction() throws Exception {
    // get the entity manager, using its factory
    EntityManagerFactory emf = emfProvider.get();
    EntityManager man = ManagedContext.getBind(EntityManager.class, emf);
    // rollback transaction
    man.getTransaction().rollback();
    // end work
    workManager.endWork();
}

And that’s it! Each test method is now within a transaction that will be rolled back when the method ends. Now all you need to do is to write the actual tests…

A Google Analytics Plugin for Nexus

Note: this post is cross-posted on Cubeia.com.

Since Firebase Community Edition uses Maven heavily, I realized I’d like to track our Nexus repository in Google Analytics. The Nexus Book says that there exists such a plugin already, but apparently now one knows where it is. So here’s my attempt to write a simple one.

If you’re an impatient kind, here’s the source code.

I created the project  in Eclipse, instructions here.

Now, let’s start off with a “repository customizer” which is Nexus extension point we’ll use to insert a custom request processor into every repository…

public class GaTrackerRepositoryCustomizer implements RepositoryCustomizer {

    private final Logger log = LoggerFactory.getLogger(getClass());

    @Inject
    @Named("gaTracker")
    private RequestProcessor gaTracker;    

    @Override
    public void configureRepository(Repository rep) throws ConfigurationException {
        log.debug("Attaching tracker to: " + rep.getName());
        rep.getRequestProcessors().put("gaTracker", gaTracker);
    }

    @Override
    public boolean isHandledRepository(Repository rep) {
        boolean b = true;
        log.info("Handles repository '" + rep.getName() + "': " + b);
        return b;    
    }
}

Not too complicated. We’re using injection to get hold of the actual tracker components and we’re inserting it into all repostories.

Wait, all repositories? Yes, and it’s problem I haven’t really figured out yet. Ideally I’d like to track “hosted” repositories only. However, we’ve configured all our builds to use a few common development “groups” for convenience. However, Nexus seems to treat groups as first class members, so even though an artifact may be deployed in a hosted repository while accessed through a group, the request processor will not get notified for the hosted repository, only the group. I tend to see “groups” as equivalent to “views” in which case I’d expect the hosted repository to be called as well, but, no…

Now, let’s create a request processor which will react when someone “reads” a path in a repository.  We’ll take it in pieces…

@Named("gaTracker")
public class GaTrackerRequestProcessor implements RequestProcessor {

    public static final String GA_TRACKER_ID = System.getProperty("cubeia.nexus.gaTrackerId");
    public static final String GA_REFERER_URL =System.getProperty("cubeia.nexus.gaRefererUrl");

    private final Logger log = LoggerFactory.getLogger(getClass());

    private final JGoogleAnalyticsTracker tracker;    

    public GaTrackerRequestProcessor() {
        if(GA_TRACKER_ID == null) {
            String msg = "Missing system property 'cubeia.nexus.gaTrackerId'";
            throw new IllegalStateException(msg);
        }
        log.info("Creating new tracker, with id: " + GA_TRACKER_ID);
        tracker = new JGoogleAnalyticsTracker("nexus", "read", GA_TRACKER_ID);
        checkRefererUrl();
        adaptLogging();
    }

[...]

We configure the plugin via system properties (not too beautiful, I know), the “tracker id” is the google tracking code id and is mandatory, and the “refer url” will be set on the call to GA if available.  We’re using the JGoogleAnalytics library to call GA for us. Also, I’m being naughty and throwing an illegal state exception if the tracker id is missing, since GA updates every 24 hours we’d like to be notified on errors early.

There’s  two methods above, one sets the logging in the tracker code to use to slf4j logger instead and the other checks for and sets the referer URL:

private void adaptLogging() {
    /*
     * Adapt the logging to use slf4j instead.
     */
    tracker.setLoggingAdapter(new LoggingAdapter() {

        public void logMessage(String msg) {
            log.debug(msg);
        }

        public void logError(String msg) {
            log.error(msg);
        }
    });
}

private void checkRefererUrl() {
    if(GA_REFERER_URL != null) {
        /*
         * If we have a referer URL we need to set this. However, the
         * tracker does not have a getter for the URL binding strategy, so
         * we'll simply create a new one, ugly, but works.
         */
        log.info("Modifying GA tracking to use referer URL: " + GA_REFERER_URL);
        GoogleAnalytics_v1_URLBuildingStrategy urlb;
        urlb = new GoogleAnalytics_v1_URLBuildingStrategy("nexus", "read", GA_TRACKER_ID);
        urlb.setRefererURL("http://m2.cubeia.com");
        // set new referer
        tracker.setUrlBuildingStrategy(urlb);
    }
}

Not too complicated eh? The only thing to note is that the only way to set the refer URL is by creating a new URL building strategy. Well, I can live with that.

Before we go on we’ll create a small subclass on FocusPoint which we’ll use for tracking. Since JGoogleAnalitycs is made primarily for applications the focus point will URL encode itself, however, that won’t work for us, so we need to override it’s getContentURI method:

/** 
 * Simple inner class that adapts the content URI to
 * not be URL-escaped.
 */
private static class Point extends FocusPoint {

    public Point(String name) {
        super(name);
    } 

    @Override
    public String getContentURI() {
        return getName();
    }
}

And finally we’ll tie it all toghether. We’ll react on “read” actions, create a URI (of form ‘//path’) and track asynchronously (which will spawn a new thread for calling GA:

public boolean process(Repository rep, ResourceStoreRequest req, Action action) {
    if(action == Action.read) {
        /*
         * 1) create path by appending repo path to repo id
         * 2) create a subclass of focus point that handles proper URI's
         * 3) track asynchronously, this will perform the tracking on a new thread
         */
        String path = rep.getId() + req.getRequestPath();
        log.debug("Tracking path: " + path);
        FocusPoint p = new Point(path);
        tracker.trackAsynchronously(p);
    } else {
        log.debug("Ingoring action '" + action + "' for: " + req.getRequestPath());
    }    
    return true;
}

And that’s it. It’s not perfect: though ideally I’d like to track hosted repositories only, I’d like to avoid tracking crawlers and I would prefer not to configure via system properties (hint for you Nexus bundle users out there, you can set system properties in the “conf/wrapper.conf” files), but I’ll leave those for a rainy day.

Here’s the source code as a Maven project.Enjoy!

Update: If you download and try to compile, you will probably be missing the JGoogleAnalytics lib. You’re welcome to use our, GA-tracked, repository if you like :-)

<repository>
  <id>cubeia-nexus</id>
  <url>http://m2.cubeia.com/nexus/content/groups/public/</url>
  <releases><enabled>true</enabled></releases>
  <snapshots><enabled>true</enabled></snapshots>
</repository>

A Java Concurrency Bug, and How We Solved It

Note: this post is cross-posted on Cubeia.com.

Everyone agrees that good debugging is critical. But it is not entirely trivial when it comes to multi-threaded applications. So here’s the story how I tracked down a nasty bug in the Java5 ReentrantReadWriteLock.

Update 15/4: To be a bit clearer, the Java bug in question is on 1) Java 5; and 2) only when using fair locks, non fairness seems to work, however, that was never an option for us, if nothing else due to this other Java bug… Thanks to Artur Biesiadowski for the heads up.

Our product Firebase is designed to offload threading issues and scalability from game developers to a generic platform. However, one of our customers experienced mysterious “freezes” when running a fairly small system load in their staging environment.

First step on the way: getting stack dumps when the system is frozen. In other words, request that the client do a “kill -3” on the server process when it’s hanged, as this dumps all threads and their current stack traces to the standard output. This we did, but only got confused by it, all threads seemed dormant and there was no visible dead-lock in the stack traces.

However, several threads were all mysteriously waiting on a read lock deep down in the server, and seemingly not getting any further, despite that fact that no-one was holding the write lock. This wasn’t conclusive though as there was one thread waiting to take the write lock and this would block the readers. But given that the server was actually frozen it looked suspicious. So my first analysis concluded:

As far as I can see, the only abnormal thing that could have caused this stack is if a thread have taken the lock (read or write) and then crashed without releasing it, however there doesn’t seem to be any place in the code not safeguarded by try/finally (where the lock is released in the finally clause).

Implied in that conclusion is of course that this might either be a normal behavior and we’re looking in the wrong direction, or that we have a more serious Java error on our hands.

There’s a lot of information to be had from a ReentrantReadWriteLock, including the threads waiting for either read or write privileges, and the thread holding a write lock (if any), but not (strangely enough) the threads actually holding a read lock. And as a reader thread can effectively block the entire lock by not unlocking while a writer is waiting, this is information you really need to know.

So the next step was to get hold of the reader threads. I did this by sub-classing the ReentrantReadWriteLock to return my own version of the read lock, which, simplified, did this:

Set readers = Collections.synchronizedSet(new HashSet());

public Set getReaders() {
    return readers;
}

public void lock() {
    super.lock();
    readers.add(Thread.currentThread());
}

public void unlock() {
    super.unlock();
    readers.remove(Thread.currentThread());
}

Given this code, we now have a set containing a snapshot of the threads holding the read lock. I then added a JMX command for printing the following information to standard out for the given read/write lock:

  1. The threads waiting for a read lock, including their stack traces.

  2. The threads waiting for the write lock, including their stack traces.

  3. Any threads holding a read lock, including their stack traces.

  4. The thread holding the write lock, including its stack trace, if any.

I shipped this patched code to the client and asked them to freeze the server with the patch, print the new debug information, and then send me the output. Which they did, and the relevant output looked like this (very much shortened):

Holding reader:
Thread[DQueue Handoff Executor Pool Thread { GAME }-1,5,main]
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
    […]

Waiting writer:
Thread[Incoming,dqueue,127.0.0.1:7801,5,Thread Pools]
    sun.misc.Unsafe.park(Native Method)
    […]

Waiting reader:
Thread[DQueue Handoff Executor Pool Thread { GAME }-1,5,main]
    sun.misc.Unsafe.park(Native Method)
    […]

See anything strange here? It appears that the same thread is both holding and waiting for a read lock at the same time. But this is supposed to be a reentrant lock. In which case…

So, the freeze was caused by this: There’s a bug in Java5 (but not in Java6) where a fair ReentrantReadWriteLock stops a reader from re-entering the lock if there’s a writer waiting. It is of course easy to write a test case for, which you can find here.

This is now submitted as a bug to Sun, but I have yet to get a confirmation and bug number.

As for Firebase, it is now patched and released to use manually tracked re-entrance for read/write locks through the entire server when running under Java5, looking, again very much simplified, like this:

private ThreadLocal count = new ThreadLocal();

public void lock() {
    if (get() == 0) {
        // don't lock if we alread have a count, in other words, only
        // really lock the first time we enter the lock
        super.lock();
    }
    // increment counter
    increment();
}

public void unlock() {
    if (get() == 0) {
        // we don't have the lock, this is an illegal state
        throw new IllegalMonitorStateException();
    } else if (get() == 1) {
        // we have the lock, and this is the “first” one, so unlock
        super.unlock();
        remove();
    } else {
        // this is not the “first” lock, so just count down
        decrement();
    }
}

// --- HELPER METHODS --- //

private void remove() {
    count.remove();
}

private int get() {
    AtomicInteger i = count.get();
    if (i == null) return 0;
    else {
         return i.intValue();
    }
}

private void increment() {
    AtomicInteger i = count.get();
    if (i == null) {
        i = new AtomicInteger(0);
        count.set(i);
    }
    i.incrementAndGet();
}

private void decrement() {
    AtomicInteger i = count.get();
    if (i == null) {
        // we should never get here...
        throw new IllegalStateException();
    }
    i.decrementAndGet();
}

The above code isn’t trivial, but shouldn’t be too hard to decode: We’re simple managing a “count” each time a thread takes the read lock, but we’re only actually locking the first time, the other times we simply increment the counter. On unlock, we decrement the counter, and if it is the “last” lock, if the counter equals 1, we do the real unlock and remove the counter.

There are two things to learn, 1) in any sufficiently complex server, the above debug information is nice to have from the start on any important lock; and 2) it really is time to migrate to Java6.

New Article: Real Time BI

Note: this post is cross-posted on Cubeia.com.

Since I’ve spent some time discussing complex event processing in various contexts the last weeks, I thought I’d write up a small article about it. And lo, here it is.I do point out in the beginning that the article is primarily targeted towards gambling networks, however it can be applied to other domains as well. Financial fraud management? Sure. Industry process supervision? No problems. And so on…

Class Loading and AOP

Note: this post is cross-posted on Cubeia.com.

I have a new article up, Debugging Class Loading with AOP, in which I briefly discuss how Aspect Oriented Programming (AOP) using AspectJ can be used to debug class loading, in particular in instances where a system have multiple class loaders.It does not cover how to use the load time weaver that comes with AspectJ. For you reference, this is roughly how to use it (for Java 5 or higher):

  1. Download AspectJ from the Eclipse site.
  2. Create a configuration file called “aop.xml” in a META-INF folder on the class path. For an example file, see below.
  3. Start your VM with: -javaagent:pathto/aspectjweaver.jar

Which is about it, the “aop.xml” file may look something like this for running the code in the article in a Tomcat 6 installation:

<code lost in blog system move>

Notice above that we include the base package of your application for weaving and the Catalina class loader packages. We’re doing this to limit the scope of the weaver, otherwise your debugging may take a bit too much time and resources.

Singletons… Again!

Perhaps I’m being a bit grumpy today, but this article rubs me the wrong way. It describes the Singleton pattern in Java but contains some problems. And one of the is a pet peeve of mine. The article claims:

  • There can only be one singleton instance per JVM == Plain wrong in every respect.
  • Singletons are good for object pooling (such as connection pooling) == Dangerous.
  • You can get away with synchronization problems with a nested inner class == Naive.

The singleton pattern does not guarantee anything regards to the number of instances of an object, and certainly not per JVM. You see, even if we disregard cloning, serialization, reflection or thread safety the following is still true: A class instance of an object is guaranteed to be unique per class loader, not JVM. In other words, if you have multiple class loaders (as do almost all application servers) you can have multiple instances of a singleton per JVM; one for each class loader.

Singletons can help guard limited resources such as object pooling. That is correct, but the implication that it is good design to use it to do so is wrong. Statically held object pools in larger and long running applications and servers are not a good idea as their lifetime cannot be directly controlled by the code. Also, if there are multiple class loaders… See the paragraph above.

And the synchronization discussion is naive as one of the comments suggests. See this, this, this or google it yourself.

So, yes, the Singleton pattern is useful, but not as massively so as many articles and tutorials suggests. In a world of serialization and class loader hierarchies it can be a real pain in the ass as well. Use with care.

Re: The Open Source Frameworks I Use

The Codist asks what open source frameworks we use. Now, he says frameworks, but it looks more like libraries and utilities. Which is fine by me as I generally tend to avoid the larger frameworks, and just as The Codist himself wrote my own web framework for my regular site.

So, without further ado, here’s some regulars:

  • Jetty – good stuff
  • c3p0 – JDBC pooling
  • log4j – logging, I don’t really like it, but…
  • jgroups – data distribution/messaging
  • JBoss Serialization – for high throughput servers
  • JBoss Cache – data replication
  • Mina – network IO framework
  • XFire – for all your SOAP needs
  • HSQLDB – just works, but I’ll probably check out H2 soon
  • Jaxen – very good xpath engine
  • Hibernate – ORM, because it is more or less standard
  • GNU Trove – primitive fast collections
  • Wicket – if I really must code an web UI

But I must admit, for my home coding I use a lot of my private libraires, I’ve got my own XML API and XPath engine. My own commons collection, UUID API, simple object -> XML lib, logging, email API, etc.