Java 8 + Light Weight Threads

The last couple of days I’ve been playing around with Quasar from the fine folks at Parallel Universe. “What’s that then?” I hear you ask. It’s an API and a runtime library that promises light weight threads for the JVM, called Fibers, and all the stuff you can do with that, like light weight channels for message passing, and a nice agent system. The result? A mid-level API for NIO operations on top of XNIO which I’ve called Slick.

My first attempt was to write a Sinatra-inspired small API for REST micro services on top of Undertow. That didn’t work out, but you need a bit of background to understand why:  In order for Fibers to be able to to their work, and do it more efficiently than normal threads, they need to have a small footprint, be cheap to create and throw away, and to be suspendable. But in order for them to be suspendable Quasar needs to save the Fiber stack as it is from the point of suspention – and when resumed the stack is put in place again and the Fiber goes on like nothing happened.

It prepares this with byte-code rewriting on startup, but you need to tell Quasar what you want suspendable and you do so with either an exception or an annotation:

Quasar will use the exception or the annotation to determine when the JVM starts what to rewrite. So any code lacking these markings will per definition not be suspendable. Imagine now a Fiber running, it’s almost exactly like a normal Thread, even the API’s are similar:

A big shout out here to Parallel Universe: the API’s are very well put together! They mimic the standard Java API’s so well that the effort to write code in a new API is almost non-existent.

As you can see above, it looks just like a normal thread, except that it’s got a return type, and yes, the execution is suspendable.

So why didn’t my Undertow experiment fly? Because in order to dispatch events in Undertow (the process of moving the action from the underlying IO thread – which you don’t want to keep busy – to a user level thread) you have to go through Undertow code which of course isn’t marked as suspendable. And when you later try to suspend the Fiber, Quasar will politely tell you that parts of your stack isn’t possible to suspend, ie. the Undertow part of the stack.

As far as I could see there was no simple way around it. So I wrote my own little NIO library instead. Life? I’ve heard of it.

If you go to Slick there’s example code, so I won’t bore you with the design choices. Nor with the quite interesting XNIO stuff. This post is about Fibers, and how they perform compared to POJT (plain old Java threads).

Do they work? Oh, yes. However, one should remember that Quasar is in version 0.5 and newly released so a bit of wonkiness is to be expected. And there was. For example, I fought hours with a strange behavior when running on Fibers (I had written the code to be able to switch back and forward between threads and Fibers): I was getting 3x the number of read notifications from XNIO as compared to when running on threads. And it completely destroyed the server. And would spike when clients disconnected in an order of magnitude. It turned out that by modifying a small bit of my code that writes output the problem disappeared.

I have no bloody clue why.

When that was over and the server seemed to work I had to figure out how to test it. Let’s not forget that POJT’s are massively optimized by now. So I was quickly CPU starved no matter what I did. But that isn’t the point with Quasar: the point is that with light threads you can scale real world scenarios easier. So I quickly threw together three different kinds of blocking (because let’s face it, all the hard problems comes from blocking) that I could run without too much effort (everything being relative).

  • Thread.sleep – This is the killer, and hopefully you’ll not use it very much. But it’s still there.
  • Asynchronous callback operations – This is what you should be doing all the time
  • Object.wait / synchronize – The usual stuff.

The Scenario
Let’s imagine a socket server, that does some work in a request/response manner, and needs an external source. It’s going to make an integration call, and this call may go to a load balanced set of services. Let’s now imagine that there’s five services behind the load balancer, and one of them is experiencing problems and have call latencies between 1.5 to 3 seconds. We should expect a roughly 20% degradation of service.

We can do this with variations on a theme:

Thread Sleep
When the blocking is done with thread sleep Quasar scores the first victory, and a decisive one at that. The thread version of the test quickly degraded and with a load of 5000 RPS (requests per second) I was out of CPU at 2500 ( I would have expected 80%, ie 4000). Quasar flew through this without a problem and scored a perfect 4000 RPS.

Btw, here’s how to do a 100 ms sleep in Quasar when you don’t know if there’s going to be a thread or a Fiber running:

The Strand is Quasar’s abstraction for a running process, and it can be either a thread or a Fiber. That’s actually super nice to have when playing around and switching back and forward between Fibers and threads.

Asynchronous Callback
This is a tie, with a slight favor to Quasar. Both parties went up to a load of 10 000 RPS where my CPU was seriously struggling (I was on an Oracle Virtual Box running Linux on 4 cores). On some of the rounds the threads bogged down and it seemed like I was getting somewhere 65-80% performance where the Fibers appeared more stable.

Of course, the thread code was using Futures. Because: duh?

Whereas the Fiber code just happily slept away the time until the operation was done. Say after me: Java is pretty efficient after all, as long as you use futures / rxJava / whatnot.

But here’s another thing to say about Fibers: the threads in this test are obviously running out of an ExecutorService (a shared thread pool to be exact). Whereas I tried to pool the Fibers and it didn’t do a blind bit of difference. So in a server loading 8K requests per second I was doing the equivalent of “new Thread” each request – so yes, they’re to efficient and cheap create indeed.

Object Wait
This is a no-go for Fibers (at least in this test) so if you really need it, you’re probably better off using threads. But really, you shouldn’t go there anyway.

Conclusion
Now then, was it easy to transition into Quasar-land? No, not really. There’s a bit of mind-fuck going on despite the API’s being so similar and as soon as you have to integrate with legacy code you’re going to have to live with less than perfect results. But if you’re writing new code, and know it’ll be asynchronous from day one (like writing, for example, completely out of thin air, a completely new NIO framework…) then it’s not too bad. Although as I said, a bit of wonkiness is expected.

So yeah, I’m impressed, it does seem like this stuff is working!

1 Comment Java 8 + Light Weight Threads

  1. Pingback: Slick with Quasar Actors | The Ironism

Leave a Reply