Trading performance for asynchronicity

An unusual conversation came up at one of the discussion groups in the day job recently. One of the interesting things that the Javascript language specification provides is a very good description of asynchronous execution that is then embodied in execution environments like NodeJS. Asynchronicity on the JVM is  emulated by an event loop mechanism on top of the usual threaded execution environment. In general if you run JVM code in a single-thread environment bad things will happen I would prefer to do it on at least two cores.

So I made the argument that if you want asynchronous code you would be better off executing code on NodeJS rather than emulating via something like Akka.

Some of my colleagues shot back that execution on NodeJS would be inferior and I didn’t disagree. Just like Erlang sometimes you want to trade raw execution performance to get something more useful out of the execution environment.

However people felt that you would never really want to trade performance for a pure asynchronous environment, which I found very odd. Most of the apps we write in the Guardian are not that performant because they don’t really need to be. The majority of our volume is actually handled by caching and a lot of the internal workloads are handled by frameworks like Elasticsearch that we haven’t written.

In follow up discussion I realised that people hadn’t understood the fundamental advantage of asynchronous execution which is that it is easier to reason about than concurrent code. Asynchronous execution contexts on NodeJS provide a guarantee that only one scope is executing at a time so whenever you come to look at an individual function you know that scope is limited entirely the block you are looking at.

Not many programmers are good at parsing and understanding concurrent code. Having used things like Clojure I have come to the conclusion that I don’t want to do concurrency without excellent language support. In this context switching to asynchronous code can be massively helpful.

Another common situation is where you want to try and achieve data locality. With concurrent code it is really easy to actually end up with net poorly performing code due to contention on contexts. Performing a logical and cohesive unit of work is arguably a lot easier in asynchronous code blocks. It should be easier to establish a context, complete a set of operations and then throw away the whole context, knowing that you won’t need to reload that context again as the task will now be complete.

It is hard to make definite statements of what appropriate solutions are for in particular situations. I do know though that performance is a poor place to start in terms of solution design. Understanding the pros and cons of execution modes matters considerably more.

Clojure, Programming

Creating Javascript with Clojure

This post is an accompaniment to my lightning talk at Clojure Exchange 2014 and is primarily a summary with lots of links to the libraries and technologies mentioned in the presentation.

The first step is to to use Wisp a compiler that can turn a Clojure syntax into pure Javascript, with no dependencies. Wisp will translate some Clojure idioms into Javascript but does not contain anything from the core libraries including sequence handling. Your code must work as Javascript.

One really interesting thing about Wisp is that it supports macros and therefore can support semantic pipelining with the threading macros. Function composition solved!

If you want the core library functionality the logical thing to add in next is a dependency on Mori which will add in data structures and all the sequence library functions you are used to with a static invocation style that is closer to Clojure syntax.

At this point you have an effective Clojure coding setup that uses pure Javascript and requires a 50 to 60K download.

However you can go further. One alternative to Mori is ImmutableJS which uses the JavaScript interfaces (object methods) for Array and Map. If you use ImmutableJS you can also make use of a framework called Omniscient that allows you develop ReactJS applications in the same way you do in Om.

ImmutableJS can also be used by TransducersJS to get faster sequence operations so either library can be a strong choice.

Clojure, Programming

Transducers at the November London Clojure Dojo 2014

One of the topics for the November ThoughtWorks dojo was transducers (something I’ve looked at before and singularly failed to get working). Tranducers will be coming to clojure.core in 1.7, the code is already in Clojurescript and core.async.

There were two teams looking at transducers, one looked more at the foundations of how transducers are implemented and the other at their performance. These are my notes of what they presented back at the dojo.

How do transducers work?

One of the key ideas underpinning transducers (and their forebears reducers) is that most of the sequence operations can be implemented in terms of reduce. Let’s look at map and filter.

(defn my-map-1 [f coll]
     (fn [acc el] (conj acc (f el))) [] coll))

(defn my-filter-1 [pred coll]
     (fn [acc el]
       (if (pred el)
         (conj acc el)
   [] coll))

Now these functions consist of two parts: the purpose of the function (transformation or selection of values) and the part that assembles the new sequence representing the output. Here I am using conj but conj can also be replaced by an implementation that uses reduce if you want to be purist about it.

If we replace conj with a reducing function (rf) that can supplied to the rest of the function we create these abstractions.

(defn my-map-2 [f]
  (fn [rf]
    (fn [acc el]
      (rf acc (f el))))

(defn my-filter-2 [pred]
  (fn [rf]
    (fn [acc el]
      (if (pred el)
        (rf acc el)

And this is pretty much what is happening when we call the single-arity versions of map and filter; in tranducers. We pass a function that is the main purpose of the operation, then a reducing function and then finally we need to do the actual transducing, here I am using reduce again but transduce does the same thing.

((my-map-2 inc) conj) ; fn
(reduce ((my-map-2 inc) conj) [] (range 3)) ; [1 2 3]

(reduce ((my-filter-2 odd?) conj) [] (range 7)) ; [1 3 5 7 9]

The team’s notes have been posted online.

How do transducers perform?

The team that was working on the performance checking compared a transduced set of functions that were composed with comp to the execution of the same functions pipelined via the right-threading macro (->>).

The results were interesting, for two or three functions performance was very similar between both approaches. However the more functions that are in the chain then the better the transduced version performs until in the pathological case there is a massive difference.

That seems to fit the promises of transducer performance as the elimination of intermediate sequences would suggest that performance stays flat as you add transforms.

There was some discussion during the dojo as to whether rewriting the historical sequence functions was the right approach and whether it would have been better to either make transducers the default or allow programmers to opt into them explicitly by importing the library like you do for reducers. The team showed that performance was consistently better with transducers (if sometimes by small margins) but also that existing code does not really need to be modified unless you previously had performance issues in which case transducers allows a simpler, direct approach to transformation chaining than was previously possible.

Closing thoughts

I suggested the transducers topic as I had singly failed to get to grips with them by myself and I was glad it sparked so much investigation and discussion. I certainly got a much better understanding of the library as a result. My thanks got to the dojo participants, particularly James Henderson.

Computer Games, Programming

ProcJam 2014 talks

Last weekend I had the chance to go to the ProcJam talks in London that were part of the ProcJam game jam. Due to some difficulties running errands and never having been to Goldsmith’s before I missed the first talk and arrived halfway through the second.

I was hoping for some tech-focussed talks about coding up procedural generation. As it turned out it wasn’t quite that kind of event but it was still pretty interesting.

Tanya Short’s talk about some of the theory behind procedural generation was quite interesting (despite a flaky Google Hangout connection). The key takeaway from her talk was to look at the structure of what you are creating. Creating believable content means creating a simulation of the creation itself while making it satisfying to explore means adding an emotional dimension.

An example she gave was of name generation, you can totally do the Elite random syllable process but to create more believable names you can introduce structure, like lineage, nicknames and titles appropriate to the society and social ranking of the character. Introduce aspects of the way that people name their children in the real world: things like fashionable names and trends that apply to characters of a certain age, repeated usage of names within families, the use of names to honour significant people in the parent’s lives. These things add emotional depth.

Tanya is working on the game Moon Hunters and she mentioned an example from the game of how personalities consisted of the following factors:

  • the results of the character’s deeds
  • the perception of others
  • how the characters contrast to the villains
  • the needs of the story and plot

I was particularly struck by the idea that the way a character behaves is bound by how they think other people expect them to behave. That’s something we see a lot in the real world and creates exactly the right emotional connection in someone who is trapped between their “true” and “public” selves.

The talk I most enjoyed was by Hazel McKendrick who is one of the developers on No Man’s Sky. For someone who doesn’t really do game programming or procedural generation it was a helpful introduction to some of the basics.

First of all she made it clear that procedural generation isn’t about randomness, abrupt jumps in the underlying numbers creates a jarring inconsistent experience for the player. In mathematical terms we don’t want to use truly random generation but instead functions that generate continuous (smooth) results. This means using noise functions rather than pseudo-random algorithms. Most people mentioned Perlin noise but Hazel also mentioned Perlin worms and Whorley noise.

She also made the great point that games are made by their special moments and rare events in the game. If the generator creates content uniformly then the world is overwhelming and might actually become boring as repetition robs the feeling of being special.

In No Man’s Sky they want the bulk of worlds to be fairly unexciting and plain and allow the player to control their experience by having the more alien or extreme worlds be at the periphery and key game world experiences at the core of the galaxy. The player determines what kind of experience they have by the depth of the region they choose to explore.

Objects in the world matter more when they exist in relationships to one another. Hazel used the example of trees, undergrowth and bushes but it also fits in with Tanya’s talk in that people are more interesting if they exist within societies, lineages and families.

Therefore the generators need to be aware of the context around them and simplest way to do that is to nest the generators so that a generator creates a set of consistent generators that then generate sympathetic output. These generators can nest to any depth and are testable as units and as chains, something that I agree with from my experience with using functional programming although Hazel is working in C++ in a slightly more conventional programming situation.

When nesting generators these way you need to check very large datasets to see that the results truly are consistent rather than frequently consistent. You can program limits on the generator to bound their output but part of the point of using procedural generation is to create things you weren’t expecting. In her slides Hazel had some examples like spaceships where it was possible to generate large catalogues of spaceships to browse visually for anomalies.

You can also write test suites to search for situations that are out of bounds and again from a functional perspective I could see the value of generative testing to exhaustively examine output to find issues with particular inputs identified as the problem.

In a question after the talk Hazel also mentioned that generators are always consistent and finite and that changes to the player’s world are held in state outside of the generators. No Man’s Sky is a mingleplayer game where players will be able to play in the same universe and share discoveries but don’t necessarily have to be in sync to play the game or impacted by the other players activities. You can explore the game at your own pace.

Hazel’s talk is on YouTube

The other interesting talk was by Mark Johnson who is working on Ultima Ratio Regum which will one day be a rogue-like exploration game about empires and the evolution of societies but at the moment is an insanely detailed world-building program that creates a world, nations within that world, their history, cities and human terrain.

At one point in the talk Mark demonstrated how it was possible to zoom into a terrain square in the world and look at the leaves in the tree.

In terms of generation Mark’s talk was similar to Hazel’s but he used the metaphor of card decks. You shuffle cards and draw and the selected card opens new card decks that can be used for choices at a more detailed level.

His example was selected the leadership model of a nation and then its economic and foreign policy. So if you select Isolationist as the foreign policy you don’t want to have Free Trade as the economic policy. Similarly religious freedom should not be available if an earlier choice was theocracy.

This cascade set of consequences even extends to where the religious buildings are placed in cities. A populist religion mean that religious buildings are scattered throughout the city, a state religion concentrates them in the governmental quarter.

Mark’s analogue metaphor was interesting and more accessible perhaps, as long as you get the card decks nested within card decks idea. He mentioned that he thought board games were the most exciting realm of game design as you can experiment relatively cheaply but as the players run the game you must be capable of expressing the rules extremely clearly so they can be understood.

One nice detail was that the game produces books about the history of nations and people within the nation. This will ultimately be how a player discovers things in the game beyond literal exploration and conversation with characters. He illustrated this by showing how he could locate the grave of a historical character from the information in that character’s biography.

While its an achievement to have a game that has that level of internal consistency at some point he does also want to allow for alternative narratives where, say, a character involved in a war between two nations would be mentioned by the histories of both but with perhaps very different perspectives on that character’s behaviour.

Mark Johnson’s talk

I’d like to complement the organisers of the talk for putting together a pretty interesting programme.


Passive-aggressive collaboration

One interesting (and depressing) aspect of post-Github open source development is the use of the pull request as a passive aggressive way of putting off potential contributors, users and testers.

I have had the experience of discovering an issue with a piece of open source and raising an issue for it only to be “invited” to contribute a reliable failing test case, with a fix and all to the project’s contribution standards.

Now the joy of open source is being able to scratch your itch and prioritising your own problems by contributing solutions.

However there are often very good reasons why the maintainers should be the ones fixing the issues.

Firstly the maintainers should be the ones who gain most from fixing issues. One cool thing about Git-based development is that forking allows you to use existing codebases but not share the views and priorities of the original project. If I disagree with the direction or design of a project I can fork it and completely change the code to match my own aesthetics and priorities.

However in most cases I agree with the direction of the maintainers and I am simply pointing out an issue or problem that maybe they haven’t encountered in their context. I could scratch my itch but it is often more effective if the maintainer took my use case into consideration and reworked the codebase to include it.


  • have more context on the code base
  • know more about the problems the code tackles
  • know their conventions and coding standards better than I do
  • have more invested in having an effective solution than I do

Looking at things like Guava which while open are effectively not open to contribution. This is a more honest approach than inviting non-trivial contributions.

Trying to take the perspective of the maintainer I know it is tedious when people do things outside of your area of interest (i.e. IE fixes when you’re not targeting that version of the browser or Windows fixes in a project targeted at UNIXes). However telling people to “fix it themselves” is not as honest as saying “that’s not our focus”. People can then decide whether they want to fork or not.


Stemming the spam

One of the features I particularly value on WordPress has been the ability to have fairly open comment threads on my blog posts. In the past Askimet used to keep these spam free and people were able to submit comments fairly freely.

Over the course of this year though, comment spam has been getting vaguely more sophisticated. The most common spam has been backlink SEO spam that consists of reasonable emails and pieces of genuine comments mixed with product keywords and generic praise.

This kind of spam seems to be massively problematic to detect. Large numbers of comments have been made to historic posts that presumably have ended up ranking well in Google.

Moderating that stuff has been a chore and if I am away from the internet for any period of time the comment were on the site for a while and this seemed to be encouraging more. I presume the bots are smart enough to detect when their comemnts have not been deleted for more than eight to twelve hours and then they pile in.

The only reasonable way forward seems to me to be switch on someone kind of pre-moderation. I’ve gone for the mildest level of control that seems to work, namely that the first post by a user will now be moderated but once someone has a valid post they will be able to post freely.

This seems to strike the right balance between allowing people to tell me I know nothing about Clojure, Scala or Javascript and avoiding people having to sign-in to post comments.

As for the creators of these ingenious bots, a decidedly slow handclap; well-played.

Clojure, Programming

London Clojure unconference July 2014 report

For the first session I was interested in trying to continue the discussion about the Clojure “sweet spot” we had had on the mailing list. But there was only a smattering of interest so we rolled it up with the discussion on how to convince people in investment banks to use Clojure.

I think Jon Pither’s approach to this is still the best which is to find a business problem and then say that you’re going to address the problem and use Clojure to solve the real problem. A pure technical argument is not really going to get buy-in from outside the developers.

A lot of organisations want to have an approved list of technologies and for institutions that have chronic and acute technical problems like banks then perhaps that is appropriate given the need for external regulation. Where these things exist I usually think it is a case of going through the bureaucratic hoops.

The approval system is not there to be opinionated but to provide oversight. Where individuals have “weaponised” the approval process to advance their view of “right” technology you need to tackle the root problem not just sneak things in as jars.

My personal view is that financial institutions have profound technology problems but that they have no incentive to address them while they continue to make a lot of money. Really their problems should be providing opportunities for new approaches but as the existing institutions have created massive barriers to entry it doesn’t happen and we’re all really just waiting for the next financial crisis to happen, maybe then…

However in the session there was a lot of discussion about whether it is appropriate for managers to determine technology choices: on the one side you want to devolve decisions to the people close to the problem, on the other programmers commonly change jobs in a shorter period that the lifespan of the software they create.

One thing I took away was that before conservative organisations adopt Clojure they will need to see widespread adoption in the companies they see as good leading indicators and the presence of a large hiring population. In these respects Scala is literally years ahead.

Our final conclusion as a group was simply that the easiest way to approve the use of Clojure was to get into management and leadership first and then do it.

For the second session I went to the discussion on React and Om. I’m looking at React currently and there were a lot of questions about what Om layers on top of the basic JS library. Anna Pawlicka provided a number of the Om answers and others chipped in with bits of React and reactive JS knowledge. I was reminded to go and look at the current state of Om and also the new tutorials. There was also some interesting talk of how to define React components, Anna used Sablono but is there still a need for JSX?

The final session of the evening was on Riemann, which in addition to be a basic introduction to what it does was a helpful reminder of the functionality that Riemann has but that I haven’t used personally. Jason Neylon mentioned that every new service they set up has a Riemann instance attached so you can just dump all events somewhere and then build dashboards dynamically as you go along (a lot better than our approach with Graphite).

Tom Crayford introduced me to the effect of clock skew on Riemann (events from the “future” relative to the Riemann server clock are dropped) and then pointed out that clock skew can actually be monitored via Riemann! Also some interesting stuff about pumping logs into Riemann and some personal experience of crazy volumes of events being successfully handled.

Just before the end of the event I dropped in to the Gorilla REPL session to see Jony Hudson demoing his amazing notebook repl that he has been using to share assignments and research with students and colleagues in his department. A really interesting application and I suspect once we get our heads round it a really interesting way of sharing problems and potential solutions as developers.

Mind slightly blown, I was personally really happy with the event and felt that I’d got a mix of advice and the kind of innovation that make the Clojure community so interesting.