Programming

Google Cloud Functions

I managed to get onto the Google Cloud Functions (GCF) alpha so I’ve had a chance to experiment with it for a while. The functionality is now in beta and seems to be generally available.

GCF is a cloud functions, functions as a service, AWS Lambda competitor. However thanks to launching after Lambda it has the advantage of being able to refine the offering rather than cloning it.

The major difference between GCF and Lambda is that GCF allows functions to be bound to HTTP triggers trivially and exposes HTTPS endpoints almost without configuration. There’s no messing around with API Gateway here.

The best way I can describe the product is that it brings together the developer experience of App Engine with the on-demand model of Lambda.

Implementing a Cloud Function

The basic HTTP-triggered cloud function is based on Express request handling. Essentially the function is just a single handler. Therefore creating a new endpoint is trivial.

Dependencies are automagically handled by use of a package.json file in the root of the function code.

I haven’t really bothered with local testing, partly because I’ve been hobby-programming but also because each function is so dedicated the functionality should be trivial.

For JSON endpoints you write a module that takes input and generates a JSON-compatible object and test that. You then marshal the arguments in the Express handler and use the standard JSON response to send the result of the module call back to the user.

Standard
Blogging, Programming, Web Applications

An overview of Javascript reactive frameworks

This post is only meant to be a snapshot of the current state of the various DOM virtualising webframeworks that are around. I’m partly publishing it to try and discover more that I may not be aware of.

Many of these frameworks trace an ancestry back to Om and React. However each one tries to deal with perceived problems with the original frameworks. The most common being that React is too heavy and opinionated while not providing a consistent data model for components. Om on the other hand is in Clojurescript and therefore represents too much to learn in terms of a new language and build process.

Libraries

Most of the libraries build on a few common building blocks that I’m not going to elaborate on here. Virtualdom was an early attempt to separate the core idea of React from the rest of the library code. Virtualdom is only concerned with creating, manipulating and stringifying DOM structures in-memory. Browser DOM APIs involving linking to the actual rendered document so managing virtual DOM is more efficient and simpler because you’re not interacting with these underlying libraries.

ImmutableJS provides a Javascript-idiom interpretation of the Clojure data structures that Om uses (and which are available as the standalone library Mori).

Omniscient

The first interesting framework to discuss is Omniscient, which as its name suggests is heavily influenced by Om but is written in Javascript and therefore does not require you to learn Clojure to use the same techniques that Om uses. Omniscient is built on top of React and ImmutableJS and uses its own library Immstruct to add reference cursors to ImmutableJS structures. Reference cursors allow a component to observe and change sections of a data structure without having to manipulate the whole thing. So for example a component can be given a single sub-key in an object that represents its state and it cannot access or change anything that is not under that key. The code can also be simplified to behave as if the sub-key was actually just the whole data object.

Omniscient doesn’t suggest an alternative to Om’s CSP, instead providing a mechanism for passing event flow functions down the component tree. You’re free to choose your own event libraries. It also means that you’re free to make your own mistakes here as no guidance is really given as to how to structure your event scheme appropriately.

Omniscient is one of the earliest frameworks to re-implement Om and therefore has one of the better sets of documentation on its Github pages. That said there’s not a lot of documentation and the framework does not have a massive community. The situation is worse in most of the other frameworks though so this might tip you over in favour of Omniscient.

Ractive

This is a bit of a Guardian shout out as the primary developer Rich Harris is a Guardian interactive developer.

Ractive (Github) is a little be different from the other frameworks as you can essentially think of it as Mustache templates backed by Observables. You declare a data-binding and write templates in normal Mustache syntax but behind the scenes Ractive is driven by changes in the data and then writes new section of DOM in-memory according to what has changed rather than DOM diff’ing.

Also Ractive sticks with two-way databinding rather than unidirectional data flow so failures in synchronisation or rendering can be problematic.

If what you want to do is render content over a Javascript data model then there is a lot in Ractive that is very compelling. It uses templates with a standard syntax that is well understood and is a soup and nuts framework that sticks to core Javascript syntax and features. However if you want to use your own event or data model you are out of luck.

Mercury

Mercury on the other hand prides itself on modularity. A microframework it attempts to create a glue layer that allows other libraries to interact in a sensible and consistent way. The default components are Virtualdom and its own observer pattern to wrap state.

Mercury’s biggest problem right now is its lack of documentation. There is an expectation that you are going to read the source code to understand what the framework is doing and how to interact with the API. I frankly think this is unrealistic. The project doesn’t currently supply the incentive to do that. Unless you have a very particular desire to avoid any framework lock-in or you want to use a very specific combination of libraries that is not supported elsewhere its hard to understand why you would invest your effort here rather than in frameworks that offer more support.

Cycle

Cycle is similarly experimental, its biggest claim is that it is truly reactive and that the rendered page is purely the result of change in state. The introduction is couched in computer science theory but it would seem that at its heart Cycle wraps RxJS and Virtualdom in a glue layer that has the programmer writing the transform sequence between the event and the DOM structure.

I think it is a positive feature that Cycle re-uses a popular library to manage its state-transitions rather than implementing yet another custom version of the Observable pattern. It also makes the framework easier to get started with if you are familiar with the Rx.

Using established libraries also makes the lack of documentation more acceptable as the Cycle readme only needs to explain how the glue works in the framework.

As something built on reactivity you have to get used to dealing with intermediate state which can be bit difficult for the beginner.

Essentially any event where the user would expect feedback means you need write the conditional structure in the output. So if the user types a character in an input box then you need to write the value of the input box to be the characters the user has typed so far. Most frameworks work at a higher level of abstraction or rather they map closer to the DOM APIs, so getting a working application means grokking the way the dataflow works.

If you’re looking for purity (and a resulting simplicity in implementation) but not to have to learn a bespoke API Cycle is nicely positioned.

WebRx

WebRx is similarly built on top of RxJS Observables but is a much fuller-fat framework that is much more a spiritual successor to Knockout than owing much to the influence Om or React.

Rather like React WebRx doesn’t really provide generalised event handling but instead has special sauce bindings for DOM events and a MessageBus system built over Rx.

It is also written in Typescript and generally looks to play well within the Microsoft ecosystem. It’s interesting to me as an example of how different a language has to be before its regarded as a barrier. Clearly the use of Typescript means there are people who will refuse to use the framework regardless of whether it works for their use case. Other people are going to be attracted exactly because it uses Typescript.

Deku

Language choices are also interesting in Deku which is another attempt to re-implement React in a superficial way.

Deku makes use of ES6 and 7 features and doesn’t aim to support a broad range of browsers (unlike say Ractive). Again that is going to rule it out for some people but this is a more interesting as now we are within dialects of the same core language. Language choice for implementing frameworks is not straightforward. What are you looking for? Conciseness? Editor support?

Deku aims to take the dom diffing approach but avoid getting caught in React’s framework and approach. In particular components are defined just as Javascript objects rather that classes and instances. Something I think makes it more elegant that normal React Components.

It does however still use JSX which is quite interesting as the framework claims to be taking a functional approach but actually uses a DSL for all its DOM construction.

The lifecycle hooks are slightly different with more hooks for different stages of the process and Deku uses some interesting function passing to send changed data down the tree to components.

Deku doesn’t take much influence from Om though. It doesn’t have sophisticated event handling and uses mutable data with generous access and callbacks on data write to do re-renders. This means bugs and state issues are no less likely to happen than with any other framework. It does adopt the single atom idea with a single tree representing the app and the app renderer being bound to the body element.

As such if you like the idea of React but don’t want to bound into its concept of how a Component should be defined but do like JSX and trust the implementors to create a better dom diff than Facebook or Virtualdom, this is the project for you.

Conclusion

I’ve only chosen a handful of frameworks to look at here, mostly based on the ones I know, I’m expecting people to point out more in the comments. I also haven’t used all of these frameworks. Road-testing all of them would be a bigger task than just trying to describe the design choices they’ve made.

The most common pattern is to try and improve the rendering time versus React by using different virtual dom difference algorithms. Usually this is combined with Observed variables that provide a Reactive component that allows changes in the data model to be conveyed to the DOM model with no coding required.

Few of the frameworks engage with the functional reactive programming paradigm by building abstract event streams or indeed any abstraction over discrete events.

The idea that the app should be a single data structure that represents the whole page seems to be gaining significant traction with several of the frameworks recommending this as an approach.

The explosion of frameworks resulting from the release of React is, I think, a positive thing. Initially it seems really daunting that you have all these choices but when you look at the real level of difference between them you can see that they are actually quite tightly coupled around a few common and core ideas and that mostly they express differences about the concerns that a framework should have which feeds into the wider conversation about micro or comprehensive frameworks.

Standard
Clojure, Programming

Creating Javascript with Clojure

This post is an accompaniment to my lightning talk at Clojure Exchange 2014 and is primarily a summary with lots of links to the libraries and technologies mentioned in the presentation.

The first step is to to use Wisp a compiler that can turn a Clojure syntax into pure Javascript, with no dependencies. Wisp will translate some Clojure idioms into Javascript but does not contain anything from the core libraries including sequence handling. Your code must work as Javascript.

One really interesting thing about Wisp is that it supports macros and therefore can support semantic pipelining with the threading macros. Function composition solved!

If you want the core library functionality the logical thing to add in next is a dependency on Mori which will add in data structures and all the sequence library functions you are used to with a static invocation style that is closer to Clojure syntax.

At this point you have an effective Clojure coding setup that uses pure Javascript and requires a 50 to 60K download.

However you can go further. One alternative to Mori is ImmutableJS which uses the JavaScript interfaces (object methods) for Array and Map. If you use ImmutableJS you can also make use of a framework called Omniscient that allows you develop ReactJS applications in the same way you do in Om.

ImmutableJS can also be used by TransducersJS to get faster sequence operations so either library can be a strong choice.

Standard
Programming

Clojurescript at London FunctionalJS

At the January’s London FunctionalJS meetup the technology under discussion and use was Clojurescript. There was an introduction to the language basics from Thomas Kristensen of Forward, which was really much more about the basics of the syntax. We then went into the dojo exercises: the choices were implementing the Todo list SPA (the Javascript world’s Pet Store), using Clojurescript with an existing Javascript framework people were already used to working with or doing some 4Clojure.

Everyone ended up doing the Todo list which is interesting in its own way. Clearly the SPA is seen as the benchmark for evaluating these kinds of technologies.

Most of the teams were able to get the basic Todo functionality done in terms of adding and removing things from the list and re-rendering it. Most teams seemed to abstract the rendering but most put the list management into the callback for the event.

Again I was interested to see that most people grasped the idea of an atom and were able to manipulate its value. Because that kind of stuff is second-nature to me now I was wondering if it would cause issue in terms of creating a modifying function rather than directly manipulating the value. The example in the setup functions of the dojo code using conj seemed to be straight-forward enough for everyone.

Identifying and deleting items seemed more problematic. Some people wanted to do it by index but for the most part matching the text of the todo-item seemed to be popular. Probably the sensible way to actually manage the items is to uuid the items to allow their underlying state to change away from the identity.

Laziness definitely caught people out, including myself! I’ve moaned about the fact that using map purely for side-effects in fact results in the form not executing. Despite this I fell into the trap again, however fortunately having encountered it before I could reverse into a quick doall.

Other teams imaginatively re-implemented doall using loop. Which I guess is testament to how easy it is to do things in a LISP.

One thing that was hellish in our team’s code and which I think cropped up in the other teams as well was the amount of set! we were applying to build up very low-level DOM calls. Right at the end I remembered that Google Closure was available to abstract some of that work away. However it still means that your knowledge of Clojure needs to be heavily supplemented by low-level DOM APIs as well as what is available in the Google Closure library (which is not the best known of libraries).

I was also wondering whether doto might not have cleaned up our code a lot. It’s an issue that a lot of Javascript mutable state is not easy to wrangle with things like threading macros that normally ease the pain. I’ve seen this in the WebGL dojos as well.

The final ugly issue of the evening was the project template that managed to both run on my machine and not run on my machine. The template was more complex that the standard SPA template as it used Compojure and Clojurescript (presumably using the former to serve static assets on localhost). Leiningen skeleton projects have to work and be reliable, otherwise potential adopters just get frustrated and quit.

The reactions were interesting, a guy on our team at the end asked why he would want to use Clojurescript. Good question. People who were doing things like building HTML5 games seemed to see the potential and advantage much more. This is an area I hadn’t really considered before but it does make a lot of sense as regular Clojure has already had a lot of success in implementing animation and complex state machines.

For me the alternating between high-level Clojure and low-level DOM APIs was painful. I’m going to be more interested in having wrappers that allow high-level programming consistently in a project. And I am going to be thinking about games more!

Standard
Clojure

Clojurescript: is it any better yet?

A while ago I wrote about how Clojurescript was a square peg being hammered into a non-existent hole to complete indifference by everyone.

So has anything changed? Well Clojurescript has continued to be developed and it seems to have lost some of the insane rough-edges it launched with, it seems possible to use it with OpenJDK now for example.

Some people have made some very cool things by using Clojurescript as a dialect for writing NodeJS code. I was particularly struck by Philip Potter's talk on Marconi.

The appearance of core.async for Clojurescript is in my view the first genuine point of interest for non-Lisp fans. It provides a genuinely compelling model for handling events in an elegant way.

David Nolen, while being modest about his contribution, has also contributed some excellent blog posts about the virtues of Clojurescript. The most essential of which is the 101 on how to get a basic Clojurescript application going with core.async. This is an excellent tutorial but also is underpinned by hard work on getting the "out of the box" developer experience slick via the Mies Leiningen template.

Let's be clear about what a difference this makes. At the London Clojure dojo we once had a dojo about using Clojurescript, after a few hours all the teams limped in with various war stories about what bits of Clojurescript they had working and what was failing and why they thought it was failing. The experience was so bad I really didn't want to do Clojurescript as a general exercise again.

In the last dojo we did Clojurescript, we used Mies and David's blogpost as a template and all the teams were able to reproduce the blogpost and some of the teams were creating enhancements based on the basic idea of asynchronous service calls.

When someone pitches a Clojurescript idea in 2014 I'm no longer in fear of a travesty. That is a massive step forward.

And that's the good news.

Clojurescript! Who is good for!

After the dojo there were some serious discussions about using Clojurescript in anger. The conversation turned eventually to GWT. In case you don't remember or have never met it GWT is essentially a browser client kit for Java developers. In London it gets used a lot by financial institutions that need rich UIs for small numbers of people. Javascript developers are unlikely to be hired by those organisations so just like Google they end up with a need but the wrong kind of skills and GWT bridges the gap. A Java developer can use a familiar language and off-the-shelf components and will end up with a perfectly serviceable rich client-side app.

There is no chance in hell that a Javascript developer is going to use GWT to build their applications.

Clojurescript feels like the same thing. Clojure developers and LISP aficionados don't know a great deal of Javascript, they can program in Clojurescript and Google Closure and it is probably going to be okay. Better in fact than if they tried to create something in an unfamiliar language with all manner of gotchas.

But there is no chance in hell that a Javascript developer is going to use Clojurescript to build their applications.

Why Coffeescript failed

The reason I say this is because Coffeescript, a great rationalisation of Javascript programming is still viewed with suspicion by Javascript developers.

I asked some at a recent Javascript tech meeting why some people didn't use it and the interesting answer was that they couldn't really understand its syntax and were effectively translating the forms into Javascript. The terseness of the language was actually off-putting because it made it harder to mentally translate what the Coffeescript program was doing.

Adding LISP and Google Closure into that mix isn't going to make that mental disconnect any easier. The truth would appear to be that Javascript developers are simply not that disenchanted with their language. Clojurescript is going to have to offer something major to get over the disadvantage of a non-curly brace language and machine-optimised generated code.

At the Clojure dojo post-mortem people talked about the fact that Clojurescript helped avoid pitfalls and unusual behaviour in Javascript. That might seem a rationale argument to a Clojurian. However it was never an argument used on the JVM. "Hey Java just has all these edge-cases, why not use a LISP variant instead?".

In both languages practitioners of the language are deeply aware of the corner cases of their language. Since they are constantly working with it they are also familiar with the best practices required to make sure you don't encounter those corner cases.

Clojure on the JVM brought power, simplicity and a model of programming that made reasoning about code paths simple.

Clojurescript has the same advantages of code structure but doesn't really give more functionality over Javascript and still has a painful inter-operation story.

Javascript: the amazing evolving language

Javascript has a strong Scheme inheritance meaning that it already contains a lot of LISP inheritance.

Also unlike Java which ended up with a specification that was in the wilderness for years, Javascript has managed to keep its language definition moving. It's sorted out its split and with aggressive language implementers in the form of the competing browsers it is rapidly adding features to the core of the languages and standards for extensions.

Javascript is almost unique that when lots of people wrote languages that compiled into Javascript the community weren't stuffy about it but actually created a specification, Source Maps, that made it easier to support generated code.

Javascript has a lot of problem areas but it is also rapidly evolving and syntactic sugar and new language constructs are adding power without necessary creating new problems or complexity. It is expansionist and ruthless pragmatic, just like Clojure on JVM in many ways.

Better the devil you know?

Visual Basic isn't the best language in the world, it's certainly not the best language for creating apps on Windows. However for organisations and programmers who have invested a lot of time in a language and a platform it normally takes a lot to get them to change.

Usually, it will take an inability to hire people to replace those leaving or the desertion of major clients before change can really be countenanced.

Javascript developers are in the same sunk-cost quandary but there is nothing on the horizon that is going to force an external change. There may be better alternatives but Javascript is one of the easiest languages to learn. It's highly interactive and its right there in the console window of this very browser!

There's no lack of demand for good looking websites and browser hackery to differentiate one web product from the next.

Regardless of the technical merits of any alternative solution offers, and we are not just talking about Clojurescript, Elm offers a similar set of advantages, the herding effect is powerful. Your investment in Javascript is far more likely to pay off that putting time into one of the alternatives, none of which have momentum.

Sometimes there are real advantages to sticking close to the devil.

Node.js

For me the most interesting area of Clojurescript is being able to write Clojure and treat V8 as an alternative runtime.

People are already noticing some odd performance characteristics where some things run better on Node than they do on JVM, most particularly around Async.

Polyglot developers are a familiar sight in the server side, knowing a variety of languages is advantage for the general programmer. Server-side Javascript is really only for those who are a one-trick pony.

It is still going to be a niche area but it is much more likely to happen than in the client-side.

Standard
Programming

Getting to grips with Generators in Javascript

One of the things that makes Python such an amazing language is its far-sighted inclusion of generators and first-order list comprehensions. Both of these things are now coming to Javascript with a healthy sense of homage in the syntax. All the examples in this post have been tested in the console of Firefox 23.

What are generators?

Generators are functions that can be called multiple times and on each call they resume execution from the state they were in when they last stopped executing. To do this they have a special keyword called yield instead of return.

Let’s consider the following trivial example:

function hello() { return "Hello"; }

hello(); // "Hello"

function lazyHello() { yield "Hello"; }

lazyHello(); // [object Generator]

var g = lazyHello(); g.next(); // "Hello"

This example might seem trivial but it actually encapsulates all the important values of generators and lazy evaluation. The difference between hello and lazyHello is that lazyHello defers execution of itself. Instead of executing it returns a Generator object that can then be evaluated for an answer later, rather like Deferreds.

This lazy evaluation means that you describe very complex or expensive calculations but not have to execute them until you need them. Now for a function that returns a single value that isn’t going to seem very exciting.

Writing infinite sequences with Generators

How can describe the world of all the positive integers? Well one way is to take the value of 1 as the start of arithmetic sum and simply add one to the previous value of the sum.

function postiveIntegers() {
  var i = 1;

  while(true) {
    yield i++;
  }
}

var g = postiveIntegers();
g.next(); // 1
g.next(); // 2

When we call next on the Generator we start executing the function, so we initialise the value of the variable and enter the loop. We then call yield and the function stops running. When we call next again we do not run the function again, we resume from the point where we stopped last time, inside the while loop.

Generators provide strong encapsulation of their data since once we have obtained a generator we can no longer access its state, just the results of its calculations. If you wanted to do this sum computation with a regular function then you would need to hold the state of the last call to the function somewhere outside the function and pass it again when you wanted to calculate the next value.

However that infinite loop is still a bit tricky and if you get it wrong then you will lock up the browser. Here’s a safer version.

function postiveIntegers(maxValue) {
  var i = 1;

  while(i <= maxValue) {
    yield i++;
  }

}

var g = positiveIntegers(2);

g.next(); // 1
g.next() // 2
g.next() // [object StopIteration]

Truly infinite sequences are really powerful but in practice you probably want to use weak-sauce versions that have some kind of bound.

Why are generators awesome?

Well firstly they allow the expression of things that would otherwise be impossible in computer code, things like very long or infinite sequences that would not fit in memory if every member had to be expressed or stored.

As part of this lazy evaluation generators are actually very economic in terms of resources, you can define a lot of rules and potential values but then only express the ones you are interested in, minimising the actual allocations.

Finally they are strongly encapsulated functions which hide their internal implementation without the need to resort to object semantics.

Standard
Programming

Keeping NPM local

I’m pretty sure that at one point NPM might not have required sudo privileges to do things but by default it now seems that people are sudo installing all over the place. It’s the equivalent of just clicking “yes” on every Windows permission dialog.

There is no particularly good reason to use sudo with a package manager. Virtualenv is tres local by default and even gem (via rbenv and rvm) can now run happily in the user space. Even if you want to share package downloads to minimise network calls then you still don’t need to install everything globally as root.

NPM is special in almost every sense of the word and it’s author recommends that you switch permissions on /usr/local to be owned by your login user. That seems kind of crazy and the kind of thing that probably would only work out for OSX users.

In fact if you are willing to build your own NodeJS it really isn’t hard to bet Node and NPM working locally and still retaining all the advantages of the “global” NPM install. Just use the standard build option of –prefix to set Node to use your home directory and just add $HOME/bin to your PATH in .bashrc or the equivalent.

Then you should be able to use npm without ever having to sudo.

My view though is that you shouldn’t have to force everything into the user space just to make sure a package manager doesn’t need sudo privileges. It would be better for everyone if NPM used a directory in home for each user since it seems to be aimed at single-user installs anyway there is not a massive saving by having packages installed in /usr/local. For the few use-cases where that would be useful then it should be an override option.

Standard