Google Cloud Functions

I managed to get onto the Google Cloud Functions (GCF) alpha so I’ve had a chance to experiment with it for a while. The functionality is now in beta and seems to be generally available.

GCF is a cloud functions, functions as a service, AWS Lambda competitor. However thanks to launching after Lambda it has the advantage of being able to refine the offering rather than cloning it.

The major difference between GCF and Lambda is that GCF allows functions to be bound to HTTP triggers trivially and exposes HTTPS endpoints almost without configuration. There’s no messing around with API Gateway here.

The best way I can describe the product is that it brings together the developer experience of App Engine with the on-demand model of Lambda.

Implementing a Cloud Function

The basic HTTP-triggered cloud function is based on Express request handling. Essentially the function is just a single handler. Therefore creating a new endpoint is trivial.

Dependencies are automagically handled by use of a package.json file in the root of the function code.

I haven’t really bothered with local testing, partly because I’ve been hobby-programming but also because each function is so dedicated the functionality should be trivial.

For JSON endpoints you write a module that takes input and generates a JSON-compatible object and test that. You then marshal the arguments in the Express handler and use the standard JSON response to send the result of the module call back to the user.


Why developers love the shiny

Developers are notorious neophiles who love adopting new technologies and designs. There is a natural bias between interest in working in emerging sectors and a general interest in novelty.

Technology as an industry also emphasises change and novelty as part of its identity with concepts like “disruption” and “innovation” being fundamental to the self-identity of the sector.

I also believe that developers are perversely incentivised to value the new over the established.

Perverse incentives

If you choose to learn an emerging language you have less history to deal with.

If you learn Java you have access to lambdas but you also have to deal with a decades worth of code that was written before lambdas were available. You’ll have to learn about interfaces, with and without implementations; inner classes; anonymous classes; dependency injection and inheritance.

If you choose to learn an emerging technology then chances are you won’t be told to RTFM as it won’t have been written yet.

Instead you’ll have the chance to interact with the technologies creators. If you make a mistake then it will be a learning experience not just for you but for the community as a whole. You’ll be united in your discovery and collaborators instead of an ignorant johnny-come-lately.

And if you learn emerging technologies you will find that conferences are eager to offer you a chance to talk about your experience. Employers will value your knowledge more than the people who added another year of experience with C#.

In every way you will find life easier and more enjoyable if you focus on emerging technology.

Falling out of love

Programmers are often like ardent lovers, in the first flush of their crush they see their new discovery as the answer to all their problems. They are evangelists and poets for their new discovery.

But over time familiarity replaces enchantment and soon we are all too aware of the flaws of tools and take for granted their utility.

Zac Tellman recently put this beautifully in a more general post about open source governance.

When our sense of possibility of what we can do with our tools is exceeded by our understanding of the constraints they have, we start to look elsewhere.

A mature legacy

Legacy is a funny word in technology. As has already been observed most of the time people are delighted to receive a legacy and the word generally has positive conotations.

For developers though the term is loaded with fear as what we are inheriting in the form of a legacy codebase is not a valuable treasure that is being handed down for safekeeping, to be enhanced and handed on to the next generation of stewards.

Instead the legacy codebase is seen as troublesome, an unwanted timebomb that is to be kept in some kind of running order and passed off to someone else as soon as possible.

Maintainers of legacy codebases are also unvalued by organisations. They are often seen as less skilled and less important than developers creating new systems. After all what they are doing is simply keeping a successful system running, they don’t need to use much imagination and they are dealing with solved problems.

Maintaining a legacy codebase often means being overlooked for promotion, pay rises or new opportunities.

In fact this view extends beyond developers, organisations do not tend to value their legacy codebases either. Product managers are equally incentivised to value the new over the old. If they add features to an existing system they are seen as less imaginative than those who create brand new systems or approaches to existing problems.

I have seen product managers deliberately sabotage legacy products so that the performance of the new solution looks better. Management often fails to look at the absolute performance of new system, just the relative numbers. People exploit that tendancy ruthlessly.

Love the new

There are more examples, however they might be better suited for dedicated blog posts in their own right. Overall though I hope I’ve illustrated that how as a profession we are encouraging people to jump on every bandwagon that is passing by.

I want to try and avoid passing a moral judgement on this. I too am a neophile, I too love novelty and would prefer to do something that I haven’t done before over perfecting my skills in a domain I know.

I just want to try and highlight the issue and move it from our unconscious to our conscious minds. Is this what we want for ourselves?

Should we rewarding maintainers of the software that pays our wages less than “rockstars” building prototypes?

Should we value simplification of our existing tools over maintaining backwards compatibility?

These are all inherently cultural issues, not technical ones. Currently we have a culture of novelty and literal innovation. I’m not sure how well it is serving us.


Using Google App Engine with Pyenv

I recently started using PyEnv to control my Python installations and make it easier to try to move more of my code to Python 3.

Google App Engine though is unapologetically Python 2.7. Google wants people to move away from the platform in favour of Google Compute custom environments and therefore has little incentive to upgrade the App Engine SDK and environments to support Python 3.

When I set my default Python to be Python 3 with PyEnv I found that despite setting a local version of Python 2.7 my App Engine instance was failing to run with an execfile is not defined exception.

The App Engine Python scripts use #!/usr/bin/env python to invoke the interpreter and for some reason PyEnv doesn’t seem to override the global setting for this despite it being correct when you check it in your shell.

After a lot of frustration and googling for an answer I haven’t found anything elegant. Instead I found this Stack Overflow answer which helpful explained that you can use #!/usr/bin/env/python2 to invoke a specific language interpreter.

Manually changing the shebang line in and solved the problem for me and got me running locally.

Obviously this is a pain if I upgrade and I feel it might be better for Google to change the scripts since they don’t have a plan to move to Python3.

Blogging, Programming

Clojure Exchange 2016

At one point during this year's Clojure Exchange I was reflecting on the numerous problems and setbacks there had been in organising the 2016 exchange with Bruce Durling and he simply replied: "Yeah it was a 2016 type of conference". So that's all I really want to say about the behind the scenes difficulties, despite the struggles I think it was a decent conference.

Personal highlights

James Reeves's talk on asynchronous Ring was an excellent update on how Ring is being adapted to enable asynchronous handlers now and non-blocking handlers in the future. I didn't know that there isn't an equivalent of the Servlet spec for Java NIO-based web frameworks.

The Klipse talk is both short and hilarious with a nicely structured double-act to illustrate the value of being able to evaluate code dynamically on a static page.

David Humphrey's talk, Log all the things was pretty comprehensive on the subject of logging from Clojure applications. It was one of those talks where you felt "well that's been sorted then".

Both Kris's keynote and Christian's Immutable back to front talked not just about the value of Clojure but how you can apply the principles of Clojure's design all across your solution.

One of the most interesting talks was a visualisation of prisoner's dilemma strategies in the browser. It was visual, experimental and informative.

Henry Garner's data science on Clojure talk was interesting again with some nice dynamic distributions and discussions of multi-arm bandit dynamic analysis. Sometimes I feel lots of the data science stuff is too esoteric with too little tangible output. This talk felt a little more relatable in terms of making dynamic variant testing less painful.


Not everything sings on the day. Daan van Berkel's talk on Rubik's Cubes suffered a technical failure that meant his presentation was not dynamically evaluating and therefore became very hard to follow. We should have tried to switch talks around or take a break and try and fix it.

The AV was a general rumbling problem with a few speakers having to have a mic switch in the middle of their talks.

Hans Hubner's talk on persistence was interesting but too quick and too subtle.

We should have had the two Spec talks closer together and earlier in the day. The things that people are doing with it are non-trivial and it is still a relatively new thing.


Spec is kind of interesting generally for the community. It has become very popular, very quickly and it is being used for all kinds of things.

One theme that came up in the conference was the idea that people wanted to share their spec definitions across the codebase. This seems a bad idea and a classic example of overreach, if someone said they defined all their domain classes in a single Java jar and shared it all across the company then you'd probably thing that is a bad idea. It's not better here because it is Clojure.

The use of Spec was also kind of interesting from a community point of view as the heaviest users of Clojure seemed to be doing the most with it. The bigger the team and the codebase the quicker people have been to adopt Spec and in some cases seem to switch from using Schema to Spec.

On the other hand the people using Clojure for data processing, web programming and things like Clojurescript have not really adopted Spec, probably because it simply doesn't add a lot of benefit for them.

So for the first time in a while we have something that requires some introduction for those new and unfamiliar with it but is being used in really esoteric ways by those making the most use of it. There is a quite a big gap between the two parts of the community.

The corridor track

Out of the UK conferences I went to Clojure Exchange felt like it had the best social pooling of knowledge outside of Scale Summit. Maybe it was because I knew more people here but the talks also had all kinds of interesting little tips. For example during Christian's talk he mentioned that S3 and Cloudfront make for one of the most reliable web API deployment platforms you can choose to use. I ended up making a huge list of links of reminders and things to follow up on. I've also included links to lots of the Github repos that were referenced during the talks.

Next year

And so with a certain inevitability we are looking to the next Clojure Exchange. We're going to have a slightly bigger program committee which should make things easier.

The other thing that we didn't really do that well this year was to try and have some talks transfer from the community talk tracks to the event. In 2017 we'll hopefully be more organised around the community and also have a series of talks that are tied in to the conference itself. If you're interested in being involved in either the organising or the talks you can get involved via London Clojurians.

See you there!

Programming, Web Applications, Work

Why can’t Forms PUT?

HTML Forms can declare a method, the HTTP verb that is used when the form is submitted, the value of this method is GET or POST.

The HTML5 spec briefly had PUT and DELETE as valid methods for the form method but has now removed them. Firefox also added support and subsequently removed them.

Recently over the course of Brexit night at The Guardian we got into a discussion about why this was the case and what the “right” way to map a form into a REST-like resource system would be.

The first piece of research was to dig into why the additional methods had been added and then removed. The answer (via Ian Hickson) was simple: PUT and DELETE have implied idempotency, the nature of form submission is that it is inherently uncacheable and therefore cannot be properly mapped onto those verbs.

So, basic problem solved, it also implies the solution for the url design for a form. A form submission represents a user submitting an untrusted data payload to a resource, this resource in turn choose to make PUT or DELETE requests but it would be dangerous to have the form do this directly.

The resource therefore is one that represents the form submission. In terms of modelling the URL I would be tempted to say that it takes the form :entity/form/submission, so for example: contact/form/submission.

There may be an argument that POSTing to the form resource represents submission so the submission part of the structure is unnecessary. In my imagination though the form resource itself represents the metadata of the form while the submission is the resource that essentially models a valid sumbission and the resource that represents the outcome of the submission.

Programming, Web Applications

AngularJS migration: PhantomJS and Angular Mocks

I have recently been upgrading a project from Angular 1.3 to 1.5 in an attempt to get the majority of our projects to a state where a migration to Angular 2 might be more likely.

The upgrade from 1.4 to 1.5 was for the most part entirely painless as the migration notes had promised. The application built and ran and none of our code seemed to be relying on any of the breaking behaviour between the versions.

There was just one problem, all our tests were failing. All the mocks were coming back as undefined with an obscure error url that didn’t really help as the advice it gave was about implementing a provider which applied to none of the mock setup that was happening in the code.

It took a bit of Googling around the problem (and hence this blog post to try and improve the situation) to find a related issue in Github that finally clued me off to the solution that we needed to update the Karma PhantomJS runner and more crucially the version of PhantomJS we were using.

As far as I can tell switching Karma to use PhantomJS 2 is a good idea irrespective of what version of Angular you are using so I think it would probably sensible to do this before you start updating Angular itself.


CloudFormation fails to create specified user

I recently had a problem with some historic CloudFormation where the user and their home directory was not being created. The problem and the solution were not complicated but my Google search returned nothing directly related to the problem except the AWS docs on the configuration syntax and there were no errors in the init log.

The user and their associated home directory were not being created which then meant when the scripting in UserData ran (which relied on a certain directory structure) I was getting a “directory not found” error.

The problem and solution are ridiculously straight-forward. The cfn-init scripts were not being installed or run. Without them configuration data in Metadata is not run, which is what the documentation in AWS::CloudFormation::Init pretty much says. I adapted this gist to install cfn-init and everything sprang into life again.

The reason I struggled so much with the problem was that I was modifying existing CloudFormation that had generated a successfully running application previous to my changes.

It took me hours to figure out that while the current version of the CloudFormation made no mention of cfn-init and yet apparently worked was simply because the necessary changes had not been checked into the source repository. Without a simple way to go back and review the actual CloudFormation config that was used (hopefully something that might change in a future version of CloudFormation) I assumed that what was missing was my knowledge and there was some other way of getting the Metadata to execute.