Programming

Using Google App Engine with Pyenv

I recently started using PyEnv to control my Python installations and make it easier to try to move more of my code to Python 3.

Google App Engine though is unapologetically Python 2.7. Google wants people to move away from the platform in favour of Google Compute custom environments and therefore has little incentive to upgrade the App Engine SDK and environments to support Python 3.

When I set my default Python to be Python 3 with PyEnv I found that despite setting a local version of Python 2.7 my App Engine instance was failing to run with an execfile is not defined exception.

The App Engine Python scripts use #!/usr/bin/env python to invoke the interpreter and for some reason PyEnv doesn’t seem to override the global setting for this despite it being correct when you check it in your shell.

After a lot of frustration and googling for an answer I haven’t found anything elegant. Instead I found this Stack Overflow answer which helpful explained that you can use #!/usr/bin/env/python2 to invoke a specific language interpreter.

Manually changing the shebang line in appcfg.py and dev_appserver.py solved the problem for me and got me running locally.

Obviously this is a pain if I upgrade and I feel it might be better for Google to change the scripts since they don’t have a plan to move to Python3.

Standard
Blogging, Programming

Clojure Exchange 2016

At one point during this year's Clojure Exchange I was reflecting on the numerous problems and setbacks there had been in organising the 2016 exchange with Bruce Durling and he simply replied: "Yeah it was a 2016 type of conference". So that's all I really want to say about the behind the scenes difficulties, despite the struggles I think it was a decent conference.

Personal highlights

James Reeves's talk on asynchronous Ring was an excellent update on how Ring is being adapted to enable asynchronous handlers now and non-blocking handlers in the future. I didn't know that there isn't an equivalent of the Servlet spec for Java NIO-based web frameworks.

The Klipse talk is both short and hilarious with a nicely structured double-act to illustrate the value of being able to evaluate code dynamically on a static page.

David Humphrey's talk, Log all the things was pretty comprehensive on the subject of logging from Clojure applications. It was one of those talks where you felt "well that's been sorted then".

Both Kris's keynote and Christian's Immutable back to front talked not just about the value of Clojure but how you can apply the principles of Clojure's design all across your solution.

One of the most interesting talks was a visualisation of prisoner's dilemma strategies in the browser. It was visual, experimental and informative.

Henry Garner's data science on Clojure talk was interesting again with some nice dynamic distributions and discussions of multi-arm bandit dynamic analysis. Sometimes I feel lots of the data science stuff is too esoteric with too little tangible output. This talk felt a little more relatable in terms of making dynamic variant testing less painful.

Disappointments

Not everything sings on the day. Daan van Berkel's talk on Rubik's Cubes suffered a technical failure that meant his presentation was not dynamically evaluating and therefore became very hard to follow. We should have tried to switch talks around or take a break and try and fix it.

The AV was a general rumbling problem with a few speakers having to have a mic switch in the middle of their talks.

Hans Hubner's talk on persistence was interesting but too quick and too subtle.

We should have had the two Spec talks closer together and earlier in the day. The things that people are doing with it are non-trivial and it is still a relatively new thing.

clojure.spec

Spec is kind of interesting generally for the community. It has become very popular, very quickly and it is being used for all kinds of things.

One theme that came up in the conference was the idea that people wanted to share their spec definitions across the codebase. This seems a bad idea and a classic example of overreach, if someone said they defined all their domain classes in a single Java jar and shared it all across the company then you'd probably thing that is a bad idea. It's not better here because it is Clojure.

The use of Spec was also kind of interesting from a community point of view as the heaviest users of Clojure seemed to be doing the most with it. The bigger the team and the codebase the quicker people have been to adopt Spec and in some cases seem to switch from using Schema to Spec.

On the other hand the people using Clojure for data processing, web programming and things like Clojurescript have not really adopted Spec, probably because it simply doesn't add a lot of benefit for them.

So for the first time in a while we have something that requires some introduction for those new and unfamiliar with it but is being used in really esoteric ways by those making the most use of it. There is a quite a big gap between the two parts of the community.

The corridor track

Out of the UK conferences I went to Clojure Exchange felt like it had the best social pooling of knowledge outside of Scale Summit. Maybe it was because I knew more people here but the talks also had all kinds of interesting little tips. For example during Christian's talk he mentioned that S3 and Cloudfront make for one of the most reliable web API deployment platforms you can choose to use. I ended up making a huge list of links of reminders and things to follow up on. I've also included links to lots of the Github repos that were referenced during the talks.

Next year

And so with a certain inevitability we are looking to the next Clojure Exchange. We're going to have a slightly bigger program committee which should make things easier.

The other thing that we didn't really do that well this year was to try and have some talks transfer from the community talk tracks to the event. In 2017 we'll hopefully be more organised around the community and also have a series of talks that are tied in to the conference itself. If you're interested in being involved in either the organising or the talks you can get involved via London Clojurians.

See you there!

Standard
Programming, Web Applications, Work

Why can’t Forms PUT?

HTML Forms can declare a method, the HTTP verb that is used when the form is submitted, the value of this method is GET or POST.

The HTML5 spec briefly had PUT and DELETE as valid methods for the form method but has now removed them. Firefox also added support and subsequently removed them.

Recently over the course of Brexit night at The Guardian we got into a discussion about why this was the case and what the “right” way to map a form into a REST-like resource system would be.

The first piece of research was to dig into why the additional methods had been added and then removed. The answer (via Ian Hickson) was simple: PUT and DELETE have implied idempotency, the nature of form submission is that it is inherently uncacheable and therefore cannot be properly mapped onto those verbs.

So, basic problem solved, it also implies the solution for the url design for a form. A form submission represents a user submitting an untrusted data payload to a resource, this resource in turn choose to make PUT or DELETE requests but it would be dangerous to have the form do this directly.

The resource therefore is one that represents the form submission. In terms of modelling the URL I would be tempted to say that it takes the form :entity/form/submission, so for example: contact/form/submission.

There may be an argument that POSTing to the form resource represents submission so the submission part of the structure is unnecessary. In my imagination though the form resource itself represents the metadata of the form while the submission is the resource that essentially models a valid sumbission and the resource that represents the outcome of the submission.

Standard
Programming, Web Applications

AngularJS migration: PhantomJS and Angular Mocks

I have recently been upgrading a project from Angular 1.3 to 1.5 in an attempt to get the majority of our projects to a state where a migration to Angular 2 might be more likely.

The upgrade from 1.4 to 1.5 was for the most part entirely painless as the migration notes had promised. The application built and ran and none of our code seemed to be relying on any of the breaking behaviour between the versions.

There was just one problem, all our tests were failing. All the mocks were coming back as undefined with an obscure error url that didn’t really help as the advice it gave was about implementing a provider which applied to none of the mock setup that was happening in the code.

It took a bit of Googling around the problem (and hence this blog post to try and improve the situation) to find a related issue in Github that finally clued me off to the solution that we needed to update the Karma PhantomJS runner and more crucially the version of PhantomJS we were using.

As far as I can tell switching Karma to use PhantomJS 2 is a good idea irrespective of what version of Angular you are using so I think it would probably sensible to do this before you start updating Angular itself.

Standard
Programming

CloudFormation fails to create specified user

I recently had a problem with some historic CloudFormation where the user and their home directory was not being created. The problem and the solution were not complicated but my Google search returned nothing directly related to the problem except the AWS docs on the configuration syntax and there were no errors in the init log.

The user and their associated home directory were not being created which then meant when the scripting in UserData ran (which relied on a certain directory structure) I was getting a “directory not found” error.

The problem and solution are ridiculously straight-forward. The cfn-init scripts were not being installed or run. Without them configuration data in Metadata is not run, which is what the documentation in AWS::CloudFormation::Init pretty much says. I adapted this gist to install cfn-init and everything sprang into life again.

The reason I struggled so much with the problem was that I was modifying existing CloudFormation that had generated a successfully running application previous to my changes.

It took me hours to figure out that while the current version of the CloudFormation made no mention of cfn-init and yet apparently worked was simply because the necessary changes had not been checked into the source repository. Without a simple way to go back and review the actual CloudFormation config that was used (hopefully something that might change in a future version of CloudFormation) I assumed that what was missing was my knowledge and there was some other way of getting the Metadata to execute.

 

Standard
Programming

First impressions of Kotlin

Kotlin is one of the next-generation languages that builds on top of Java. It’s kind of a post-Scala and Groovy language that comes from JetBrains and therefore has a lot of static functionality that enables great tooling to be built on top of it.

It has been in development for a while but it is now getting a big push in terms of marketing as it approaches version one. I have noticed this a lot in terms of Android development, where Google and Oracle’s legal wrangle over the JDK code used in Android applications offers an opportunity for people who want great bytecode compatibility and post-Java 6 features but who cannot upgrade their Java version.

Caveats

This blog post is purely based on going through the tutorials and koans for Kotlin and not any production experience I have. This post is more a summary of my initial evaluation of whether to spend more time with this language.

Key features

Kotlin aims to have great interoperability with Java but aims to reduce boilerplate coding and eliminate certain classes of error within pure Kotlin code.

The Java legacy

Kotlin’s symbiotic relationship with Java means that fundamentally you have a language that has all of Java’s quirks and legacy and adds to it a new layer of syntax and complexity. Essentially Kotlin is syntax-sugar on Java so deep that it is like the inch-high frosting on a cupcake.

Scala has also had a strong influence on Kotlin but disappointingly this means that many of the quirky aspects of Scala have been transplanted to Kotlin. Most particularly Scala’s val and var system of maintaining compatibility with Java’s fundamentally mutable variable system.

Like a lot of object-orientated languages with lambda support, functions like filter or map are on the data and take a lambda. So you chain operations together in a trainwreck-style or if you don’t like that then you have to introduce intermediate variables. I prefer collection manipulations to be their own standalone functions which take a sequence or iterable and the lambda. This allows partial or deferred application.

What’s good about Kotlin?

Kotlin has all the higher-order function functionality that you would expect along with a straight-forward declaration and package-style namespacing.

It has some “annotation functions” that allow you to package data objects in the same way as Scala case classes.

If you limit yourself to functions and data then you have a compact language with the power to do meaningful work.

It reminds me a lot of Groovy but is typed and compiled and is more in the camp of “if it compiles it will work”.

Unsurprisingly the tooling in IntelliJ is excellent and it is easy to write and navigate around the code.

The extension functions allow a way of enhancing or bespoking code you don’t own that is more elegant than Scala’s implicit magic. The function declarations attach to the type and compiler magic introduces an implicit this. By comparison with implicit there is much less runtime magic and if you are using IntelliJ then the declarations are easy to navigate.

The type system

Over half the koans are concerned with type-compatibility with Java, in particular issues with generics and extension methods. Type inference didn’t seem that good or bad, you have to declare the types of parameters and the return type of functions, which is par for the course. I didn’t come across any confusing type errors although the extension methods sometimes had confusing scoping issues if I didn’t declare them correctly.

Rather like Groovy, Kotlin has decided to retain null compatibility with Java but uses Option and some built-in operators to allow some type-safety around nulls. I found the new operators to be more confusing that simple null-checking as they do some type-changing from Option[T] to T conditional on the Option being Some[T], otherwise the expression doesn’t get evaluated.

In theory this means you write code that accesses nested, potentially null attributes of an object in a single line without risking a Null Pointer Exception. In practice though it seemed just as likely that the code execution would get vetoed which meant that you have a subtle code branch after each use of a null-checking operator.

I’m not sure the special operators added any real value to the normal API for Option, they are less explicit in their behaviour and they really seem more concerned with reducing line count when interacting with legacy code.

So most of Kotlin’s typing seems concerns with retro-fitting fixes to the underlying Java type system. It certainly doesn’t seem to have an declared interest in having more sophisticated or powerful types.

Final thoughts: Scala versus Kotlin

Scala in many ways is much more ambitious than Kotlin but in outcomes they are very similar. Both fundamentally want to retain compatibility with Java including mutable variables, null, mutable collections and the Java type system. Both add higher-order functions and a system for extending code that you don’t own.

Obviously Scala is the earlier language and therefore a lot of what Kotlin is doing is feature matching.

The thing that separates them is really what purpose you are using them for. If you are looking for an actively developed language that is fundamentally an enhanced Java with modern features then Kotlin has better tooling and a more explicit extension system.

If you are looking for a richer type system that allows you to express behaviour as the result of the application of types or you are into category theory then Kotlin isn’t going to do anything for you and Scala is still the better choice.

Standard
Blogging, Programming, Web Applications

An overview of Javascript reactive frameworks

This post is only meant to be a snapshot of the current state of the various DOM virtualising webframeworks that are around. I’m partly publishing it to try and discover more that I may not be aware of.

Many of these frameworks trace an ancestry back to Om and React. However each one tries to deal with perceived problems with the original frameworks. The most common being that React is too heavy and opinionated while not providing a consistent data model for components. Om on the other hand is in Clojurescript and therefore represents too much to learn in terms of a new language and build process.

Libraries

Most of the libraries build on a few common building blocks that I’m not going to elaborate on here. Virtualdom was an early attempt to separate the core idea of React from the rest of the library code. Virtualdom is only concerned with creating, manipulating and stringifying DOM structures in-memory. Browser DOM APIs involving linking to the actual rendered document so managing virtual DOM is more efficient and simpler because you’re not interacting with these underlying libraries.

ImmutableJS provides a Javascript-idiom interpretation of the Clojure data structures that Om uses (and which are available as the standalone library Mori).

Omniscient

The first interesting framework to discuss is Omniscient, which as its name suggests is heavily influenced by Om but is written in Javascript and therefore does not require you to learn Clojure to use the same techniques that Om uses. Omniscient is built on top of React and ImmutableJS and uses its own library Immstruct to add reference cursors to ImmutableJS structures. Reference cursors allow a component to observe and change sections of a data structure without having to manipulate the whole thing. So for example a component can be given a single sub-key in an object that represents its state and it cannot access or change anything that is not under that key. The code can also be simplified to behave as if the sub-key was actually just the whole data object.

Omniscient doesn’t suggest an alternative to Om’s CSP, instead providing a mechanism for passing event flow functions down the component tree. You’re free to choose your own event libraries. It also means that you’re free to make your own mistakes here as no guidance is really given as to how to structure your event scheme appropriately.

Omniscient is one of the earliest frameworks to re-implement Om and therefore has one of the better sets of documentation on its Github pages. That said there’s not a lot of documentation and the framework does not have a massive community. The situation is worse in most of the other frameworks though so this might tip you over in favour of Omniscient.

Ractive

This is a bit of a Guardian shout out as the primary developer Rich Harris is a Guardian interactive developer.

Ractive (Github) is a little be different from the other frameworks as you can essentially think of it as Mustache templates backed by Observables. You declare a data-binding and write templates in normal Mustache syntax but behind the scenes Ractive is driven by changes in the data and then writes new section of DOM in-memory according to what has changed rather than DOM diff’ing.

Also Ractive sticks with two-way databinding rather than unidirectional data flow so failures in synchronisation or rendering can be problematic.

If what you want to do is render content over a Javascript data model then there is a lot in Ractive that is very compelling. It uses templates with a standard syntax that is well understood and is a soup and nuts framework that sticks to core Javascript syntax and features. However if you want to use your own event or data model you are out of luck.

Mercury

Mercury on the other hand prides itself on modularity. A microframework it attempts to create a glue layer that allows other libraries to interact in a sensible and consistent way. The default components are Virtualdom and its own observer pattern to wrap state.

Mercury’s biggest problem right now is its lack of documentation. There is an expectation that you are going to read the source code to understand what the framework is doing and how to interact with the API. I frankly think this is unrealistic. The project doesn’t currently supply the incentive to do that. Unless you have a very particular desire to avoid any framework lock-in or you want to use a very specific combination of libraries that is not supported elsewhere its hard to understand why you would invest your effort here rather than in frameworks that offer more support.

Cycle

Cycle is similarly experimental, its biggest claim is that it is truly reactive and that the rendered page is purely the result of change in state. The introduction is couched in computer science theory but it would seem that at its heart Cycle wraps RxJS and Virtualdom in a glue layer that has the programmer writing the transform sequence between the event and the DOM structure.

I think it is a positive feature that Cycle re-uses a popular library to manage its state-transitions rather than implementing yet another custom version of the Observable pattern. It also makes the framework easier to get started with if you are familiar with the Rx.

Using established libraries also makes the lack of documentation more acceptable as the Cycle readme only needs to explain how the glue works in the framework.

As something built on reactivity you have to get used to dealing with intermediate state which can be bit difficult for the beginner.

Essentially any event where the user would expect feedback means you need write the conditional structure in the output. So if the user types a character in an input box then you need to write the value of the input box to be the characters the user has typed so far. Most frameworks work at a higher level of abstraction or rather they map closer to the DOM APIs, so getting a working application means grokking the way the dataflow works.

If you’re looking for purity (and a resulting simplicity in implementation) but not to have to learn a bespoke API Cycle is nicely positioned.

WebRx

WebRx is similarly built on top of RxJS Observables but is a much fuller-fat framework that is much more a spiritual successor to Knockout than owing much to the influence Om or React.

Rather like React WebRx doesn’t really provide generalised event handling but instead has special sauce bindings for DOM events and a MessageBus system built over Rx.

It is also written in Typescript and generally looks to play well within the Microsoft ecosystem. It’s interesting to me as an example of how different a language has to be before its regarded as a barrier. Clearly the use of Typescript means there are people who will refuse to use the framework regardless of whether it works for their use case. Other people are going to be attracted exactly because it uses Typescript.

Deku

Language choices are also interesting in Deku which is another attempt to re-implement React in a superficial way.

Deku makes use of ES6 and 7 features and doesn’t aim to support a broad range of browsers (unlike say Ractive). Again that is going to rule it out for some people but this is a more interesting as now we are within dialects of the same core language. Language choice for implementing frameworks is not straightforward. What are you looking for? Conciseness? Editor support?

Deku aims to take the dom diffing approach but avoid getting caught in React’s framework and approach. In particular components are defined just as Javascript objects rather that classes and instances. Something I think makes it more elegant that normal React Components.

It does however still use JSX which is quite interesting as the framework claims to be taking a functional approach but actually uses a DSL for all its DOM construction.

The lifecycle hooks are slightly different with more hooks for different stages of the process and Deku uses some interesting function passing to send changed data down the tree to components.

Deku doesn’t take much influence from Om though. It doesn’t have sophisticated event handling and uses mutable data with generous access and callbacks on data write to do re-renders. This means bugs and state issues are no less likely to happen than with any other framework. It does adopt the single atom idea with a single tree representing the app and the app renderer being bound to the body element.

As such if you like the idea of React but don’t want to bound into its concept of how a Component should be defined but do like JSX and trust the implementors to create a better dom diff than Facebook or Virtualdom, this is the project for you.

Conclusion

I’ve only chosen a handful of frameworks to look at here, mostly based on the ones I know, I’m expecting people to point out more in the comments. I also haven’t used all of these frameworks. Road-testing all of them would be a bigger task than just trying to describe the design choices they’ve made.

The most common pattern is to try and improve the rendering time versus React by using different virtual dom difference algorithms. Usually this is combined with Observed variables that provide a Reactive component that allows changes in the data model to be conveyed to the DOM model with no coding required.

Few of the frameworks engage with the functional reactive programming paradigm by building abstract event streams or indeed any abstraction over discrete events.

The idea that the app should be a single data structure that represents the whole page seems to be gaining significant traction with several of the frameworks recommending this as an approach.

The explosion of frameworks resulting from the release of React is, I think, a positive thing. Initially it seems really daunting that you have all these choices but when you look at the real level of difference between them you can see that they are actually quite tightly coupled around a few common and core ideas and that mostly they express differences about the concerns that a framework should have which feeds into the wider conversation about micro or comprehensive frameworks.

Standard