Work

The gold-plated donkey cart

I'm not sure if he came up with the term but I'm going to credit this idea to James Lewis who used it in relation to a very stuck client we were working on at ThoughtWorks.

The golden donkey cart is a software delivery anti-pattern where a team ceases to make step changes to their product but instead undergoes cycles of redevelopment of the same product making every more complex and rich iterations of the same feature set.

So a team creates a product, the donkey cart, and it's great because you can move big heavy things around in it. After a while though you're used to the donkey cart and you're wondering if things couldn't be better. So the team gets together and realise that they could add suspension so the ride is not so bumpy and you can get some padded seats and maybe the cart could have an awning and some posts with rings and hooks to make it easier to lash on loads. So donkey cart v2 is definitely better and it is easier and more comfortable to use so you wonder, could it be better yet.

So the team gets back together and decides that this time they are going to build the ultimate donkey cart. It's going to be leather and velvet trim, carbon fibre to reduce weight, a modular racking system with extendible plates for cargo. The reins are going to have gold medallions, it's going to be awesome.

But it is still going to be a donkey cart and not the small crappy diesel truck that is going to be the first step on making donkey carts irrelevant.

The gold-plated donkey cart is actually a variant on both the Iron Law of Oligarchy and the Innovator's Dilemma.

The donkey cart is part of the valuable series of incremental improvements that consists of most business as usual. Making a better donkey cart makes real improvements for customers and users.

The donkey cart team is also there to create donkey carts. That's what they think and talk about all the time. It is almost churlish to say that they are really the cargo transport team because probably no-one has ever expressed their purpose or mission in those terms because no-one else has thought of the diesel truck either.

Finally any group of people brought together for a purpose will never voluntarily disband itself. They will instead find new avenues of donkey cart research that they need to pursue and there will be the donkey cart conference circuit to be part of. The need for new donkey cart requirements will be self-evident to the team as well as the need for more people and time to make the next donkey cart breakthrough, before one of their peers does.

Standard
Web Applications

State of the Browser 2014

I haven’t been to State of the Browser before. It is a very cheap one day conference during the weekend on the topic of web standards and the web in general.

Conway Hall, the venue is a beautiful place and very recommended. However the grand aura of humanist lectures did remind you how lame most slide-based presentations are. Shut out the light, we can’t see the cat gif!

The theme and topics of the conference are vague and therefore there was a lot of variety in the talks. More than half were coming from professional vendor advocates and while slick and enjoyable there was a palpable sense of yearly objectives being ticked off. Community communication, check; reminder of organisation mission, check. The rest of the talks were pretty crappy though so its not all roses in the community either.

I’ve put down a few immediate reaction thoughts but I thought I would try and formulate some general takeaways.

Firstly the meaning of the web is very vague, there was an attempt to formulate the meaning of a “web platform” but it floundered a bit. The difficulty is not really what is the web, which is fundamentally unchanged since its inception, but rather what are all the companies doing when they try and build and expand on web?

Essentially what do browser vendors talk about when they talk about the web? To them the web is the input that the browser will accept. Microsoft, Mozilla, Opera and Google were all represented along with Telefonica who are making a big bet on Firefox OS.

One key theme was the belief that affordable smartphones (say below £50 to by and presumably close to £10 a month to run) are imminent and they will herald a new wave of traffic and content consumption. I feel that broadening on-demand access to the web is a good opportunity but the value of this audience, beyond hopefully buying data plans that are more expensive than talk minutes and text bundles, was utterly unproven and seemed an issue of no concern to the speakers.

One interesting thing about web development is that it is a place where visual design, technology and content creation collide into one huge grope box orgy where everything gets mixed up with everything else.

The visual design of the web was mentioned more than a few times and a lot of the standards work was essentially about delivering more fidelity to conceptual designs. It’s interesting that this is seen as fundamentally good thing rather than being interrogated. Perhaps it was discussed in earlier years.

There was also an interesting division in what people saw as their responsibilities. Javascript is now sufficiently complex that there is stratification and specialisation even with this niche. “Glass” people do UX, HTML and CSS, Javascript people do MVC “backend” work and performance and literally no-one is thinking about how the server could make any of this easier.

There was a dispiriting sense from a technology perspective of people hitting everything in sight with a golden hammer made of HTML/CSS/JS. About a fifth of the things discussed on stage boiled down to “a written standard for accessing OS capabilities based on an implementation of that standard”. It makes you appreciate things like Linux where there is pressure to actually tackle root problems and needs rather than layering hack on hack. The acceptance of the diabolic state of touch detection is an example, leading to the suggestion that you should progressive enhance on the detection of mouse events. I mean after all why use a filesystem abstraction when you could just iterate over /dev yourself?

The same paucity of leadership came up on the issue of HTTP 2 where it became clear that the vendors regard it as a way of dealing with the overhead of HTTP connections not really as a way to create the right kind of networking for the new activity we want to perform online.

It was also nice to see not one but two “standards” for defining viewport relative sizes: vw in the viewport spec (which seems very sensible and progressive by the way) and w in the picture/srcset responsive images standard.

There were a few moments when people seemed to touch on a better way of doing things, for example, declarative programmatic rules for layout; but these were rare. Maybe it’s just not that kind of conference.

In terms of talks the clear standout was Martin Beeby’s talk on what the Internet Explorer team have been doing to remove bottlenecks from their rendering. Most of the stuff was sensible and straight-forward but the detail on GPU interaction was fascinating, particularly on picture loading.

One massive problem with the conference was the weird idea that speakers weren’t going to take questions after their talks. Martin mentioned that buffers between the browser and the GPU were small and I would have loved to have know whether than was an intrinsic limitation or not. The lack of ability to follow up on issues diminished the utility of all the talks.

Other than that the walkthroughs of specifications of viewport, service workers (particularly the caching API) and the picture tag were all helpful. Andreas Bovens’s talk also had a helpful review of pixel density and its new related units.

The talks were filmed, I have no idea whether they will posted at some point but those are the ones I’d recommend.

The ticket was very cheap but the main issue of the conference was the time it takes. The programming is very baggy, I felt if all the talks had been halved in length and the panel discussion chopped to make room for post-talk questions there would have been a really good long afternoon of material.

I’ll probably give it another go next year but be a bit more ruthless about what talks to attend.

Standard
Web Applications

Better than Freemium

The new Kickstarted blogging platform Ghost has an interesting payment model. At the free tier you have full access to the platform but you are allowed zero views of the content you create.

Normally with blogging software you want to encourage as many page views as you can get to help promote your platform. The Ghost approach is an interesting way of dealing with the issue of trying to explain your product and have people try it without resorting to free tiers or advertising-supported freemium.

However it also means that you are making an open-ended commitment to the platform, if you ever stop paying then all your content disappears off the internet.

Posthaven is more appealing because it makes explicit promises about the persistence of your content. On the other hand as a replacement for Posterous it has less need to explain its proposition.

Having encountered the issue of offering free trials at Wazoku I wondered whether what we were really learning was that our product wasn’t simple enough to pitch a minimum subscription.

Making people pay something, no matter how notional is a more effective way of gathering feedback than the analytics and subjective feedback of free trials.

This pretty cool blog post on removing free plans at Trak.io makes a load of really good points about what kinds of thing goes wrong with free plans and the freemium model.

I’m not sure what the answer is to people not understanding your product but free trials are not the answer. The ultimate feedback on your product is whether someone will pay for it or not.

Standard
Programming

Why I’m excited about Julia

Julia is a new language that promises (or threatens) to take the best of all existing languages and put them into one fantastic package.

As a Python fan I found the syntax immediately appealing and familiar. However there are a number of Python niggles that appear to be improved in Julia.

Don’t like significant whitespace? Julia is using a end keyword.

function add(x, y) x + y end

Find Python’s keyword initialising rule counter-intuitive? Julia re-initialises on invocation.

function test_init(a, b = [1])
println(a, b)
push!(b, a)
println(b)
end

test_init(5) # Prints 5, 1, 1, 5

test_init(7) # Prints 7, 1, 1, 7

Wish that sequence operations were more consistent?

a = [1, 2, 3] # 3-element Array{Int64,1}: (via type inference)
shift!(a) # 1
pop!(a) # 3

Julia also comes with a shell which has all the niceties of ipython.

Julia is typed but so far the inferencing for primitives has been good enough. For collections you only seem to need a type declaration if the collection is initially empty.

Julia is also trying to do a lot with parallel processing and performance. I haven’t looked into that so much because that isn’t my usecase but so far the requirements for efficient typing aren’t causing any ceremony in the code. There is also some innate immutability in things like tuples but apart from that it is mutable by default with an immutable keyword applied to data structures that allows compiler and GC optimisation.

So Julia is a language that is immediately accessible to all Pythonistas, currently has less surprises than Python and is typed but enough inference to avoid them getting in the way. Pretty exciting!

Standard
Gadgets

The Doxie handheld scanner

Doxie makes handheld scanners, which seemed the perfect replacement for my flatbed scanner that is currently without working drivers for any modern OS.

I opted for the DoxieOne even though my scanning is primarily for Linux and there it wasn’t going to use all the integration and software the device comes with.

Doxie’s failure to integrate with Linux is a bit baffling but the basic behaviour is absolutely acceptable. It connects as a standard storage device by USB and you can just use all your usual tools against the device’s drive.

The scans produced are good quality and unlike a flatbed it will scan any length of document as long as it is roughly A4 or less in width. Of course you can’t use it to scan books (although there is a mini-flatbed version) if that matters to you.

The only real difficulty in using the Doxie is that an slight misalignment when you first engage the roller can get very pronounced by the end of the scan and the result is a picture at an angle that is hard to fix up in an image editor. If I realise that something is feeding incorrectly I tend to rescan as it is easier.

I was looking for a portable scanner that is easy to store and interact with and the Doxie meets the bill in an elegant way.

Standard
Programming

The state of microservices

One of the liveliest sessions at Scale Summit was the one on microservices where opinions flowed free and fast in a rapidly rotating fishbowl.

There were several points of interest that I took away and I thought I would write them up here as some of the key questions. In another post I’ll talk about the problems with microservices that haven’t been solved yet.

Are we doing microservices yet?

Some people in the room had already adopted microservices, the reasons given included: breaking down or trying to change functionality monolith codebases, trying to scale up an existing applications or architectures or actually as an existing best practice that should be applied more generally.

What is a microservice?

A few people wanted to get back to this. Showing that while it is a handy term it isn’t universally understood or agreed.

I personally think a microservice is a body of code whose purpose and implementation is easy to understand when studied, adheres to the Unix philosophy of doing one thing well and which can be re-implemented quickly if needed.

Are microservices just good practice or righteous SOA?

Well naturally you can regard every new concept as being a distillation of previous good practice. However the point of naming something is to try and make it easy to triangulate on a meaning in conversation.

Won’t microservices just be corrupted by consultants and architects?

Yes, everything popular gets corrupted in time. I’m okay with that because in the interval we have a handy term to describe some useful patterns of solution design.

Don’t microservices complicate operations?

One attendee put it well: microservices are recursive so if the operations team are going to support them then they should be in the business of creating a service that deploys services.

Some people felt that for microservices to work developers and teams had to have full stack ownership and responsibility but I felt that was trying to smuggle devops in under the microservices banner.

I think microservices are best deployed on a platform and that platform defines what a deployable service is and can be responsible for shutting down misbehaving services.

Such a scheme allows for other aspects of the Unix way to be applied such as man pages, responding to –help and other useful conventions.

The platform can check whether these conventions have been met before deploying the service.

Isn’t microservices just a way to discuss the granularity of a service?

Yes in a way, although there are a few other practices that make up a successful application of microservices you can just think about it as being a way of checking the responsibility boundaries of your service and how easy it would be to replace.

If your service has multiple responsibilities and is difficult to replace easily then it might have problems.

A lot of people wanted to use AWS an an example of good services without microservices. I think AWS is a good example of service implementation: each part has good boundaries and there are a few implementations of the key APIs. Things like the AWS security functionality is a good example of how you have to work hard to avoid having services rely on other services, and the result isn’t elegant as a result.

I would argue that public-facing APIs are probably where you want to composite microservices to provide a consistent facade onto the functionality though.

As other delegates pointed out, isolating services makes them easier to scale. If starting a server in the EC2 API requires more resources than shutting it down you might prefer to scale up just the creation service rather than many instances of the whole API which are unlikely to be used or consume resources.

As ever, horses for courses, you’re going to know your domain better than I do.

Don’t microservices cause problems as well as solve them?

Absolutely, choosing the wrong solution at the wrong time is going to cause problems. Zealously over-applying microservices or applying the idea to absurd levels is not going to have a happy outcome.

A guess a good point is that we know our problems with existing service implementations. We don’t know what problems there are with microservices or whether they have logical and simple solutions. However they are helping us solve some of our known problems.

Aren’t microservices simply REST-ful services done right?

The most common form of microservice today is probably one implemented via HTTP and JSON. However this form isn’t prescriptive. ProtocolBuffers might be a better choice for an internal exchange format and ZeroMQ might be a better choice for a transport.

I also think that message queues are a good basis for microservices with micro consumers and producers focussing on tight message types.

See also my mini-list of microservice myths which has more on this subject.

Should we be doing microservices?

I suspect that doing microservices for the sake of ticking a solution buzzword is bad. However I think microservices seem a pretty good solution to a problem I know I’ve seen in a fast-moving domain where you are trying to innovate without creating a maintenance burden or large legacy.

Standard
Programming

Scale Summit 2014

Scale Summit is the new Scale Camp, an unconference aimed at bringing the same kind of topics as you might expect at Velocity.

This was the first Scale Summit, the venue was excellent as was the food (especially the bacon rolls, from Eden apparently) and supply of drink. Scale Summit happens under Chatham House rules so there’s no attribution of what is said which allows the attendees to be really frank and also for people to be free with what they really know rather than hedging and trying to be “on message”. It makes for a fascinating gathering.

The sessions varied in their organisation but all focussed on discussion between the participants. I managed to go to the Elasticsearch session, which was interesting for the practical boundaries that people were finding and also the operational knowledge. On the subject of using ES as the primary application store, the feeling seemed to be “not yet”, but there was also some words of wisdom about separating out document stores and search functionality and not finding a superficial unity in the two purposes.

The microservices session was a fast and furious fishbowl, easily the liveliest event and one that is going to require a post in its own right. It was interesting to see that the room split into practitioners and people who were sceptical that microservices were a thing or held value over conventional service development.

After lunch I sat in on what can be done to get frontend testing off the critical path to production (not much now but clearly more effort needs to be made), distributed DOS attacks on transactional sites (not as scary as I imagined but again we have to be thinking about how this works), distributed data stores (good war stories, felt better informed for going), getting ops and developers to work together and Linux containers (definitely going to try Docker now).

I had quite a few questions going into the event and while I didn’t get all the answers I hoped for I did at least establish that smart people don’t have simple answers to them either which is reassuring. It’s hard to tell in the heat of it all whether you’re on the edge of things doing things that are pushing the boundaries or simply over-complicating your situation.

The attendees were nicely mixed and from a range of backgrounds, ops, architecture and developers were all well-represented so you felt you were seeing a rounded situation.

The unconference format left me wanting more rather than feeling I had had enough. The openess was amazing and I am planning on being there next year.

Standard