Programming, Web Applications, Work

Why can’t Forms PUT?

HTML Forms can declare a method, the HTTP verb that is used when the form is submitted, the value of this method is GET or POST.

The HTML5 spec briefly had PUT and DELETE as valid methods for the form method but has now removed them. Firefox also added support and subsequently removed them.

Recently over the course of Brexit night at The Guardian we got into a discussion about why this was the case and what the “right” way to map a form into a REST-like resource system would be.

The first piece of research was to dig into why the additional methods had been added and then removed. The answer (via Ian Hickson) was simple: PUT and DELETE have implied idempotency, the nature of form submission is that it is inherently uncacheable and therefore cannot be properly mapped onto those verbs.

So, basic problem solved, it also implies the solution for the url design for a form. A form submission represents a user submitting an untrusted data payload to a resource, this resource in turn choose to make PUT or DELETE requests but it would be dangerous to have the form do this directly.

The resource therefore is one that represents the form submission. In terms of modelling the URL I would be tempted to say that it takes the form :entity/form/submission, so for example: contact/form/submission.

There may be an argument that POSTing to the form resource represents submission so the submission part of the structure is unnecessary. In my imagination though the form resource itself represents the metadata of the form while the submission is the resource that essentially models a valid sumbission and the resource that represents the outcome of the submission.

Web Applications, Work

Why don’t online publishers use https?

Why don’t big publishers use https instead of https? The discussion comes up every three to six months at the Guardian and there seems to be no technical barrier to doing this. There has been a lot of talk about where the secure termination happens and how to get certificates onto the CDN but there seem to be good answers to all the good questions. There doesn’t seem to be any major blockers or even major disadvantages in terms of network resources.

So why doesn’t it happen? Well public content publishers are dependent for the most part on advertising and online advertising is a total mess.

Broken and miss-configured advertising is a major source of issues and the worst aspect of the situation is that you really don’t have much control over what is happening. When you call out to the ad server you essentially yield control to whatever the ad server is going to do.

Now your first-level campaigns, the stuff that are in-house, premium or bespoke campaigns are usually designed to run well on the site and issues with this are often easy to fix because you can talk to your in-house advertising operations team.

However in a high-volume site this is a tiny amount of the advertising you run because you tend to have a much larger inventory (capacity to serve ads) in practice than you can sell. That is generally because supply of online advertising massively outstrips demand.

The way the discrepancy is made good is via ad exchanges which are really clever pieces of technology that try to find the best price for available both publisher and ad buyer. Essentially the ad exchanges try to establish a spot price for an available ad slot amongst all the campaigns the buyers have set up.

However you have virtually no say over what the format of the advert the exchange is going to serve up. The bundle of content that makes up the ad is called the “creative” and might be a simple image but more likely is a script or iframe that is going to load the actual advert, run personalisation and tracking systems.

You have no real control as to what the creatives are and they certainly haven’t been written with your site in mind and most probably security is a very minimal concern compared to gathering marketing information on your view.

So if the creative contains any security breaking rule or any resource that is not also https they you get a security exception on the site. The customer then blames you for being insecure.

One of our consumer products, which do all run under https, ran ads and every other month this issue would come up. In the end we decided that the value of the subscription was more than the value of any advertising that was undermining the image of being secure and reliable so we took the advertising off.

And therefore until agencies and ad exchanges change their policies so that ads are only served off https this situation is unlikely to change. Ironically there is no reason for ads to be served off https since they don’t want to be cached and wants to do lots of transactional stuff with the client anyway.

If the online advertising business went secure-only then online publishers would be able to follow them. Until then public pages are likely to remain on http.


The gold-plated donkey cart

I'm not sure if he came up with the term but I'm going to credit this idea to James Lewis who used it in relation to a very stuck client we were working on at ThoughtWorks.

The golden donkey cart is a software delivery anti-pattern where a team ceases to make step changes to their product but instead undergoes cycles of redevelopment of the same product making every more complex and rich iterations of the same feature set.

So a team creates a product, the donkey cart, and it's great because you can move big heavy things around in it. After a while though you're used to the donkey cart and you're wondering if things couldn't be better. So the team gets together and realise that they could add suspension so the ride is not so bumpy and you can get some padded seats and maybe the cart could have an awning and some posts with rings and hooks to make it easier to lash on loads. So donkey cart v2 is definitely better and it is easier and more comfortable to use so you wonder, could it be better yet.

So the team gets back together and decides that this time they are going to build the ultimate donkey cart. It's going to be leather and velvet trim, carbon fibre to reduce weight, a modular racking system with extendible plates for cargo. The reins are going to have gold medallions, it's going to be awesome.

But it is still going to be a donkey cart and not the small crappy diesel truck that is going to be the first step on making donkey carts irrelevant.

The gold-plated donkey cart is actually a variant on both the Iron Law of Oligarchy and the Innovator's Dilemma.

The donkey cart is part of the valuable series of incremental improvements that consists of most business as usual. Making a better donkey cart makes real improvements for customers and users.

The donkey cart team is also there to create donkey carts. That's what they think and talk about all the time. It is almost churlish to say that they are really the cargo transport team because probably no-one has ever expressed their purpose or mission in those terms because no-one else has thought of the diesel truck either.

Finally any group of people brought together for a purpose will never voluntarily disband itself. They will instead find new avenues of donkey cart research that they need to pursue and there will be the donkey cart conference circuit to be part of. The need for new donkey cart requirements will be self-evident to the team as well as the need for more people and time to make the next donkey cart breakthrough, before one of their peers does.


Agile: are scrummasters the masters?

One of the fault lines in modern Agile development remains the purpose and application of process. For me the fundamental conflict between a developer and a “scrummaster” is to do with what the main purpose of that role is. Scrummasters often profess a servant manager role for themselves while actually enacting a traditional master hierarchical function.

The following is the acid test for me. The servant manager is one who takes the work I am doing and expresses it in a form that allows people outside the team to understand what I am doing, the progress I have made on it and make predictions about when my work will be complete.

The traditional manager instead tries to control my work so that it fits neatly into the reporting tools that they want to use. They don’t hesitate to interfere, manipulate and control to make their life easier with their own superiors.

Calling yourself a servant manager but then telling people how to structure their work is paying lipservice to a popular slogan while continuing a strand of managerial behaviour that has been proven to fail for decades.

Software, Work

Up-front quality

There has been a great exchange on the London Clojurians mailing list recently talking about the impact of a good REPL on development cycles. The conversation kicks into high-gear with this post from Malcolm Sparks although it is worth reading it from the start (membership might be required I can’t remember). In his post Malcolm talks about the cost of up-front quality. This, broadly speaking, is the cost of the testing required to put a feature live, it is essentially a way of looking at the cost that automated testing adds to the development process. As Malcolm says later: “I’m a strong proponent of testing, but only when testing has the effect of driving down the cost of change.”.

Once upon a time we had to fight to introduce unit-testing and automated integration builds and tests. Now it is a kind of given that this is a good thing, rather like a pendulum, the issue is going too far in the opposite direction. If you’ve ever had to scrap more than one feature because it failed to perform then the up-front quality cost is something you consider as closely as the cost of up-front design and production failure.

Now the London Clojurians list is at that perfect time in its lifespan where it is full of engaged and knowledgeable technologists so Steve Freeman drops into the thread and sensibly points out that Malcolm is also guilty of excess by valuing feature mutability to the point of wanting to be able to change a feature in-flight in production, something that is cool but is probably in excess of any actual requirements. Steve adds that there are other benefits to automated testing, particularly unit testing, beyond guaranteeing quality.

However Steve mentions the Forward approach, which I also subscribe to, of creating very small codebases. So then Paul Ingles gets involved and posts the best description I’ve read of how you can use solution structure, monitoring and restrained codebases to avoid dealing with a lot of the issues of software complexity. It’s hard to boil the argument down because the post deserves reading in full. I would try and summarise it as the external contact points of a service are what matters and if you fulfil the contract of the service you can write a replacement in any technology or stack and put the replacement alongside the original service.

One the powerful aspects of this approach is that is generalises the “throw one away” rule and allows you to say that the current codebase can be discarded whenever your knowledge of the domain or your available tools change sufficiently to make it possible to write an improved version of the service.

Steve then points out some of the other rules that make this work, being able to track and ideally change consumers as well. Its an argument for always using keys on API services, even internal ones, to help see what is calling your service. Something that is moving towards being a standard at the Guardian.

So to summarise, a little thread of pure gold and the kind of thing that can only happen when the right people have the time to talk and share experiences. And when it comes to testing, ask whether your tests are making it cheaper to change the software when the real functionality is discovered in production.


Agile software development defers business issues

My colleague Michael Brunton-Spall makes an interesting mistake in his latest blog post:

much of our time as developers is being completely wasted writing software that someone has told us is important.  Agile Development is supposed to help with this, ensuring that we are more connected with the business owners and therefore only writing software that is important.

Most Agile methodologies actually don’t do what Michael says here. Every one I’ve encountered in the wild treats it as almost axiomatic that there exists someone who knows what the correct business decision is. That person is then given a title, “product owner” for example and then is usually assigned responsibility for three things: deciding what order work is to be done, judging whether the work has been done correctly and clarifying requirements until they can be reduced to a programming exercise.

That’s why it was liberating to come across System Thinking which does try to take a holistic approach and say that any organisation is only really as good as its worst performing element. Doing that does not eliminate all the process improvements in development that Agile can provide but also illustrates that a great development team doing the wrong thing is a worse outcome than a poor development team doing the right thing.

The invention of the always correct product owner was a neat simplification of a complex problem that I think was probably designed to avoid having multiple people telling a development team different requirements. Essentially by assigning the right to direct the work of the development team to one person the issue of detail and analysis orientated developers getting blown off-course by differing opinions was replaced by squabbling outside the team to try and persuade the decision maker. Instead of developer versus business the problem was now business versus business.

Such a gross simplification has grave consequences as the “product owner” is now a massive point of failure and few software delivery teams can effectively isolate themselves from the effects of such a failure. I have heard the excuse “we’re working on the prioritised backlog” several times but I’ve never seen it protect a team from a collectivised failure to deliver what was really needed.

Most Agile methodologies essentially just punt and pray over the issue of business requirements and priorities, deferring the realities of the environment in the hoping of tackling an engineering issue. Success however means to doing what Michael suggests and trying to deal with the messy reality of a situation and providing an engineering solution that can cope with it.


Breaking the two-week release cycle

I gave a lightning talk about some of the work I did last year at the Guardian to help break the website out of the two-week release cycle and make it possible to switch to a feature-release based process. It’s the first time I’ve given a public talk about it although I have discussed with friends and obviously within the Guardian as well where we are still talking about how best to adopt this.

I definitely think that feature-releasing is the the only viable basis for effectively software delivery, whether you are doing continuous delivery or not.

In a short talk there’s a lot you have to leave out but the questions in the pub afterwards were actually relatively straight-forward. The only thing I felt I didn’t necessarily get across (despite saying it explicitly) was that this work was done on the big Enterprise Java monolith at the Guardian. We aren’t talking about microapps or our new mobile platform (although they too are released on a feature basis rather than on a cycle) we are talking about the application that is sometimes referred to as the “Monolith”. It was really about changing the world to make it better rather than avoid difficulty and accepting the status quo.

Feature-releasing has real benefits for supporting and maintaining software. On top of this, if you want to achieve collective team effort then focussing on a feature it going to better rather than doing a swath of work in a mini-waterfall “sprint”. The team stands a better chance of building up a release momentum and cadence and from that building up stakeholder confidence and a reputation for responsive delivery.