Programming

Sustainable web development

This article about delivering a sustainable small website is one of the most inspiring things I’ve read in weeks. It made me think about the true costs of the web revolution and made me think again about the environmental claims we make about moving paper processes online. Green energy and clean websites are entirely possible but will people pay the premium for it?

Finally like all truly great pieces it offered some practical insight into problems I have been grappling with in my work. The observation that small, fast loading server-rendered pages are the same as client-side SPAs but more energy efficient is a great insight.

However the best was the idea of simplifying the navigation into its own page. This one thing is a piece of genius that hundreds of man-years of UX research have failed to discover.

Thinking about the energy cost of our websites is also a great proxy for thinking about the cost basis for our infrastructure and where we want to put our processing effort.

Standard
Software

Passive-aggressive collaboration

One interesting (and depressing) aspect of post-Github open source development is the use of the pull request as a passive aggressive way of putting off potential contributors, users and testers.

I have had the experience of discovering an issue with a piece of open source and raising an issue for it only to be “invited” to contribute a reliable failing test case, with a fix and all to the project’s contribution standards.

Now the joy of open source is being able to scratch your itch and prioritising your own problems by contributing solutions.

However there are often very good reasons why the maintainers should be the ones fixing the issues.

Firstly the maintainers should be the ones who gain most from fixing issues. One cool thing about Git-based development is that forking allows you to use existing codebases but not share the views and priorities of the original project. If I disagree with the direction or design of a project I can fork it and completely change the code to match my own aesthetics and priorities.

However in most cases I agree with the direction of the maintainers and I am simply pointing out an issue or problem that maybe they haven’t encountered in their context. I could scratch my itch but it is often more effective if the maintainer took my use case into consideration and reworked the codebase to include it.

Maintainers:

  • have more context on the code base
  • know more about the problems the code tackles
  • know their conventions and coding standards better than I do
  • have more invested in having an effective solution than I do

Looking at things like Guava which while open are effectively not open to contribution. This is a more honest approach than inviting non-trivial contributions.

Trying to take the perspective of the maintainer I know it is tedious when people do things outside of your area of interest (i.e. IE fixes when you’re not targeting that version of the browser or Windows fixes in a project targeted at UNIXes). However telling people to “fix it themselves” is not as honest as saying “that’s not our focus”. People can then decide whether they want to fork or not.

Standard
Software

In praise of fungible developers

The “fungibility” of developers is a bit of hot topic at the moment. Fungibility means the ability to substitute one thing for another for the same effect; so money is fungible for goods in modern economies.

In software development that means taking a developer in one part of the organisation and substituting them elsewhere and not impacting the productivity of either developer involved in the exchange.

This is linked to the mythical “full-stack” developer by the emergence of different “disciplines” within web software development, usually these are: devops, client-side (browser-based development) and backend development (services).

It is entirely possible for developers to enter one of these niches and spend all their time in it. In fact sub-specialisations in things like responsive CSS and single-page apps (SPA) are opening up.

Now my view has always been that a developer should always aspire to have as broad a knowledge base as possible and to be able to turn their hand to anything. I believe when you don’t really understand what is going on around your foxhole then problems occur. Ultimately we are all pushing electric pulse-waves over wires and chips and it is worth remembering that.

However my working history was pretty badly scarred by the massive wave of Indian outsourcing that happened post the year 2000 and as a consequence the move up the value-chain that all the remaining onshore developers made. Chad Fowler’s book is a pretty good summary of what happened and how people reacted to it.

For people getting specialist pay for niche work, full-stack development doesn’t contain much attraction. Management sees fungibility as a convenient way of pushing paper resources around projects and then blaming developers for not delivering. There are also some well-written defences of specialisation.

In defence of broad skills

But I still believe that we need full-stack developers and if you don’t like that title then let’s call them holistic developers.

Organisations do need fungibility. Organisations without predictable demand or who are experiencing disruption in their business methodology need to be flexible and they need to respond to situations that are unexpected.

You also need to fire drill those situations where people leave, fall ill or have a family crisis. Does the group fall apart or can it readjust and continue to deliver value? In any organisation you never know when you need to change people round at short notice.

Developers with a limited skill set are likely to make mistakes that someone with a broader set of experiences wouldn’t. It is also easier for a generalist developer to acquire specialist knowledge when needed than to broaden a specialist.

Encouraging specialism is the same as creating knowledge silos in your organisation. There are times when this might be acceptable but if you aren’t doing it in a conscious way and accompanying it with a risk assessment then it is dangerous.

Creating holistic developers

Most organisations have an absurd reward structure that massively benefits specialists rather than generalists. You can see that in iOS developer and mobile responsive web CSS salaries. The fact that someone is less capable than their colleagues means they are rewarded more. This is absurd and it needs to end.

Specialists should be treated like contractors and consultants. They have special skills but you should be codifying their knowledge and having them train their generalist colleagues. A specialist should be seen as a short-term investment in an area where you lack institutional memory and knowledge.

All software delivery organisations should practice rotation. Consider it a Chaos Monkey for your human processes.

Rotation puts things like onboarding processes to the test. It also brings new eyes to the solution and software design of the team. If something is simple it should make sense and be simply to newcomer, not someone who has been on the team for months.

Rotation applies within teams too. Don’t give functionality to the person who can deliver it the fastest, give it to the person who would struggle to deliver it. Then force the rest of the team to support that person. Make them see the weaknesses in what they’ve created.

Value generalists and go out of your way to create them.

Standard
Clojure

Clojurescript: is it any better yet?

A while ago I wrote about how Clojurescript was a square peg being hammered into a non-existent hole to complete indifference by everyone.

So has anything changed? Well Clojurescript has continued to be developed and it seems to have lost some of the insane rough-edges it launched with, it seems possible to use it with OpenJDK now for example.

Some people have made some very cool things by using Clojurescript as a dialect for writing NodeJS code. I was particularly struck by Philip Potter's talk on Marconi.

The appearance of core.async for Clojurescript is in my view the first genuine point of interest for non-Lisp fans. It provides a genuinely compelling model for handling events in an elegant way.

David Nolen, while being modest about his contribution, has also contributed some excellent blog posts about the virtues of Clojurescript. The most essential of which is the 101 on how to get a basic Clojurescript application going with core.async. This is an excellent tutorial but also is underpinned by hard work on getting the "out of the box" developer experience slick via the Mies Leiningen template.

Let's be clear about what a difference this makes. At the London Clojure dojo we once had a dojo about using Clojurescript, after a few hours all the teams limped in with various war stories about what bits of Clojurescript they had working and what was failing and why they thought it was failing. The experience was so bad I really didn't want to do Clojurescript as a general exercise again.

In the last dojo we did Clojurescript, we used Mies and David's blogpost as a template and all the teams were able to reproduce the blogpost and some of the teams were creating enhancements based on the basic idea of asynchronous service calls.

When someone pitches a Clojurescript idea in 2014 I'm no longer in fear of a travesty. That is a massive step forward.

And that's the good news.

Clojurescript! Who is good for!

After the dojo there were some serious discussions about using Clojurescript in anger. The conversation turned eventually to GWT. In case you don't remember or have never met it GWT is essentially a browser client kit for Java developers. In London it gets used a lot by financial institutions that need rich UIs for small numbers of people. Javascript developers are unlikely to be hired by those organisations so just like Google they end up with a need but the wrong kind of skills and GWT bridges the gap. A Java developer can use a familiar language and off-the-shelf components and will end up with a perfectly serviceable rich client-side app.

There is no chance in hell that a Javascript developer is going to use GWT to build their applications.

Clojurescript feels like the same thing. Clojure developers and LISP aficionados don't know a great deal of Javascript, they can program in Clojurescript and Google Closure and it is probably going to be okay. Better in fact than if they tried to create something in an unfamiliar language with all manner of gotchas.

But there is no chance in hell that a Javascript developer is going to use Clojurescript to build their applications.

Why Coffeescript failed

The reason I say this is because Coffeescript, a great rationalisation of Javascript programming is still viewed with suspicion by Javascript developers.

I asked some at a recent Javascript tech meeting why some people didn't use it and the interesting answer was that they couldn't really understand its syntax and were effectively translating the forms into Javascript. The terseness of the language was actually off-putting because it made it harder to mentally translate what the Coffeescript program was doing.

Adding LISP and Google Closure into that mix isn't going to make that mental disconnect any easier. The truth would appear to be that Javascript developers are simply not that disenchanted with their language. Clojurescript is going to have to offer something major to get over the disadvantage of a non-curly brace language and machine-optimised generated code.

At the Clojure dojo post-mortem people talked about the fact that Clojurescript helped avoid pitfalls and unusual behaviour in Javascript. That might seem a rationale argument to a Clojurian. However it was never an argument used on the JVM. "Hey Java just has all these edge-cases, why not use a LISP variant instead?".

In both languages practitioners of the language are deeply aware of the corner cases of their language. Since they are constantly working with it they are also familiar with the best practices required to make sure you don't encounter those corner cases.

Clojure on the JVM brought power, simplicity and a model of programming that made reasoning about code paths simple.

Clojurescript has the same advantages of code structure but doesn't really give more functionality over Javascript and still has a painful inter-operation story.

Javascript: the amazing evolving language

Javascript has a strong Scheme inheritance meaning that it already contains a lot of LISP inheritance.

Also unlike Java which ended up with a specification that was in the wilderness for years, Javascript has managed to keep its language definition moving. It's sorted out its split and with aggressive language implementers in the form of the competing browsers it is rapidly adding features to the core of the languages and standards for extensions.

Javascript is almost unique that when lots of people wrote languages that compiled into Javascript the community weren't stuffy about it but actually created a specification, Source Maps, that made it easier to support generated code.

Javascript has a lot of problem areas but it is also rapidly evolving and syntactic sugar and new language constructs are adding power without necessary creating new problems or complexity. It is expansionist and ruthless pragmatic, just like Clojure on JVM in many ways.

Better the devil you know?

Visual Basic isn't the best language in the world, it's certainly not the best language for creating apps on Windows. However for organisations and programmers who have invested a lot of time in a language and a platform it normally takes a lot to get them to change.

Usually, it will take an inability to hire people to replace those leaving or the desertion of major clients before change can really be countenanced.

Javascript developers are in the same sunk-cost quandary but there is nothing on the horizon that is going to force an external change. There may be better alternatives but Javascript is one of the easiest languages to learn. It's highly interactive and its right there in the console window of this very browser!

There's no lack of demand for good looking websites and browser hackery to differentiate one web product from the next.

Regardless of the technical merits of any alternative solution offers, and we are not just talking about Clojurescript, Elm offers a similar set of advantages, the herding effect is powerful. Your investment in Javascript is far more likely to pay off that putting time into one of the alternatives, none of which have momentum.

Sometimes there are real advantages to sticking close to the devil.

Node.js

For me the most interesting area of Clojurescript is being able to write Clojure and treat V8 as an alternative runtime.

People are already noticing some odd performance characteristics where some things run better on Node than they do on JVM, most particularly around Async.

Polyglot developers are a familiar sight in the server side, knowing a variety of languages is advantage for the general programmer. Server-side Javascript is really only for those who are a one-trick pony.

It is still going to be a niche area but it is much more likely to happen than in the client-side.

Standard
Web Applications, Work

The myth of “published” content

Working at the Guardian you often end up having conversations with people about the challenges you face in scaling to meet the often spiky traffic you get in online media. One thing that comes up again and again is the idea that content, once published is essentially static. Now there is a lot to be said for this as digital journalism sticks pretty close to a lot of the conventions of print media; copy is often culled from the print version and follows the 24 hour media cycle quite strongly.

However what is often surprising is the amount of edits a piece of content receives, particularly if it is not a print feature article. The initial version of an article is often the mandatory information and a few paragraphs sufficient to get across the basic story. It then goes through a number of revisions that often happen while the article is draft. Often but not always.

Once the article gets published online though it triggers a new wave of edits as language gets cleaned up and readers, editors and lawyers all descend on it. Editors now have a lot more tools to see what the reaction of the audience to a piece of content is and see how it is playing in social media. You also have articles picked up externally and that means making sure the article works as a landing page.

Naturally stories often develop their own momentum that requires you to switch from a single piece to a set of stories that are approaching different aspects of the overall reporting. You then need to link the different pieces of content together to form a logic package of content.

One thing that is interesting is looking at how many articles are changed after seven days. It is a surprising number as new stories often create a need to create a historic context and often historical stories look dusty in the light of breaking events. We have also had strange things happen with social news where aggregating sites pick up some story that was overlooked at the time.

All of this means that you cannot naively treat content as static but in fact means that you have an interesting decaching problem as it is true that content doesn’t change much, until it does start changing and then it needs to reflect the changes reasonably rapidly if you want to be picked up by things like Google.

 

Standard
Java

EuroClojure 2012 Day 1

So there were definitely two big themes in the talks on the first day of the conference.

The first has been about how to use event-based systems to create flexible aggregate data models. All speakers seem to have settled on a reduce or foldLeft approach for creating the aggregate but there have been two models put forward already CQRS and a kind of Aggregate query bus but really it seems that responsibility for accepting event data and allow querying and access to aggregated views seem to be responsibilities in the same system.

The other thing has been creating query systems using logical predicates. The were no less than three generic query systems put forward: core.logic for low-level flexible implementations that identify either data or results and two general query libraries: one from Datomic and the other from Cascalog.

Standard
Web Applications

Good magic, bad magic

Philip Potter pinged me his post on Sinatra magic during the week. Mark Needham’s comment and code on solving the mocking problem is good advice to the problem as posed.

At Wazoku where we use the often equally magical Bottle framework we don’t use top-down TDD but instead outside-in functional tests (with no funky runners as we don’t need CI). This solves the whole magic issue by shifting the attention to what the public interactions of the application are. This is one of the massive benefits of using a microapp HTTP/JSON/REST-like architecture. I could flip the API from Bottle to Django or Compojure or Sinatra and my test suite can keep on rocking and telling me whether the behaviour my consumers are relying on is correct.

The major thing I felt when reading through Philip’s post was the massive amount of effort that was going into testing relatively simple behaviour. This is a bit of anti-pattern with Agile developers (or perhaps it is part of the mastery thing where rote “correct” behaviour is modified by experience and judgement). One of the massive advantages of using something like Sinatra is that you can get a whole web app with rich behaviour into less than 200 lines. If you then create thousands of lines of test code and battle with the magic for hours on end you’ve completely destroyed your productivity.

If you have a code base that you expect to be large and highly contested by a large development team you need good, layered testing and to use frameworks that support that. If you have an app that is small and when its done it is done then there is no need to agonise as to whether it was done “right”.

The idea that top-down TDD is the only correct way to write software is corrosive. When faced with a generally poorly skilled and educated workforce it is good to have rules. I have imposed a certain style of TDD on a group myself because it gives a good framework for work and achieves very consistent output.

However with skilled people on small scale projects you can kill yourself by imposing arbitrary rules. I love Sinatra and while I might be equivocal about magic I think it is ridiculous to moan about it if you are using something as unicorn-packed as Ruby. For example Philip was trying to use RSpec mocks and stubs to do his TDD. The result is kind of saying that you’re disappointed that your “good” magic for testing didn’t work with the “bad” magic of a DSL for web applications. Even if your RSpec code passed its tests you still haven’t said anything particularly deep about the production behaviour of your application as your unit testing environment was severely compromised by the manipulations of your mocking framework.

So my rule of thumb is: if its simple, do it; if it was simple, functionally test it; if it was never really simple then test-drive it with suitable tools.

Standard
Java, Programming

Python as a post-Java language

I’m a UNIX-based developer and since 2000 I have been working mainly with Java and then JVM languages. When Java 7 slipped I made no real secret of the fact that Java was in a lot of trouble. The post-Oracle world though looks even worse with a lack of clarity of what in the core ecosystem is free, open source and liability free.

Clojure and to a less extent Scala are great steps forward so I don’t feel the burning need for a Java 7/8 whatever. However a moribund or tainted JVM is a major problem and so I’m now thinking about what the post-Java escape route looks like. On the web front it is pretty obvious, Python and Ruby are great languages with great frameworks for developing web-based application. For the server-side heavy lifting it is a lot less clear, people are talking about Google Go but that does feel quite low-level, I’m not sure I’m ready to go back to pointer wrangling even with memory-management. It feels like something you’d build a tool out of not an application. Mono feels like more of the same problems of wrestling with big companies with vested interests, if you are going to do that then why not try and sort out the OpenJDK?

As the title of the post suggests the language I am most inclined towards right now is Python. It is a really concise but clear language that on UNIX systems comes with an amazingly comprehensive set of libraries and which has a virtual environment and dependency management that is on a par with RVM and gem.

The single issue that comes up is performance, what I have been finding that for 80% of the work I am doing performance is okay and I’m producing a fraction of the code I would normally have to create. For that last 20% maybe I am going to have to look at something like Go or (god forbid) go back to C but I would much prefer to see a Clojure or Scala that could run on top of something like LLVM. I also have some hope of smarter people than me making progress on a JIT for Python that might take 20% down to a figure where performance would matter so much to me I wouldn’t mind sweating to make it happen.

Standard
Java, Ruby, Web Applications, Work

Working with Rails

I have recently struggled to try and get a prototype web application done with Ruby on Rails. It has been a really great experience and like all such experiences consists of both good and bad moments. Things that are particularly striking coming from a Java Web environment:

  • Integrated console
  • No recompile/build/deploy cycle
  • No mix and match components
  • No real programming language skill required.

It is also remarkable how quickly people revert to the level of thought, planning and execution that you might use for a shell script. Even I have found myself getting caught up in “playing” with the app interactively rather than being focussed on creating behaviour and functionality in a structured way.

Elaborating on these points; in Java web development the first issue is usually building up your component stack. Some people just choose SpringĀ  but those people are idjits! If you are working purely in a web tier without any need for any backend interaction then it is probably a good bet but generally you want to put some thought into how you are going to assemble your various stacks. You tend to have to choose your MVC web framework, your persistence framework and your service framework (if appropriate). You often have to give some thought to your caching and messaging frameworks if they are relevant.

With Rails you just use the appropriate Active Module or Rails built-in features. If there is something that is going to ease some pain for you then it is going be operating as a plugin. That’s it, no framework holy war. It also means that there are a lot of applications that are just not going to be suitable for a Rails application as all those Java components have various strengths for different environments. I certainly wouldn’t want to tackle a legacy database with ActiveRecord for example. Not that it couldn’t be done but I wouldn’t want to do it.

The zero-turnaround is impressive after the build-deploy-check cycle. It was a wow factor last year when I saw the Phobos framework being demonstrated by Sun and its not lost any of its shine. By separating its deployment environments so ruthlessly Rails is able to deliver a really positive developer experience.

The interactive console takes a bit of getting used to in terms of faking browser requests but once you get used to doing so it is another tool that you wonder how you’ve managed without for so long. It’s much more intuitive to use than setting up a remote Java debugging instance (although admittedly you do similar tasks in both). It allows you to scratch those itchy “why?” questions.

And finally that lack of programming skill? A bit controversial perhaps? Well I feel that in recent weeks what I have been learning is how to manipulate Rails. The fact that it is implemented on Ruby may allow a lot of features to be implemented in the way they that they are but you are very rarely called upon to show Ruby madskillz. Instead the majority of the time you are simply plugging little customisations into the Rails framework. It is called Rails for a reason after all, when you are on them you are amazingly productive but if you can’t package your problem into a Rails solution then you are out of luck, you’re going to have to develop your own solution and that is going to be hard work. I have been trying to develop my Ruby skills (that is a whole other story) but the truth is you can bang out something that is acceptable with very little Ruby knowledge. You can go a lot of the way purely by mastering Ruby’s hash syntax.

So have I been converted? Well for all its problems I do have a hankering to get back to my great big Java applications with their holy wars and heavyweight processing. After all being on the Rails is fine for getting things done quickly but it can feel claustrophobic. I am also really glad that after getting distracted by the whole JSF controversy the challenge Rails presents to the status quo of web development means that a lot of the Java frameworks are starting to respond to the real problems faced in web development and ensuring that the easy stuff should be easy.

Standard