Software

Passive-aggressive collaboration

One interesting (and depressing) aspect of post-Github open source development is the use of the pull request as a passive aggressive way of putting off potential contributors, users and testers.

I have had the experience of discovering an issue with a piece of open source and raising an issue for it only to be “invited” to contribute a reliable failing test case, with a fix and all to the project’s contribution standards.

Now the joy of open source is being able to scratch your itch and prioritising your own problems by contributing solutions.

However there are often very good reasons why the maintainers should be the ones fixing the issues.

Firstly the maintainers should be the ones who gain most from fixing issues. One cool thing about Git-based development is that forking allows you to use existing codebases but not share the views and priorities of the original project. If I disagree with the direction or design of a project I can fork it and completely change the code to match my own aesthetics and priorities.

However in most cases I agree with the direction of the maintainers and I am simply pointing out an issue or problem that maybe they haven’t encountered in their context. I could scratch my itch but it is often more effective if the maintainer took my use case into consideration and reworked the codebase to include it.

Maintainers:

  • have more context on the code base
  • know more about the problems the code tackles
  • know their conventions and coding standards better than I do
  • have more invested in having an effective solution than I do

Looking at things like Guava which while open are effectively not open to contribution. This is a more honest approach than inviting non-trivial contributions.

Trying to take the perspective of the maintainer I know it is tedious when people do things outside of your area of interest (i.e. IE fixes when you’re not targeting that version of the browser or Windows fixes in a project targeted at UNIXes). However telling people to “fix it themselves” is not as honest as saying “that’s not our focus”. People can then decide whether they want to fork or not.

Standard
Blogging

Stemming the spam

One of the features I particularly value on WordPress has been the ability to have fairly open comment threads on my blog posts. In the past Askimet used to keep these spam free and people were able to submit comments fairly freely.

Over the course of this year though, comment spam has been getting vaguely more sophisticated. The most common spam has been backlink SEO spam that consists of reasonable emails and pieces of genuine comments mixed with product keywords and generic praise.

This kind of spam seems to be massively problematic to detect. Large numbers of comments have been made to historic posts that presumably have ended up ranking well in Google.

Moderating that stuff has been a chore and if I am away from the internet for any period of time the comment were on the site for a while and this seemed to be encouraging more. I presume the bots are smart enough to detect when their comemnts have not been deleted for more than eight to twelve hours and then they pile in.

The only reasonable way forward seems to me to be switch on someone kind of pre-moderation. I’ve gone for the mildest level of control that seems to work, namely that the first post by a user will now be moderated but once someone has a valid post they will be able to post freely.

This seems to strike the right balance between allowing people to tell me I know nothing about Clojure, Scala or Javascript and avoiding people having to sign-in to post comments.

As for the creators of these ingenious bots, a decidedly slow handclap; well-played.

Standard
Clojure, Programming

London Clojure unconference July 2014 report

For the first session I was interested in trying to continue the discussion about the Clojure “sweet spot” we had had on the mailing list. But there was only a smattering of interest so we rolled it up with the discussion on how to convince people in investment banks to use Clojure.

I think Jon Pither’s approach to this is still the best which is to find a business problem and then say that you’re going to address the problem and use Clojure to solve the real problem. A pure technical argument is not really going to get buy-in from outside the developers.

A lot of organisations want to have an approved list of technologies and for institutions that have chronic and acute technical problems like banks then perhaps that is appropriate given the need for external regulation. Where these things exist I usually think it is a case of going through the bureaucratic hoops.

The approval system is not there to be opinionated but to provide oversight. Where individuals have “weaponised” the approval process to advance their view of “right” technology you need to tackle the root problem not just sneak things in as jars.

My personal view is that financial institutions have profound technology problems but that they have no incentive to address them while they continue to make a lot of money. Really their problems should be providing opportunities for new approaches but as the existing institutions have created massive barriers to entry it doesn’t happen and we’re all really just waiting for the next financial crisis to happen, maybe then…

However in the session there was a lot of discussion about whether it is appropriate for managers to determine technology choices: on the one side you want to devolve decisions to the people close to the problem, on the other programmers commonly change jobs in a shorter period that the lifespan of the software they create.

One thing I took away was that before conservative organisations adopt Clojure they will need to see widespread adoption in the companies they see as good leading indicators and the presence of a large hiring population. In these respects Scala is literally years ahead.

Our final conclusion as a group was simply that the easiest way to approve the use of Clojure was to get into management and leadership first and then do it.

For the second session I went to the discussion on React and Om. I’m looking at React currently and there were a lot of questions about what Om layers on top of the basic JS library. Anna Pawlicka provided a number of the Om answers and others chipped in with bits of React and reactive JS knowledge. I was reminded to go and look at the current state of Om and also the new tutorials. There was also some interesting talk of how to define React components, Anna used Sablono but is there still a need for JSX?

The final session of the evening was on Riemann, which in addition to be a basic introduction to what it does was a helpful reminder of the functionality that Riemann has but that I haven’t used personally. Jason Neylon mentioned that every new service they set up has a Riemann instance attached so you can just dump all events somewhere and then build dashboards dynamically as you go along (a lot better than our approach with Graphite).

Tom Crayford introduced me to the effect of clock skew on Riemann (events from the “future” relative to the Riemann server clock are dropped) and then pointed out that clock skew can actually be monitored via Riemann! Also some interesting stuff about pumping logs into Riemann and some personal experience of crazy volumes of events being successfully handled.

Just before the end of the event I dropped in to the Gorilla REPL session to see Jony Hudson demoing his amazing notebook repl that he has been using to share assignments and research with students and colleagues in his department. A really interesting application and I suspect once we get our heads round it a really interesting way of sharing problems and potential solutions as developers.

Mind slightly blown, I was personally really happy with the event and felt that I’d got a mix of advice and the kind of innovation that make the Clojure community so interesting.

Standard
Games

Feral Vector 2014

I decide to take a break from my regular technology concerns and take a day off to visit indie games conference Feral Vector. The conference program was packed through the whole day with 20 minute talks (making it a little tricky to judge when to leave for things like lunch). The programming was really good and the shorter format made it feel more lively that other recent events I’ve been at like State of the Browser.

The venue was the Crypt on the Green aka the crypt of St.James in Clerkenwell, which is a pretty great venue and particularly in the summer has the advantage of having the church grounds/public park. In terms of layout though the talks were in the sideroom with the far larger room being given over the game demos. So the talks were packed all day while the main room always felt empty. A swap on the day might have worked a lot better. The small mezzanine used as a tea room also ended up feeling like a sauna.

In terms of the demos I liked the folk games tutorial of Turtle Wushu and I real liked Night in the Woods which felt like a platforming Slackers that was in the same cultural space as Gone Home. Hohokum was very weird, it definitely has that play feel but lacks enough feedback to make you feel like you’re actually interacting with the world.

I didn’t see all the talks so I’m going to talk about the ones that I liked. Standouts were Tim Hunkin on the arcade game booths he builds. The units are witty takes on conventional games of physical strength and dexterity. Adam Hay gave a great overview of how music and audio design has developed in videogames and was the closest that the event came to a technical talk. The explanations of the different synth chips versus sampled sound was interesting along with the way that sound design was initially a technical challenge due to hardware but then becomes a simulation challenge once hardware ceases to be a limit.

There were a few performance pieces (and quite a few journalists or writers in here): Christos Reid oscillated between confessional and an analysis of autobiographical games, making lots of good points but never really being clear about what any of it meant. Alice O’Conner mixed spoken word performance of mod readme files with her confession that she was losing interest in gaming and an awkward attempt to contextualise the readme file writers; the recital element was the strongest. Hannah Nicklin gave the strongest performance on the subject of how games break for her but the strength of the performance robbed the analysis of power as you ended up appreciating the delivery but feeling that the material lacked the depth and reflection it deserved. You felt that it was more about hitting a beat than exploring an idea. Near the end of the day James Parker did a puppet show Q&A that took a few easy pop shots but was also laugh out loud funny; his turn actually did the best job of marrying form and material.

Tammy Nicholls talked about world building and how it is valuable to both game depth and play as well as the commercial aspects of intellectual property but never really got into the details so I felt I was hearing half an argument. I often find that a good narrative and deep background carries you through some poor gameplay but perhaps it is undervalued in terms of game development.

Luke Whittaker talked about working on the game Lumino city and why for this game and previous game Lume the studio have focussed on physical materials translated into a game format. While the details of laser-cutting cardboard to make the city were fascinating I’m not sure really whether any meaningful justification for the approach was offered except the aesthetic which seems to be partly a nostalgia for a certain era of animation. However there’s no denying that the aesthetic is unique and its worth looking through the screenshots on the site.

There was also an interesting piece on combining art styles by SFB Games and collaborator Catherine Unger which again had a little bit of technical detail as to the issues and the solutions.

Finally there was a talk about physical puzzle rooms, a genre I don’t like even in digital format, but it did mention the interesting intersection between immersive, participative theatre and physical gaming. This was relatively new ground for me (although obviously people have raved about Secret Cinema). I was interested by the idea of things like 2.8 days later and the Heist. Not enough to want to participate yet but definitely more curious about the possibilities.

I think the talks were all recorded, although the room was often in darkness to make the projection work so I’m not sure how that worked out.

Standard
Programming

Programming as Pop Culture

The “programming is pop culture” quote has been doing the rounds from a 2004 interview with Alan Kay in terms of the debate on the use of craft as a metaphor for development. Here’s a recap:

…as computing spread out much, much faster than educating unsophisticated people can happen. In the last 25 years or so, we actually got something like a pop culture, similar to what happened when television came on the scene and some of its inventors thought it would be a way of getting Shakespeare to the masses. But they forgot that you have to be more sophisticated and have more perspective to understand Shakespeare. What television was able to do was to capture people as they were.

On the face of it this is a snotty quote from someone who feels overlooked; but then I am exactly one of those unsophisticated people who entered a democratised medium and is now rediscovering the past!

As a metaphor though in trying analyse how programmers talk and think about their work and the way that development organisations organise what they do then the idea that programming is pop culture is powerful, relevant and useful.

I don’t necessarily think that pop culture is necessarily derogatory. However in applying it to programming I think that actually you have to accept that it is negative. Regular engineering for example doesn’t discuss the nature of its culture, it is much more grounded in reality, the concrete and stolen sky as Locke puts it.

Architecture though, while serious, ancient and storied is equally engaged in its own pop culture. After all this is the discipline that created post-modernism.

There are two elements to programming pop culture that I think are worth discussing initially: fashion and justification by existence.

A lot of things exist in the world of programming purely because they are possible: Brainfuck, CSS versions of the Simpsons, obfuscated C. These are the programming equivalent of the Ig Nobles, weird fringe activities that contribute little to the general practice of programming.

However ever since the earliest demoscene there has been a tendency to push programming to extremes and from there to playful absurdity. These artifacts are justified through existence alone. If they have an audience then they deserve to exist, like all pop culture.

Fashion though is the more interesting pop culture prism through which we can investigate programming. Programming is extremely faddish in the way it adopts and rejects ideas and concepts. Ideas start out as scrappy insurgents, gain wider acceptance and then are co-opted by the mainstream, losing the support of their initial advocates.

Whether it is punk rock or TDD the patterns of invention, adoption and rejection are the same. Little in the qualitative nature of ideas such as NoSQL and SOA changes but the idea falls out of favour usually at a rate proportional to the fervour with which it was initially adopted.

Alpha geeks are inherently questors for the new and obscure, the difference between them and regular programmers is their guru-like ability to ferret out the new and exciting before anyone else. However their status and support creates an enthusiasm for things. They are tastemakers, not critics.

Computing in general has such fast cycles of obsolescence that I think its adoption of pop culture mores is inevitable. It is difficult to articulate a consistent philosophical position and maintain it for years when the field is in constant churn and turmoil. Programmers tend to attach to concrete behaviour and tools rather than abstract approaches. In this I have sympathy for Alan Kay’s roar of pain.

I see all manner of effort invested in CSS spriting that is entirely predicated on the behaviour of HTTP 1.1 and which will all have to be changed and undone when the new version HTTP changes file downloading. Some who didn’t need the micro-optimisation in initial download time will have better off if they ignored the advice and waiting for the technology to improve.

When I started programming professional we were at the start of a golden period of Moore’s Law where writing performant code mainframe-style was becoming irrelevant. Now at the end of that period we still don’t need to write performant code, we just want easy ways to execute it in parallel.

For someone who loves beautiful, efficient code the whole last decade and a half is just painful.

But just as in music, technical excellence doesn’t fill stadiums. People go crazy for terrible but useful products and to a lesser degree for beautiful but useless products.

We rediscover the wisdom of the computer science of the Sixties and Seventies only when we are forced to in the quest to find some new way to solve our existing problems.

Understanding programming as pop culture actually makes it easier to work with developers and software communities than trying to apply an inappropriate intellectual, academic, industrial or engineering paradigms. If we see the adoption, or fetishism, of the new as a vital and necessary part of deciding on a solution then we will not be frustrated with change for its own sake. Rather than scorning over-engineering and product ivory towers we can celebrate them as the self-justifying necessities of excess that are the practical way that we move forward in pop culture.

We will not be disappointed in the waste involved in recreating systems that have the same functionality implemented in different ways. We will see it as a contemporary revitalisation of the past that makes it more relevant to programmers now.

We will also stop decrying things as the “new Spring” or “new Ruby on Rails”. We can still say that something is clearly referencing a predecessor but we see the capacity for the homage to actually put right the flaws in its ancestor.

Pop culture isn’t a bad thing. Pop culture in programming isn’t a bad thing. But it is a very different vision of our profession that the one we have been trying to sell ourselves, but as Kay says maybe it better captures who we really are.

Standard
culture

Peak Peak

The Guardian recent published a little spattering of articles talking about such frivolous things as peak beard and peak craft beer. While this is a cute way of poking fun at current trends I worry that it is devaluing a useful term.

Most people are using “peak X” to mean simply “X suffers diminishing returns”. Namely that at some point supply of a product, be it facial hair or small-batch beer, exceeds demand and its over-supply actually diminishes demand.

The original form of the term, peak oil, refers to the moment when you maximise the conversion of a finite resource. The point about peak oil is that once you’ve hit it you can no longer achieve the same output again. After the peak, the value of the resource begins to rise due to its scarcity and the diminishing availability of the resource starts to outweigh the efficiency gains in its conversion.

Although it seems that in terms of popular usage the misapplication of “peak” seems to be winning, I think it would be a shame if people start misunderstanding the original meaning of the term due to the misapplication to renewal resources.

Standard
Gadgets, Programming, Python

Creating the Guardian’s Glassware

For the last two months on and off I’ve been developing the Guardian’s Glassware in conjunction with my colleague Lindsey Dew.

Dealing with secret-pre-alpha hardware has at times being interesting but the actual process of writing services using the Mirror API is actually pretty straight-forward.

Glass applications are divided into native, using the Glass SDK, and service-based Glassware using Mirror. Mirror is built on web-friendly technologies such as HTTP, JSON and OAuth2. It also follows the Google patterns for APIs so things like authentication, discovery and the client libraries are all as you would expect if you’ve used a modern Google API before.

For our project, which was focussed on trying to create a sensible, useful newsfeed to Glass, we went with Mirror. If you want to do things like geolocation or picture and video upload then you’ll want to go native.

For various reasons we had a very narrow initial window for development. Essentially we had to start and finish in May. Our prototyping was done with a sample app from Google (you can use Mirror without an actual device), the Mirror playground and a lot of imagination.

When we actually got our Glass devices it took about a week to get my head round what the usecase was. I was thinking that it was like a very lightweight mobile phone but it is much more pervasive with lots of light contact points. I realised that we could be more aggressive about pushing information out and could go for larger sets of stories (although that was dialled back a bit in the final app to emphasise editorial curated content).

Given the tight, fixed deadline for an unknown product the rest of the application was build using lots of known elements. We used a lot of the standard Glass card templates. We used Python on Google App Engine to simplify the integration service and because we had been building a number of apps on that same stack. The application has a few concerns:

  • performing Google Authentication flow
  • polling the Guardian’s Content API and our internal Notification platform
  • writing content to Mirror
  • handling webhook callbacks from Mirror
  • storing a user’s saved stories

We use Content API all the time and normally we are rendering it into widgets or pages but here we are just transforming JSON into JSON.

The cards are actually rendered by our application and the rendered content is packaged into the JSON along with a text representation. However rendering according to the public Glass stylesheet and the actual device differed, and therefore checking the actual output was important.

The webhooks are probably best handled using deferred tasks so that you are handing off the processing quickly and limiting the concern to just processing the webhook’s payload.

For the most part the application is a mix of Google stock API code and some cron tasks that reads a web API and writes to one.

Keeping the core simple meant it was possible to iterate on things like the content mix and user interactions. The need to verify everything in device served as a limiting factor.

Glass is a super divisive technology, people are very agitated when they see you wearing it. No-one seems to have an indifferent opinion about them.

Google have done a number of really interesting things with Glass that are worth considering from a technology point of view even if you feel concerned about privacy and privilege.

Firstly the miniaturisation is amazing. The Glass hardware is about the size of a highlighter and packs a camera, memory, voice synth, wifi and bluetooth. The screen is amazingly vivid and records and plays video well. It has a web browser that seems really capable of standard HTML rendering.

The vocal recognition and command menus are really interesting and you feel a little bit space age when you fire off a Google query and get the information you’re looking for read back to you in seconds.

Developing with the Mirror API is really interesting because it solves the Android fragmentation issue. My application talks to Mirror, not to the native device. If Google want to change the firmware, wire protocol or security they can without worrying about how it will affect the apps. If they do have to make breaking changes then can use the standard webapi versioning they already use.

Unlike most of the Guardian projects this one has been embargoed before the UK launch but it is great to see it out in the open. Glass might not be the ultimate wearable tech answer; just as the brick phones didn’t directly point to the iPhone. Glass is a bold device and making the Guardian’s journalism available on a new platform has been an interesting test of our development processes and an interesting challenge to the idea of what web-capable devices are (just as the Pixel exposed some flaky thinking about what a touch device is).

What will be interesting from here is how journalists will use Glass. Our project didn’t touch on how you can use Glass to share content from the scene, but the Glass has powerful capabilities to capture pictures and video hands-free and deliver it back to desk editors. There’s already a few trials planned in less stressful feature pieces and it will be interesting to see if people find the interface intuitive and more convenient that firing up their phone.

Standard