Category Archives: Software

Software

Volunteering at State of Open 2025

I volunteered at the State of Open conference this month, the conference is put on by the Open Source technology advocacy groups OpenUK.

Volunteering allowed me to sit in on a few sessions. AI was obviously a hot topic. There is naturally a lot of unhappiness at what constitutes the idea of an “open” foundation model; it isn’t just the code and the model weights, there’s also an interest in the training corpus and any human refinement process that might be used.

It is reasonable to assume that a lack of transparency in training data is because a lot of it has been illegally obtained. The conference did have a discussion group on the UK government’s consultation on copyright and training material, one that critics have said represents a transfer of wealth from creators to technologists.

Overall though it felt that there was more unhappiness than solutions. The expectation seems to be that companies will be able to train their models on whatever material they want and can obtain.

This unhappiness rang again for the other topic I heard a lot about which was maintainer well-being and open source community health. Maintainers feel stretched and undervalued, companies have been withdrawing financial support and informal, volunteer-run organisations handle conflict poorly within its own pool of collaborators leading to people leaving projects where they feel criticised and undervalued.

The good news is that people’s belief in the importance and value of openness, transparency and collaboration is still strong. The speakers at the conference were here to share because they want to help others and believe in the power of shared efforts and knowledge.

Becoming a volunteer

Someone asked me about how you volunteer for the conference and to be honest it was pretty straight-forward, I saw an invitation on LinkedIn, filled out a Google Form and then just turned up to the briefings and did the jobs I was asked to do. If I have the time then I think it is always worth volunteering to help out at these kinds of events as while you might not be able to get to see everything you want it also means you have something meaningful to be doing if the schedule is kind of ropey.

You also get to interact with your fellow volunteers which is much more fun that going to a conference alone.

Links

  • Astronomer Apace Airflow as a service
  • dbt a tool for transforming data
  • Tessl a start up looking to switch from coding as we know it today to specification-driven development

Talk recommendations

This is purely based on what I was able to see.

Standard
Software

Thoughts on the ethical use of LLMs

Large language models (LLMs) have felt fraught with issues. Let’s start with the environmental impact which has been completely disastrous and has essentially led to Big Tech ditching all their Net Zero promises. Vast amounts of speculative money have led to truly insane amounts of energy being spent creating models that don’t have strongly differentiated capabilities. Between this and cryptocurrency you’d be surprised to discover that electricity is not free and actually has consequences to its use.

Then there is the question of the corpus, mass ingestion of content for training AIs combined with obfuscation on the original of that material has resulted in a toxic situation for people who feel they have been taken advantage of and dubious legal situation for people using the output of such models.

The inherent flaws of the models’ probabilistic nature (hallucination, non-determinism) combined with user’s flawed mental models of what is happening is causing all kinds of strange fallout.

Finally there are the way that LLMs are being applied to problems, namely without any discretion or thought as to whether they have any relevance to the situation in hand. Again that glut of money at a time when most businesses are being squeezed by interest rates means that what gets used is what funders are excited about not what users need.

Now I’m not anti-AI or LLM in principle. I think there are some strong use-cases: summarisation, broad textual or structured document analysis and light personalisation. All machine models have infinite patience for user interaction and it seems humans prefer the tone of model generated content to that created by humans (which creates the burning question, why?) (2025-01-16: this article on how cognitive biases feed into interpretations of chat bot interactions seems relevant but it also includes an important reminder that the models are human ranked and tuned before they are released so I think it is natural that high agreeability would score well and unfriendly models would be binned). I think LLMs with access to vast amounts of information help put a floor under people’s understanding of problems and how to tackle things which is why individual subscriptions have been more popular than institutional ones.

However the foundation under these valid use cases needs to be sound and it currently it isn’t.

The new models by Pleais show that it is possible to do a better job of these problems. By having clearer information about the provenance of the information and the terms under which the team were allowed to use it. They have also been open about the carbon cost of training the model.

There still remain questions about the carbon cost of running the model and some about what the researchers mean about generating additional material for their corpus but this feels like the minimum that the bigger players should be offering.

The clarity over the training set should help alleviate the concerns people have about content being exploited with componsation or permission. Clear carbon figures mean we can start to compare the cost of creating new models and start to measure the efficiency of such efforts. Such a consideration should maybe be a factor in deciding whether a training process should be continued or not.

Privacy concerns can be alleviated by running models locally as well as insisting on greater clarity in the terms of service of the cloud providers (something I think Amazon moved closer towards with their Nova models).

I believe it is possible to address the genuine concerns people have about LLMs and to benefit from their use but the problems need to be acknowledge and addressed in a way that the mad scramble for AI gold simply has not done so far.

Standard
Software

Halfstack London 2024

Halfstack is a really interesting conference which is increasingly heading into a space that is unrepentantly about hobby projects, the love of technology and amateurism. But also a few talks by professional developer advocates or relationship managers.

It happens on Brick Lane and has an open bar in the afternoon. You might think that this is either hip or insufferable. Both opinions might be right.

In terms of work relevant material there was an interesting if often incoherent talk on the phases of the event loop which seems to be a popular topic but which was full of new information for me (being pretty basic I suppose); a sales pitch for the developer experience in Deno and using Tensorflow JS to do image recognition in mode.

Christian Heilmann has switched from giving talks on developer tools in the browser to the state of developer employment and this time highlighted the dilemma facing junior developer roles. While demand has fallen back for developers (compared to the incredible growth in demand for the previous five to ten years) it has done dramatically for entry-level roles and less so for experienced developers.

When the industry turns off the pipeline like this the effects tend to take years to feed through, as experienced people retire or switch to other roles there are less people taking their place as entrants have responded to the market signal and are doing something else.

The industry gamble here is that AI is going to make up the gap but the risk is whether the people using the AI are going to have a deep enough understanding of what is being created that they can support and maintain the result.

Maintaining codebases was in a way the theme of the talk, with all the emphasis on producing more code with the help of AIs is anyone thinking about what will happen to these digital products in five years time?

The real highlight of the day was a talk that combined the history of the 808 and 909 with a reminder of how crazy some of the browser API support is. Did you know that your browser probably supports MIDI?

According to the talk, you can read the slides online, the 808 and 909 were both flops on release that became classics after hitting the bargain bin so that a different kind of musician could access them and apply a different aesthetic sense to their capabilities.

The talk then used web APIs to recreate the 808 sound with samples via the ToneJS library and to trigger them with a USB connected device (less well supported). That was followed up with a mini-sequencer that was good enough to do a little live performance.

The day ended with a talk on using technology in murder mystery parties which was a bit crazy and obsessional and interesting in the way that people who have Gone Too Far can be. There was a bit where a trunk was being wired up to the mains where I thought the biggest danger might be the risk of death by homemade electronics.

Tickets for 2025 are available and next year’s conference and it has just recently been confirmed that enough pre-sales have been made to ensure the venue can be booked and that therefore next year’s conference is definite. In a period of declining sponsorship and stretched personal budgets that’s a vote of confidence in the conference from its audience.

Standard
Software

OpenUK: What the fork do we do now?

I attended an excellent talk organised by OpenUK about open-source forks recently.

Dawn Foster gave the context about why forks happen and a few historical examples of when forks overtake their originating projects and why that sometimes doesn’t happen and a few occasions when both the original project and the fork end up in different niches.

This was all given life by James Humphries’ talk about OpenTofu and the problems the consortium of former competitors have had getting their project off the ground. One immediate lesson learned was not to just take the head of the forked project but look for a stable milestone to branch from so you don’t immediately inherit someone else’s work in progress.

Another interesting observation was that commits direct to main without the context of a PR were often harder to understand. Change control processes attract a lot of passion but it was interesting that fast moving projects with direct to main changes are harder for newcomers to understand and maintain.

One problem I found particularly interesting was the change in licensing on Terraform’s registry project which meant the fork had to construct an alternative registry very quickly. They had advice from several other projects including Homebrew and were able to quickly bring up a registry that could be community-maintained and with a contribution of network costs from Cloudflare very low-cost.

Hashicorp clearly changed the terms to not allow competitors to use their registry and maybe that’s valid but it demonstrated that it is the ecosystem that makes forks difficult rather the actual code of the tool.

The project removed the tool’s telemetry early on to help respect user privacy but then then that makes it harder to explain whether their fork is finding traction or not. Looking at the traffic into the registry is a proxy for the volume of usage. This balance between privacy and metrics is an interesting one in open projects.

Standard
Software

Great software delivery newsletters

I currently subscribe to a number of great newsletters around technology and software delivery. While the Fediverse is also a great place to pick up news and gossip I have found that there is something really valuable in having a regular curated round up of interesting articles. It may be no surprise that the consistently great newsletters are produced by people who are engaged in consultancy. I think they inevitably get exposed to trends and concerns in the industry and also can commit the time to writing up their thoughts and reflecting on their chosen content.

Pat Kua‘s Level Up focuses on technical leadership and tends to have good pieces around human factors, managing yourself and creating good systems for delivery. It also often has advice pieces for people coming into technical management or leadership.

John Cutler’s The Beautiful Mess focuses on Product but is also great on strategy and importantly is always focused on getting to a better product process by emphasising collaboration and breaking down barriers between functional silos. I also enjoy reading how he approaches putting together alternatives to roadmaps and strategy documents. I think he has the best sense on how to use things like metrics and North Stars.

Emily Weber writes Posts from Awesome Folk has a focus on management, leadership, consensus building and healthy organisation cultures. As the title suggests it offers a carefully curated selection of posts that are often longer form and are generally from expert practitioners.

Michael Brunton-Spall‘s Cyber Weekly is your one stop shop for news on security and analysis of the key issues of the day.

Simon Willison‘s newsletter is more recent and feels more like a very long blog that is getting pushed into the newsletter format. Despite this Simon is one of the most creative and independent developers you could read and he was early into the LLM and generative AI and has lots of interesting insight into what you can do with these models and what works and what doesn’t. He’s also an (intimidating) role model for what independent, solo devs can achieve.

I have a lot of other subscriptions (and indeed a lot of people seem to be starting newsletters currently) so I will probably need to do a follow up to this post in a couple of months if I see that people are posting consistently useful things. One general thing to point out is that if I’m working on a particular technology (like Django, Go or React) I’ll often subscribe to the weekly community news roundups to get a feel for what’s happening. However I find the volume of links and items is overwhelming if you don’t have a specific interest or purpose in reading through them so I relegate them to RSS when I’m not actively working with them and have a more occasional catchup.

Standard
Software

Futurespectives

Futurespectives seem to be a much rarer practice than retrospectives. I learnt about using Futurespectives when I working with ThoughtWorks and I’ve used them a few times but I can’t seem to find a great online resource to introduce people to the idea. Liz Keogh’s advice on Futurespectives is probably the best I’ve found (beyond a lot of retro as a service companies writing marketing blog material about them).

One reason for their lack of adoption is that they require a certain amount of speculative imagination and that sometimes doesn’t come easily to developers who are very rooted in the realities of their work and sometimes think it is fanciful to speculate about the future.

However if you can persuade people to engage then I find this exercise to be very valuable for surfacing concerns and getting delivery teams to align on the broad shape of their approach to the work. It often sparks conversations that are being suppressed particular if people are being pressed for “commitments” on the upcoming work.

As with retrospectives, generating responses to the initial questions is best done independently and the consolidation of the individual answers and the discussion of what they reveal is best done collectively.

I ask the following questions but as with all practices it is often worth investing some time in trying to figure out what the purpose of the exercise is and what questions would best elicit responses that drive the conversation forward.

As a facilitator the frame for these questions are: “Imagine that we have completed our project. It has been a success even if at times it may have been hard work. The project meets the requirements and is working well in production. Our solution may be different from what we imagine today but we were able to adopt new ideas successfully. The team is happy and satisfied with how the work has gone and we didn’t need to make any excessive requests on their time and skills.”

  • How were we successful?
  • What problems did we have to overcome?
  • What are we proud about what we’ve done?

I ask people to generate responses from their perspective alone although they are free to speculate about how other teams and people will have helped or contributed along the way.

If people are struggling with the exercise I sometimes try to provide some starting questions. How do you feel about the project being complete? What do you feel satisfied to have done? What went better than you were expecting? How do other people feel about the work when they are talking to you about it?

Again the frame for all these questions is that the project has been successful (despite any doubts the participant may have now); the engineering mindset needs to accept that as definite thing and that the problem to be solved is: how was it successful despite this doubts? How were the problems solved or mitigated?

This last part is the critical step because it typically allows people to apply unconventional problem solving ideas. Typically people who are worried about a future problem cannot get past it if they feel it is unsurmountable however if you tell them that someone else has already solved it then just knowing that a solution exists allows people to reframe the problem and overcome their block on what the answer may be.

During the consolidation phase of the exercise, you bring the individual answers together and play them back to the group as a whole. This element is exactly the same as facilitating a regular retrospective. Try and ensure that any explanation of people’s ideas that the group needs is done during this phase. Often people are more aligned than they think but if there are any sharp disagreements in the approach they will typically come out now and its important that participants don’t reject any ideas at this stage because they will just return to their existing mindset.

Pay particular attention to similar ideas using different language, this can indicate that people are probably approaching the problems in a similar way but aren’t yet communicating enough to have shared ideas or a collaborative design approach. If there’s a lot of this it may be worth setting up a follow up to just review and consolidate the current state of play in the project. It may be that preparation is being rushed and the team isn’t having enough time to work together.

After creating and consolidating the initial input we now look at the three questions in a different way to help us generate actions from the futurespective. I sum up how we move from our imagined future to actions today in the following way for each question:

  • How do we realise our expected paths to success? What needs to happen to start towards that outcome? (Make true)
  • It is likely that we will encounter our anticipated problems, how can we minimise the impact they will have on us? (De-risk)
  • How can we ensure we have pride in our work? (Achieve)

At this point the session is more of a facilitated free-for-all with the initial phase being open to all ideas and suggestions. Some really common actions are that technical leaders realise they need to share more information on their vision and ideas with the rest of the team. It is also really common that when several people anticipate the same problem that prototyping, testing or training can be done very early on in the project plan to remove the problem or shift it a better understood class of problem.

The pride question often has actions that are associated with process, quality and shared standards and beliefs. Often though ideas about collaboration and the “team contract” come into play. Leaders can explain what others can rely on them for and what they want from the rest of the team. People can share fears in a way that allows people in authority to acknowledge that they share the same fears in a safe way. The format encourages not just the expression of fear but how will we manage our anxiety about the upcoming work.

In many ways if you’ve facilitated a retrospective you have all the skills that are required to run a futurespective, the tricky thing is about getting the participants in the right frame of mind.

In terms of measuring the impact of a successful futurespective you should be able to see a move from analysis to action and a growth of shared language and outcomes. Perceptions between the key participants of the project should be positive as they are already imagining a successful partnership ahead.

Standard
Programming, Software, Work

Defining the idea of “software engineering”

I have been reading Dave Farley’s Modern Software Engineering. Overall it’s a great read and thoroughly recommended (I’m still reading through it but I’ve read enough to know it is really interesting and a well-considered approach to common problems in development).

One of the challenges Dave tackles is to try and provide a definition of what software engineering actually is. This is actually a pretty profound challenge in my view. I’ve often felt that developers have usurped the title of engineer to provide a patina of respectability to their hacky habits. Even in Dave’s telling of the origin of the term it was used to try and provide parity of esteem with hardware engineers (by no lesser figure than Margaret Hamilton).

In large organisations where they have actual engineers it is often important to avoid confusion between what Dave categorises as Design and Production engineering. Software engineering sits in the world of design engineering. Software is malleable and easy to change unlike a supply chain or a partially completed bridge. Where the end result of the engineering process is an expensive material object Dave points that it is common to spend a lot of time modelling the end result and refining the delivery process for the material output as a result of the predictions of the model. For software to some extent our model is the product and we can often iterate and refine it at very low cost.

Dave proposes the following definition of engineering:

Engineering is the application of an empirical, scientific approach to finding efficient, economical solutions to practical problems.

Dave Farley, Modern Software Engineering

This definition is one I can live with and marries my experience of creating software to the wider principles of engineering. It also bridges across the two realms of engineering, leaving the differences to practices rather than principles.

It is grounded in practicality rather than aloof theories and it emphasises that capacities drive effective solutions as much as needs. This definition is a huge step forward in being able to build consensus around the purpose of a software engineer.

Standard
Software

State of the Browser 2021

This is the first in-person conference I’ve been to since the pandemic and since it normally clashes with PyCon UK this is also the first State of the Browser that I’ve been too in a while.

As a high-level pitch for the conference it a chance to hear from standard makers and browser developers about their thoughts on the web, web standards and issues in web development.

The conference had an audience of probably a third of what I felt it had the last time I attended in person. There was not issue with distancing and you could add a stickers to your attendee batch to nix photography and to ask for people to keep their distance.

Usually the chance to socialise and network is a major part of the conference experience but once I was there I realised that I didn’t really want to spend the time required to get to know someone new while COVID is as prevalent as it is, not attend the generous post-conference drinks.

Which made me wonder why I was there at all. The answer, on reflection, is that being physically present meant that I was actually present for the talks as well. I’ve bought tickets for virtual events earlier in the year and I still haven’t watch the videos.

By physically turning up I did pay more attention and I did engage and learn more than I did virtually.

I found a few things about the conference frustrating though. Firstly a number of the speakers weren’t there and instead had recorded a talk so being at the conference ended up being a collective video watch without being able to control the video and skip the boring bits. Also there were no questions from the audience because that was being handled on Discord. Now most of my Discord is taken up with gaming because, y’know that’s what Discord pretty much is for the most part. So I wasn’t able to see that side of things because I didn’t have time to set up some kind of work account. But generally whether it was Slack or something else I kind of think having the questions on the conference chat meant that the talks were actually lectures and where the speakers weren’t that proficient with their delivery it made the talks more boring.

So at the end of the experience I have no idea as to whether my attendance was a good idea or not. I probably would have been distracted at home but at least I could have sorted out Discord and have watched the pre-recorded videos in a more comfortable environment (I certainly could of dodged the morning torrential rain).

But when there was a good in-person speaker it was great. Rachel Andrew was the standout for managing a review of the history of layout systems while also previewing the thinking of the standards groups. In particular drawing a fascinating line between the necessity of the contains CSS directive to the ability to be able to look forward to container queries. Stephanie Stimac shared similar insight into what the future may hold for the development of the Form elements and their backwards-compatible codification and customisation.

Alex Russell offered a rebuttal of the locked down mobile ecosystems from a capitalist perspective but failed to really offer remedies given that this overall is a capitalist failure.

In a totally different vibe Heydon Pickering did a talk about requiring people to switch off Javascript to read his blog. It was closer to standup and I did laugh out loud several times although trying to explain what made it funny and entertaining has proven highly difficult.

Rachel Andrew is one of the people behind Notist which a few people were using to share slide links. I hadn’t heard of it before and I can see it’s pretty handy compared to trawling Youtube trying to figure out if some talk you half remember has been posted there.

Overall I think it was worth the effort, I felt I got outside my bubble for a while and felt a bit more connected to the efforts that are still ongoing to safeguard and advance the web as a whole.

Standard
Software, Work

The problem with developer job titles

Job titles are hard. This exchange on Twitter prompted a few thoughts that I couldn’t quite fit into a few smart-arsed twits.

In Chris’s Tweet he mentions that engineer is a co-opted title and that engineering is a discipline in its own right which most software groups don’t subscribe to because they aren’t really trying to do engineering. Not that this is a criticism but there is a massive difference between building a bridge or a road and creating a new web service. For a start there is a lot more established practice, science and understanding in physical engineering and more established and understood formal qualifications.

When I briefly worked in government helping create the Digital Careers framework people who were associated with Defence rightly objected to the confusion between software “engineering” and other engineers within professional frameworks. No-one is going to ask a developer to fix the combat damage on an airfield. I’ve previously joked that if we were honest we’d talk about “software overengineers” given that most developers struggle to find the simplest thing that works.

For the framework we settled on “developer” for people who wrote code and inconsistently used “engineering” for operational roles. I think on the basis that they created “infrastructure” where maybe the analogy makes a sort of sense.

I would also have gotten rid of “architect” if I’d had a chance; for exactly the same confusion but that term was too deeply embedded and still is a badge of prestige within the industry. Even now in the commercial world I have experienced hires wanting to be involved in “architecture” (and sadly not wanting to help me remodel my ground floor).

In Chris’s tweet he asks about what happened to the title “Programmer”. When I started in the industry this was indeed the coveted title and ideally I still think of myself this way even though it’s blatantly not true in the same way now.

However the issue with being a programmer is that jobs that literally involve just programming are few and far between. When I started in the industry the experienced developers were people who were at the tail-end of mainframe programming and a bit of what they were doing was still persuading machines to perform the tasks that were needed. The end was already in sight for pure programming jobs though. Some of my first professional programming work involved networking, a slightly dirty topic for the mainframe types.

Nowadays the emphasis is on understanding the domain space you are working on as well as the technical aspects of programming. I prefer the term “developer” (as others do) with the implication of being someone who develops systems of value via the medium of technology.

However that term also has its problems. When I worked at the Guardian I had a personal SEO battle with the Pune-based property development group for the search term “Guardian developers”. That battle seems to have been won now via sub-domain. This seems to be true more generally and now it is property developers who are having to use the prefix “property” on their job titles.

For a new profession not even past it’s first century, creating our professional lexicon is always going to be hard but in borrowing titles so shamelessly we are always creating problems for ourselves.

Programmer is probably the closest, truest name for what most of us do at the core of our role. For web developers though, assembling the typical bricolage of libraries and tooling is often an exercise in minimal programming and maximum duct taping. Perhaps it fairest to say that we are “software assemblers”, expect that might get confused with, you know, assemblers. Painful.

So in the end most of us are expected to bring capability in programming within teams that are creating technological systems of value. As long as programmers realise that programming is not the activity of value in itself then maybe we don’t need to worry so much about titles.

Standard
Programming, Software, Web Applications, Work

Prettier in anger

I’ve generally found linting to be a pretty horrible experience and Javascript/ES haven’t been any exception to the rule. One thing that I do agree with the Prettier project is that historically linters have tried to perform two tasks to mixed success: formatting code to conventions and performing static analysis.

Really only the latter is useful and the former is mostly wasted cycles except for dealing with language beginners and eccentrics.

Recently at work we adopted Prettier to avoid having to deal with things like line-lengths and space-based tab sizes. Running Prettier over the codebase left us with terrible-looking cramped two-space tabbed code but at least it was consistent.

However having started to live with Prettier I’ve been getting less satisfied with the way it works and Prettier ignore statements have been creeping into my code.

The biggest problem I have is that Prettier has managed its own specific type of scope creep out of the formatting space. It rewrites way too much code based on line-size limits and weird things like precedent rules in boolean statements. So for example if you have a list with only one entry and you want to place the single entry on a separate line to make it clear where you intend developers to extend the list Prettier will put the whole thing on a single line if it fits.

If you bracket a logical expression to help humans parse the meaning of the statements but the precedent rules mean that brackets are superfluous then Prettier removes them.

High-level code is primarily written for humans, I understand that the code is then transformed to make it run efficiently and all kinds of layers of indirection are stripped out at that point. Prettier isn’t a compiler though, it’s a formatter with ideas beyond its station.

Prettier has also benefited from the Facebook/React hype cycle so we, like others I suspect, are using it before it’s really ready. It hides behind the brand of being “opinionated” to avoid giving control over some of its behaviour to the user.

This makes using Prettier a kind of take it or leave it proposition. I’m personally in a leave it place but I don’t feel strongly enough to make an argument to remove from the work codebase. For me currently tell Prettier to ignore code, while an inaccurate expression of what I want it to do, is fine for now while another generation of Javascript tooling is produced.

Standard