Category Archives: Programming

Programming

Enterprise programming 2023 edition

Back in the Naughties there were Enterprise Java Beans, Java Server Pages, Enterprise Edition clustered servers, Oracle databases and shortly thereafter the Spring framework with its dependency injection wiring. It was all complicated, expensive and to be honest not much fun to work with. One of the appeals of Ruby on Rails was that you could just get on an start writing a web application rather than staring at initialisation messages.

During this period I feel there was a big gap between the code that you wrote for living and the code you wrote for fun. Even if you were writing on the JVM you might be fooling around with Jython or Groovy rather than a full Enterprise Java Bean. After this period, and in particular post-Spring in everything, I feel the gap between hobby and work languages collapsed. Python, Ruby, Scala, Clojure, all of these languages were fun and were equally applicable to work and small-scale home projects. Then with Node gaining traction in the server space then gap between the two worlds collapsed pretty dramatically. There was a spectrum that started with an inline script in a HTML page that ran through to a server-side API framework with pretty good performance characteristics.

Recently though I feel the pendulum has been swinging back towards a more enterprisey setup that doesn’t have a lot of appeal for small project work. It often feels that a software delivery team can’t even begin to create a web application without deploying on a Kubernates cluster with Go microservices being orchestrated with a self-signing certificate system and a log shipping system with Prometheus and Grafana on top.

On the frontend we need an automatically finger-printing statically deployed React single-page app, ideally with some kind of complex state management system like sagas or maybe everything written using time-travelable reactive streams.

Of course on top of that we’ll need a design system with every component described in Storybook and using a modular class-based CSS system like Tailwind or otherwise a heavyweight styled component library based on Material design. Bonus points for adding React Native into this and a CI/CD system that is ideally mixes a task server with a small but passionate community with a home-grown pipeline system. We should also probably use a generic build tool like Bazel.

And naturally our laptop of choice will be Apple’s OSX with a dependency on XCode and Homebrew. We may use Github but we’ll probably use a monorepo along with a tool to make it workable like Lerna.

All of this isn’t much fun to work on unless you’re being paid for it and it is a lot of effort that only really pays off if you hit the growth jackpot. Most of the time this massive investment in complex development procedures and tooling simply throws grit into the gears of producing software.

I hope that soon the wheel turns again and a new generation of simplicity is discovered and adopted and that working on software commercially can be fun again.

Standard
Programming, Python

Transcribing podcasts with Google’s Speech to Text API

I don’t really listen to podcasts, even now when I have quite a long commute. I generally read faster than I can listen and prefer to read through transcripts than listen, even when the playback speed is increased. Some shows have transcripts and generally I skim read those when available to see if it would be worth listening to segments of the podcasts. But what about the podcasts without transcripts? Well Google has a handy Speech to Text API so why not turn the audio into a text file and then turn it into a HTML format I can read on the phone on the tube?

tldr; the API is pretty much the same one as generates the Youtube automatic subtitling and transcripts. It can just about create something that is understandable as a human but its translation of vernacular voices is awful. If Youtube transcripts don’t work for you then this isn’t a route worth pursuing.

Streaming pods

I’m not very familiar with Google Cloud Services, I used to do a lot of App Engine development but that way of working was phased out in favour of something a bit more enterprise friendly. I have the feeling that Google Cloud’s biggest consumers are data science and analysis teams and the control systems intersect with Google Workspace which probably makes administration easier in organisations but less so for individual developers.

So I set up a new project, enabled billing, associated the billing account with a service account, associated the service account with the project and wished I’d read the documentation to know what I should have been doing. And after all that I created a bucket to hold my target files in.

You can use the API to transcribe local audio files but only if they are less than 60 seconds long. I needed to be using the long running asynchronous invocation version of the API. I also should have realised that I need to write the transcription to a bucket too, I ended up using the input file name with “.json” attached but until I started doing that I didn’t realise that my transcription was failing to recognise my input.

Learning the API

One really nice feature Google Cloud has is the ability to run guided tutorials in your account via CloudShell. You get a step by step guide that can simply paste the relevant commands to your shell. Authorising the shell to access the various services was also easier than generating credentials locally for what I wanted to do.

Within 10 minutes I had processed my first piece of audio and had a basic Python file setup. However the test file was in quite an unusual format and the example was the synchronous version of the API.

I downloaded a copy of the Gettysburg address and switched the API version but then had my CloudShell script await the outcome of the transcoding.

Can you transcribe MP3?

The documentation said yes (given a specific version) and while the client code accepted the encoding type, I never got MP3 to work and instead I ended up using ffmpeg to create FLAC copies of my MP3 files. I might have been doing something wrong but I’m not clear what it was and the job was accepted but it was returning an empty JSON object (this is where creating files for the output is much more useful that trying to print an empty response).

FLAC worked fine and the transcript seemed pretty on the money and converting the files didn’t seem that much of a big deal. I could maybe do an automatic conversion later when the file hit the bucket if I needed to.

However after my initial small files I found that waiting for the result of the API call resulted in hitting a timeout on the execution duration within the shell. I’ve hit something like this before when running scripts over Google Drive that copied directories. I didn’t have a smart solution then (I just skipped files that already existed and re-run the jobs a lot) and I didn’t have one now.

Despite the interactive session timing out the job completed fine and the file appeared in the storage bucket. Presumably this would have been where it would have been easier to be running the script locally or on some kind of temporary VM. Or perhaps I should have been able to get the run identifier and just have checked the job using that. The whole asynchronous execution of jobs in Google Cloud is another area where what you are meant to do is unclear to me and working on this problem didn’t require me to resolve my confusion.

Real audio is bobbins

So armed with a script that had successfully rendered the Gettysburg address I switched the language code to British English, converted my first podcast file to FLAC and set the conversion running.

The output is pretty hilarious and while you can follow what was probably being said it feels like reading a phonetic version of Elizabethan English. I hadn’t listened to this particular episode (because I really don’t listen to podcasts, even when I’m experimenting on them) but I did know that the presenters are excessively Northern and therefore when I read the text “we talk Bob” I realised that it probably meant “we are talking bobbins”. Other gems: “threw” had been rendered as “flu” and “loathsome” as “lord some”. Phonentically if you know the accent you can get the sense of what was being talked about and the more mundane the speech the better the transcription was. However it was in no way an easy read.

I realised that I was probably overly ambitious going from a US thespian performing a classic of political speechwriting to colloquial Northern and London voices. So next I chose a US episode, more or less the first thing I could get an MP3 download of (loads of the shows are actually shared on services that don’t allow you access to the raw material).

This was even worse because I lacked the cultural context but even if I had, I have no idea how to interpret “what I’m doing ceiling is yucky okay so are energy low-energy hi”.

The US transcript was even worse than the British one, partly I think because the show I had chosen seems to have the presenters talking over one another or speaking back and forth very rapidly. One of them also seems to repeat himself when losing his chain of thought or wanting to emphasise something.

My next thought was to try and find a NPR style podcast with a single professional presenter but at this point I was losing interest. The technology was driving what content I was considering rather than bringing the content I wanted to engage with to a different medium.

You Tube audio

If you’ve ever switched on automatic captioning in Youtube then you’ve actually seen this API in action, the text and timestamps in the JSON output are pretty much the same as what you see in both the text transcript and the in-video captioning. My experience is that the captioning is handy in conjunction with the audio but if I was fully deaf I’m not sure I would understand much about what was going on in the video from the auto-generated captions.

Similarly here, the more you understand the podcast you want to transcribe the more legible the transcription is. For producing a readable text that would reasonably represent the content of the podcasts at a skim reading level the technology doesn’t work yet. The unnatural construction of the text means you have to quite actively read it and put together the meaning yourself.

I had a follow-up idea of using speech to text and then automated translation to be able to read podcasts in other languages but that is obviously a non-starter as the native language context is vital for understanding the transcript.

Overall then a noble failure; given certain kinds of content you can actually create pretty good text transcriptions but as a way of keeping tabs on informal, casual audio material, particularly with multiple participants this doesn’t work.

Costs

I managed to blow through a whole £7 for this experiment which actually seemed like a lot for two podcasts of less than an hour and a seven minute piece of audio. In absolute terms though it is less than proverbial avocado on toast.

Future exploration

Meeting transcription technology is meant to be pretty effective including identifying multiple participants. I haven’t personally used any and most of the services I looked at seemed aimed at business and enterprise use and didn’t seem very pay as you go. These however might be a more viable path as there is clearly a level of specialisation that is needed on top of the off-the-shelf solutions to get workable text.

Links

Standard
Programming, Work

August 2023 month notes

I have been doing a GraphQL course that is driven by email. I can definitely see the joy of having autocompletion on the types and fields of the API interface. GraphQL seems to have been deployed way beyond its initial use case and it will be interesting to see if its a golden hammer or genuinely works better than REST-based services outside the abstraction to frontend service. It is definitely a complete pain in the ass compared to HTTP/JSON for hobby projects as having to ship a query executor and client is just way too much effort compared to REST and more again against maybe not doing a Javascript app interface.

I quite enjoyed the course, and would recommend it, but it mostly covered creating queries so I’ll probably need to implement my own service to understand how to bind data to the query language. I will also admit that while it is meant to be quite easy to do each day I ended up falling behind and then going through half of it on the weekend.

Hashicorp’s decision to change the license on Terraform has caused a lot of anguish on my social feeds. The OpenTerraform group has already announced that they will be creating a fork and are also promising to have more maintainers than Hashicorp. To some extent the whole controversy seems like a parade of bastards and it is hard to choose anyone as being in the right but it makes most sense to use the most open execution of the platform (see also Docker and Podman).

In the past I’ve used CloudFormation and Terraform, if I was just using AWS I would probably be feeling smug with the security of my vendor lock-in but Terraform’s extensibility via its provider mechanisms meant you could control a lot of services via the same configuration language. My current work uses it inconsistently which is probably the worst of all worlds but for the most part it is the standard for configuring services and does have some automation around it’s application. Probably the biggest advantage of Terraform was to people switching clouds (like myself) as you don’t have to learn a completely new configuration process, just the differences with the provider and the format of the stanzas.

The discussion of the change made we wonder if I should look at Pulumi again as one of the least attractive things about Terraform is its bizarre status as not quite a programming language, not quite Go and not quite a declarative configuration. I also found out about Digger which is attempting to avoid having two CI infrastructures for infrastructure changes. I’ve only ever seen Atlantis used for this so I’m curious to find out more (although it is such an enterprise level thing I’m not sure I’ll do much than have an opinion for a while).

I also spent some time this month moving my hobby projects from Dataset to using basic Pyscopg. I’ve generally loved using Dataset as it hides away the details of persistence in favour of passing dictionaries around. However it is a layer over SQLAlchemy which is itself going through some major point revisions so the library in its current form is stuck with older versions of both the data interaction layer and the driver itself. I had noticed that for one of my projects queries were running quite slowly and comparing the query time direct into the database compared to that arriving through the interface it was notable that some queries were taking seconds rather than microseconds.

The new version of Psycopg comes with a reasonably elegant set of query primitives that work via context managers and also allows results to be returned in a dictionary format that is very easy to combine with NamedTuples which makes it quite easy to keep my repository code consistent with the existing application code while completely revamping the persistence layer. Currently I have replaced a lot of the inserts and selects but the partial updates are proving a bit trickier as dataset is a bit magical in the way it builds up the update code. I think my best option would be to try and create an SQL builder library or adapt something like PyPika which I’ve used in another of my projects.

One of the things that has surprised me in this effort is how much the official Python documentation does not appear in Google search results. Tutorial style content farms have started to dominate the first page of search results and you have to add a search term like “documentation” to surface it now. People have been complaining about Google’s losing battle with content farms but this is the first personal evidence I have of it. Although I always add “MDN” to my Javascript and CSS searches so maybe this is just the way of the world now, you have to know what the good sites are to find them…

Standard
Programming

London Django Meetup May 2023

Just one talk this time and it was more of a discussion of the cool things you can do with Postgres JSON fields. These are indeed very cool! Everything I wanted to do with NoSQL historically is now present in a relational database without compromise on performance or functionality, that is an amazing achievement by the Postgres team.

The one thing I did learn is that all the coercion and encoding information is held in the Django model and query logic which means you only have basic types in the column. I previously worked on a codebase that used SQLAlchemy and a custom encoder and decoder which split custom types into a string field with the Python type hint (e.g. Decimal, UUID) and the underlying value. By comparison with the Django implementation which appears to just use strings this is a leaky abstraction where the structure of the data is compromised by the type hint.

Using the Django approach would have been easier when using direct SQL on the database and followed the principle of least surprise.

The speaker was trying to make a case for performing aggregate calculations in the database but via the Django ORM query language which wasn’t entirely convincing. Perhaps if you have a small team but the resulting query language code was more complex that the underlying query and was quite linked to the Postgres implementation so it felt that maybe a view would have been a better approach unless you have very dynamic calculations that are only applied for a fixed timespan.

It was based on an experience report so it clearly worked for the implementing group but if felt like the approach strongly coupled the database, the web framework and the query language.

Standard
Programming

Version management with asdf

I typically use languages that are unmanageable without being able to version the language release you are dealing with (Python and Javascript). I have also been historically bad at keeping up to date with releases and therefore ending up with code that sometimes doesn’t run at all (Rust and Scala).

asdf is a version manager to rule them all. It provides a common set of commands to manage language dependencies (and the installation of different language versions) but has a plugin interface that different languages can use to bring in language specific concerns.

As a user you just need to learn one set of commands to manage all languages; implementations can build on a stable core system and simply focus on their requirements. Everyone is a winner.

One top of that instead of having multiple hidden files for multi-language projects (usually Javascript and some other language) you now have one file with all the language definitions in.

The only complication I’ve found is retraining myself to the new command set and remembering which commands work on asdf itself (things like updating the tool itself, setting specific versions in different scopes and managing the language plugins themselves) and which work on the plugins (installing new versions). The plugins also have no requirement to be consistent amongst themselves so in some you can specify “lts” as a target for example or “latest”. Others require the full three digit semantic version. These conventions seem to have come from the tools the plugins are replacing.

Overall though I think retraining myself to learn a single tool is probably going to be easier than having an increasing number of per language systems.

Standard
London, Programming, Web Applications, Work

Halfstack on the Shore(ditch) 2022

This is the first time the conference has been back at Cafe 1001 since the start of the Pandemic and my first HalfStack since 2021’s on the Shore event.

In some ways Halfstack can seem like a bit of an outlandish conference but generally things that are highly experimental or flaky here turn up in refined mainstream forms three to five years later. Part of the point of the event is to question what is possible with the technologies we have and what might be possible with changes that are due in the future. Novelty, niche or pushing the envelope talks are about expanding the conversation about what is possible.

The first standout talk this year was by Stephanie Shaw about Design Systems. It tries to make the absurdist argument that visual memes meet all the criteria to be a design system before looking at what are the properties of a good design system that would disqualify memes. The first major point that resonated with me was that design systems are hot and lots of people say they have them when what they actually have are design principles, a component library or an illustration of UI variant behaviour.

I was also impressed that the talk had a slide dedicated to when a design system would be inappropriate. Context always matters in terms of implementing ideas in organisations and it is important to understand what the organisation needs and capabilities that are required to get value from an idea. Good design systems provide a strong foundation for rapid, consistent development and should demonstrate a clear return on the investment in them.

One of the talks that has stayed with me the longest was one that was about things that can be done now. I’ve seen Chris Heilmann talk about dev tools at previous conferences but this time the frame of the talk was different and was about using dev tools in the browser to make the web sane again. He reminded me that you can use the dev tools to edit the page. Annoying pop-up? Delete it! Right-click hijacked? Go into the handler bindings and unbind the customer listener. Auto-playing video? Change it’s attributes or again just delete the whole thing. He also did explain some new things that I wasn’t aware of such as the ability to take a screenshot of a specific node from within the DOM inspector. I’ve actually used that a few times since in my work.

There was an impromptu talk that was grounded in a context that was a little hard to follow (maintaining peer to peer memes in a centralised internet apocalypse I think) but was about encoding images into QR codes that included an explanation of how QR codes actually work and encode information (something I didn’t know). The speaker took the image data, transformed it into a series of QR codes, then had a website that displayed the QR codes in sequence and a web app that used a phone camera to scan the codes and reassemble the image locally. The scanning app was also able to understand where in the sequence the QR code was which created a kind of scanning line effect as it built up the image which was very cool to watch.

There were three talks that all involved a significant amount of simultaneous interaction and each using slightly different methods but clearly the theme was having many people together on a webpage interacting in near real time.

The first thing to say is that I took a decent but relatively low-powered Pinebook laptop to the conference as I thought I would just need something simple to take notes and look things up on the internet, maybe code along with some Javascript. All of the interactive demos barely worked on it and the time to be active was significantly longer than say the attendees with the latest Macs. I think the issue was a combination of having really substantial downloads (which appeared not to be cached so refreshing the browser was fatal) but also just massive requirements on CPU in the local synchronisation code.

The first was by a pro developer relations person, Jo Franchetti, who works for Ably and who used the Ably API. Predictably this was the best working (and looking) demo with a fun Halloween theme around the idea of an ouija board or, more technically, trying to spell out messages by averaging all the subscribers’ mouse movements to create a single movement over the screen. However even using a commercial API, probably having no more than 25 connections and a single-screen UI my laptop still ground to a halt and had significant lag on the animations. It did look great projected on the big screen though.

Jo’s talk introduced me to an API I hadn’t heard of before scrollTo (part of a family of scrolling APIs). This is an example of how talks about things on the edge of the possible often come back to things that are more practical day to day.

James Allardice and Ross Greenhalf had the least successful take on the multiuser extension and in terms of presentation style seemed to be continuing an offstage squabble in front of everyone. I get the impression that they were very down on what they had been able to achieve and were perhaps hoping for a showcase example to promote their business.

Primarily they didn’t get this because they were bizarrely committed to AWS Lambda as the deployment platform. Their idea was to do a multiplayer version of Pong and it kind of worked, except the performance was terrible (for everyone this time, not just me). This in turn actually created a more fun experience that what they had intended to build as the lag meant you needed to be quite judicious in when you sent your command (up or down) to the server as there was a tendency to overshoot with too many people sending commands as ball approached and then another as they were waiting for the first one to take effect. You needed to slow down your reaction cycle and try and anticipate what other people would be doing.

The game also only lasted for the duration of a Lambda timeout of a single execution run as the whole thing was run in the execution memory of a single Lambda instance. This was a consequence of the flawed design but again it wasn’t hard to imagine how Lambda could be quite effective here as long as you’re not using web sockets for the push channel. It feels like this kind of thing would probably be pretty trivial in something like Elixir in a managed container but was a bit of a uphill battle in a Javascript monolith Function as a Service.

The most creative multi-user demo was by Mynah Marie (aka Earth to Abigail who has been a performer at previous Halfstacks) who used Estuary to create a 15 person online jam session which was surprisingly harmonious for a large group with little in the way of being able to monitor your own sound (I immediately had more empathy for any musician who has asked the desk for less drums in their monitor). However synchronisation was again a big problem, not only did other people paste over my loops but also after leaving the session one of my loops remained stubbornly playing until killed by the admin despite me not being able to access the session again, I was given a new user identity and no-one seemed able to reconnect with the orphan session.

Probably the most mindblowing technical talk was by Ulysses Popple about his tool Nodessey which is both a graph editor or notebook and a way to feed values into nodes that can then visualise the input they are receiving from their parent nodes. It reminded me a bit of PureData. I found following the talk, which was a mixture of notes and live-coded examples, a bit tricky as its an unusual design and trying to follow how the data structure was working while also trying to follow the implementation was tricky for me.

One thing I found personally interesting is that Nodessey is built on top of a minimal framework called Hyperapp which I love but have never seen anyone else use. I now see that I have very much underestimated the power of the framework and I want to start trying to use it more again.

Michele Riva did a talk about the use of English in programming languages which had a helpful introduction to programming languages that had been created in non-English languages. As an English speaker you tend to not need to ever leave the US-led universe of English based languages but it was interesting to see how other language communities had approached making programming accessible for non-English speakers. There was a light touch on non-alphabetic languages and symbolic languages like J (and of course brainfuck).

Perhaps the most practical talk of the conference was by Ante Barić around browser extensions. I’ve found these really valuable for creating internal organisation tooling in a very lightweight way but as Chris Heilmann reminded us in his talk too many extensions end up hammering browser performance as they all attempt to intercept the network requests and render cycle. The talk used a version of Clippy to create annoying commentary on the websites you were visiting but it had some useful insight into what is happening with browser extensions and future plans from both the Google and Mozilla teams as well as practical ways to build and use them.

Ante mentioned a tool that I was previously unaware of called web-ext that is a Mozilla project but which might be able to build out Chrome extensions in the future and gives a simplified framework for putting together extensions.

General notes

Food and drink is available when you want it just by showing the staff your conference lanyard. Personally I think it is great when conferences are able to be so flexible around letting people eat when they want to and avoiding the massive queues for food that typically happen when you try and cram an entire conference into a buffet in 90 minutes. I think it also helps include people who may have particular eating patterns that might not easily fit into scheduled tea and lunch breaks. It also makes it feel less like school.

In terms of COVID risk, the conference was mostly unmasked and since part of the appeal is the food and drink I felt like I wasn’t going to be changing my risk very much by wearing a mask during the talk sections. The ventilation seemed good (the room could be a bit cold if you were sitting in the wrong place) and there was plenty of room so I never had to sit right next to someone. This is probably going to remain a conference that focuses on in-person socialising and therefore isn’t going to appeal to everyone. Having a mask mandate in the current environment would take courage. The open air “beach” version of the conference on the banks of the Thames would probably be more suitable for someone looking to avoid indoor spaces.

Going back?

Halfstack is a lot of fun and I’ve booked my super early-bird for this year I think it offers a different balance of material compared to most web and Javascript conferences. This year I learnt practical things I could bring to my day job and was impressed by what other people have been able to achieve in theirs.

Standard
Programming, Software, Work

Defining the idea of “software engineering”

I have been reading Dave Farley’s Modern Software Engineering. Overall it’s a great read and thoroughly recommended (I’m still reading through it but I’ve read enough to know it is really interesting and a well-considered approach to common problems in development).

One of the challenges Dave tackles is to try and provide a definition of what software engineering actually is. This is actually a pretty profound challenge in my view. I’ve often felt that developers have usurped the title of engineer to provide a patina of respectability to their hacky habits. Even in Dave’s telling of the origin of the term it was used to try and provide parity of esteem with hardware engineers (by no lesser figure than Margaret Hamilton).

In large organisations where they have actual engineers it is often important to avoid confusion between what Dave categorises as Design and Production engineering. Software engineering sits in the world of design engineering. Software is malleable and easy to change unlike a supply chain or a partially completed bridge. Where the end result of the engineering process is an expensive material object Dave points that it is common to spend a lot of time modelling the end result and refining the delivery process for the material output as a result of the predictions of the model. For software to some extent our model is the product and we can often iterate and refine it at very low cost.

Dave proposes the following definition of engineering:

Engineering is the application of an empirical, scientific approach to finding efficient, economical solutions to practical problems.

Dave Farley, Modern Software Engineering

This definition is one I can live with and marries my experience of creating software to the wider principles of engineering. It also bridges across the two realms of engineering, leaving the differences to practices rather than principles.

It is grounded in practicality rather than aloof theories and it emphasises that capacities drive effective solutions as much as needs. This definition is a huge step forward in being able to build consensus around the purpose of a software engineer.

Standard
Programming

State of the Browser 2022

I’ve attended a few of these conferences and have always found them helpful. This year it had relocated to the Barbican Centre with the food and drink area overlooking the beautiful Conservatory there, great choice as a venue.

The conference was a hybrid in-person/online event that I think could serve as a model for other conferences that seem to only be focusing on their return to in-person. Due to other commitments I wasn’t able to be at the venue all day and so at lunchtime I headed home and picked up a few of the rest of the talks on the livestream. It was great to have the flexibility and made the whole conference more accessible.

Talks-wise it was interesting as ever and a little bit less inward looking or niche interest that it has been in the past. There were the usual mix of upcoming standards and challenges in implementing them, how to apply techniques to the current broad mainstream of browsers and a little bit of evangelism for playfulness and environment impact.

One of my key takeaways was on this last point; using an image CDN that can do automatic content negotiation to use an efficient modern image standard has a huge carbon saving. It feels a bit crazy that so many companies are still serving fixed sizes and formats off things like Cloudfront and S3.

Bruce Lawson kicked off the event with a good historical perspective talk on the history of standards (and the struggle to create and maintain them) and brought the issues of standardisation through the search for technical solutions to the world of regulation and better digital policy. Engaging with law makers is a more realistic way to improve the online world that the search of technical solutions to social problems.

More practically we can hope that Apple will be compelled as a digital gatekeeper to allow competition on browser implementations on its platform and maybe even fund its Safari team properly to have better compatibility with the general web standards on iOS. I felt it was nice for a recognition that government organisations can be engaged and willing to listen and that progress can be made be working together rather than outside of regular power structures.

Probably the best talk I heard was “Be the browser’s mentor not it’s micromanager” by Andy Bell this talk neatly encompassed two major ideas: the first was the way that layout systems in CSS have advanced to the point where you are describing structure and allowing the layout manager to actually decide the rendering and secondly on how digital design approaches have managed to fall between the abstractions of the grid system and the precise layout of magazine style layout.

By leaning on the layout engines the amount of CSS we have to write is much more minimal than the micromanaging fussiness typical to component design systems. It is also more powerful and expressive, avoiding the overly complex muddle that is often associated with component style systems but also not going too far down the class frenzy of utility class systems.

Sophie Koonin taught me how to use the prefers-reduced-motion preference via the medium of late 90s website chaos. A good example of the mixture of fun and practical content.

I also enjoyed Alistair Shepherd‘s talk which had a few technical bits and pieces but managed to bridge the themes of the conference by wanting to create a personal website that first and foremost reflected his personal interests and then used the tech to deliver the vision he had for himself. Although the idea to have websites that vary according to the time of date is quite an interesting idea.

I didn’t catch the last few talks so I’m hoping to be able to watch them when they come to YouTube (or maybe some federated alternative!).

Overall still one of the necessary conferences to catch for web technology and now easier to engage with than ever before.

Links

Standard
Programming

Keeping the batteries included

Python is well-known for being a “batteries included” language which means it comes with a rich variety of modules that work right out of the box. However this recent post about the regular expression library shows the problems that can occur from shipping such a wide variety of code as part of the language core. Maintaining a wide-ranging codebase is challenging as is keeping it aligned to a language’s release cadence.

The problems suggest that really the language should focus just on the core tools and syntax of the language with everything else being on a different release cycle. However by itself that core isn’t too useful for pragmatic coding purposes. Curation of code and having a sensible selection of libraries is a challenge. Some languages like Clojure and Elm have a controlled ecosystem of libraries that are adjacent to a small language core. However here the difference between curation and gatekeeping is fine and it feels like only languages with a large community can do it effectively.

Perhaps the answer is to have a basic implementation of core functionality in the core but to use the language documentation to suggest alternative libraries. This moves the problem to the documentation team but hopefully this is a simpler arena to both maintain and curate.

Standard
Programming

PR Reviews: not the messiah, not a naughty boy

On Tech Twitter there seems to be a lot of chat recently about how the Github Pull Request (PR) review process is “broken” or underserves internal teams (an example) that have strong collaboration practices such as pairing or mob coding.

Another objection raised is the idea of internal gatekeeping within development groups, I’m not sure I fully follow the argument but I think it runs along the lines that the PR review process allows powerful, influential members of the group to enforce their views over the others.

This is definitely a problem but frankly writing AST tools linked to the “merge to master” checks is probably a more controlling tool than spending your time policing PRs.

With open source projects often all the action happens at the pull request because the contributors often have no other interaction. Proponents are right to point out that if this works for open source projects do you really need to put effort into other practices upstream of the PR? Opponents are also right to point out that adopting the work practice of an entirely different context into your salaried, work context is crazy. They are both right, individual organisations need to make deliberate decisions using the challenges of both side to the way they are working.

I’m not sure how much of this is a fightback by pairing advocates (that’s how I interpret this thread). There is a general feeling that pairing as a practice has declined.

In my work the practice is optional. As a manager I’ve always known that in terms of output delivery (which before people object, may be important particularly if you’re dealing with runway to meet salary) pairing is best when you’re increasing the average rather than facilitating the experienced.

I think even with pairing you’d want to do a code review step. Pairs are maybe better at avoid getting into weird approaches to solutions than an individual but they aren’t magical and if you are worried about dominant views and personalities pairing definitely doesn’t help solve that.

So I’d like to stick up for a few virtues of the Pull Request Review without making the argument that Pull Requests are all you need in your delivery process.

As an administrator of policy who often gets asked about Software Development LifeCycles (SDLC) as a required part of good software governance. It is handy to have a well documented, automated review process before code goes into production. It ticks a box at minimum disruption and there isn’t an alternative world where you’re just streaming changes into production on the basis of automated testing and production observability anyway.

As a maintainer of a codebase I rely on PRs a lot as documentation. Typically in support you handle issues on much more software than you have personally created. Pairing or mob programming isn’t going to work in terms of allowing me to support a wide ranging codebase. Instead it’s going to create demand for Level 3 support.

Well-structured PRs often allow me to understand how errors or behaviour in production relate to changes in code and record the intent of the people making the change. It makes it easier to see situations that were unanticipated or people’s conception of the software in use and how that varies from actual use.

PR review is also a chance for people outside the core developers to see what is happening and learn and contribute outside of their day to day work. Many eyes is not perfect but it is a real thing and people who praise teaching or mentoring as a way to improve technique and knowledge should be able to see that answering review questions (in a suitable form of genuine inquiry) is part of the same idea.

PRs also form a handy trigger for automated review and processes. Simple things like spelling checks allow a wider range of people to contribute to codebases fearlessly. Sure you don’t need a PR to use these tools but in my experience they seem more effective with the use of a stopping point for consideration and review of the work done to date.

Like a lot of things in the fashion-orientated, pendulum swinging world of software development good things are abandoned when no longer novel and are exaggerated to the point that they become harmful. It’s not a world known for thoughtful reflection and consideration. But saying that pull request reviews undermine trust and cohesion in teams or a formulaic practice without underlying benefit seems unhelpfully controversial and doctrinal.

Standard