Category Archives: Programming

Programming, Python

London Python Coding Dojo February 2025

The Python coding dojo is back and this time it allows AI assisted coding which means that some of the standard katas become trivial and instead the challenges have to be different to either require different problems to be combined in an interesting way or have a very hard problem that doesn’t have a standard solution.

The team I was in worked on converting image files to ASCII art (with a secondary goal of trying to create an image that would work with the character limit of old-school Twitter).

We used ChatGPT and ran the code in Jupyter notebooks. To be honest ChatGPT one-shotted the answer, clearly this is a thing that has many implementations. Much of the solution was as you would expect, reading the image and converting it to greyscale. The magic code is this line (this is regenerated from Mistral rather than the original version).

ascii_chars = "@%#*+=-:. "

This string is used to map the value of each pixel to a character. It is really the key to a good solution in terms of the representation of the image and also when we tried to refine the solution to add more characters this was the bit of the code that went wrong as the generated code tends not to understand that the pixel mapping depends on the length of this string. A couple of versions of the code had an indexing issue as it kept the original mapping calculation but changed the size of the string.

On the one hand the experience was massively deflating, we were probably done in 15 or 20 minutes. Some of the team hadn’t used code assistance this way before so they got something out of it. Overall though I’m not sure what kind of learning experience we were having and whether the dojo format really helps build learning if you allow AI-assistance.

If the problems become harder to allow for the fact that anything trivial is already in the AI databank then the step up into understanding the problem as well as the output is going to be difficult for beginners.

There’s lots to think about here and I’m not sure there are any easy answers.

Standard
Programming

Don’t use Postgres Enums for domain concepts

If you’ve ever read a piece of advice about using Postgres Enums you’ve probably read not to use them and to use a peer table with foreign key constraints instead. If you’ve ever seen a Typescript codebase in the wild recently chances are that this advice has absolutely been ignored and enums are all over the place.

I’m not really sure why this should be when even Typescript discourages the use of enums itself. I think it a combination of a spurious sensation of type safety combined with a desire to think about a table with a simple column of constrained values that maps to an object with a limited set of typed constants. My main theory is that the issue is that values in a Check constraint are difficult to map into a Typescript ORM.

But to be clear there’s no ORM in any language that magically makes using enums painless. The issues stem from the Postgres implementation which is often just made worse by bad ORM magic trying to hide the problem through complex migrations.

Domain ideas change, enums shouldn’t

First of all I want to be clear that my intent isn’t to complain about Postgres enums. They have completely valid use cases and if you want to describe units of measure or ordinal weekday mappings then there’s nothing particularly wrong with them or their design. Anything that is a fundamental immutable concept is probably a great fit for the current enums model.

My issue is with mapping domain values onto these enums. We all know that business concepts and operations are subject to change, these ideas are far more malleable than the speed of light or the freezing point of water. Therefore they should be easy to change when our understanding of the domain changes.

And this is where the recommendation to use foreign keys instead of enums comes in. Changing a row in a table is lot easier than trying to migrate a enum. Changing a label, adding and removing rows, all of them become easier and follow existing patterns for managing relational data. You can also expose these relationships are a configuration layer without having to make changes to the database definition library.

Changing enums in Postgres

In Postgres you can expand an enum but if you want to delete or rename a value then you actually end up deleting the enum entirely and recreating it.

While you’re deleting it you have to really block all operations on anything that uses that enum, any mutation is going to be a nightmare to handle. For some organisations that don’t allow this kind of migration this would be a dealbreaker already.

And then you have value consistency, what do you do if you’re removing a value? Typically there is a bit of a dance where you create two enumerations, one representing the current valid values and another representing the future value set. You then edit the values of the column and swap over the types.

Overall the entire process feels crazy when you could just be editing parent rows like any other piece of relational data.

What about language enums?

I don’t feel that exercised about enums expressed in source code and used within a well-defined context. I’m not sure you want to try and persist them to the storage layer but if you do then having a well-defined process for doing so like a mapper or repository could do the heavy lifting of seeing whether the code values and the storage values are in sync.

If enums make your code easier to work with then that’s fine, let the interface deal with synchronisation and have a plan on how the code should work if it is out of sync with its external collaborators.

However if you are using Typescript then do look at the advice on using constant objects versus enums.

But please don’t encode a malleable domain concept in an immutable data storage implementation.

Standard
Programming

Type stripping in Node

I tried the new type stripping feature which is enabled by default in Node 23.6.0. Essentially the type information is stripped from the code file and things that require transformation are not supported by default yet. You’re essentially writing Typescript and running Javascript. This is essence of the Typescript promise that there’s nothing in the language that isn’t Javascript (ignore enums).

On the surface of it this might seem pointless but the genius of it is that all the source files are Typescript and all tooling works as normal. There’s just no complicated build or transformation step. I have some personal projects that were written as Javascript but which were using typing information via JSDoc. Type stripping is much less complicated and the type-checking and tooling integration is much richer.

I’m planning to try and convert my projects and see if I hit any issues but I feel that this should be the default way to use Typescript now and that it is probably worth looking to try and upgrade older projects. The gulf of functionality between Node 18 and 23 is massive now.

Standard
Programming

Notes on calling Rust from Javascript

Although the tutorial looks great and comprehensive it describes a process that is surprisingly cumbersome. If you switch to the modern web approach things seem to get a lot simpler. This simply requires you to set the target of the output to be web.

With this done and a little bit of boilerplate (which I need to extract and make reusable somehow) which consisted of a static page, a one-line dynamic import Javascript file (which I’m not sure is needed) and a variant of the generated bootstrap file I was able to write any pure function using numbers or strings in Rust and then use it in short-order in the static page.

wasm-pack outputs a lot of files, there is the WASM binary itself which seems straight-forward, a couple of Typescript definition files which I presume is to help integrate the output into Typescript projects if you’re shipping the output as a module. There are two Javascript files that seem to define the Javascript version of the Rust functions and the value conversion between the two languages. I’m not sure why there are two, I might need to do a clean build and see what comes out. Maybe the smaller Javascript file actually contains everything needed for the web.

The way things feel at the moment is that if you’re stuck with a build issue the errors and the solution can feel incomprehensible but once it works it works really well and it is shockingly easy to export callable functions.

Apart from trying to find the minimum viable build I also need to try and implement something that is a bit more computationally challenging in the Rust function and get a sense of what the overhead of the Web Assembly call is because it feels relatively minimal.

Standard
Programming

What I learnt publishing an Eleventy site to Github Pages

I have an Eleventy site in a Github repo and I want to publish it, what could be more logical or easier than pushing it to Github Pages?

Well, the process was relatively easy but there was still enough gotchas to make it worth recording the lessons learned. First off the Eleventy documentation for Github Pages is not great, I ended up using the Starter Template for Jekyll combined with the Vite documentation but I think a few of my problems came from mashing up my sources.

First things first, you have to manually set your repo’s project Pages setting to Github Actions for anything to happen. I thought the Github Actions could somehow set this up via the configure Pages action but it is a cart before horse situation.

I had quite a few obscure YAML parsing errors and you don’t get any more detail back than “your file is wrong”. I found the action linter invaluable but I also faced a problem that my syntax problem was being reported on the job name but the actual problem was further down from the reported line. Cutting and pasting segments into the linter eventually allowed me to track down the problematic statement and get a parsed file.

Permissions on a job are not additive to the base permissions but override them. I thought I was adding a permission at the job level but I was in fact resetting them.

Having the permissions wrong resulted in an obscure error message Unable to get ACTIONS_ID_TOKEN_REQUEST_URL env variable which is the authorisation endpoint for an API action. The reason it is unavailable is that the permission isn’t set for the execution context.

Github Pages will publish to <Github username>.github.io/<repo name> which means that by default all the Eleventy generated links will be wrong. You need to use the Base plugin (confusing named to suggest an affinity to the base element) but totally different.

Concurrency was interesting and people seem to do a lot of different things. My conclusion is that the deployment job should only have one instance but shouldn’t be cancelled. If you have a build job that can be split by branch and should be cancelled if a new job is triggered.

Having different concurrency rules seems to be a big reason for splitting up the build and deployment activities, otherwise just having one end to end job seems easier to work with.

I should probably have gone with a starter template (you get offered them when you switch on Github Actions for a repository), unless you’re using a specific tool the static template seems the best. From here you just need to replace the artifact upload action with a build and upload step.

Standard
Programming

Juxt 24

Juxt hold an occasional conference and this edition was focused on Fintech which isn’t an area that I know really well but have dabbled in a bit.

Opening keynote

Fortunately the opening talk by Fran Bennett of the Ada Lovelace Institute was on AI and draw a parallel between the Post Office/Fujitisu debacle and the current level of credulity around the potential of generative AI. I particularly liked this (paraphrased) quote.

Computer systems operate within existing systems of power

If we choose to believe the myth of infallible machines over fallible humans then injustices like the Horizon scandal will just occur again and again.

Eliminating non-determinism

Allen Rohner of Griffin Bank offered a talk on improving testing systems by taking aim firmly at “flaky” tests and attributing them to non-deterministic and side-effecting behaviour either in the system under test or in the testing code itself.

He used the example of the FoundationDB testing strategy and a focus on invariant behaviour to facilitate automated generative testing. The practical twist he offered on this was Griffin’s use of stateful proxies that can also be part of generated testing to provide something strong than mocks or stubs in integration testing.

I think the key takeaway though was to change the way that you think about unreliable tests and consider changing the system to solve the problem rather than hacking around the tests.

Workflows in service clothing

Phill Barber‘s talk on workflows was one of my favourites of the day. Perhaps because I wasn’t expecting to enjoy it and partly because his argument in favour of workflows and orchestrated workflows (over choreographed events) was persuasive. He also didn’t try to deny the problems there can be with workflows: like only being able to visually design them and then exporting them to source control and never delivering the ability to for non-technical to change the system.

He tackled the key issues of the “workflow black hole effect” head on by putting the workflows inside the service boundaries. This approach also minimises the complexity and rigidity that can come from orchestration as you are talking about a few dedicated flows within a service. The orchestration rules are hidden from the service callers and therefore remain malleable.

He also suggested something interesting in that when a collaborating service becomes too anaemic and the balance of functionality ends up in the workflow side you can eliminate the service entirely and allow the workflow to access the datastore associated with the service. In the example given this essentially eliminated a feature light microservice in his example and instead brought the data ownership into a service with broader responsibilities. I would be interested if the idea would extend to multiple data ownerships but the thought only occurred to me well after the event.

He mentioned nflow as an embeddable (JVM-based) open source workflow engine that allows configuration in code.

Monoliths, monoliths, monoliths!

Everyone was of one mind that you should start each development with a monolith as Vlad Yatsenko, CTO of Revolut, put it, the service should be just one box on the system diagram. No-one was fundamentally against microservices but the preference was clear for “right-sized” services divided by organisation or operational properties and to decompose the monolith into the services rather than trying to jump straight to a distributed system.

Magic versus abstraction

In the questions section of the final talk by Zohor Melamed, Harry Percival asked the question about what the difference was between a great abstraction and the “magic” behaviour that Zohor had railed against in his talk. Again paraphrasing the response:

The difference between magic and a good abstraction is that the abstraction doesn’t shape the solution.

Bad abstractions are like async and await, good abstractions are like Docker which genuinely does not leak the details of the running container.

Conclusion

Thanks to Malcolm and Jon for the invite, it was an interesting line up, even for someone for whom the “buy side” is a mystery.

Standard
Programming

Redis: not one fork but two

Redis made a license change (see Hashicorp before them) and as day follows night forks duly appeared. Although excitingly this time there are two alternatives to choose from Valkey, which seems to have more corporate support and Redict which is more independent and is being championed by the person behind Sourcehut who is more than a bit of a Marmite person.

It was also interesting to see that both projects opted for an “io” domain despite the ethical issues associated with it (a balanced summary if you’re unfamiliar). It is a shame that the “dev” domain hasn’t proved a bit more popular.

Standard
Programming

How to call instance methods by name on a class in Typescript

I recently wanted to parameterise a test so that it included the method to test as a parameter.

This is easy in Javascript:

const myClass = new MyClass();

['methodA', 'methodB'].forEach((methodName) => myClass[methodName]();

But when you try this naively in Typescript it fails with a message that the class cannot be indexed by the type string.

The method interfaces of the class actually form a type check that needs to be satisfied and this led me to the keyof operator which forms this type.

As I was working on a test I didn’t need a strict type check so I could simply declare my string as keyof(MyClass) and this resolved the type complaint.

If the code was actually in the production paths then I would be a bit warier of simply casting and would probably try to avoid dynamic programming because it feels like it working around the type-checking that I wanted by using Typescript in the first place.

I’m not sure how I expected this to work but I was kind of expecting the type-checker to be able to use the class definition to make the check rather than using a more generic reflection that works for objects too but at the cost of having to have more annotation of your intent.

Standard
Programming, Work

December 2023 month notes

Web Components

I really want to try and understand these better as I think they are offering a standards-based, no-build solution for components combined with a better way of dropping in lightweight vanilla JS interactivity to a page where I might have used AlpineJS before now.

I’m still at the basic learning stage but I’ve been hopping around the Lean Web Club tutorials to get a sense of the basics. One of the things that is already interesting is that Web Components wrap their child HTML is quite a clear and scoped way so you can use them quite easily to mix server rendered content with runtime dynamic content. I haven’t found an elegant way to do that with other frameworks.

Scoping and Shaping

I attended an online course by John Cutler which was a pretty good introduction to idea of enabling constraints. Most times I like to attend courses and classes to learn something but every now and then it feels good to calibrate on what seems obvious and easy and understand other people’s struggles with what seems basic stuff.

A few takeaways: being a good stakeholder is an underrated skill and being clear about the boundaries of what you’re willing to accept is important to allow teams working on problems to be successful. If someone says they can’t work with your constraints then its not a good fit, if no-one can work with your constraints then you either need to do the work yourself or give up on it.

The most insightful piece of the meeting for me came around the psychology of leaders in the new economy where profits are more important than growth and experimentation. John’s theory is that this pressure makes it harder for executive teams to sign off on decisions or to give teams a lot of leeway in approaching the problem. To provide meaningful feedback to executing teams senior stakeholders feel they need more information and understanding about the decisions they are making and the more hierarchical an organisation the more information needs to go up the chain before decisions can come back down.

Before zero interest rates there used to be a principle that it wasn’t worth discussing something that wouldn’t make back the cost of discussing it. Maybe rather than doing more with less we should be trying to get back to simple not doing things unless they offer a strong and obvious return.

How I learned to love JS classes

I have never really liked or seen the point in Javascript’s class functionality. Javascript is still a prototype-based language so the class syntax is basically complex syntax sugar. React’s class-based implementation was complex in terms of how the class lifecycle and scope interacted with the component equivalent so I was glad to see it replaced by stateless components. However classes are pretty much the only way that you can work with Web Components so I’ve been doing a lot more of them recently than previously.

I’ve also been dropping them into work projects although it raises some interesting questions when you’re using Typescript as the difference between a class and an interface is quite blurry there. Presumably classes should either have static elements or also encapsulate behaviour to make the inheritance meaningful otherwise it’s simply an interface that the implementing class needs to provide.

Standard
Programming

Halfstack on the Shore(ditch) 2023

Self-describing as an “anti-conference” or the conference that you get when you take all the annoying things about conferences away. It is probably one of the most enjoyable conferences I attend on a regular basis. This year is was in a new venue quite close to the previous base at Cafe 1001 which was probably one of my favourite locations for a conference.

The new venue is a small music venue and the iron pillars that fill the room were awkward for sightlines until I grabbed a seat at the front. The bar opened at midday and was entirely reasonable but the food was not as easily available as previously available but you were still able to walk to the nearby cafe and show your conference badge if you wanted.

Practical learnings

Normally I would say that HalfStack is about the crazy emergent stuff so I was surprised to actually learn a few things that are relevant to the day job (admittedly I have been doing a lot more backend Javascript than I was previously). I was quite intrigued to see some real-world stats that the Node’s in-built test runner is massively faster than Jest (which maybe should not be so surprising as it does some crazy things). I’ve been using [Bun]() recently which does have a faster runner and it makes TDD a lot more fun that with the normal Jest test runner.

I also learnt that NODE_ENV is used by library code to conditionally switch on paths in their code. This is obviously not a sound practice but the practical advice was to drop variables that map to environments completely and instead set parameters individually as per standard 12 factor app practice. I think you can refine that with things like dotenv but I’m basically in agreement. Two days later I saw a bunch of environment-based conditional code in my own workplace source code.

It was also interesting to see how people are tackling their dependency testing. It felt like the message is that your web framework should come with mocks or stubs for testing routing and requests as standard and that if it doesn’t then maybe you should change your framework. That feels a bit bold but that’s only because Javascript is notorious for having anaemic frameworks that offer choice but instead deliver complexity and non-trivial decisions. On reflection it seems like having a built-in unit testing strategy for your web framework seems like a must-have feature.

Crazy stuff

There was definitely less crazy stuff than in previous years. A working point of sale system including till management based on browser APIs was all quite practical and quite a good example of why you might want USB and serial port access within the browser.

There was also a good talk about converting ActionScript/Flash to Javascript and running emulation of old web games although that ultimately turned out to be a way of making a living as commercial games companies wanted to convert their historic libraries into something that people could continue to use rather than being locked away in a obsolete technology.

The impact of AI

One of the speakers talked about using ChatGPT for designing pitches (the generated art included some interesting interpretations our how cat claws work and how many claws they have) and I realised listening to it that for some younger people the distilled advice and recommendations that the model has been fed is exactly the kind of mentoring that they have desired. From a negative perspective this means an endless supply of non-critical ideas and suggestions that require little effort on the user’s part; just another way to avoid having to do some of the hard work of deliberative practice. On the positive side a wealth of knowledge that is now available to the young in minutes.

While I might find the LLMs trite, for people starting their careers the advice offered is probably more sound that their own instincts. There also seems to be some evidence appearing that LLMs can put a floor under poor performance by correctly picking up common mistakes and errors. At a basic level they are much better at spelling and grammar than non-native speakers for example. I don’t think they have been around long enough to have reliable information though and we need to decide what basic performance of tasks looks like.

I wonder what the impact will be on future conference talks as ChatGPT refines people to a common set of ideas, aesthetics and structures. Probably it will feel very samey and there will be a desire to have more quirky individual ideas. It feels like a classic pendulum swing.

Big tech, big failings

Christian Heilmann’s talk was coruscating about the failures of big tech during the acute phase of the COVID pandemic and more generally about being unable to tackle the big problems facing humanity and instead preferring to focus on fighting for the attention economy and hockey stick growth that isn’t sustained. He also talked about trying to persuade people that they don’t have to work at FAANGS to be valid people in technology.

His notes for this talk are on his blog.

Final thoughts

Chat GPT might need me to title this section as a conclusion to avoid it recommending that I add a conclusion. HalfStack this year is happening at a strange time for programming and the industry. There wasn’t much discussion of some topics that would have been interesting around the NodeJS ecosystem such as alternative runtimes and the role of companies, consultancy and investment money in the evolution of that ecosystem. The impact of a changed economic environment was clear and in some cases searing but it was a helpful reminder that it is possible to find your niche and make a living from it. You don’t necessarily need to hustle and try and make it big unless that is what you really want to do.

The relaxed anti-conference vibe felt like a welcome break from the churn, chaos and hamster wheel turning that 2023 has felt like. I’ve already picked up my tickets for next year.

Links

Standard