Tag Archives: Month notes

Month notes

April 2024 month notes

I’ve read quite a few people complaining about the continuing degradation of Google search results but this month I genuinely started to notice issues with search results about programming and system design. There’s always been a bit of game playing in the top position but the problem I noticed was that the later results feature a lot of recycling of the same information (and in my case incorrect or irrelevant information) so that there was really only one result on the front page.

There was also a lot of Medium links which itself is getting increasing unusable if you don’t want to have an account or engage in whatever pop up activity Medium thinks is going to boost its monthly active users.

Search alternatives

I’ve started using Ecosia for its green credentials and because it seems to have results that are less gamed (although W3 Schools is still too prominent). I also gave Codemate Bot a go, which is essentially a tailored LLM. It seemed a bit better than Gemini and a few times gave the right answer faster than Google searching. However follow up questions were pretty terrible and conventional LLMs seemed to be better at refining.

This is going to be a bit of a painful ongoing task I think.

Online learning

I’ve been revisiting some Javascript and Typescript basics recently because both languages have changed since I originally encountered them and some new features have replaced previous conventions. I prefer text-based learning because I find it much easier to skim over areas that I know that it is to fast-forward through a video. I therefore have been using Educative and Lean Web Club. Lean Web Club is primarily Web Standards based Javascript and a bit of CSS, its small projects and bite-sized explanations are pretty handy but it lacks an internal search for when you can’t quite remember where something is located. It has been handy for seeing examples of how low-level ES Modules work, Web Components and also getting an overview of the different storage APIs that exist (and which ones haven’t been deprecated!).

Educative is broader in its content and works with different content providers to adapt their material to the platform. Therefore the style is a bit more variable particularly in the granularity of the course topics. It features mini-quizzes and again the quality is a bit variable but it does try to use different means to consolidate learning.

Like everything today, Educative has an LLM element which means it can ask open-ended questions that you reply to with free text and then your answer is evaluated. This seems pretty handy for things like interviewing and testing how clear your explanations are. However just like interviewing it can suffer from unclear questions.

For example in one question about distributed systems it wanted more detail on handling distribution across geographic regions but was unclear about whether there was meant to be a global identity service for all regions or the service was meant to be independently distributed so regions were compatible but still globally unique. There wasn’t really a way to tease that out of the LLM and even the “ideal” answer wasn’t very clear on the preferred approach.

What is awesome in Educative (and credit to MDN because it also has this feature in its documentation and I use it a lot there too) is that it has interactive code examples inline that you can edit and play around with. This allows you to see the effect of the code which is often easier than reading about what it is meant to do and you can play around to confirm your understanding of what is happening.

There were lots of Typescript modules I wish had read before I encountered them in the real world: membership of interfaces and its associated type checkers and when basic type inference fails for example.

Rust Nation pre-conference talks

I went to a community meetup of preview talks from the Rust Nation conference that was held last month. The most interesting talk was this one about the culture of purity in Rust around the use of unsafe and in fact how if this desire to be memory-safe is to be realised there needs to be work in some of the core libraries that the language community uses. I thought Tim did a good job of combining practical research with a plea for a more tolerant community.

Standard
Month notes, Work

March 2024 month notes

Dependabot under the hood

I spent a lot more time this month than I was expecting with one of my favourite tools Github’s Dependabot. It started when I noticed that some of the projects were not getting security updates that others were. I know it possible for updates to be suspended on projects that neglect their updates for too long (I should really archive some of my old projects) but checking the project settings confirmed that everything was setup correctly and there was nothing that needed enabling.

Digging in I wondered how you are meant to view what Dependabot is doing, you might think it is implemented as an Action or something similar but in fact you access the information through the Insights tab.

Once I found it though I discovered that the jobs had indeed been failing silently (I’m still not sure if there’s a way to get alerted about this) because we had upgraded our Node version to 20 but had set the option engine-strict on. It turns out that Dependabot runs on its own images and those were running Node 18. It may seem tempting to insist that your CI uses the same version as your production app but in the case of CI actions there’s no need to be that strict, after all they are just performing actions in your repository management that aren’t going to hit your build chain directly.

Some old dependencies also caused problems in trying to reconcile their target version, the package.json Node engine and the runtime Node version. Fortunately these just highlighted some dependency cruft and depreciated projects that we just needed to cut out of the project.

It took a surprising amount of time to work through the emergent issues but it was gratifying to see the dependency bundles flowing again.

Rust

I started doing the Rustlings tutorial again after maybe a year in which I’d forgotten about it (having spent more time with Typescript recently). This is a brilliant structured tutorial of bite-sized introductions to various Rust concepts. Rust isn’t that complicated as a language (apart from its memory management) but I’ve found the need to have everything right for the code to compile means that you tend to need to devote dedicated time to learning it and it is easy to hit some hard walls that can be discouraging.

Rustlings allows you to focus on just one concept and scaffolds all the rest of the code for you so you’re not battling a general lack of understanding of the language structure and just focus on one thing like data structures or library code.

Replacing JSX

Whatever the merits of JSX it introduces a lot of complexity and magic into your frontend tooling and I’ve seen a lot of recommendations that it simply isn’t necessary with the availability of tagged string literals. I came back to an old Preact project this month that I had built with Parcel. The installation had a load of associated security alerts so on whim I tried it with ViteJS which mostly worked except for the JSX compilation.

Sensing a yak to shave I started to look at adding in the required JSX plugin but then decided to see if I really needed it. The Preact website mentioned htm as an alternative that had no dependencies. It took me a few hours to understand and convert my code and I can’t help but feel that eliminating a dependency like this is probably just generally a good idea.

The weirdest thing about htm is how faithful it is to the JSX structure, I was expecting something a bit more, well, HTML-ly but props and components pretty much work exactly how they do in JSX.

Postgres news

A Postgres contributer found a backdoor into SSH that required an extensive amount of social engineering to achieve. If you read his analysis of how he discovered it then it seems improbable that it would have been discovered. Some people have said this is a counterpoint to “many eyes make bugs shallow” but the really problem seems to be how we should be maintaining mature opensource projects that are essentially “done” and just need care and oversight rather than investment. Without wanting to centralise open source it feels like foundations actually do a good job here by allowing these kind of projects to be brought together and have consistent oversight and change management applied to them.

I read the announcement of pgroll which claims to distil best practice for Postgres migrations regarding locks, interim compatibility and continuous deployment. That all sounds great but the custom definition format made me feel that I wanted to understand it a little better and as above, who is going to maintain this if it is a single company’s tool?

Postgres was also compiled into WASM and made available as an in-memory database in the browser, which feels a bit crazy but is also awesome for things like testing. It is also a reminder of how Web Assembly opens up the horizons of what browsers can do.

Hamstack

Another year, another stack. I felt Hamstack was tongue in check but the rediscovery of hypermedia does feel real. There’s always going to be a wedge of React developers, just like there will be Spring developers, Angular developers or anything else that had a hot moment at some point in tech history. However it feels like there is more space to explore web native solutions now than there was in the late 2010s.

This article also introduced me to the delightful term “modulith” which perfects describes the pattern that I think most software teams should follow until the hit the problems that lead to other solution designs.

Standard
Work

February 2024 month notes

Postgres

Cool thing of the month is pgmem which is a NodeJS in-memory database with a Postgres compatible API. It makes it easy to create very complete integration or unit tests covering both statement testing and object definitions. So far everything that has worked with pgmem has been flawless in both Docker-ised Postgres instances and CloudSQL Postgres.

The library readme says that containers for testing are overkill and it has delivered on that claim for me. Highly recommended.

Less good has been adventures in CloudSQL’s IAM world. A set of overlapping work requirements means that the conventional practices of using roles and superuser permissions is effectively impossible so I’ve been diving deeper than I’ve ever expected to go into the world of Postgres’s permission model.

My least favourite discovery this month has been that it is possible to successfully grant a set of permissions to a set of users that generates no errors (admittedly via a Terraform module; I need to check whether the Postgres directly complains about this) but also gets denied by the permission system.

The heart of the problem seems to be that the owner of the database objects defines the superset of permissions that can be accessed by other users but that you can happily grant other users permissions outside of that superset without error except when you try to use that permission.

The error thrown was reported on a table providing a foreign key constraint so there were more than a few hours spent wondering why the user could read the other table but then get permission denied on it. The answer seemingly being that the insert into the child table triggers the permission violation but that the validation of the constraint in the constraining table triggers the permission system.

I’m not sure any of this knowledge will ever be useful again because this setup is so atypical. I might try and write a DevTo article to provide something for a future me to Google but I’m not quite sure how to phrase it to match the query.

Eager initialisation

I learnt something very strange about the Javascript test data generation FakerJS this month but it just a specific example of libraries that don’t make an effort to lazy load their functionality. I’ve come across this issue in Python where it affected start times in on-demand code, Java where the assumption that initialisation is a one-time cost meant that multiple deployments a day meant the price was never amortised and now I’ve encountered it in Javascript.

My takeaways are that it is important to [set aggressive timeouts](https://nodejs.org/api/cli.html#–test-timeout) on your testing suite rather than take the default of no timeouts.. This only surfaced because some fairly trivial tests using the Faker data couldn’t run in under a second which seemed very odd behaviour.

Setting timeouts also helps surface broken asynchronous testing and makes it less tedious to wait for the test suite to fail or hang.

Standard
Work

October 2023 month notes

I’ve been learning more about Postgres as I have been moving things from Dataset to Psycopg3. It is kind of ridiculous the kind of things you can do with it when strip away the homogenising translation layer of things like ORMs. Return a set of columns from your update? No problem. Upsert? Straight-forward.

However after completing a CONFLICT clause I received a message that no conflict was possible on the columns I was checking and I discovered that I had failed to add a Primary Key to the table when I created it. It probably didn’t matter to the performance of the table as it was a link table with indexes on each lookup column but I loved the way that the query parsing was able to do that level of checking on my structure.

Interestingly I had a conflict clause in my previous ORM statement I was replacing and it had never had an issue so presumably it was doing an update then insert pattern in a transaction rather than using native features. For me this shows how native solutions are often better than emulation.

Most of the apps I’ve converted to direct use of queries are feeling more responsive now (including the one I use to draft these posts) but I’m not 100% certain whether this is because of switch to lower-level SQL or because I’ve been fixing the problems in the underlying relational model that were previously being hidden from me.

We’re going to need a faster skateboard

I have been thinking a lot about the Gold-plated Donkey Cart this month. When you challenge problems with solutions you often first have a struggle to get people to admit that there is a problem and even if it is admitted then often the first response is to try and patch or amend the existing solution than consider whether the right response might be.

We have additive minds so this tendency to patch what is existing is natural but sometimes people aggressively defend the status quo, even when it is counter-productive to their overall success.

Weakly typed

I’ve had some interesting experiences with Typescript this month, most notably an issue with a duplicated package which resulted in code that has been running in production for months but which has either not been correctly typed or has been behind the intended version by maybe four major versions. Typescript is interesting amongst type-hinted languages in that it has typing files that are often supplied separately from the code itself and in some cases which exist independently of the code itself. My previous experience of Python typing for example stopped the checker at the boundaries of third-parties and therefore only applied to the code you are writing yourself.

I’m uncertain of the value of providing type files for Javascript libraries as the compile-time and runtime contexts seem totally different. I found a Javascript dependency that had a completely broken unit test file and on trying to correct it I found that it couldn’t have the behaviour that the tests were trying to verify. Again I wondered about how this code was working in production and predictably it turned out that the executed code path never included the incorrectly specified behaviour. Dynamic code can be very resilient and at the same time a time bomb waiting to happen no matter what your

I think Typescript code would be better off if it was clearer that any guarantees of correctness can only be provided for the code you have totally under your control and which is being compiled and checked by you.

Frozen in time

I’ve been thinking a lot as well about a line from this talk by Killian Valkhof where he mentions that our knowledge on how to do things often gets frozen based on how we initially learnt to do things. For developers who learnt React for frontend will be the future people who learnt to do frontend via jQuery. I’ve been looking at Web Components which I thought were pretty terrible when they first came out but now look delightfully free of complex build chains and component models.

But more fundamentally it has made me think about when I choose or reject things am I doing so based on their inherent qualities in the present moment or based on the moment in time when I first learnt and exercised those skills. For CSS for example I’m relatively old-fashioned and I have never been a fan of the CSS-in-JS idea. However I think this approach, while maybe being outside contemporary preferences, is sound. Sound CSS applies across any number of frontend component models and frameworks and the work that goes into the CSS standards is excellent where as (ironically) the limitations of Javascript frameworks to express CSS concepts means that often it is a frozen subset that is usable.

I’ve never been entirely comfortable with Docker or Kubernates though and generally prefer PaaS or “serverless” solutions. Is that because I enjoyed the Heroku developer experience and never really understood the advantages of containerisation as a result.

Technology is fashion and therefore discernment is a critical quality for developers. For most developers though it is not judgement that they manifest but a toxic self-belief in the truth of whatever milieu they entered into the industry in. As I slog through my third decade in the profession doubt is something that I feel strongly about my opinions and trying to frame my judgements in the evidence and reasoning available now seems a valuable technique.

Standard
Work

September 2023 month notes

I tried the Kitten framework as I was quite surprised to see it request permission to change my privileged port permissions on install. I had to read through the post and its related posts before I realised that the nature of restricted ports is so ingrained in my I never asked whether it was genuinely a security risk to have them accessible from userspace. I would recommend taking a look through the posts linked to from the above post because when I realised that often my answer to port restrictions is to sudo onto them and that is a bit of weird way of not actually being secure.

I haven’t done much with Kitten, just working my way through the tutorial. The static serving is fine, the dynamic pages are a bit odd and the default ability to retain server state feels very odd.

I’ve also been continuing to try and learn Koa although if this wasn’t related to work I wouldn’t be bothering. Taking a look at the State of Javascript indicates that Express is the runaway winner and all other frameworks are pretty esoteric.

As an aside the state of in-page linking in the State of Javascript is embarrassing, the page structure is really complicated and doesn’t seem able to assign a simple id to a section tag.

Koa is from the school of “everything is a plugin” so out of the box it is completely anaemic and you have the zero-joy experience of trying to figure what libraries and plugins you should use. Most of the core plugins haven’t been updated in years, which is good in terms of stability but makes it hard to understand which libraries are actually unmaintained and which are fundamental. I much prefer the Python approach of having batteries included but being able to swap things out if you have particular needs.

One thing that Koa does differently to Express is to use a combined Context object instead of explicit request and response objects. I don’t think that is really very helpful and I did manage to mix the concept up with Go Contexts. Koa contexts are just a big ol’ object that includes some default values and the response kind of magically fires after all the middleware has fired. I feel it is a bit of step backwards in terms of clarity. My guess is that it makes it easier for plugins to add functions into the context object rather than having to explicitly import them and use them within the handler code.

I’m building a basic old school webapp so I needed some templating and that was a bit of journey in terms of what is popular but Nunjacks is work-friendly and based on jinja2 so it feels very familiar.

I’ve been slowly continuing to replace my various Python database libraries with a simpler and faster set of string queries executed through psycopg3. Next on the chopping block is Pony, which while it is relatively enjoyable as an ORM is needlessly clever in its use of generators and lambdas to do queries. I found a broken query and despite reading through the documentation I couldn’t fix it. If you already know SQL an abstraction has to be pretty powerful to be worth the overhead on things like queries which are fundamental a string and a map of data bindings and not much more.

I attended the State of the Browser conference this month and it was a good edition that balanced the input of practicioners, browser makers and had practical technical advice and reminders. It also managed to limit itself to only one non-technical related talk. I’ll write up a few notes in a separate post but this felt like a great return on the time invested.

I also discovered Rosie Pattern Language this month, a parser based alternative to regular expressions, I was intrigued but it lacks a helpful tutorial or introduction article so it has gone into the backlog to investigate later.

I started reading the book The Logic of Failure this month and I’m about halfway through it. It is a fascinating read and describes a series of experiments done with computer simulations of various situations from a town, a sub-Saharan ecosystem to a fridge with a broken thermometer. The outcomes are then mapped to the participants voiced thoughts to try and identify patterns of behaviour and the underlying rationales that drive them. Obviously the goal of reading such books is to try and temper the causes of failure in yourself but some of the problems the book highlights such the behaviour of complex inter-related components and exponential behaviour are just things that all humans are bad at.

Standard