Tag Archives: unconference

London, Work

Will humans still create software?

Recently I attended the London CTOs Unconference: an event where senior technical leaders discuss various topics of the day.

There were several proposed sessions on AI and its impact on software delivery and I was party of a group discussing the evolution of AI-assisted development and looking at the how this would change and ultimately what role people thought there would be for humans in software delivery (we put the impacts of artificial general intelligence to one side to focus on what happens with the technologies we currently have).

The session was conducted under Chatam House equivalent rules so I’m just going to record some of the discussion and key points in this post.

Session notes

Currently we are seeing automating of existing processes within the delivery lifecycle but there are opportunities to rethink how we deliver software that makes better use of the current generative AI tools and perhaps sets us up to take advantage of better options in future.

Rejecting a faster horse

Thinking about the delivery of the whole system rather than just modules of code, configuration or infrastructure allows us to think about setting a bigger task than simply augmenting a human. We can start to take business requirements in natural language, generate product requirement documents from these and then use formal methods to specify the behaviour of the system and verify that the resulting software that generative AI creates meets these requirements. Some members of the group had already been generating such systems and felt it was a more promising approach than automating different roles in the current delivery processes.

Although these methods have existed for a while they are not widely used currently and therefore it seems likely that reskilling will be required by senior technical leaders. Defining the technical outcomes they are looking for through more formal structures that work better with machines requires both knowledge and skill. Debugging is likely to move from the operation of code to the process of generation within the model and leading to an iterative cycle of refinement of both prompts and specifications. In doing this, the recent move to expose more chain of thought information to user is helpful and allows the user to refine their prompt when they can see flaws in the reasoning process of the model.

The fate of code

We discussed whether code would remain the main artefact of software production and we didn’t come to a definite conclusion. The existing state of the codebase can be given as a context to the generation process, potentially refined as an input in the same way as retrieval augmented generation works.

However if the construction of the codebase is fast and cheap then the value of code retention was not clear, particularly if requirements or specifications are changing and therefore an alternative solution might be better than the existing one.

People experimenting with whole solution generation do see major changes between iterations; for example where the model selects different dependencies. For things like UIs this matters in terms of UX but maybe it doesn’t matter so much for non-user facing things. If there is a choice of database mappers for example perhaps we only care that the performance is good and that SQL injection is not possible.

Specifications as artefacts

Specifications and requirements need to be versioned and change controlled exactly as source code is today. We need to ensure that requirements are consistent and coherent, which formal methods should provide, but analysis and resolution of differing viewpoints as to the way system works will remain an important technical skill.

Some participants felt that conflicting requirements would be inevitable and that it would be unclear as to how the generating code would respond to this. It is certainly clear that currently models do not seem to be able to identify the conflict and will most probably favour one of the requirements over the others. If the testing suite is independent than behavioural tests may reveal the resulting inconsistencies.

It was seen as important to control your own foundation model rather than using external services. Being able to keep the model consistent across builds and retain a working version of it decouples you from vendor dependencies and should be considered as part of the build and deployment infrastructure. Different models have different strengths (although some research contradicts this anecdotal observation). We didn’t discuss supplementation techniques but we did talk about priming the code generation process with coding standards or guidelines but this did not seem to be a technique currently in use.

For some participants using generative AI was synonymous with choosing a vendor but this is risky as one doesn’t control the lifespan of such API-based interactions or how a given model might be presented by the vendor. Having the skills to manage your own model is important.

In public repositories it has been noted that the volume of code produced has risen but quality has fallen and that there is definitely a trade-off being made between productivity and quality.

This might be different in private codebases where different techniques are used to ensure the quality of the output. People in the session trying these techniques say they are getting better results than what is observed in public reporting. Without any way of verifying this though people will just have to experiment for themselves and see if they can improve on the issues seen in the publicly observed code.

When will this happen in banks?

We talked a little bit about the rate of adoption of these ideas. Conservative organisations are unlikely to move on this until there is plenty of information available publicly. However if an automated whole system creation process works there are significant cost savings associated with it and projects that would previously have been ruled out as too costly become more viable.

What does cheap, quick codebases imply?

We may be able to retire older systems with a lot of embedded knowledge in them far more quickly than if humans had to analyse, extract and re-implement that embedded knowledge. It may even be possible to recreate a mainframe’s functionality on modern software and hardware and make it cheaper than training people in COBOL.

If system generation is cheap enough then we could also ask the process to create implementations with different constraints and compare different approaches to the same problem optimising for cost, efficiency or maintenance. We can write and throw away many times.

What about the humans?

The question of how humans are involved in the software delivery lifecycle, what they are doing and therefore what skills they need was unclear to us. However no-one felt that humans would have no role in software development but that it was likely to be different to the skill set the made people successful today.

It also seemed unlikely that a human would be “managing” a team of agents if the system of specification and constraints was adopted. Instead humans would be working at a higher level of abstraction with a suite of tools to deliver the implementation. Virtual junior developers seemed to belong to the faster horse school of thinking.

Wrapping up the session

The session lasted for pretty much the whole of the unconference and the topic often went broad which meant there were many threads and ideas that were not fully resolved. It was clear that there are currently at least two streams of experimentation: supplementing existing human roles in the software delivery cycle with AI assistance and reinventing the delivery process based on the possibilities offered by cheap large language models.

As we were looking to the future we most discussed the second option and this seems to be what people have in mind when they talk about not needing experienced software developers in future.

In some ways this is the technical architect’s dream in that you can start to work with pure expressions of solutions to problems that are faithfully adhered to by the implementer. However the solution designer now needs to understand how and why the solution generation process can go wrong and needs to verify the correctness and adherence of the final system. The non-deterministic nature of large language models are not going away and therefore solution designers need to think carefully about their invariants to ensure consistency and correctness.

There was a bit of an undertow in our discussions about whether it was a positive that a good specification almost leads to a single possible solution or whether we need to allow the AI to confound our expectation of the solution and created unexpected things that met our specifications.

The future could be a perfect worker realising the architect’s dream or it could be more like a partnership where the human is providing feedback on a range of potential solutions provided by a generation process, perhaps with automated benchmarking for both performance and cost.

It was a really interesting discussion and an interesting snapshot of what is happening in what feels like an area of frenzied activity from early adopters, later adopters probably can afford to give this area more time to mature as long as they keep an eye on whether the cost of delivery is genuinely dropping in the early production projects.

Standard
Events

Barcamp 13

Barcamp 13 is a general tech and nerdery unconference in London. In its current incarnation it is a one day event at an academy school next to the Tottenham Hotspur stadium.

Although most of the topics were technology related the analog sessions were amongst the most memorable, in particular the Cèilidh session was really fun and a total change in energy. I wasn’t expecting to dance when arrived. I also enjoyed the Minimal Viable Zine session which was about making single sheet zines for communicating urgent information indirectly but person to person.

The work relevant sessions included Nested CSS, which I’ve now started to adopt for my CSS work and looking at how to apply retrospectively to my hobby projects.

I also went to a session about tackling polarisation which was quite interesting as it had both self-proclaimed leftists and Hungarian fans of Orban. I was curious as to why the convener felt that polarisation was new and a problem. The answer the group came up with was that if the polarisation results in shrinking the envelope of who are considered people or a reduction in the rights of people in society then you are potentially talking about life and death. We’ve seen this in the treatment of both refugees and trans rights.

A few things that came through in this session was that there was a strong belief in the power of media (traditional and social) to change social attitudes. I think that would be something interesting to follow up on.

There was also a session on AI-generated music which was interesting but also worrying and which I think probably deserves a post in its own right.

I learned some things, I had fun, I enjoyed chatting to the other attendees and I managed to not be on fire. It was a super interesting day and I will definitely make an effort to get to the next one.

Standard
Clojure, Programming

London Clojure unconference July 2014 report

For the first session I was interested in trying to continue the discussion about the Clojure “sweet spot” we had had on the mailing list. But there was only a smattering of interest so we rolled it up with the discussion on how to convince people in investment banks to use Clojure.

I think Jon Pither’s approach to this is still the best which is to find a business problem and then say that you’re going to address the problem and use Clojure to solve the real problem. A pure technical argument is not really going to get buy-in from outside the developers.

A lot of organisations want to have an approved list of technologies and for institutions that have chronic and acute technical problems like banks then perhaps that is appropriate given the need for external regulation. Where these things exist I usually think it is a case of going through the bureaucratic hoops.

The approval system is not there to be opinionated but to provide oversight. Where individuals have “weaponised” the approval process to advance their view of “right” technology you need to tackle the root problem not just sneak things in as jars.

My personal view is that financial institutions have profound technology problems but that they have no incentive to address them while they continue to make a lot of money. Really their problems should be providing opportunities for new approaches but as the existing institutions have created massive barriers to entry it doesn’t happen and we’re all really just waiting for the next financial crisis to happen, maybe then…

However in the session there was a lot of discussion about whether it is appropriate for managers to determine technology choices: on the one side you want to devolve decisions to the people close to the problem, on the other programmers commonly change jobs in a shorter period that the lifespan of the software they create.

One thing I took away was that before conservative organisations adopt Clojure they will need to see widespread adoption in the companies they see as good leading indicators and the presence of a large hiring population. In these respects Scala is literally years ahead.

Our final conclusion as a group was simply that the easiest way to approve the use of Clojure was to get into management and leadership first and then do it.

For the second session I went to the discussion on React and Om. I’m looking at React currently and there were a lot of questions about what Om layers on top of the basic JS library. Anna Pawlicka provided a number of the Om answers and others chipped in with bits of React and reactive JS knowledge. I was reminded to go and look at the current state of Om and also the new tutorials. There was also some interesting talk of how to define React components, Anna used Sablono but is there still a need for JSX?

The final session of the evening was on Riemann, which in addition to be a basic introduction to what it does was a helpful reminder of the functionality that Riemann has but that I haven’t used personally. Jason Neylon mentioned that every new service they set up has a Riemann instance attached so you can just dump all events somewhere and then build dashboards dynamically as you go along (a lot better than our approach with Graphite).

Tom Crayford introduced me to the effect of clock skew on Riemann (events from the “future” relative to the Riemann server clock are dropped) and then pointed out that clock skew can actually be monitored via Riemann! Also some interesting stuff about pumping logs into Riemann and some personal experience of crazy volumes of events being successfully handled.

Just before the end of the event I dropped in to the Gorilla REPL session to see Jony Hudson demoing his amazing notebook repl that he has been using to share assignments and research with students and colleagues in his department. A really interesting application and I suspect once we get our heads round it a really interesting way of sharing problems and potential solutions as developers.

Mind slightly blown, I was personally really happy with the event and felt that I’d got a mix of advice and the kind of innovation that make the Clojure community so interesting.

Standard
Programming

Scale Summit 2014

Scale Summit is the new Scale Camp, an unconference aimed at bringing the same kind of topics as you might expect at Velocity.

This was the first Scale Summit, the venue was excellent as was the food (especially the bacon rolls, from Eden apparently) and supply of drink. Scale Summit happens under Chatham House rules so there’s no attribution of what is said which allows the attendees to be really frank and also for people to be free with what they really know rather than hedging and trying to be “on message”. It makes for a fascinating gathering.

The sessions varied in their organisation but all focussed on discussion between the participants. I managed to go to the Elasticsearch session, which was interesting for the practical boundaries that people were finding and also the operational knowledge. On the subject of using ES as the primary application store, the feeling seemed to be “not yet”, but there was also some words of wisdom about separating out document stores and search functionality and not finding a superficial unity in the two purposes.

The microservices session was a fast and furious fishbowl, easily the liveliest event and one that is going to require a post in its own right. It was interesting to see that the room split into practitioners and people who were sceptical that microservices were a thing or held value over conventional service development.

After lunch I sat in on what can be done to get frontend testing off the critical path to production (not much now but clearly more effort needs to be made), distributed DOS attacks on transactional sites (not as scary as I imagined but again we have to be thinking about how this works), distributed data stores (good war stories, felt better informed for going), getting ops and developers to work together and Linux containers (definitely going to try Docker now).

I had quite a few questions going into the event and while I didn’t get all the answers I hoped for I did at least establish that smart people don’t have simple answers to them either which is reassuring. It’s hard to tell in the heat of it all whether you’re on the edge of things doing things that are pushing the boundaries or simply over-complicating your situation.

The attendees were nicely mixed and from a range of backgrounds, ops, architecture and developers were all well-represented so you felt you were seeing a rounded situation.

The unconference format left me wanting more rather than feeling I had had enough. The openess was amazing and I am planning on being there next year.

Standard