Tag Archives: rest

Month notes

December 2024 month notes

Not a whole lot to report on due to this being holiday season.

Colab

I started using Google’s Colab for quick Python notebooks. It’s pretty good and the notebook files integrate into regular Drive. Of course there is always the fear that Google will cancel it at a moment’s notice so I might look at the independent alternatives as well.

I’ve been looking at simulation code recently and it has been handy to run things outside a local setup and across laptops.

tRPC

You can’t spend much time in Typescript world without using Zod somewhere in your codebase. Zod was created by Colin McDonnell and this month I read an old blog post of his introducing the ideas behind tRPC. The post or essay is really interesting as it identifies a lot of problems that I’ve seen with GraphQL usage in projects (and to be fair some OpenAPI generated code as well).

It is quite rare to see a genuine REST API in the commercial world, it is more typical to see a REST and HTTP influenced one. GraphQL insists on a separation of the concepts of read (query) and write (mutation) which makes it more consistent than most REST-like interfaces but it completely fails to make use of HTTP’s rich semantics which leaves things like error handling as a bit of joke.

Remote Procedure Calls (RPC) preceded both REST and GraphQL and while the custom protocols and stub generators were dreadful the mental model associated with RPC is actually pretty close to what most developers actually do with both REST and GraphQL. They execute a procedure and get a return result.

Most commercial-world REST APIs are actually a kind of RPC over HTTP using JSON. See the aside in the post about GraphQL being RPC with a schema.

Therefore the fundamental proposition of the post seems pretty sound.

The second strong insight is that sharing type definitions is far preferable and less painful than sharing generated code or creating interface code from external API definitions (I shudder when I see a comment in a codebase that says something like “the API must be running before you build this code”). This is a powerful insight but one that doesn’t have a totally clean answer in the framework.

Instead the code reaches out to import the type definition from the server by having the local codebase available in some agreed location. I do think this is better than scraping a live API and type sharing code is clearly less coupled than sharing data structures but I’m not sure it is quite the panacea being claimed.

What it undoubtedly does improve on is generated code, generated code is notorious hard to read, leads to arguments about whether it should be version controlled or not and when it goes wrong there is almost inevitably the comparison dance between developers who have working generated code and those who don’t. Having a type definition that is version controlled and located in one place is clearly a big improvement.

I’ve only seen a few mentions of commercial use of tRPC and I haven’t used it myself. It is a relatively small obscure project but I’d be interested in reading production experience reports because on the face of it it does seem to be a considered improvement over pseudo-REST and GraphQL interfaces.

God-interfaces

The article did also remind me of a practice that I feel might be an anti-pattern but which I haven’t had enough experience so far to say for sure. That is taking a generated type of a API output and using it as the data type throughout the client app. This is superficially appealing: it is one consistent definition shared across all the code!

There are generally two problems I see with this approach, firstly is protocol cruft (which seems to be more of a problem with GraphQL and automagic serialisation tools) which is really just a form of leaky abstraction; secondly, if a data type is a response structure from a query type of endpoint then the response often has a mass of optional fields that continuously accrue as new requirements arrive.

You might be working on a simple component to do a nicely formatted presentation of a numeric value but what you’re being passed are twenty plus fields, none of which might exist or have complex dependencies between one another.

What I’ve started doing, and obviously prefer, is to try and isolate the “full fat” API response at the root component or a companion service object. Every other component in the client should use a domain typed definition of its interface.

Ideally the naming of the structures in the API response and the client components would allow each domain interface to be a subset of the full response (or responses) if the component is used across different endpoints.

In Typescript terms this means components effectively define interfaces for their parameters and passing the full response object to the component works but the code only needs to describe the data actually being used.

My experience is that this has led to code that is easier to understand, is easier to modify and is less prone to breaking if the definition of the API response changes.

The death of the developer

I’ve been reading this Steve Yegge post a lot as well The Death of the Stubborn Developer. Steve’s historical analysis has generally been right which gives me a lot of pause for thought in this post. He’s obviously quite invested in the technology that underpins this style of development though and I have a worry that it is the same kind of sales hustle that was involved in crypto. If people don’t adopt this then how is the investment in this kind of assisted coding going to be recouped.

Part of what I enjoy about coding is the element of craft involved in putting together a program and I’m not sure that the kind of programming described in the post is the kind of thing I would enjoy doing and that’s quite a big thing given that it has been how I’ve made a living up until now.

Standard
Programming, Web Applications, Work

Why can’t Forms PUT?

HTML Forms can declare a method, the HTTP verb that is used when the form is submitted, the value of this method is GET or POST.

The HTML5 spec briefly had PUT and DELETE as valid methods for the form method but has now removed them. Firefox also added support and subsequently removed them.

Recently over the course of Brexit night at The Guardian we got into a discussion about why this was the case and what the “right” way to map a form into a REST-like resource system would be.

The first piece of research was to dig into why the additional methods had been added and then removed. The answer (via Ian Hickson) was simple: PUT and DELETE have implied idempotency, the nature of form submission is that it is inherently uncacheable and therefore cannot be properly mapped onto those verbs.

So, basic problem solved, it also implies the solution for the url design for a form. A form submission represents a user submitting an untrusted data payload to a resource, this resource in turn choose to make PUT or DELETE requests but it would be dangerous to have the form do this directly.

The resource therefore is one that represents the form submission. In terms of modelling the URL I would be tempted to say that it takes the form :entity/form/submission, so for example: contact/form/submission.

There may be an argument that POSTing to the form resource represents submission so the submission part of the structure is unnecessary. In my imagination though the form resource itself represents the metadata of the form while the submission is the resource that essentially models a valid sumbission and the resource that represents the outcome of the submission.

Standard
Programming

The state of microservices

One of the liveliest sessions at Scale Summit was the one on microservices where opinions flowed free and fast in a rapidly rotating fishbowl.

There were several points of interest that I took away and I thought I would write them up here as some of the key questions. In another post I’ll talk about the problems with microservices that haven’t been solved yet.

Are we doing microservices yet?

Some people in the room had already adopted microservices, the reasons given included: breaking down or trying to change functionality monolith codebases, trying to scale up an existing applications or architectures or actually as an existing best practice that should be applied more generally.

What is a microservice?

A few people wanted to get back to this. Showing that while it is a handy term it isn’t universally understood or agreed.

I personally think a microservice is a body of code whose purpose and implementation is easy to understand when studied, adheres to the Unix philosophy of doing one thing well and which can be re-implemented quickly if needed.

Are microservices just good practice or righteous SOA?

Well naturally you can regard every new concept as being a distillation of previous good practice. However the point of naming something is to try and make it easy to triangulate on a meaning in conversation.

Won’t microservices just be corrupted by consultants and architects?

Yes, everything popular gets corrupted in time. I’m okay with that because in the interval we have a handy term to describe some useful patterns of solution design.

Don’t microservices complicate operations?

One attendee put it well: microservices are recursive so if the operations team are going to support them then they should be in the business of creating a service that deploys services.

Some people felt that for microservices to work developers and teams had to have full stack ownership and responsibility but I felt that was trying to smuggle devops in under the microservices banner.

I think microservices are best deployed on a platform and that platform defines what a deployable service is and can be responsible for shutting down misbehaving services.

Such a scheme allows for other aspects of the Unix way to be applied such as man pages, responding to –help and other useful conventions.

The platform can check whether these conventions have been met before deploying the service.

Isn’t microservices just a way to discuss the granularity of a service?

Yes in a way, although there are a few other practices that make up a successful application of microservices you can just think about it as being a way of checking the responsibility boundaries of your service and how easy it would be to replace.

If your service has multiple responsibilities and is difficult to replace easily then it might have problems.

A lot of people wanted to use AWS an an example of good services without microservices. I think AWS is a good example of service implementation: each part has good boundaries and there are a few implementations of the key APIs. Things like the AWS security functionality is a good example of how you have to work hard to avoid having services rely on other services, and the result isn’t elegant as a result.

I would argue that public-facing APIs are probably where you want to composite microservices to provide a consistent facade onto the functionality though.

As other delegates pointed out, isolating services makes them easier to scale. If starting a server in the EC2 API requires more resources than shutting it down you might prefer to scale up just the creation service rather than many instances of the whole API which are unlikely to be used or consume resources.

As ever, horses for courses, you’re going to know your domain better than I do.

Don’t microservices cause problems as well as solve them?

Absolutely, choosing the wrong solution at the wrong time is going to cause problems. Zealously over-applying microservices or applying the idea to absurd levels is not going to have a happy outcome.

A guess a good point is that we know our problems with existing service implementations. We don’t know what problems there are with microservices or whether they have logical and simple solutions. However they are helping us solve some of our known problems.

Aren’t microservices simply REST-ful services done right?

The most common form of microservice today is probably one implemented via HTTP and JSON. However this form isn’t prescriptive. ProtocolBuffers might be a better choice for an internal exchange format and ZeroMQ might be a better choice for a transport.

I also think that message queues are a good basis for microservices with micro consumers and producers focussing on tight message types.

See also my mini-list of microservice myths which has more on this subject.

Should we be doing microservices?

I suspect that doing microservices for the sake of ticking a solution buzzword is bad. However I think microservices seem a pretty good solution to a problem I know I’ve seen in a fast-moving domain where you are trying to innovate without creating a maintenance burden or large legacy.

Standard
Programming, Ruby, Scripting, Web Applications

Sinatra and Haml

On Monday I was meant to be learning how to use the Grails framework (following up the impressive Grails site that Sky has launched) but I instead got distracted into scratching another itch, using Sinatra with JRuby. Since I was doing a little REST application I thought I would also give Haml a go as it promised a far easier way of generating HTML responses.

Installing both Gems was easy as ever and Sinatra was really easy to understand conceptually. Put the HTTP method name, the URI pattern you want to match and then the return value of the block is what goes back to the client in the response.

So for example to map “/hello” to the plain text response “Hello World” you simply have.

get "/hello" do
  "Hello World"
end

Fire up the script (with necessary requires) and a HTTP server is set up and running. It really couldn’t be simpler. You can use a :symbol in the URI matcher and then access it through the params hash. Within half an hour I was starting to add resources into my served HTML and I felt like master of the REST UNIVERSE.

Sinatra is a really smart piece of code that makes it simple to write a basic web application. I have loads of ideas of how it could be useful but one of my first thoughts was that it actually does a good job of solving the issue of GUI platforms.

My next project is to see how it works with posting data back to the app but on the face of it it all seems straight-forward.

Haml is another story. It is undoubtedly a good idea and on the right track. A DSL for creating HTML the method html for example creates a block html tag  while p “Hello World” creates a paragraph tag with the parameter as the content. Hash parameters sent to the method become attributes of the class.

It is much quicker than generating the HTML by hand but often not by much. That is because Haml has not found the same easy metaphor for code and content that Sinatra has. It is very picky, constantly harping about two-space indentation (no more, no less, don’t dare use tabs you bastardo!) for example, it sometimes throws a stack trace that failed to make it to Sinatra leading to a blank screen on the browser and hunt into the console for a clue as to what has gone wrong.

The documentation is to poor to figure out why the library doesn’t accept what you are doing and I am still baffled as to whether the module handles inline tags or whether you are meant to devolve all of that to Textile. Passing parameters to Haml templates seems unnecessarily complex (the only way I could get it to work is with normal string interpolation, it’s a solution but it seems to break the DSL concept) and mixing inline formatting with parameters baffles me still.

Haml makes more sense to me than RHTML style templating but its claim to simplicity and enjoyment seems to come only with a deep understanding and long experience with using it. It took me 20 minutes to fall in love with Sinatra but it took 20 minutes for me to get a working Haml page. And I didn’t enjoy myself doing it.

Standard