Programming

PyPika

PyPika is a DSL for creating SQL queries. It works by generating SQL from definitions that you create inline rather than by interpreting models or data classes or structures.

It makes it easy to define minimum projections and is also straight-forward to bind data to as you simply provide the values into the query that is generated rather than binding values to a query definition.

Once you have the query object constructed you call str on the object to obtain the actual SQL which you then need to pass to some execution context.

This also means that it is easy to test what SQL you are generating (although really using a DSL like this means you should really be trusting the library).

I have only had one real problem with the statements I have used to date and that is around the Postgres `RETURNING` statement that allows a generated key to be returned to the caller of an INSERT.

Apart from this using Pypika has been better than using a Python template or writing raw SQL.

Standard
Programming

Sustainable web development

This article about delivering a sustainable small website is one of the most inspiring things I’ve read in weeks. It made me think about the true costs of the web revolution and made me think again about the environmental claims we make about moving paper processes online. Green energy and clean websites are entirely possible but will people pay the premium for it?

Finally like all truly great pieces it offered some practical insight into problems I have been grappling with in my work. The observation that small, fast loading server-rendered pages are the same as client-side SPAs but more energy efficient is a great insight.

However the best was the idea of simplifying the navigation into its own page. This one thing is a piece of genius that hundreds of man-years of UX research have failed to discover.

Thinking about the energy cost of our websites is also a great proxy for thinking about the cost basis for our infrastructure and where we want to put our processing effort.

Standard
Programming, Software, Web Applications, Work

Prettier in anger

I’ve generally found linting to be a pretty horrible experience and Javascript/ES haven’t been any exception to the rule. One thing that I do agree with the Prettier project is that historically linters have tried to perform two tasks to mixed success: formatting code to conventions and performing static analysis.

Really only the latter is useful and the former is mostly wasted cycles except for dealing with language beginners and eccentrics.

Recently at work we adopted Prettier to avoid having to deal with things like line-lengths and space-based tab sizes. Running Prettier over the codebase left us with terrible-looking cramped two-space tabbed code but at least it was consistent.

However having started to live with Prettier I’ve been getting less satisfied with the way it works and Prettier ignore statements have been creeping into my code.

The biggest problem I have is that Prettier has managed its own specific type of scope creep out of the formatting space. It rewrites way too much code based on line-size limits and weird things like precedent rules in boolean statements. So for example if you have a list with only one entry and you want to place the single entry on a separate line to make it clear where you intend developers to extend the list Prettier will put the whole thing on a single line if it fits.

If you bracket a logical expression to help humans parse the meaning of the statements but the precedent rules mean that brackets are superfluous then Prettier removes them.

High-level code is primarily written for humans, I understand that the code is then transformed to make it run efficiently and all kinds of layers of indirection are stripped out at that point. Prettier isn’t a compiler though, it’s a formatter with ideas beyond its station.

Prettier has also benefited from the Facebook/React hype cycle so we, like others I suspect, are using it before it’s really ready. It hides behind the brand of being “opinionated” to avoid giving control over some of its behaviour to the user.

This makes using Prettier a kind of take it or leave it proposition. I’m personally in a leave it place but I don’t feel strongly enough to make an argument to remove from the work codebase. For me currently tell Prettier to ignore code, while an inaccurate expression of what I want it to do, is fine for now while another generation of Javascript tooling is produced.

Standard
Programming, Python

403 Forbidden errors with Flask and Zappa

One thing that tripped me up when creating applications with Zappa was an error I encountered with form posting that seems to have also caught out several other developers.

The tldr is that if you are getting a 403 Forbidden error but your application is working locally then you probably have a URL error due to the stage segment that Zappa adds to the URL of the deployed application. You need to make sure you are using url_for and not trying to write an absolute path.

The stage segment

Zappa’s url structure is surprisingly complicated because it allows you to have different versions of the code deployed under different aliases such as dev, staging and production.

When running locally your code doesn’t have the stage prefix so it is natural to use a bare path, something like flask.redirect(‘/’) for example.

If you’re using the standard form sequence of GET – POST – Redirect then everything works fine locally and remotely until the raw redirect occurs remotely and instead of getting a 404 error (which might tip you off to the real problem more quickly) you get a 403 forbidden because you are outside the deployed URL space.

If you bind a DNS name to a particular stage (e.g. app-dev.myapp.com) then the bare path will work again because the stage is hidden behind the CloudFront origin binding.

Always use url_for

The only safe way to handle URLs is for you to delegate all the path management and prefixing to Zappa. Fortunately Flask’s in-built url_for function, in conjunction with the Zappa wrapper can take care of all the grunt work for you. As long as all your urls (both in the template and the handlers) use url_for then the resulting URLs will work locally, on the API Gateway stages and if you bind a DNS name to the stage.

If this is already your development habit then great, this post is irrelevant to you but as I’ve mostly been using Heroku and App Engine for my hobby projects I’d found myself to be in the habit of writing the URLs as strings, as you do when you write the route bindings.

Then when the error occurred I was checking the URL against my code, seeing that they matched and then getting confused about the error because mentally I’d glossed over the stage.

Standard
Programming

Google Cloud Functions

I managed to get onto the Google Cloud Functions (GCF) alpha so I’ve had a chance to experiment with it for a while. The functionality is now in beta and seems to be generally available.

GCF is a cloud functions, functions as a service, AWS Lambda competitor. However thanks to launching after Lambda it has the advantage of being able to refine the offering rather than cloning it.

The major difference between GCF and Lambda is that GCF allows functions to be bound to HTTP triggers trivially and exposes HTTPS endpoints almost without configuration. There’s no messing around with API Gateway here.

The best way I can describe the product is that it brings together the developer experience of App Engine with the on-demand model of Lambda.

Implementing a Cloud Function

The basic HTTP-triggered cloud function is based on Express request handling. Essentially the function is just a single handler. Therefore creating a new endpoint is trivial.

Dependencies are automagically handled by use of a package.json file in the root of the function code.

I haven’t really bothered with local testing, partly because I’ve been hobby-programming but also because each function is so dedicated the functionality should be trivial.

For JSON endpoints you write a module that takes input and generates a JSON-compatible object and test that. You then marshal the arguments in the Express handler and use the standard JSON response to send the result of the module call back to the user.

Standard
Programming

Why developers love the shiny

Developers are notorious neophiles who love adopting new technologies and designs. There is a natural bias between interest in working in emerging sectors and a general interest in novelty.

Technology as an industry also emphasises change and novelty as part of its identity with concepts like “disruption” and “innovation” being fundamental to the self-identity of the sector.

I also believe that developers are perversely incentivised to value the new over the established.

Perverse incentives

If you choose to learn an emerging language you have less history to deal with.

If you learn Java you have access to lambdas but you also have to deal with a decades worth of code that was written before lambdas were available. You’ll have to learn about interfaces, with and without implementations; inner classes; anonymous classes; dependency injection and inheritance.

If you choose to learn an emerging technology then chances are you won’t be told to RTFM as it won’t have been written yet.

Instead you’ll have the chance to interact with the technologies creators. If you make a mistake then it will be a learning experience not just for you but for the community as a whole. You’ll be united in your discovery and collaborators instead of an ignorant johnny-come-lately.

And if you learn emerging technologies you will find that conferences are eager to offer you a chance to talk about your experience. Employers will value your knowledge more than the people who added another year of experience with C#.

In every way you will find life easier and more enjoyable if you focus on emerging technology.

Falling out of love

Programmers are often like ardent lovers, in the first flush of their crush they see their new discovery as the answer to all their problems. They are evangelists and poets for their new discovery.

But over time familiarity replaces enchantment and soon we are all too aware of the flaws of tools and take for granted their utility.

Zac Tellman recently put this beautifully in a more general post about open source governance.

When our sense of possibility of what we can do with our tools is exceeded by our understanding of the constraints they have, we start to look elsewhere.

A mature legacy

Legacy is a funny word in technology. As has already been observed most of the time people are delighted to receive a legacy and the word generally has positive conotations.

For developers though the term is loaded with fear as what we are inheriting in the form of a legacy codebase is not a valuable treasure that is being handed down for safekeeping, to be enhanced and handed on to the next generation of stewards.

Instead the legacy codebase is seen as troublesome, an unwanted timebomb that is to be kept in some kind of running order and passed off to someone else as soon as possible.

Maintainers of legacy codebases are also unvalued by organisations. They are often seen as less skilled and less important than developers creating new systems. After all what they are doing is simply keeping a successful system running, they don’t need to use much imagination and they are dealing with solved problems.

Maintaining a legacy codebase often means being overlooked for promotion, pay rises or new opportunities.

In fact this view extends beyond developers, organisations do not tend to value their legacy codebases either. Product managers are equally incentivised to value the new over the old. If they add features to an existing system they are seen as less imaginative than those who create brand new systems or approaches to existing problems.

I have seen product managers deliberately sabotage legacy products so that the performance of the new solution looks better. Management often fails to look at the absolute performance of new system, just the relative numbers. People exploit that tendancy ruthlessly.

Love the new

There are more examples, however they might be better suited for dedicated blog posts in their own right. Overall though I hope I’ve illustrated that how as a profession we are encouraging people to jump on every bandwagon that is passing by.

I want to try and avoid passing a moral judgement on this. I too am a neophile, I too love novelty and would prefer to do something that I haven’t done before over perfecting my skills in a domain I know.

I just want to try and highlight the issue and move it from our unconscious to our conscious minds. Is this what we want for ourselves?

Should we rewarding maintainers of the software that pays our wages less than “rockstars” building prototypes?

Should we value simplification of our existing tools over maintaining backwards compatibility?

These are all inherently cultural issues, not technical ones. Currently we have a culture of novelty and literal innovation. I’m not sure how well it is serving us.

Standard
Programming

Using Google App Engine with Pyenv

I recently started using PyEnv to control my Python installations and make it easier to try to move more of my code to Python 3.

Google App Engine though is unapologetically Python 2.7. Google wants people to move away from the platform in favour of Google Compute custom environments and therefore has little incentive to upgrade the App Engine SDK and environments to support Python 3.

When I set my default Python to be Python 3 with PyEnv I found that despite setting a local version of Python 2.7 my App Engine instance was failing to run with an execfile is not defined exception.

The App Engine Python scripts use #!/usr/bin/env python to invoke the interpreter and for some reason PyEnv doesn’t seem to override the global setting for this despite it being correct when you check it in your shell.

After a lot of frustration and googling for an answer I haven’t found anything elegant. Instead I found this Stack Overflow answer which helpful explained that you can use #!/usr/bin/env/python2 to invoke a specific language interpreter.

Manually changing the shebang line in appcfg.py and dev_appserver.py solved the problem for me and got me running locally.

Obviously this is a pain if I upgrade and I feel it might be better for Google to change the scripts since they don’t have a plan to move to Python3.

Standard