Category Archives: Scripting

Python, Web Applications

Deploying Python apps to Epio

I recently got my beta access to ep.io, the Python application deployment platform. I had the chance today to have a play around and try out some deployments so I thought I would try and give my view on the experience before. I’ve deployed Python apps to Heroku and Gondor before so those services form my reference points here.

So firstly, there’s a command-line client that you install via pip and you effectively deploy to the platform via a client-command, SSH keys and what looks like git on the server-side. This is more like Gondor than Heroku (which is intimately linked to git). It means you have your choice of source control and if you want to be a Python purist you never need to step outside of Python for everything you are doing.

Applications consist of essentially one configuration file that states where the WSGI application is and what the requirements file is. Compared to Gondor it is a very simple setup but it did feel that it could be even simpler if it made convention-based assumptions such as the requirements file being called requirements.txt, for example.

Leveraging WSGI and configuration this way gives a very flexible platform and I was able to get both Flask and Bottle to work (the former very quickly because it has documentation, the latter via trial and error that might require its own blog-post). I didn’t have time to try Django but I felt pretty confident that I could get whatever framework I wanted working once I understood the basic setup.

Unlike Heroku, Epio provides a fixed framework for executing the apps. It seems you will be running behind NGINX and Gunicorn. Both are good choices and I certainly like them but if you want to play around with different servers like Tornado or CherryPy you may prefer Heroku’s more open deployment model. I did like the way that you can use the configuration file to have NGINX serve static content directly.

Epio naturally has less of an ecosystem than Heroku but has Solr, Postgres and Redis out of the box. All solid choices and covering off the majority of what I would need. I was certainly grateful that I didn’t have to grapple with remote database administration and could prototype apps with just Redis.

Deployment and logging have kind of rough edges. Being able to access logs directly from the application page was a win for me, however when I was struggling to define the WSGI entrypoint correctly it seemed as if the application wasn’t being really compiled until the first request comes in. I would see an entry confirming a new deployment but then nothing until I hit the app. I think there should be some kind of sanity check of what you have uploaded to see whether it will even run.

Right now epio is providing a Python-based cloud deployment platform with a sensible set of supplementary services and low opinion about the source control system to you use. It feels like if this had been around at the start of the year it would have blown me away. However now there is more competition and therefore questions of price and ease of use will matter in terms of  how compelling it is to use the service.

If you do Python web development I would definitely recommend you sign up for beta and give it a go yourself as it seems a very solid prototyping platform. If you are not a Ruby and Git fan then you may well love what is on offer here because it is already very convenient, makes few demands on you and gets your web app public in minutes.

Standard
Programming, Python

Django and JSON stores: a match in heaven

My current project is using CouchDB as its store and Django to implement the web frontend. When you have a JSON store such as CouchDB then Python is a natural complement due to its brilliant parsing of JSON into native data structures and its powerful dictionary data type that makes it really easy to work with maps.

In a previous project using Python and Mongo we used Presentation objects to provide domain logic on top of the raw data maps but this time around I wanted to try and cut out a layer and just work with maps as much as possible (perhaps the influence of the Clojure programming I’ve been doing on the side).

However this still leaves two problems that need addressing. Firstly Django templates generally handle dictionaries like a dream allowing to address them with the standard dot syntax. However both Mongo and Couch use leading underscores to indicate “special” variables and this clashes with the Python convention of having a leading underscore indicate a private member of the class. The most immediate time you encounter this is when you want to use the id of a document in a url and the naive doc._id does not work.

The other problem is the issue of legitimate domain logic. In Wazoku we want to use people’s names if they have supplied them and fallback to their email if they haven’t supplied their name.

The answer to both of these problems (without resorting to an intermediary object) is Django’s filters. The necessary logic can be written in a few lines of Python that simply examines the dictionary and does the necessary transformation to derive the id or user’s name. This is much lighter than the corresponding Presentation solution.

Standard
Programming, Python, Ruby

Truly open classes

Here’s an interesting observation, I needed to write a little script to automate some number calculating for me. I was wondering whether to do it in Ruby or Python. I’m doing a lot of Python at the moment so I felt I ought to give Ruby a little go. Share some of the love.

However the solution I had in mind really didn’t work with Ruby because while Ruby has open classes it has a comparatively fixed idea of attributes. In Python you can set attributes very freely on any object so I have got in the habit of creating something and then enhancing by applying a function. Example? Okay.

def make_captain(actor):
actor.rank = "Captain"
return actor

class Person:
pass

captain = make_captain(Person())

So this little trick doesn’t work, or rather is much more difficult to do in Ruby as Ruby, at is dynamic heart, is a language that believes in object-orientation and that classes should encapsulate rather than being little collections of data. You can use instance_variable_get/set but it lacks the elegance of the Python syntax.

In Ruby it would be easier to define the attributes in the class using the existing metaprogramming constructs and then have a class method to generate the content (effectively encapsulating my script logic).

Now this isn’t a straight “Ruby sux, no Python sux more” post. Between Scala, Clojure and Python I have been doing a lot more in a functional style that depreciates objects as anything more than value carriers. The Ruby vision of a class would give me something with a stronger sense of purpose and encapsulation, something that is hard to benefit from in a script for a particular purpose.

What is going to be interesting this year is trying to identify when the value of a piece of code is in the structure of it’s data-definition (i.e. objects) versus its process (functions). Having had a think about it I should perhaps rewrite my script to use some OO modelling because it may answer similar requirements down the line. However from a strict Lean/Waste point of view I should have gone with the Python solution as Ruby was imposing a restriction on me while providing benefits that I was unlikely to realise.

Standard
Python, Web Applications, Work

Declare for simplicity

I was struggling with an issue today on a template that was blowing up due to missing keys in a template data hash. At first I tried to write some conditional code in the template. That ended up being quite ugly so then I was trying to find a way to condition the template on whether the key was present.

Then it struck me that the issue was really the missing key. As the hash is prepared in Python code the original implementation frugally avoided creating the entry if the underlying data wasn’t present. While this is clever and minimal it actually just pushes the complexity out of the Presenter logic and into the template where it is much worse because you can only interact with the Presentation’s abstraction of the original data.

The problem here is that dynamic languages sometimes allow you to be too clever. In a declared structure system you have to declare all your data and provide sensible defaults. This makes writing the templating solution at lot easier as you never have a missing data problem to deal with (you still have headaches with duff defaults but that is a different post).

So I went back to the presenter and declared an empty list under the key that had been causing me grief. Bingo! My failure went away and the default behaviour of not generating any content for the empty list kicked off, reducing my template back to the unconditional single-line I had originally.

Standard
Ruby, Web Applications

Haml, Sinatra and template variables

Haml advertises itself as offering a haiku for HTML; that might mean it is short, beautiful and expressive or it could mean it is enigmatic, unsatisfying and cryptic. After a period away I have forgotten how Ruby-centric Haml is. If you want to include parameters in a Haml template within Sinatra you need to bind a map to the parameter locals.

haml :index, :locals => {:hello => “World”}

The easiest way to then include the data within a tag seems to be to use Ruby string interpolation

%p= “Hello #{locals[:hello]}”

There may be more elegant ways to do this but it seems relatively idiomatic. It is just hard to adjust to after months working on very restrictive templating systems.

Standard
Groovy, Programming

Low Expectations for the Build

I attended the talk on Gradle by Hans Doktor tonight and while I found myself agreeing that Maven is wholly unsatisfactory I did end up thinking that actually our expectations of build tools in the Java space are really low. What kind of things does Gradle offer us? Proper event interception, genuine integration with the build lifecycle, build targets dynamically defined at runtime, a directed cyclic dependency graph.

Looking at the list some of things you can’t believe are not part of our standard build package. We should be able to know when a build starts and stops and be able to attach code to those events. We should have decent target resolution that avoids duplication of tasks.

Gradle is head and shoulders about the morass that is Maven and is clearly superior to the ageing but faithful Ant but that it manages to be so on so little functionality is a shame.

Standard
Python, Web Applications

Juno, a micro web framework for Python

I love microframeworks and I love Python to it therefore follows that I love the minimal web frameworks you can get for Python like CherryPy. However as soon as I heard about Juno, a clone of my favourite microframework Sinatra I had to give it a go.

For me the archetypal application for Sinatra is Git-Wiki and therefore to kick the tires on Juno I decided to try and do the same thing with Bazaar. Hence:  Bzr-Wiki on Juno!

You need Juno, Jinja2 and SQLAlchemy as well the source from Launchpad. Once you have it all then got to the repo directory and type bzr init. You should then be good to fire up wiki.py and muck around with it.

It is slightly ugly and lacks access to revision history, diffs and user tracking but apart from that it is a surprisingly functional wiki. It is also bi-directional in that you can add files to the Bazaar repo and they get reflected in the app.

So what was Juno like to work with? Well overall I thought this was the best Python microframework I have used so far. I really like the idea of decorating methods to avoid having to generate a mapping table. The syntax is terse and comprehensible, the conventions around the framework made sense to me.

By comparison with its progenitor I think I really missed the autoloading/dynamic evaluation that allows you to change code in Sinatra and have it immediately take effect. The function of the Request decorator was initially quite obscure (it binds all HTTP verbs to that method, if you want to map GET to another method you must specify all verbs independently) and I am still not sure it is right. I think the most specific decorator should take precedence. Other than that I think the framework ports a lot of the concepts from Sinatra in a sympathetic way.

The dependency on SQL Alchemy is also really clunky. If you specify that you are not using a database (as is the case here) then it is annoying to have to download a dependency and makes installation on Windows a pain I didn’t even want to try and tackle.

Juno is really promising though and I look forward to it developing. I think it would be a real delight to use it in an environment like Google App Engine.

Standard
Programming, Ruby, Web Applications

Why does Heroku matter?

Heroku is an amazing cloud computing service that I think is incredibly important but which really isn’t getting the visibility it should be at the moment.

Why does Heroku matter?

Firstly it gets deployment right. What is deployment in Heroku? It is pushing a version of your app to the cloud. That’s it. The whole transformation into a deployable application slug is handled by the Heroku, pushing it out to your dynamos is handled by Heroku. Everything is handled by Heroku except writing your application.

Secondly, it is the first cloud/scalable computing targeted directly at Ruby that has the right pricing structure to encourage experimentation and innovation. It has the low barrier to entry that Google App Engine has but requires no special code and imposes no special restrictions.

Thirdly it does the right thing with your data. It respects your right to access, mine and manipulate your data and provides a mechanism for doing this easily.

For me Heroku has set a new benchmark for Cloud Computing and build and deployment in the enterprise as well.

Standard
Ruby, Scripting, Web Applications

Semi-static CMSs

You know you have a good idea when there are half a dozen implementations of it. All essentially the same but filled with subtle nuance. The latest idea I have stumbled across is the “semi-static” website generator. The idea is so painfully obvious I cannot believe that it has taken us until 2009 to do it properly.

What is a semi-static website? Well it’s actually something like this blog. I spend ages writing and editing the articles but once I publish something I don’t generally change it unless it has factual errors. I need something that allows me to make changes but my content easily but my content doesn’t really benefit in anyway from being stored in a database or being generated dynamically on the fly because it is going to spend most of its life not being changed.

If you look at a lot of CMS systems the same is true. People want to be able to edit and reorganise material easily but they are rarely changing it more than once a week. The majority of the time the pages are static.

If you look at a number of CMS or Wiki systems they are actually quite complicated. Before you can get going and generate your content (which remember is the core of what you want to do) you might have to setup a database and get the application deployed to a webserver.

What the semi-static website generator does is provide a set of scripts that allow you to edit and create content and then “compile” your site. The result is set of static HTML, Javascript and CSS files that can be deployed to your static webserver (which is generally pretty easy to setup, most servers will come with Apache all ready to go).

When you combine it with a bit of DVCS magic deployment is even easier. You just checkin your new pages and then update the webserver copy of the content.

So if you want to get into this kind of content management what options do you have? Well you have a lot of choice indeed. Almost all of them are Ruby projects and they have a certain similarity to the Rails concept of code generation. You will be issuing commands like “create page”, “build site” and so on. You operate at one remove from it.

There are many, many, many choices but for myself I chose three to look at Nanoc, Webby and Jekyll. Nanoc I came across via Proggit and was struck by the simplicity and clarity of the documentation.

Installation of all three is via gems but there are some differences here. Nanoc is a very simple Gem and works with HTML and Erb by default, if you want to use a markup language or different templating language you will have to install the gem yourself. This means it also installs easily under JRuby too.

Webby by comparison is a very noisy install, I think it may have its dependencies slightly wrong as it installs a version of rake, flexmock and rspec which are surely development dependencies rather than anything I as a user would need to use for the framework. It also uses hpricot which means you can’t install it under JRuby.

Jekyll has a similarly demanding set of requirements but will install under JRuby. Jekyll did feel a lot like a personal solution as I was installing it. Textile is great but if I am going to be using Markdown why do I need to install RedCloth?

Overall I liked the nanoc install experience most.

Moving on to using them. Nanoc and Webby are very similar in that they use generators to create new elements in the site. Jekyll seems to use convention instead. At this point it is worth mentioning that there are some big differences between the three products in their intentions. Nanoc is a general website creator but it does have a certain amount of opinion as to how to do that. For example it generates very nice clean website urls by putting each page in a directory by itself and then creating the page as an index.html page within that directory. Make sure you have your webserver configured appropriately.

Webby is pretty flexible but comes with some built-in templates that will be build you a basic page, a tumblog style site, a normal blog and a presentation. That was a pretty neat feature and one the other frameworks didn’t have.

However the negative side to this is that the templating is based on Blueprint; which is both excellent as Blueprint is a great way of designing pages and a pain as Blueprint is actually pretty involved in terms of understanding the grid concept and what the various span-x css classes translate to. I felt that Webby did a great job of offering cross-browser styling that looked good but that in doing so it didn’t make the simple stuff simple. With my trial site my sidebar came out on the bottom of the page and I didn’t really know how to get on the right-hand side of the page. In nanoc I could just float: right it, in Webby I felt I had to understand the grid that had been created for me. I’m not sure it really allowed you to use any CSS knowledge someone might already have but the benefit is you know your site is going to look good across browsers and in print and so on. For more professional site development this might actually sway you to select Webby.

Jekyll is primarily a tool for publishing a blog style site so it has a lot of built-in support for special information relating to posts but in terms of layouts it uses general HTML templating like Nanoc does. It allows  for the creation of little chunks of reusable content across pages via includes but lacks some of the flexibility of Nanoc’s tag helpers. While focussing on blog publishing Jekyll actually encompasses a powerful set of processing tools for content that effectively allows you to use it as CMS. Jekyll is used to process GitPages on GitHub, a real demonstration of its power beyond just blogging.

For what I wanted to do I initially felt that Nanoc was the best choice, it’s flexible, powerful and generates nice sites. Webby is powerful but didn’t expose that power to me in a way that made me feel in-control. However when I started to get to grips with Jekyll I was really amazed at how powerful the use of a few conventions were. Jekyll processes your site’s pages effectively in-place. It can choose to process your files if they match a naming convention (such as having a content extension like textile or markdown) and it uses an embedded YAML section in the file to do things like provide the title of the page or select the layout that the content should be displayed within. While Nanoc produces fancy URLs Jekyll made it easy to build more conventional websites consisting of clusters of pages.

Jekyll uses a small number of special directories (indicated by a leading underscore) to hold things like layouts, cross-page snippets and posts (if you are using the dated posting functionality). However for a simple site the number of directories and configuration files was significantly less than Nanoc. In fact you could get by with just a master page layout, which would require just one special directory and page within it. Everything that should have been simple was with Jekyll and the result was very close to a conventional website you might have coded by hand but extremely fast to create. It also uses just one command to press the pages and fire up a local server to view the pages. Both Webby and Nanoc seemed to make pressing and previewing slightly harder than it needed to be and at least once I forget to execute the Nanoc commands in the right order.

So, conclusions… Going forward I will be using both Nanoc and Jekyll. Webby I would be willing to consider but I don’t really have a task for it that would reward the additional effort I have to invest in its capabilities. If I just want to bang out pages for a website but minimise my effort and make it easy to abstract common content then I will definitely be using Jekyll. It is a joy to use. For sites that are maybe more based around articles or grouped content then I think I will be using Nanoc. The default URL construction is nice for hierarchical data (as long as you can stand the many directories) and there’s a sense that the configuration files make it easy to make changes that affect the site as a whole. However it does use a lot of scaffolding to achieve the effect and there are a lot of exposed options that could have been reduced by adopting some conventions Jekyll style.

Standard
Groovy, Java, Programming, Scripting, Software

Working with Groovy Tests

For my new project Xapper I decided to try and write my tests in Groovy. Previously I had used Groovy scripts to generate data for Java tests but I was curious as to whether it would be easier to write the entire test in Groovy instead of Java.

Overall the experience was a qualified “yes”. When I was initially working with the tests it was possible to invoke them within Eclipse via the GUnit Runner. Trying again with the more recent 1.5.7 plugin, the runner now seems to be the JUnit4 one and it says that it sees no tests, to paraphrase a famous admiral. Without being able to use the runner I ended up running the entire suite via Gant, which was less than ideal, because there is a certain amount of spin-up time compared to using something like RSpec’s spec runner.

I would really like all the major IDEs to get smarter about mixing different languages in the same project. At the moment I think Eclipse is the closest to getting this to work. NetBeans and Intellij have good stories around this but it seems to me to be a real pain to get it working in practice. I want to be able to use Groovy and Java in the same project without having Groovy classes be one of the “final products”. NetBeans use of pre-canned Ant files to build projects is a real pain here.

Despite the pain of running them though I think writing the tests in Groovy is a fantastic idea. It really brought it home to me, how much ceremony there is in conventional Java unit test writing. I felt like my tests improved when I could forget about the type of a result and just assert things about the result.

Since I tend to do TDD it was great to have the test run without having to satisfy the compiler’s demand that methods and classes be there. Instead I was able to work in a Ruby style of backfilling code to satisfy the runtime errors. Now some may regard this a ridiculous technique but it really does allow you to provide a minimum of code to meet the requirement and it does give you the sense that you are making progress as one error after another is squashed.

So why use Groovy rather than JRuby and RSpec (the world’s most enjoyable specification framework)? Well the answer lies in the fact that Groovy is really meant to work with Java. Ruby is a great language and JRuby is a great implementation but Groovy does a better job of dealing with things like annotations and making the most of your existing test libraries.

You also don’t have the same issue of context-switching between different languages. Both Groovy and Scala are similar enough to Java that you can work with them and Java without losing your flow. In Ruby, even simple things like puts versus println can throw you off. Groovy was created to do exactly this kind of job.

If the IDE integration can be sorted out then I don’t see any reason why we should write tests in Java anymore.

Standard