Programming, Ruby, Scripting, Web Applications

Sinatra and Haml

On Monday I was meant to be learning how to use the Grails framework (following up the impressive Grails site that Sky has launched) but I instead got distracted into scratching another itch, using Sinatra with JRuby. Since I was doing a little REST application I thought I would also give Haml a go as it promised a far easier way of generating HTML responses.

Installing both Gems was easy as ever and Sinatra was really easy to understand conceptually. Put the HTTP method name, the URI pattern you want to match and then the return value of the block is what goes back to the client in the response.

So for example to map “/hello” to the plain text response “Hello World” you simply have.

get "/hello" do
  "Hello World"
end

Fire up the script (with necessary requires) and a HTTP server is set up and running. It really couldn’t be simpler. You can use a :symbol in the URI matcher and then access it through the params hash. Within half an hour I was starting to add resources into my served HTML and I felt like master of the REST UNIVERSE.

Sinatra is a really smart piece of code that makes it simple to write a basic web application. I have loads of ideas of how it could be useful but one of my first thoughts was that it actually does a good job of solving the issue of GUI platforms.

My next project is to see how it works with posting data back to the app but on the face of it it all seems straight-forward.

Haml is another story. It is undoubtedly a good idea and on the right track. A DSL for creating HTML the method html for example creates a block html tag  while p “Hello World” creates a paragraph tag with the parameter as the content. Hash parameters sent to the method become attributes of the class.

It is much quicker than generating the HTML by hand but often not by much. That is because Haml has not found the same easy metaphor for code and content that Sinatra has. It is very picky, constantly harping about two-space indentation (no more, no less, don’t dare use tabs you bastardo!) for example, it sometimes throws a stack trace that failed to make it to Sinatra leading to a blank screen on the browser and hunt into the console for a clue as to what has gone wrong.

The documentation is to poor to figure out why the library doesn’t accept what you are doing and I am still baffled as to whether the module handles inline tags or whether you are meant to devolve all of that to Textile. Passing parameters to Haml templates seems unnecessarily complex (the only way I could get it to work is with normal string interpolation, it’s a solution but it seems to break the DSL concept) and mixing inline formatting with parameters baffles me still.

Haml makes more sense to me than RHTML style templating but its claim to simplicity and enjoyment seems to come only with a deep understanding and long experience with using it. It took me 20 minutes to fall in love with Sinatra but it took 20 minutes for me to get a working Haml page. And I didn’t enjoy myself doing it.

Standard
Programming, Ruby, Scripting

CouchDB

What is CouchDB?

CouchDB is a dedicated document based database that kind of puts it in the same space as Exist, Xindice and Oracle Berkeley XML Db. What makes it very different is that instead of building around XML it is built around Javascript. The document storage format is JSON and the query language is Javascript. Something like Exist uses a minimum of well-formed XML for data and XPath or XQuery for querying.

Now the merits and flaws of the various markup languages is really holy war and I don’t want to get into it too much. I think it is enough to say that CouchDB aims to be lightweight in implementation without being lightweight in features or performance. JSON is very popular in the web space and by focussing on making common cases easier it is less work to use than XML. The more complex the data or specification requirements though the less viable it is as a solution.

Using JSON means that stored data is extremely flexible in structure, unlike relational data you can have very gappy or bitty information and not have a problem. Take something like a customer relationship management system. This is often a poor fit to relational data because you tend to discover information in small tranches. Initially a lead might be a first name and a telephone number. As the relationship develops you add a surname, a company name, a position, an collection of issues or queries and so on. JSON is a really good way of capturing this evolving picture of things.

Using Javascript to query this data surprised me initially, I feel pretty comfortable with my limited Javascript skills and therefore didn’t have a problem with it as a choice. I’ll come back to this later in the post but actually the idea sounds strange but is a natural fit when you use it. Again it is about the right tool for the right job but in the context of the data you are using in CouchDB creating a query by combing iterators, if statements and the functional programming map function is a better fit than trying to adapt SQL.

The final component of CouchDb is the functional language Erlang which is used to service the HTTP REST interface that CouchDB uses to provide an interface to the database. Erlang provides very cheap, lightweight concurrency that provides a good fit for handling lots of HTTP requests.

The interesting thing is that CouchDB automatically provides the REST front end that you tend to build on top of Java object stores like JBoss Cache or Coherence (which I have also been looking at recently). With those products you tend to stay native until you need to share data with other languages or systems and then you tend to REST serve it out as XML with a lightweight HTTP server. If you can see that need ahead of time then CouchDB might well be compelling due to the built-in support.

Installing

After reading about CouchDB I wanted to take a look at this new document database in a bit more detail. I managed to install the base tarball from the Google Code page easily. I then cut and pasted the Ruby sample REST code from the CouchDb Wiki into irb and on running the server I found that I could store and retrieve documents really easily. The combination of interactive Ruby and the REST interface made it easy to play around with data objects.

However when trying to run the tests via the administration web page only four of the tests would pass. The only error message was just an Error Code of 136. Since I seemed able to store and pull documents (and could confirm that the datafile had been generated and had data) I put it down to alpha flakiness and pushed on.

It was only when I tried executing ad-hoc queries and kept having them fail that I realised what was linking my problems. Every time the main Erlang server handed off a query to the Javascript interpreter the process died. Erlang is really resilient to thread death so there was zero impact from this.

CouchDB uses the Mozilla Spidermonkey libraries to construct its own View Server program which is configured via the couch.ini file. Having double-checked that the interpreter was there, was executable and could read the main.js script it was passed I was baffled. I decided to pull down the code base from SVN and then rebuild it. Building from SVN is a bit more involved than the tarfile and its worth double-checking the documentation on the wiki before you get stuck in. Having built and installed myself a new copy I still had the same problem and was feeling pretty cheesed off. With no other source of help I headed to IRC and checked out the Freenode #couchdb channel. There Christopher Lenz gave me a good steer that he had had a problem with Erlang’s HIPE (high performance virtual machine) on Debian.

I had been using Erlang 5.5.5 which I think I used apt-get to obtain. Downloading the latest Erlang release (5.6.1, they release pretty frequently) I tried building that from source and hung when configure tried to check for floating-point exceptions. Running configure with disable-hipe allowed configure to complete and then Erlang was straight-forward to build and install.

Restarting the CouchDB server I found my queries were working and all but one test (“conflicts” fails on an error assertion and this is apparantly a known issue) passing, hurrah!

The machine I had been working on was a Feisty Fawn Ubuntu instance so I loaded up Gutsy Gibbon in VM Ware Player and tried building Erlang and CouchDb there. Erlang was fine, even with Hipe. For CouchDB I used the latest Erlang I had built and used apt-get to obtain the Spidermonkey and ICU depdencies. Everything worked out of the box with this combination.

This problem highlights a few of CouchDB’s weaknesses. Firstly it is all bleeding edge stuff, Erlang is bounding along and so is CouchDB. Secondly there is no test suite in the source distribution that can help troubleshoot issues. Finally the error logging is tricky to understand for a neonate. I’m certain that having an EUnit test suite would have directed me towards the fact that the Javascript was failing much quicker than using trial and error and IRC.

The final issue was where CouchDB stores its logs and datafiles by default. These default to /usr/local/var which is not really where I want to store data like this. /var is a more natural choice and FHS seems to agree. You can change this when building in the etc/couch.ini.tpl file of the source directory but it would be nicer to have a more natural choice by default.

I also tried to install CouchDB on OSX without using a packaging mechanism but instead referring to an instance of Spidermonkey 1.7 I had on my system. While I could get configure to accept the Javascript library it wouldn’t recognise the jsapi.h header, maybe because configure doesn’t define the right Macros when it tries to build the test file.

Working with CouchDB

My first experience of CouchDB’s web interface was that it was good-looking but that the error messages, unresponsive links, Firebug warnings and occasional Javascript pop-ups were all signs that it was a work in progress. On fixing my passthrough to Javascript I had a very different experience of a slick and good looking interface that uses amazingly responsive AJAX to provide a really good experience.

Learning how to generate queries for example is easy with the in-built query browser (I’ll save the actual syntax for a later post) as the feedback from the system is very quick. Similarly creating new databases, documents and fields is actually not torturous but slick and quick.

I’m certain this is because Erlang and AJAX form a natural partnership in creating and servicing small requests that need to be handled quickly. I may be wrong but this is certainly my most positive AJAX webservice experience to date.

The CouchDB wiki contains information on getting started with various languages and I choose Ruby. It was a simple cut and paste into irb and then it was straight-forward to interact with database from the shell and the web client.

One of CouchDB’s more unusual feature seems to stem from Erlang’s concurrency model. Data is assigned a revision and in addition to applying multiple changes in order you can also view previous revisions. That’s a pretty weird feature compared to most of the current datastores. You also need to refer to a revision if you want to update a record. If the revision of the target of the update has changed when you submit your changes your changes get refused. The revision mechanism is also the basis of the replication mechanism although there isn’t enough documentation to understand when the replication pairs check their revision ids.

Incremental data-construction is easy via the web-gui but you need to fiddle a little bit to get the revision number to target if doing it programatically. Presumably a library would make that easier going. For information that has little inherent structure or is very gappy or incremental then CouchDB is a great data store and is currently occupying a niche as far as I know.

Standard
Ruby, Scripting

Dumb Ruby Gems mistake

So on and off over the last week I have been trying to get RedCloth to work in both Ruby and JRuby. Despite getting the gem I kept getting a failure when I tried to require ‘redcloth’. When I finally got Textile support working in Java before I had managed to weave the two or three lines of Textile magic the tutorials promised I shrugged and took the task off my GTD list.

Except that today my stubborn side told me to take a look at the Ruby Gems documentation after seeing an answer to a similar problem on a mailing list. Sure enough there in Chapter 3 was the explanation that you need to require Ruby Gems before the actual Gem you want to use.

Heading to irb and sure enough:

require 'rubygems'
require 'redcloth'

Does the trick and I am finally appreciating the library. Now maybe I’m slow or perhaps the same code on the RedCloth page should have included a full script that would regardless of what was in your RUBYOPT environment variable.

Standard
Java, Ruby, Web Applications, Work

Working with Rails

I have recently struggled to try and get a prototype web application done with Ruby on Rails. It has been a really great experience and like all such experiences consists of both good and bad moments. Things that are particularly striking coming from a Java Web environment:

  • Integrated console
  • No recompile/build/deploy cycle
  • No mix and match components
  • No real programming language skill required.

It is also remarkable how quickly people revert to the level of thought, planning and execution that you might use for a shell script. Even I have found myself getting caught up in “playing” with the app interactively rather than being focussed on creating behaviour and functionality in a structured way.

Elaborating on these points; in Java web development the first issue is usually building up your component stack. Some people just choose Spring  but those people are idjits! If you are working purely in a web tier without any need for any backend interaction then it is probably a good bet but generally you want to put some thought into how you are going to assemble your various stacks. You tend to have to choose your MVC web framework, your persistence framework and your service framework (if appropriate). You often have to give some thought to your caching and messaging frameworks if they are relevant.

With Rails you just use the appropriate Active Module or Rails built-in features. If there is something that is going to ease some pain for you then it is going be operating as a plugin. That’s it, no framework holy war. It also means that there are a lot of applications that are just not going to be suitable for a Rails application as all those Java components have various strengths for different environments. I certainly wouldn’t want to tackle a legacy database with ActiveRecord for example. Not that it couldn’t be done but I wouldn’t want to do it.

The zero-turnaround is impressive after the build-deploy-check cycle. It was a wow factor last year when I saw the Phobos framework being demonstrated by Sun and its not lost any of its shine. By separating its deployment environments so ruthlessly Rails is able to deliver a really positive developer experience.

The interactive console takes a bit of getting used to in terms of faking browser requests but once you get used to doing so it is another tool that you wonder how you’ve managed without for so long. It’s much more intuitive to use than setting up a remote Java debugging instance (although admittedly you do similar tasks in both). It allows you to scratch those itchy “why?” questions.

And finally that lack of programming skill? A bit controversial perhaps? Well I feel that in recent weeks what I have been learning is how to manipulate Rails. The fact that it is implemented on Ruby may allow a lot of features to be implemented in the way they that they are but you are very rarely called upon to show Ruby madskillz. Instead the majority of the time you are simply plugging little customisations into the Rails framework. It is called Rails for a reason after all, when you are on them you are amazingly productive but if you can’t package your problem into a Rails solution then you are out of luck, you’re going to have to develop your own solution and that is going to be hard work. I have been trying to develop my Ruby skills (that is a whole other story) but the truth is you can bang out something that is acceptable with very little Ruby knowledge. You can go a lot of the way purely by mastering Ruby’s hash syntax.

So have I been converted? Well for all its problems I do have a hankering to get back to my great big Java applications with their holy wars and heavyweight processing. After all being on the Rails is fine for getting things done quickly but it can feel claustrophobic. I am also really glad that after getting distracted by the whole JSF controversy the challenge Rails presents to the status quo of web development means that a lot of the Java frameworks are starting to respond to the real problems faced in web development and ensuring that the easy stuff should be easy.

Standard
Ruby, Scripting

Operator Overloading? Nada!

So today I got a perfect example of why operator overloading is such a nightmare.

What do you think this piece of Ruby means?

a << @b

The short answer is that it means nada. It will depend on both a and b and even if you learn what one instance of the operator means it will serve as no guide to identical operators for different types. This one operator resulted in a bug every time it was used and I don’t think it represents any value over a more verbose method call.

Standard
Groovy, Java, Python, Ruby, Scripting

Groovy or JRuby?

Martin Fowler blogged about the question a couple of days ago and ever since I have pondering that maybe it is not really the right question to ask.

I currently toodle between Jython, JRuby and Groovy for various reasons and I am an expert in none. The interesting thing I have found is that it is hard to pick one and just focus on that. To some extent they overlap heavily: they are all cross-platfrom, they are all dynamic, they all integrate with Java API stack I’ve committed to memory.

The first thing to say is that I am interested in scripting languages for prototyping and admin style scripting. I have never used Rails and the Grails data-model means that you need a specific kind of project to work on. If you want to use a particular product and that is only on one platform then that kind of makes your decision for you.

Each language has its own strengths, from my point of view I would categorise them in the following way. Jython has Python’s readability and solid language design, JRuby has Sun’s support, excellent community contributed library code and is very dynamic, Groovy is mini-Java, so it’s easy to learn and most importantly it has a functioning interactive console.

The last point might seem a bit weird, what about jirb and jython‘s interactive mode? Well Martin makes a very important point in his post about the purpose of these ports. Both JRuby and Jython aim to stay faithful to their source languages and be able to run code from their parent C implementations while expanding the API by accessing the Java libraries. Groovy on the other hand stays close to Java syntax and is the only one of the three that allows you to cut and paste code from a regular Java application and then play around with it in an interactive session. That is a very powerful and compelling feature.

Almost all the Groovy I do either comes from wanting to leverage or understand a piece of Java code.

All my Jython work on the other hand is about wanting to automate administration or manual tasks in a clear and concise fashion. Python’s dynamic data structures help, but so does zxJDBC the Jython specific database library that mixes DBI with JDBC to create a highly portable but simple database connectivity solution with no boilerplate!

JRuby on the other hand is something that only really comes up because Alpha Geeks love it. The syntax is gnarled and there is a significant learning curve before someone from a Java background can get “The Ruby Way” of things. The new integration of JRuby into NetBeans though makes developing in the language a comparative snap and I would suspect that JRuby will be a valid choice of application development language alongside Java now. The choice will be driven by the problems you are trying to solve not because one language is inherently “better” than the other.

Standard
Python, Ruby, Scripting

Starting Ruby and Python

This weekend I had a chance to start some Ruby and Python programming. My favourite scripting language to date is Perl and in terms of Java scripting I do actually like Groovy (although it has rapidly fallen out of favour with the Java crowd). Out of the two I have done currently done more complex stuff with Ruby but one thing is pretty immediate and that is that Python is very much on the back foot when it come to OOP scripting.

That’s not necessarily a bad thing though as actually scripting tends to be procedural and my Python script seemed to do a similar amount of work without the involvement of OO code. If I wanted something done quickly I would probably go for Python in the same way I go for Perl now. However if I thought something had legs and needed to be supported and expanded I would definitely go with Ruby.

One thing that both languages seem to have issues with is their big libraries. Most of my time with both languages was spent looking up how to perform some action. I was certain that neither of the languages required me to roll my own but I felt the situation was a lot like C and C++. C, like Perl, has the advantage of being syntactically very compact. You have to learn very little to know the language completely. Of course that means it lacks a certain power that comes from building on the capabilities of the language.

Perl answers this problem with modules which are a good idea as you only have to learn about standard libraries in little module sized chunks that match the problem you’re working on.

Python is kind of similar, in the sense there isn’t a lot in the core, but different in that, like C++ (and Java), you need to know your standard library if you are really going to get stuff done effectively.

Ruby is an altogether different beast. To get anything done you have to grapple with what feels, from my Java-centric point of view, like a very arbitrary API that is attached to every object. If you know the capabilities of a type then you have access to a lot of power. If you don’t, then you are swearing, scratching your head and reaching for the API description in the Pickaxe book.

One key thing that I feel is holding me back is the lack of variable typing. Having dynamically typed variables is cool, unless of course I know that I only want one type of variable (say an Integer). Maybe the code reflection and auto-completion features of Java are influencing me here but I do think it is helpful to be able to specify a type for a variable and then have the programmer’s editor or IDE be able to determine the API available to you.

Oh and yes, I might be making this complaint because I have already been caught out by the lack of automatic conversion between strings and numbers.

Overall my opinion at the moment is that Python is like Perl only cleaner and more legible. I think it would be really good for protoyping an application with a big team because there is a lot more consistency in the syntax and the style.

However Ruby’s object-orientation does mean that it would be an obvious choice for trying out OO Java ideas. Particularly if something like JRuby helped bring the benefits of both languages together (as well as making the most of my heavy investment in memorising huge chunks of the Java API).

Standard