Category Archives: Scripting

Java, Programming, Python, Scripting, Web Applications, Work

JMeter or The Grinder, so which one is better, like?

Or for the benefit of Google: Apache JMeter versus The Grinder. Fight!

A while ago I got paid to put these two tools head to head and I think it’s been long enough that the people who put up the money will have got the benefit now.

I had used The Grinder in a previous job where it had done excellent service finding out what level of peak load our site could handle. It was convenient in that you script it in Jython because Jython was our chosen scripting language.

JMeter on the other hand was a whole new thing to me. It is a kind of graphical programming interface where you drop controls into a tree structure and the framework generates threads that run through the tree generating interactions with the target application via things like HttpClient.

The scripting versus component structure is a pretty major difference between the two and is probably one of the key things that is going to help you choose between two products that are both, to be honest, pretty mature, capable pieces of software.

If you don’t feel comfortable with Python or Jython then Grinder is likely to be out. JMeter is much more friendly to non-programming testers. However this decision isn’t as clear as you might think, JMeter actually includes Javascript functions, access to BeanShell and its own expression language. As we created more complex traffic simulations we found ourselves having extremely complex expressions that often mixed a scripting language with an expression evaluation. In some ways The Grinder is more honest in telling you up front that you are going to have to program some of this stuff yourself.

Another big differentiator is in reporting, if you need to generate reports then JMeter is a much better choice as it has components that collect and collate information and you can dump data to XML for XSLT transformation into pretty much any output format you want. It is also possible to run JMeter headless once you have the test scripts so it is possible to encorporate it into a continuous integration process. Debugging slightly flaky applications is much easier with JMeter because you can easily capture a lot of result information and then just delete the data gathering component once the test flow is working.

A lot of the functionality of both products overlaps: they can both be used a browser proxy and be used to capture scripts as a user “drives” around the site. These captures are likely to form the basis of your test plans unless you have a very clear URL scheme with little session state.

Both use thread pools to generate load and both tend to get blocked on your application before they max out their local machine (although both are capable of using 90% of memory and CPU). JMeter has a few nice options around randomly ramping up the thread activation (recreating a spike in usage more closely). The Grinder tends to give more bang for the buck as its runtime is very simple and minimal.

Both are also capable of using distributed agents and collating the data from these agents back to the coordinator.

In many ways there isn’t a “bad” choice to be made between them. So what might sway your choice one way or another? Well JMeter is an Apache project and that might make a difference for some people as you have all those good things like project governance and a good chance the project is going to continue to be developed and enhanced. JMeter was also the only project to have a test suite last time I checked out both code bases.

The Grinder does have one unbeatable quality in my opinion and that’s flexibility. When you look at the features listed on the websites you might think that JMeter does all this cool stuff with HTTP, SOAP and POP3 and the like and that The Grinder is comparatively light in the feature set.

The opposite is actually true, as The Grinder website points out the The Grinder can test anything with a Java API. There really is no limit to what you can make it do or how you can execute your test plan. In fact most of the things I’ve complained about with The Grinder are fixable from within the test scripts  (what I am really complaining about is that some things like capturing HTML output per run is something that should be available as part of the standard package). On the other hand I felt that creating a new JMeter component was actually quite tricky as you have to deal with both the GUI aspects and the actual functionality you are trying to create in your test component.

If you really want to have total control of your concurrent volume testing then The Grinder is probably the best product for you.

Perhaps the last topic for consideration is The Grinder’s use of Jython. Python is a great language and you get a lot of power for some very concise code. Just as some people are going to find it off-putting others are actually going to find it very compelling.

Okay so yet another weak “horses for courses” post on the InterWeb. Perhaps the easiest summary to end on is that your test teams will probably take a shine to JMeter while your developer teams who are also building scale tests are probably going to like The Grinder; unless they dislike Python or learning a new language.

Standard
Programming, Python, Ruby

Mocking Random

Mocking calls to random number generators is a useful and important technique. Firstly it gives you a way into testing something that should operate randomly in production and because random number generation comes from built-in or system libraries normally it is also a measure of how well your mocking library actually works.

For Ruby I tend to use RSpec and its in-built mocking. Here the mocking is simple, of the form (depending on whether you are expecting or stubbing the interaction):

Receiver.should_receive(:rand)
Receiver.stub!(:rand)

However what is tricky is determining what the receiver should be. In Ruby random numbers are generated by Kernel so rand is Kernel.rand. This means that if the mocked call occurs in a class then the class is the receiver of the rand call. If the mocked call is in a module though the receiver is properly the Kernel module.

So in summary:

For a class: MyClass.should_receive(:rand)
For a module: Kernel.should_receive(:rand)

This is probably obvious if you a Ruby cognoscenti but is actually confusing compared to other languages.

In Python random functions are provided by a module, which is unambiguous but when using Mock I still had some difficultly as to how I set the expectation. Mock uses strings for the method called by the instance of the item for the mock anchor.  This is how I got my test for shuffling working in the end.

from mock import Mock, patch_object
import random

mock = Mock()
class MyTest(unittest.TestCase):
    @patch_object(random, 'shuffle', mock)
    def test_shuffling(self):
            thing_that_should_shuffle()
            self.assertTrue(mock.called, 'Shuffle not called')

You can see the same code as a technicolor Gist.

This does the job and the decorator is a nice way of setting out the expectation for the test but it was confusing as to whether I am patching or patch_object’ing and wouldn’t it be nice if that mock variable could be localised (maybe it is and I haven’t figured it out yet).

Next time I’m starting some Python code I am going to give mocktest a go. It promises to combine in-built mock verification which should simplify things. It would be nice to try and combine it with Hamcrest matchers to get good test failure messages and assertions too.

Standard
Python, Web Applications

Django in 24 hours

Last Saturday afternoon I decided to learn Django. It was 2pm on the first day of SiCamp 2008 in London and being the only developer in the room at that point I decided that I should do whatever I felt would be the best option to get an application running by 2pm the next day.

Previously I have done some Google App Engine and the experience convinced me to give Django a go after I found myself, by intuition, creating a GAE project structure of handlers.py (views), models.py and a directory called templates that contained templates. I was then disappointed to find the whole world had got there before me.

So, Django in 24 hours, baptism of fire. What do I think now looking back on the experience?

Everyone has told me that the Django documentation is good and I think I have to concur. Not everything is so clear that when you’re speed reading in one window and typing in another it works first time but importantly nothing in the documentation is actually wrong. When stuff is not working a second, careful look at the documentation got me back on track.

Importantly Django’s core model of web development is sound and intuitive. My editor had around ten files open for the project and the flow of adding something to the application did naturally flow from url to handler to view to model. Maybe the only quibble I have is that the views.py file is deceptively named in MVC terms.

The core of the framework is amazingly concise, I spent the majority of my time thinking about the problem not about the framework API. Binding a URL to function made sense, having to specify a template instead of having one inferred from the method name was maybe my one criticism in the method handler but on the plus side it does allow for flexibility in handling requests. Handing off from one request handler to another was very easy.

Django templates are both amazing and annoying. The syntax and principles are amazing, it was easy to play around with the pages and the template inheritance was really powerful for avoiding duplication. However when I transferred the application from self-serving to mod_python the template generation was very wobbly when compiling changes from the file system. Of course this could also have been mod_python but it was the latest 3 series stable source compiled for the machine. I’ve used Jinja2 previously when generating HTML in Python and might be tempted to stick with it in future.

Django models are great, I hate ORM but I really liked syntax for defining persistence properties and I liked the way that you don’t have the fact that you are really dealing with SQL hidden away from you. It genuinely seemed a more convenient way of expressing the data model rather than an OO wallpaper over relational data storage. I didn’t feel the need to add domain logic to the models but I felt like it wasn’t really polluting the model to do that either.

One thing that didn’t work at all was changing the relations between models; it took me two or three attempts to finally model the relationships between the data concepts. Each time I changed a Foreign Key or Many to Many relationship I ended up deleting the database (SQLite3) as I couldn’t figure how to migrate from the old schema to the new.

One reason for choosing Django was the idea that I wouldn’t have to write the backend code as the admin stuff would be right there for me. It took me a while to get the 1.0 admin to fire up but once it was running it did perform as advertised. One of the attractive things about the application was that the data model followed the conceptual language of the solution in a really powerful way. You could use the admin interface to have a Devotee perform Devotion to an Action. My geek excitement peaked anyway, YMMV.

So them’s the highlights of the experience. Overall Django delivered me a rapid web development process in an intuitive, powerful way and lived up to nearly all of the claims made on the tin. Deploying to Apache/mod_python was painful but most of the pain surrounded the infrastructure of my box (multiple versions of Python, Apache config files) and my lack of mad Apache admin skillz.

I would happily tackle another project in it again.

Perhaps of interest is how the Django development experience matches up against Rails or GAE  which would have been the other obvious choices. GAE would have been very similar but the deployment would have been better and I wouldn’t have had any automatic admin. In retrospect it may have been a better choice for a hack party type event. It certainly would have been my choice for personal projects for easy of deployment but now I have one Django app running perhaps that isn’t as relevant any more. Certainly the thing that has kept me from GAE before, the pain of data migration, doesn’t seem that better in Django (except that you control the datastore and its contents).

Compared to Rails?

  • Admin is much more awesome than scaffolding.
  • Django ORM is much less complex than Active Record, all the data required to create, deploy and use the object is in one place. Django doesn’t have Migrations but has its own brand of database versioning pain.
  • RSpec is awesome (despite its monkey patching of Object) so you aren’t going to beat Rails for easy testing.
  • Django templates are more powerful and easier to use than Erb but you have a lot of Ruby templating options so it’s hard to make a complete comparision. They probably both similar in the sense that you can find a template library that suits your preferences. Django is the purely solution out of the box.
  • Routing and Controllers are much less involved in Django than Rails.
  • Django is less opinionated about how you structure your application directories, which I like.
  • Django doesn’t bake-in AJAX components but is “batteries included”, Rails probably generates better Web2.0 style apps for less effort.
  • Finally Django uses only a few code generators because its basic structure is far less involved. It also generates far less “stuff” for each MVC element which I quite like as I don’t tend to use everything Rails generates.

Okay detailed analysis over, what’s the high-level view? Django and Rails are similar experiences but I think the major differences between them are almost what you could say about Python and Ruby. In Django you are going to get simplicity, clarity and a real choice of how you plug your infrastructure components together. In Rails you are going to get magic up front which is cool but also magic at the back end, which is not cool. Ultimately I think the answer is how opinionated you like your software. Well punk? How opinionated do you like it?

Standard
Ruby

Ruby Manor

So first things first. Ruby Manor was a huge success and a real credit to James and Murray who created and ran it. Thank you guys very much for doing it.

The talks really varied a lot, and ironically I went away most interested in Rabbit MQ and Neo4J, both of which were only tangentially related to Ruby. George Palmer’s talk on Nanite was probably the most interesting of the day but during it I picked up a sense that the audience was mostly focussed on web development within a Rails space.

This reached some kind of nadir in Alex Maccaw’s frankly unreal implementation of recommendation analysis as a Rails plugin which really did look like someone hammering a square peg through a round hole. It did however generate some interesting post conference blogging.

I also enjoyed the Shoes and Monkeybars talk although they did end up confirming my reservations about both libraries. Monkeybars really seems quite a complex setup and kind of lays an MVC framework on top of Swing which already has an MVC like framework. However it does use Matisse which is still an amazing constraints based GUI builder. Shoes is awesome, until you get stuck – then there isn’t enough feedback to understand why your awesome has disappeared. Seriously fun to play with though.

So it has been proven you can have a cheap, community-led conference in London for Ruby. Is it now time for someone to step forward and organise the Python Snake Pit?

Standard
Groovy, Programming, Software

Using Gant to build Java Projects

I know I said I would be taking a step by step introduction to Gant in my last post on the subject but sometimes the devil drives and you need things done in a less systematic way.

Recently I have been building Java and Scala projects with Gant. I think it has been a successful exercise so I am just going to jump on and show you some example buildfiles. The first one is going to be a Java project. This project is obviously toy code but I think if you just download the sample project and start filling in your own code (it’s an Eclipse project, Intellij and NetBeans should both import it successfully) you will be happy with how little you have to change the build file.

First have a look at the Gant file itself and then I am going to talk about the things that I think make Gant such a powerful and productive tool. The first thing to point out is the line count, this represents a complete build file for a Java SE project in under 100 lines. That includes the Hello World target I’ve left in as an echo test. Gant might have a slightly weird syntax if you are unfamiliar with Groovy but it isn’t verbose.

Targets and dependencies were in the last post so this time the new thing is using AntBuilder. Any method call that starts with Ant is a invocation of an Ant task. These are nothing but thin wrappers around the normal Ant task (and in fact I usually write them using the Ant Task documentation). Things that are attributes in Ant XML become hash properties in the parameter list. Things that would normally be nested elements are calls to the enclosing builder.

One area where Gant wins big is the way you can mix normal variables, Groovy string interpolation and Ant properties. Declaring the directory paths as Groovy variables near the head of the file allows me to create new paths via interpolation and assign the variable to the properties of an Ant task. Ant XML has properties and macro interpolation but this is both clearer and easier.

I am also using the Gant built-in clean task and in the course of using it, Groovy’s operator overloading for lists. I’m not a huge fan of operator overloading but if it’s clear enough here then it is great to have a DRY list assignment.

I also like the way that the directories are created in the init task. This kind of closure looping over a list again shows some of the power and conciseness that can be achieved by a language rather than a configuration file. If you don’t read Groovy then the each method iterates over the list its attached to and each item resulting from the iteration is stored in the whimsical “it” local variable.

Standard
Programming, Ruby, Scripting, Web Applications

Sinatra example

Since the Sinatra Project website currently seems to have been hijacked and directed to spam (the RubyForge page seems to be fine and is still a good introduction in its own right) I wanted to post an example of how to get going with Sinatra and also to highlight what makes it such a different approach to web frameworks. Here’s the example code on CodeDumper.

I wrote this with JRuby (for the cross-platform win). You’ll need to install the Sinatra, Haml and Mongrel Gems to get it to run.

It uses two styles, first there is the REST-ful extraction of parameters from the url and then there is the form POST submission. In both cases the code is pretty much the same as Sinatra extracts all parameters into the params hash.

I’ve inlined the Haml to make the example simple but normally the view templates would be extracted out of the code into separate files.

Sinatra is sometimes described as being as a web DSL rather than a web framework, it seems apt as it eschews MVC separation but instead by attaching code to routes directly it allows Controller code to be tiny and to delegate appropriately rather than putting in a heavyweight structure that might result in more frameworking code than actual “doing” code.

However one of the things I really like is something you can’t show in a code fragment and that is the ability to dynamically interact with your web application on the fly. Start up Sinatra, change the code of the file you’re running and your changes are reflected immediately. End the compile, deploy, view cycle! It kind of makes web programming fun again.

Standard
Ruby, Web Applications

Haml: Round Deux

So, with a comment on an earlier post about Haml, saying that it had had a major update I decided to give it another go. This time in conjunction with the web framework Ramaze.

While Haml has definitely improved but there was a definite sensation of returning to an abusive relationship. Haml is very cool but it can also be a colossol bitch to use. Mostly because of its whitespace lovin’ ways. Care to punt as to what the root cause of this issue was:

Illegal nesting: content can’t be both given on the same line as %form and nested within it.

I’d put a space character between my attribute curly braces and the form tag declaration:

%form {:action => 'url'}

when I should have put:

%form{:action => 'url'}

Do I really need this level grief just to avoid writing closing tags?

Haml does, as its advocates say, bring a clean and clear syntax to the table but it also brings a bundle of its own issues along the way. Particularly time consuming is having to adjust indents when placing content at different layers. The strict two spaces rule means you have to adjust blocks and even if you set up your browser to do soft tabs at two spaces (just for one syntax) it is still easier to play around with markup than Haml.

One feature I did like a lot since last time is that variable interpolation is a lot easier now:

%p== My variable goes #{@here}

Inline tags are still hopeless but that’s true of mostly of the lightweight templating systems.

Standard
Ruby

Still waiting for a Bulk Gems update…

I need to post about this again because life was just getting impossible on my cheapo virtual box. Well, I say cheap but it probably still has 75Mb of free memory that was just being hosed by RubyGems.

I tried updating to Ruby 1.8.7 but that had no effect because the culprit is really Gem itself.

The final solution (after naturally getting off 1.8.6, do it today…) turned out to be building and installing Gem from the 2.0 source tarball. Since getting away from 1.1.1 my Gem experience has been a lot smoother. Ramaze and the monsterous Merb installed smartly and without fuss. Don’t suffer for longer than you have to!

Standard
Groovy, Java, Programming, Scripting, Web Applications

Towards Groovier Projects

My latest project has witnessed an influx of Groovy. The project buildfile is run by Gant, there are Groovlets providing lightweight pages but it is in the test folders that the Groovy has made its most insidious advances.

Markup builder makes fragments of HTML to test, Groovy’s built-in Sql is beginning to setup and check the results of data operations at a functional level. Soon I plan to implement a Groovy Struts 2 result and then it will begin to replace Freemarker.

Standard
Programming, Python

Setting up a Mercurial work environment

So today I had a frustrating experience trying to get a Mercurial environment going for the current project I am working on. I am convinced for the kind of work we are doing the distributed branch model is going to be the right solution and I had been using a local Mercurial instance during my tech spiking earlier on the product.

Having built Python and Mercurial on the server and created a Mercurial user with the SSH keys of the individual developers I thought it would be easy sailing this morning. Not a bit of it and all of the problems stemmed from the lack of by default support for SSH in Windows.

Like most people I use the Putty Tool Suite for my SSH needs on windows. However simply aliasing putty to ssh doesn’t work. There are some key command switches that are different (including the trivial but annoying -P instead of -p for port).

Coming from a Java Subversion background I am used to having a portable library to take care of my ssh protocol needs. I also use OSX for my home development and that obviously provides a UNIX ssh command-line implementation, hence hassle-free SSH based source control.

The command line tool turned out to be the root of all my problems, both Eclipse and NetBeans Mercurial plugins don’t provide an implementation of the Mercurial client, instead they delegate it to the Mercurial client on the host OS, that in turns delegates ssh protocol invocations to the command line SSH.

The solution is simple once you know it, in the Mercurial.ini file you can alias ssh to Putty’s plink executable. There is in fact an example in the Mercurial book. Better still you don’t have to specify the key used if you are using Pageant.

However getting the first station to work was incredibly hard work. I even downloaded Cygwin just to get a more normal Unix ssh but by that point I had put a typo into the ssh alias and was getting a very weird error whenever I invoked Mercurial and I’d lost the forest in the trees.

At one stage I even gave Bazaar a go, hoping that SFTP might cure all my ills. However Bazaar uses Paramiko for it’s SSH support and on Windows that was failing due to its inability to find an OS source of entropy.

Server-side Python also let me down a bit as I was having an issue with the zlib module and after similar experiences with Ruby I knew that this would be because I must have compiled Python before zlib. Despite cleaning and re-configuring Python it still didn’t build the zlib module and in then end I had to go and run configure within the zlib module manually prior to a full Python build. This is exactly the same issue I have had with Ruby, what is the problem with this library?

Once you’ve done it once Mercurial repositories are actually easier to setup and manage than SVN (and that itself is pretty easy once you’ve done it a few times) and if you are working in a UNIX environment then they are extremely compelling.

However on Windows you are currently going to either do a lot of preparatory reading or be ready to set aside six hours to prevent frustration. In my case it probably didn’t help that I am also the one selling the utility of Mercurial. The pioneers always have the arrows in their backs; on the other hand they also get to better places long before anyone else.

Standard