Programming, Python

How does the patch decorator in Mock work?

I tend to use Mock more as a stubbing library rather than for mocking. The patch decorator is pretty handy in terms of this as it takes care of all the resetting once your stubbed test has run making it easy to have a test where a dependency returns an empty list, followed by a single-entry list and so on.

However I often forget how exactly it works so I’ve decided to write up my latest remembering of how to do this (via John Hartley’s help and reminders) so I have something to look up next time I forget.

The first thing is that the patch decorator takes a string that represents the fully qualified name of the stub/mock you want to create. In a Django app for example that means you should include the app name at the root. The name also reflects the local name of an imported item. Something I commonly do wrong is to bind to the absolute name, say ‘random.choice’ rather than ‘myapp.mymodule.random.choice’. If you are in the situation where your stub is correct when you call it directly but never happens when you run the code under test I am pretty sure that naming will be at the root of your problems 95% of the time.

For each string argument you have in patch you also need to define a parameter to the test function, this will contain the actual Mock object and is what you use to actually stub the value to what you want it to be for the test. Use names that make sense here, stub_db, fake_file_reader not just mock1, mock2 and so on.

With these relatively few reminders in place you should now be in a position to stub simply with Mock!

Standard
Python, Web Applications

Deploying Python apps to Epio

I recently got my beta access to ep.io, the Python application deployment platform. I had the chance today to have a play around and try out some deployments so I thought I would try and give my view on the experience before. I’ve deployed Python apps to Heroku and Gondor before so those services form my reference points here.

So firstly, there’s a command-line client that you install via pip and you effectively deploy to the platform via a client-command, SSH keys and what looks like git on the server-side. This is more like Gondor than Heroku (which is intimately linked to git). It means you have your choice of source control and if you want to be a Python purist you never need to step outside of Python for everything you are doing.

Applications consist of essentially one configuration file that states where the WSGI application is and what the requirements file is. Compared to Gondor it is a very simple setup but it did feel that it could be even simpler if it made convention-based assumptions such as the requirements file being called requirements.txt, for example.

Leveraging WSGI and configuration this way gives a very flexible platform and I was able to get both Flask and Bottle to work (the former very quickly because it has documentation, the latter via trial and error that might require its own blog-post). I didn’t have time to try Django but I felt pretty confident that I could get whatever framework I wanted working once I understood the basic setup.

Unlike Heroku, Epio provides a fixed framework for executing the apps. It seems you will be running behind NGINX and Gunicorn. Both are good choices and I certainly like them but if you want to play around with different servers like Tornado or CherryPy you may prefer Heroku’s more open deployment model. I did like the way that you can use the configuration file to have NGINX serve static content directly.

Epio naturally has less of an ecosystem than Heroku but has Solr, Postgres and Redis out of the box. All solid choices and covering off the majority of what I would need. I was certainly grateful that I didn’t have to grapple with remote database administration and could prototype apps with just Redis.

Deployment and logging have kind of rough edges. Being able to access logs directly from the application page was a win for me, however when I was struggling to define the WSGI entrypoint correctly it seemed as if the application wasn’t being really compiled until the first request comes in. I would see an entry confirming a new deployment but then nothing until I hit the app. I think there should be some kind of sanity check of what you have uploaded to see whether it will even run.

Right now epio is providing a Python-based cloud deployment platform with a sensible set of supplementary services and low opinion about the source control system to you use. It feels like if this had been around at the start of the year it would have blown me away. However now there is more competition and therefore questions of price and ease of use will matter in terms of  how compelling it is to use the service.

If you do Python web development I would definitely recommend you sign up for beta and give it a go yourself as it seems a very solid prototyping platform. If you are not a Ruby and Git fan then you may well love what is on offer here because it is already very convenient, makes few demands on you and gets your web app public in minutes.

Standard
Work

Google Apps and App Engine

If you use Google Apps to provide you with email then you should also really be thinking about enabling and using Google App Engine as well. Internal applications are much easier to deliver to the business as a whole and having a ready-made platform makes it easier to try out ideas that previously would have been impractical.

The first advantage is that Google Apps that are bound into your domain allow you to create something that is easy to access for an existing user (no additional login is required) but also gives you peace of mind that you are exposing virtually zero surface area for attack.

The second is that for Python at least it is easy to access a very full featured environment with a minimum of code. Want to send emails, have task queues, access to memcache, serve static content? It is all a YAML configuration line or import away.

I love services like Heroku but a lot of internal apps have relatively light usage and benefit from the batteries included approach rather than combining various plugins. It makes it easy to switch between different approaches and react to different demands.

Standard
Programming, Python

Django and JSON stores: a match in heaven

My current project is using CouchDB as its store and Django to implement the web frontend. When you have a JSON store such as CouchDB then Python is a natural complement due to its brilliant parsing of JSON into native data structures and its powerful dictionary data type that makes it really easy to work with maps.

In a previous project using Python and Mongo we used Presentation objects to provide domain logic on top of the raw data maps but this time around I wanted to try and cut out a layer and just work with maps as much as possible (perhaps the influence of the Clojure programming I’ve been doing on the side).

However this still leaves two problems that need addressing. Firstly Django templates generally handle dictionaries like a dream allowing to address them with the standard dot syntax. However both Mongo and Couch use leading underscores to indicate “special” variables and this clashes with the Python convention of having a leading underscore indicate a private member of the class. The most immediate time you encounter this is when you want to use the id of a document in a url and the naive doc._id does not work.

The other problem is the issue of legitimate domain logic. In Wazoku we want to use people’s names if they have supplied them and fallback to their email if they haven’t supplied their name.

The answer to both of these problems (without resorting to an intermediary object) is Django’s filters. The necessary logic can be written in a few lines of Python that simply examines the dictionary and does the necessary transformation to derive the id or user’s name. This is much lighter than the corresponding Presentation solution.

Standard
Programming, Python, Ruby

Truly open classes

Here’s an interesting observation, I needed to write a little script to automate some number calculating for me. I was wondering whether to do it in Ruby or Python. I’m doing a lot of Python at the moment so I felt I ought to give Ruby a little go. Share some of the love.

However the solution I had in mind really didn’t work with Ruby because while Ruby has open classes it has a comparatively fixed idea of attributes. In Python you can set attributes very freely on any object so I have got in the habit of creating something and then enhancing by applying a function. Example? Okay.

def make_captain(actor):
actor.rank = "Captain"
return actor

class Person:
pass

captain = make_captain(Person())

So this little trick doesn’t work, or rather is much more difficult to do in Ruby as Ruby, at is dynamic heart, is a language that believes in object-orientation and that classes should encapsulate rather than being little collections of data. You can use instance_variable_get/set but it lacks the elegance of the Python syntax.

In Ruby it would be easier to define the attributes in the class using the existing metaprogramming constructs and then have a class method to generate the content (effectively encapsulating my script logic).

Now this isn’t a straight “Ruby sux, no Python sux more” post. Between Scala, Clojure and Python I have been doing a lot more in a functional style that depreciates objects as anything more than value carriers. The Ruby vision of a class would give me something with a stronger sense of purpose and encapsulation, something that is hard to benefit from in a script for a particular purpose.

What is going to be interesting this year is trying to identify when the value of a piece of code is in the structure of it’s data-definition (i.e. objects) versus its process (functions). Having had a think about it I should perhaps rewrite my script to use some OO modelling because it may answer similar requirements down the line. However from a strict Lean/Waste point of view I should have gone with the Python solution as Ruby was imposing a restriction on me while providing benefits that I was unlikely to realise.

Standard
Java, Programming

Python as a post-Java language

I’m a UNIX-based developer and since 2000 I have been working mainly with Java and then JVM languages. When Java 7 slipped I made no real secret of the fact that Java was in a lot of trouble. The post-Oracle world though looks even worse with a lack of clarity of what in the core ecosystem is free, open source and liability free.

Clojure and to a less extent Scala are great steps forward so I don’t feel the burning need for a Java 7/8 whatever. However a moribund or tainted JVM is a major problem and so I’m now thinking about what the post-Java escape route looks like. On the web front it is pretty obvious, Python and Ruby are great languages with great frameworks for developing web-based application. For the server-side heavy lifting it is a lot less clear, people are talking about Google Go but that does feel quite low-level, I’m not sure I’m ready to go back to pointer wrangling even with memory-management. It feels like something you’d build a tool out of not an application. Mono feels like more of the same problems of wrestling with big companies with vested interests, if you are going to do that then why not try and sort out the OpenJDK?

As the title of the post suggests the language I am most inclined towards right now is Python. It is a really concise but clear language that on UNIX systems comes with an amazingly comprehensive set of libraries and which has a virtual environment and dependency management that is on a par with RVM and gem.

The single issue that comes up is performance, what I have been finding that for 80% of the work I am doing performance is okay and I’m producing a fraction of the code I would normally have to create. For that last 20% maybe I am going to have to look at something like Go or (god forbid) go back to C but I would much prefer to see a Clojure or Scala that could run on top of something like LLVM. I also have some hope of smarter people than me making progress on a JIT for Python that might take 20% down to a figure where performance would matter so much to me I wouldn’t mind sweating to make it happen.

Standard
Programming

Redis 2, worth the hype

So I’ve flirted a little bit with Redis but never really had anything that fitted it’s solution profile, until this week!

I needed to cross reference postcodes and their corresponding longitude and latitudes. I tried a few other solutions but in the end I decided that a postcode was a nice normalisable key (it just needs the country details adding in) and that since I had thousands of records to relate, I really valued speed above everything else.

Redis (in conjunction with the Python client library) tore through the data set in terms of both inserting approximately 1.7M records and reading roughly 300K entries. I also liked using the hash functionality to store the long and lat under the same key rather than having to write my own logic to create the pairing.

Redis really helped solved my problem and lived up to its promises. It definitely has a place in my toolkit from now on.

Standard
Python, Web Applications

Juno, a micro web framework for Python

I love microframeworks and I love Python to it therefore follows that I love the minimal web frameworks you can get for Python like CherryPy. However as soon as I heard about Juno, a clone of my favourite microframework Sinatra I had to give it a go.

For me the archetypal application for Sinatra is Git-Wiki and therefore to kick the tires on Juno I decided to try and do the same thing with Bazaar. Hence:  Bzr-Wiki on Juno!

You need Juno, Jinja2 and SQLAlchemy as well the source from Launchpad. Once you have it all then got to the repo directory and type bzr init. You should then be good to fire up wiki.py and muck around with it.

It is slightly ugly and lacks access to revision history, diffs and user tracking but apart from that it is a surprisingly functional wiki. It is also bi-directional in that you can add files to the Bazaar repo and they get reflected in the app.

So what was Juno like to work with? Well overall I thought this was the best Python microframework I have used so far. I really like the idea of decorating methods to avoid having to generate a mapping table. The syntax is terse and comprehensible, the conventions around the framework made sense to me.

By comparison with its progenitor I think I really missed the autoloading/dynamic evaluation that allows you to change code in Sinatra and have it immediately take effect. The function of the Request decorator was initially quite obscure (it binds all HTTP verbs to that method, if you want to map GET to another method you must specify all verbs independently) and I am still not sure it is right. I think the most specific decorator should take precedence. Other than that I think the framework ports a lot of the concepts from Sinatra in a sympathetic way.

The dependency on SQL Alchemy is also really clunky. If you specify that you are not using a database (as is the case here) then it is annoying to have to download a dependency and makes installation on Windows a pain I didn’t even want to try and tackle.

Juno is really promising though and I look forward to it developing. I think it would be a real delight to use it in an environment like Google App Engine.

Standard
Java, Programming, Work

The Java Developer’s Dilemmia

I believe that Java developers are under a tremendous amount of pressure at the moment. However you may not feel it if you believe that Java is going to be around for a long time and you are happy to be the one maintaining the legacy apps in their twilight. Elliotte Rusty Harold has it right in the comments when someone says that there are a lot of Java jobs still being posted. If you enjoy feasting off the corpse then feel free to ignore the rest of this post because it is going to say nothing to you.

Java is in a tricky situation due to competition on all fronts. C# has managed to rally a lot of support. Some people talk nonsense about C# being what Java will look like in the future. C# is what Java would like if you could break backwards compatibility and indeed even runtime and development compatibility in some cases (with Service Packs). C# is getting mind share by leapfrogging ahead technology-wise at the expense of its early adoptors. Microsoft also does a far better job of selling to IDE dependent developers and risk-adverse managers.

Ruby and Python have also eaten Java’s lunch in the web space. When I am working on web project for fun I work with things like Sinatra, Django and Google App Engine. That’s because they are actually fun to work with and highly productive. You focus on your problem a lot sooner than you do in Java.

The scripting languages have also done a far better job of providing solutions to the small constant problems you face in programming. Automating tasks, building and deployment, prototyping. All these things are far easier to do in your favourite scripting language than they are in Java which will have to wait for JDK7 for a decent Filesystem abstraction for example.

Where does this leave Java? Well in the Enterprise server-side niche, where I first started to use it. Even there though issues of concurrency and performance are making people look to things like Erlang and JVM alternatives like Scala and Clojure.

While, like COBOL and Fortran there will always be a market for Java skills and development. The truth is that for Java developers who want to create new applications that lead in their field; a choice about what to do next is fast approaching. For myself I find my Java projects starting to contain more and more Groovy and I am very frustrated about the lack of support for mixed Java/Groovy projects in IDEs (although I know SpringSource is putting a lot of funding into the Eclipse effort to solve the problem).

If a client asks for an application using the now treadworn combination of Spring MVC and Hibernate I think there needs to be a good answer as to why they don’t want to use Grails which I think would increase productivity a lot without sacrificing the good things about the Java stack. Companies doing heavy lifting in Java ought to be investigating languages like Scala, particularly if they are arguing for the inclusion of properties and closures in the Java language spec.

Oracle’s purchase of Sun makes this an opportune moment to assess where Java might be going and whether you are going to be on the ride with it. It is hard to predict what Oracle will do, except that they will act in their perceived economic interest. The painful thing is that whatever you decide to do there is no clear answer at the moment and no bandwagon seems to be gaining discernible momentum. It is a tough time to be a Java developer.

Standard
Programming, Python, Ruby

Mocking Random

Mocking calls to random number generators is a useful and important technique. Firstly it gives you a way into testing something that should operate randomly in production and because random number generation comes from built-in or system libraries normally it is also a measure of how well your mocking library actually works.

For Ruby I tend to use RSpec and its in-built mocking. Here the mocking is simple, of the form (depending on whether you are expecting or stubbing the interaction):

Receiver.should_receive(:rand)
Receiver.stub!(:rand)

However what is tricky is determining what the receiver should be. In Ruby random numbers are generated by Kernel so rand is Kernel.rand. This means that if the mocked call occurs in a class then the class is the receiver of the rand call. If the mocked call is in a module though the receiver is properly the Kernel module.

So in summary:

For a class: MyClass.should_receive(:rand)
For a module: Kernel.should_receive(:rand)

This is probably obvious if you a Ruby cognoscenti but is actually confusing compared to other languages.

In Python random functions are provided by a module, which is unambiguous but when using Mock I still had some difficultly as to how I set the expectation. Mock uses strings for the method called by the instance of the item for the mock anchor.  This is how I got my test for shuffling working in the end.

from mock import Mock, patch_object
import random

mock = Mock()
class MyTest(unittest.TestCase):
    @patch_object(random, 'shuffle', mock)
    def test_shuffling(self):
            thing_that_should_shuffle()
            self.assertTrue(mock.called, 'Shuffle not called')

You can see the same code as a technicolor Gist.

This does the job and the decorator is a nice way of setting out the expectation for the test but it was confusing as to whether I am patching or patch_object’ing and wouldn’t it be nice if that mock variable could be localised (maybe it is and I haven’t figured it out yet).

Next time I’m starting some Python code I am going to give mocktest a go. It promises to combine in-built mock verification which should simplify things. It would be nice to try and combine it with Hamcrest matchers to get good test failure messages and assertions too.

Standard