Category Archives: Work

Software, Work

Agile must be destroyed

Let’s stop talking about the post-Agile world and start creating it.

Agile must be destroyed not because it has failed, not because we want to return to Waterfall or to the unstructured, failing, flailing projects of the past but because it’s maxims, principles and fixed ideas have started to strangle new ideas. All we need to take with us are the four principles of the Agile Manifesto. Everything else are just techniques that were better than what preceded them but are not the final answer to our problems.

There are new ideas that help us to think about our problems: a philosophy of flowing and pulling (but not the training courses and books that purport to solve our problems), the principles of software craftmanship, “constellation” architecture, principles of Value that put customers first not our clients. There are many more that I don’t know yet.

We need to adopt them, experiment with them, change and adapt them. And when they too ossify into unchallengeable truisms we will need to destroy them too.

Standard
Python, Web Applications, Work

Declare for simplicity

I was struggling with an issue today on a template that was blowing up due to missing keys in a template data hash. At first I tried to write some conditional code in the template. That ended up being quite ugly so then I was trying to find a way to condition the template on whether the key was present.

Then it struck me that the issue was really the missing key. As the hash is prepared in Python code the original implementation frugally avoided creating the entry if the underlying data wasn’t present. While this is clever and minimal it actually just pushes the complexity out of the Presenter logic and into the template where it is much worse because you can only interact with the Presentation’s abstraction of the original data.

The problem here is that dynamic languages sometimes allow you to be too clever. In a declared structure system you have to declare all your data and provide sensible defaults. This makes writing the templating solution at lot easier as you never have a missing data problem to deal with (you still have headaches with duff defaults but that is a different post).

So I went back to the presenter and declared an empty list under the key that had been causing me grief. Bingo! My failure went away and the default behaviour of not generating any content for the empty list kicked off, reducing my template back to the unconditional single-line I had originally.

Standard
Programming, Work

Code Coverage 90% meaningless

I have always been quite sceptical about the value of code coverage metrics. Most of the teams I have seen who have been productive and produced code with few defects in production have not used code coverage as a metric. Where code coverage tends to be obsessively tracked it seems to be more as a management reporting metric (often linked to “Quality”) and rarely seems to correlate with lower defects or malleable software instead it often appears in low-collaboration or low trust environments.

Code coverage has most benefit in an immature unit testing environment or in a “test-after” team. With test-after you have to have code coverage to ensure that people remembered to test all the possible execution paths. My personal preference is to push TDD as a practice in preference to code coverage because a side-effect is that you get 100% code coverage.

Code coverage is also quite a different beast to static or complexity analysis of code bases. Static analysis is a useful tool and some complexity measures actually make good indicators of the “quality” of the code base. It is also not the same as instrumented code, which is invaluable with dealing with code you’ve inherited or to discover how much of the codebase actually gets used in production.

Standard
Java, Programming, Work

The Java Developer’s Dilemmia

I believe that Java developers are under a tremendous amount of pressure at the moment. However you may not feel it if you believe that Java is going to be around for a long time and you are happy to be the one maintaining the legacy apps in their twilight. Elliotte Rusty Harold has it right in the comments when someone says that there are a lot of Java jobs still being posted. If you enjoy feasting off the corpse then feel free to ignore the rest of this post because it is going to say nothing to you.

Java is in a tricky situation due to competition on all fronts. C# has managed to rally a lot of support. Some people talk nonsense about C# being what Java will look like in the future. C# is what Java would like if you could break backwards compatibility and indeed even runtime and development compatibility in some cases (with Service Packs). C# is getting mind share by leapfrogging ahead technology-wise at the expense of its early adoptors. Microsoft also does a far better job of selling to IDE dependent developers and risk-adverse managers.

Ruby and Python have also eaten Java’s lunch in the web space. When I am working on web project for fun I work with things like Sinatra, Django and Google App Engine. That’s because they are actually fun to work with and highly productive. You focus on your problem a lot sooner than you do in Java.

The scripting languages have also done a far better job of providing solutions to the small constant problems you face in programming. Automating tasks, building and deployment, prototyping. All these things are far easier to do in your favourite scripting language than they are in Java which will have to wait for JDK7 for a decent Filesystem abstraction for example.

Where does this leave Java? Well in the Enterprise server-side niche, where I first started to use it. Even there though issues of concurrency and performance are making people look to things like Erlang and JVM alternatives like Scala and Clojure.

While, like COBOL and Fortran there will always be a market for Java skills and development. The truth is that for Java developers who want to create new applications that lead in their field; a choice about what to do next is fast approaching. For myself I find my Java projects starting to contain more and more Groovy and I am very frustrated about the lack of support for mixed Java/Groovy projects in IDEs (although I know SpringSource is putting a lot of funding into the Eclipse effort to solve the problem).

If a client asks for an application using the now treadworn combination of Spring MVC and Hibernate I think there needs to be a good answer as to why they don’t want to use Grails which I think would increase productivity a lot without sacrificing the good things about the Java stack. Companies doing heavy lifting in Java ought to be investigating languages like Scala, particularly if they are arguing for the inclusion of properties and closures in the Java language spec.

Oracle’s purchase of Sun makes this an opportune moment to assess where Java might be going and whether you are going to be on the ride with it. It is hard to predict what Oracle will do, except that they will act in their perceived economic interest. The painful thing is that whatever you decide to do there is no clear answer at the moment and no bandwagon seems to be gaining discernible momentum. It is a tough time to be a Java developer.

Standard
Java, Programming, Python, Scripting, Web Applications, Work

JMeter or The Grinder, so which one is better, like?

Or for the benefit of Google: Apache JMeter versus The Grinder. Fight!

A while ago I got paid to put these two tools head to head and I think it’s been long enough that the people who put up the money will have got the benefit now.

I had used The Grinder in a previous job where it had done excellent service finding out what level of peak load our site could handle. It was convenient in that you script it in Jython because Jython was our chosen scripting language.

JMeter on the other hand was a whole new thing to me. It is a kind of graphical programming interface where you drop controls into a tree structure and the framework generates threads that run through the tree generating interactions with the target application via things like HttpClient.

The scripting versus component structure is a pretty major difference between the two and is probably one of the key things that is going to help you choose between two products that are both, to be honest, pretty mature, capable pieces of software.

If you don’t feel comfortable with Python or Jython then Grinder is likely to be out. JMeter is much more friendly to non-programming testers. However this decision isn’t as clear as you might think, JMeter actually includes Javascript functions, access to BeanShell and its own expression language. As we created more complex traffic simulations we found ourselves having extremely complex expressions that often mixed a scripting language with an expression evaluation. In some ways The Grinder is more honest in telling you up front that you are going to have to program some of this stuff yourself.

Another big differentiator is in reporting, if you need to generate reports then JMeter is a much better choice as it has components that collect and collate information and you can dump data to XML for XSLT transformation into pretty much any output format you want. It is also possible to run JMeter headless once you have the test scripts so it is possible to encorporate it into a continuous integration process. Debugging slightly flaky applications is much easier with JMeter because you can easily capture a lot of result information and then just delete the data gathering component once the test flow is working.

A lot of the functionality of both products overlaps: they can both be used a browser proxy and be used to capture scripts as a user “drives” around the site. These captures are likely to form the basis of your test plans unless you have a very clear URL scheme with little session state.

Both use thread pools to generate load and both tend to get blocked on your application before they max out their local machine (although both are capable of using 90% of memory and CPU). JMeter has a few nice options around randomly ramping up the thread activation (recreating a spike in usage more closely). The Grinder tends to give more bang for the buck as its runtime is very simple and minimal.

Both are also capable of using distributed agents and collating the data from these agents back to the coordinator.

In many ways there isn’t a “bad” choice to be made between them. So what might sway your choice one way or another? Well JMeter is an Apache project and that might make a difference for some people as you have all those good things like project governance and a good chance the project is going to continue to be developed and enhanced. JMeter was also the only project to have a test suite last time I checked out both code bases.

The Grinder does have one unbeatable quality in my opinion and that’s flexibility. When you look at the features listed on the websites you might think that JMeter does all this cool stuff with HTTP, SOAP and POP3 and the like and that The Grinder is comparatively light in the feature set.

The opposite is actually true, as The Grinder website points out the The Grinder can test anything with a Java API. There really is no limit to what you can make it do or how you can execute your test plan. In fact most of the things I’ve complained about with The Grinder are fixable from within the test scripts  (what I am really complaining about is that some things like capturing HTML output per run is something that should be available as part of the standard package). On the other hand I felt that creating a new JMeter component was actually quite tricky as you have to deal with both the GUI aspects and the actual functionality you are trying to create in your test component.

If you really want to have total control of your concurrent volume testing then The Grinder is probably the best product for you.

Perhaps the last topic for consideration is The Grinder’s use of Jython. Python is a great language and you get a lot of power for some very concise code. Just as some people are going to find it off-putting others are actually going to find it very compelling.

Okay so yet another weak “horses for courses” post on the InterWeb. Perhaps the easiest summary to end on is that your test teams will probably take a shine to JMeter while your developer teams who are also building scale tests are probably going to like The Grinder; unless they dislike Python or learning a new language.

Standard
Work

One Year at ThoughtWorks

It’s now been a year since I joined Thoughtworks and  for me a year at ThoughtWorks is like two at any other company I’ve ever worked at. I have learned so much since I joined I really feel like I’m a very different person to the one who joined. I’ve met lots of really smart people who are doing really interesting things but all of whom have been generous and unstinting with their knowledge, experience and advice when asked.

This openess is the thing that really sets the culture apart from so many other firms. Knowledge seems to have value only when shared and people are generally so enthusiatic about the things they know they are really eager to help you understand things. In most companies knowledge is power and hoarded carefully, divested for maximum gain with the grace of a man having a tooth extracted.

The other cultural aspect that has been really different is that there is tremendous peer pressure to be excellent. If you are cutting corners or hacking something that’s convenient but flawed or just riding on your opinion then someone is going to call you on it.

When I’m working on client-owned projects I often think about different approaches and then wonder how I would feel if I had to justify the solution I chose to my TW collegues. It’s interesting because it gives you a strength to stand up against weak solutions and weak answers, even if you’re not actually working with anyone from ThoughtWorks.

So it’s been a good year generally and certainly the best I have had working for a company rather than doing my own thing. However ThoughtWorks isn’t perfect because any aggregation of individuals requires compromise and the biggest problem with ThoughtWorks is how you handle that.

One of the obnoxious things you can come across is the idea that you should be grateful to work for ThoughtWorks (particularly held I think amongst those who have only worked at TW or the City and other consultancies). ThoughtWorks has problems and each individual has to balance the benefits of getting to work with so many amazing people against an organisation that can’t really resolve its central dichotomies. Is it the Employee owned company that is delivering excellence or is it the home to the best knowledge workers who are revolutionising IT?

The trouble with not deciding exactly what the company wants to offer its employees as the vision is that every decision ends up infuriating half of your smart, emotionally invested and highly motivated workforce because it doesn’t fit with their interpretation of the ThoughtWorks ideal.

So it has been a great year and an experience I would recommend to anyone. My closing thoughts about ThoughtWorks is that it is a company that hasn’t said “No” to anything I have wanted to do. There isn’t always a lot of support and it is more forgiveness than permission but I feel that ThoughtWorks has helped me be a better person because it has given me the chance to do things that other organisations simply shut down in an arbitrary and off-hand way.

So thanks Roy and everyone else who works and has worked at ThoughtWorks for creating that opportunity for me.

Standard
Work

What next for Carbon Co-op?

A few people have asked me about what is happening next with Carbon Co-op. The short answer is that I have no idea. I had never heard of it before Saturday and I am not very personally invested in it. Probably the best thing to do if you are interested in the future of the business idea is follow the founder on Twitter and ask him.

Now as far as the code that is posted on Launchpad is concerned. I’m going to leave the Sicamp branch exactly as it was when I pushed at 2pm on Sunday. That’s where we go to.

I don’t really like the thought of it but I am tempted to do a screencast to explain the design of the app and show the site and basically all the stuff that wasn’t included in the actually SiCamp presentation. We did a lot of work! We should have shown it!

In terms of the trunk branch I have a few technical questions left of Django that I might use the project to experiment on. These are:

  • Markdown support in descriptions
  • AJAX support
  • Using MySql in the deployed mod_python version (with that done I could open up the app with the understanding it doesn’t do anything).
  • Introducing mocked testing as you would do for a Rails app and off the back of this some refactoring of the views section
  • Replace the Django Template Loader with the Jinja2 equivalent

In terms of the domain problems we set out to solve on the weekend itself, there are two that didn’t get resolved. Firstly there is the Locality issue: who is near who. For the weekend I was going to simply hack it that if the first segment of the post code was the same then those people counted as being in the same area. I’d like to do something more elegant and in depth but that’s another post.

Secondly there is the “ratcheting” idea that was kind of fundamental but I’m not sure really got highlighted in the pitch. The idea is that the app should always be exhorting you to get more people to join and giving you milestones to go for. This means once you have identified all the people in a Locality you need to figure what Actions they are taking and how many people in each Locality are Devoted to the Action.

You then need to see if the Action has Tiers with higher Thresholds and tell the user to try and get their friends and neighbours to commit to the next Tier.

Once you have all the numbers of Ongoing Actions, Devotees and people in the Locality you can then start doing some awesome stuff like altering the postion of the Actions to promote Actions that are close to crossing the next Threshold and so on.

If all that doesn’t make sense then don’t worry because I probably need to explain the design somewhere.

Standard
Java, Web Applications, Work

Better, Faster, Weaker

I went to the Developers and Startups Meetup last week (look it up on Meetup if you’re interested it was a good event). One thing that struck me was that all the entrepreneurs were mostly interested in PHP and Ruby on Rails (there was a significant rump of ASP and .Net which I can only put down to C#’s language hotness at the moment). Java in particular was poorly regarded.

I was intrigued by what was causing the negative vibe as while I can understand that Ruby on Rails is the new tech hotness I couldn’t really see the appeal of PHP over something like Java that has a decent stack that is practically free. Talking to a few people I got the sense that PHP was seen as a “getting things done” language. Something that got you to a tangible real product very quickly. Rails seemed to have much the same quality but I think a lot of people felt that Rails skills were more expensive and in a lot of cases they wanted a thin web layer over an existing service backend that already existed.

Of course both PHP, Rails and the Java Web Stack all make various compromises to do what they do. Where Rails is opinionated, Java probably isn’t opinionated enough. If you put all your code into page (Model 1 stylee) then I’m pretty certain that writing JSPs in J2EE 5 is probably as quick to put something together.

Of course one thing that hugely hampers Java as a productive web stack is the compile and deploy loop. There is none of the immediacy of something like RoR that allows you to make changes very quickly in response to feedback. The tradeoff is of course that the code needs to be interpreted and you have to add some kind of caching to avoid the performance hit of always having to evaluate code.

But given that there are a group of people who need to have code quickly so they can sell something to their customers or investors and that they are willing to make all kinds of compromises about scale and maintainance costs is the Java platform really out of the question? I ask this because I think I have been involved in quite responsive Java web teams where feature changes have been matters of days not weeks or months.

I think that perhaps the issue is partially just mental. Enterprise software development tends to take place in a risk-adverse environment where certainty or process is valued over rapid delivery. “Best practice” in this environment tends to sacrifice timeliness. In the smaller business space this might well be a practice turned into dogma.

That said though some Java Frameworks introduce major barriers to just getting stuff done. Any framework that forces you to produce XML based interaction flow is probably distracting you from generating functionality that people want and will pay for. Routing isn’t a trivial part of an application but that’s all the more reason to box clever with it. Any framework that doesn’t make it simple to inject services into classes is making life too hard. Ditto any framework that forces you to work with a particular Expression Language or Templating product.

I am wondering if what I am looking for is actually the weakest Java framework. Rather than having functionality for all circumstances (and having to configure and setup it all up) I want a very basic REST framework which I can then add templating and persistence according to my need. It still wouldn’t have the nice scaffolding of RoR but it would mean that each element of the web app would be absolutely vital to its function. Weakness would generate the power of vitality and reduce the obfuscation of bloat.

Standard
Software, Web Applications, Work

String Templates, or what I learned from Python and doing nothing

It’s an ill wind that blows no-one any good. The same is true of projects (although money generally helps more here; it’s an ill project that is making no-one any money).

I’m currently meant to be doing some work on Accessibilty for some new HTML pages. I thought it would be pretty easy but I was really wrong and it is changing the whole way I look at the View part of the (deceased) MVC web paradigm.

On my last project I was looking at things like Groovy’s Markup Builder and marvelling at how my collegues managed to put together a 30 line Freemarker template that did some pretty compex HTML assembly. In my spare time I have been looking at Haml as a way of escaping the verbosity and monotony of XHTML and to have the code guarantee the correctness of my page structure to avoid validation grind.

That’s because in those projects I was a developer/web designer. I wanted accurate, compliant HTML with minimum effort and which was easy to style without having awful CSS hacks.

On my current project I’m in the utterly baffling (for me anyway) world of .Net. There is no way that I can understand the huge variety of C#, XML and templating overrides that make up my current project. Having code generate HTML is a massive barrier to me being productive because, while I know a far bit of HTML having to root around an entire Visual Studio project to find the fragment that generates the problematic Div element you actually want to work with means I spend the whole day knowing nothing about .Net rather than applying the knowledge I do have.

Now some people are going to say that having a wacky Component model is different from having a nice templating language but look at something like Haml or Freemarker. The former is concise and fun and full of obscure rules; the latter is tremendously powerful, more firmly rooted in HTML and not much less obscure. For power users I agree, they are the bomb. They are a massive barrier to entry though, in a way that HTML just isn’t. People may do HTML badly but they rarely don’t do it at all.

This experience put combined with using Django/Jinja/Google App Engine is leading to me have a huge rethink about the way templating and views are put together. Passing a map of parameters to a template that is essentially exactly the way the output will look on the final device is obviously the way this problem should be tackled.

To try and get the HTML to generate in the current project I spent a day trying to get: SQLServer, BizTalk, Active Directory and Windows MQ to work together. This is utter madness and can only have been created by programmers who have no idea how to collaborate with non-programmers.

Why should I be trying to install BizTalk when what I want to do is actually generate some sample HTML so we can have a quick check of WAI standards? It should be possible to define some fixture data and then just generate HTML from the templates. It really shouldn’t be hard.

This experience is really changing the way I think about web frameworks. I am already determined to learn String Template, then I am going to look at whether my current favourite frameworks allow me to use it. I’m going to look at frameworks that ask you to put the HTML template next to the Java code. I want to know if I can put those templates in the same heirarchy that the actual website uses.

In short if I need to work with people outside the project team on a web project again, how can I get all the good things about templates and combine them with both simplicity and intuition as to how a website is organised?

Standard
Work

Against Trust

Work is generally better in high-trust environments, it is one of those success breeding success things. However you cannot simply get to high-trust because it is mutually beneficial to both parties. During the trust gaining stage things must be proven.

One of the nice things about working in IT (rather like studying for a Maths degree) is that things don’t have to be taken on trust. Sometimes it is convenient to but if needed to you can often make the effort to show that some thing happens exactly as you said it does. That’s the joy of unit-testing for example. Scala language advocates have also talked about the determinate nature of its static code checking. The code has to be right, otherwise it wouldn’t compile.

So in general I like trust, because its productive; but I don’t like to trust because I don’t feel I should have to. It should always be possible to show that what you are saying is true. Of course in consultancy that kind of proof is hard to come by so instead of absolute proof you instead have to come up with metrics that you can measure to allow you to fail fast instead of wasting a lot of time (and trust!) on an idea that isn’t going to pan out. This means you do not have to ask people to trust your ideas, merely agree some criteria for judging success and failure at minimum cost.

These ideas are some touchstones of how I have come to see effective IT within organisations. They are also what I feel informs me in a recent debate going through ThoughtWorks about Pay Transparency. You can find an effective starting point in Matt Buckland’s blog and what I think is an excellent statement of why Pay Transparency is a good idea in Francisco’s blog.

For me trust is good thing but should never be a necessary thing. Matt’s final point in his post is the most important as I think it is what really informs his view on pay transparency:

In all of these discussions the culture of transparency is held as the ultimate goal of an organisation and I’d wonder if this is the case. In my opinion it is a culture of trust we should strive for. Employees who feel they are paid fairly, because they have effectively sold the “exploitation of their labour” with full knowledge of the market in which that labour is sold will be better able to realise their own position … but instead trust they are given a fair salary based on their own personal circumstance and that the company they work for will aid them in their strive to grow and develop as an individual – salary, the nuts and bolts measure of their value, will become secondary.

From my point of view I want a culture of transparency because I believe that from it a culture of trust emerges. I don’t want to have to trust anyone, I would rather they proved what they say. In exactly the same way as I have to build trust with clients. In exactly the same way as a client perhaps trusts my code but can inspect my test reports at any time.

In my experience the processes and procedures around pay in the industry have been exceedingly opaque. There are so many variables and factors that pay is effectively arbitrary. I have often thought that pay really only reflects the ability of an organisation to function effectively without you.

In this context salary transparency is really the most important thing you can do to build trust around pay as few organisations can articulate the process behind salaries except that it is down to the abilities of individuals to negotiate for themselves and devil take the hindmost.

This post title says “Against Trust”, but it is really “Against Blind Trust” which is what people who reject Pay Transparency but forward no alternative mechanism are really asking for.

Standard