Tag Archives: riemann

Clojure, Programming

London Clojure unconference July 2014 report

For the first session I was interested in trying to continue the discussion about the Clojure “sweet spot” we had had on the mailing list. But there was only a smattering of interest so we rolled it up with the discussion on how to convince people in investment banks to use Clojure.

I think Jon Pither’s approach to this is still the best which is to find a business problem and then say that you’re going to address the problem and use Clojure to solve the real problem. A pure technical argument is not really going to get buy-in from outside the developers.

A lot of organisations want to have an approved list of technologies and for institutions that have chronic and acute technical problems like banks then perhaps that is appropriate given the need for external regulation. Where these things exist I usually think it is a case of going through the bureaucratic hoops.

The approval system is not there to be opinionated but to provide oversight. Where individuals have “weaponised” the approval process to advance their view of “right” technology you need to tackle the root problem not just sneak things in as jars.

My personal view is that financial institutions have profound technology problems but that they have no incentive to address them while they continue to make a lot of money. Really their problems should be providing opportunities for new approaches but as the existing institutions have created massive barriers to entry it doesn’t happen and we’re all really just waiting for the next financial crisis to happen, maybe then…

However in the session there was a lot of discussion about whether it is appropriate for managers to determine technology choices: on the one side you want to devolve decisions to the people close to the problem, on the other programmers commonly change jobs in a shorter period that the lifespan of the software they create.

One thing I took away was that before conservative organisations adopt Clojure they will need to see widespread adoption in the companies they see as good leading indicators and the presence of a large hiring population. In these respects Scala is literally years ahead.

Our final conclusion as a group was simply that the easiest way to approve the use of Clojure was to get into management and leadership first and then do it.

For the second session I went to the discussion on React and Om. I’m looking at React currently and there were a lot of questions about what Om layers on top of the basic JS library. Anna Pawlicka provided a number of the Om answers and others chipped in with bits of React and reactive JS knowledge. I was reminded to go and look at the current state of Om and also the new tutorials. There was also some interesting talk of how to define React components, Anna used Sablono but is there still a need for JSX?

The final session of the evening was on Riemann, which in addition to be a basic introduction to what it does was a helpful reminder of the functionality that Riemann has but that I haven’t used personally. Jason Neylon mentioned that every new service they set up has a Riemann instance attached so you can just dump all events somewhere and then build dashboards dynamically as you go along (a lot better than our approach with Graphite).

Tom Crayford introduced me to the effect of clock skew on Riemann (events from the “future” relative to the Riemann server clock are dropped) and then pointed out that clock skew can actually be monitored via Riemann! Also some interesting stuff about pumping logs into Riemann and some personal experience of crazy volumes of events being successfully handled.

Just before the end of the event I dropped in to the Gorilla REPL session to see Jony Hudson demoing his amazing notebook repl that he has been using to share assignments and research with students and colleagues in his department. A really interesting application and I suspect once we get our heads round it a really interesting way of sharing problems and potential solutions as developers.

Mind slightly blown, I was personally really happy with the event and felt that I’d got a mix of advice and the kind of innovation that make the Clojure community so interesting.

Standard
Clojure

Getting started with Riemann stream processing

Riemann is a great application for dealing with event processing but it doesn’t have a lot of documentation or newbie friendly tutorials. There are some cool pictures that explain the principles of the app but nothing beyond that. At some point I want to try and contribute some better documentation to the official project but in the meantime here’s a few points that I think are useful for getting started.

I’m assuming that you’ve followed these instructions to get a working Riemann installation and you’ve followed the instructions on how to submit events to Riemann via the Ruby Riemann client interface.

At this point you want to start making your own processing rules and it is not clear how to start.

Well the starting point is the idea of streams when an event arrives in Riemann it is passed to each stream that has been registered with what is called the core. Essentially a stream is a function that takes an event and some child streams and these functions are stored in a list in the core atom under the symbol :streams.

Okay let’s look at an example. The first obvious thing you want to do is print out the events that you are sending to Riemann. If you’ve got the standard download open the etc/riemann.config file, set the syntax for the file to be Clojure, as this is read into Clojure environment in the riemann/config namespace and you can use full Clojure syntax in it. In the config file add the following at the end. Now either run the server or if it is running reload the config file with kill -HUP <Riemann PID>.

(streams prn)

prn is a built-in function that will print an event and pass it on to following streams.

In irb let’s issue an event:

r << {host: "rrees.me", service: "posts", metric:  5}

You should see some output in the Riemann log along the following lines.


#riemann.codec.Event{:host "rrees.me", :service "posts", :state nil, :description nil, :metric 5, :tags nil, :time 1366450306, :ttl nil}

I’m going to assume this has worked for you. So now let’s see how events get passed on further down the processing chain. If we change our streams function to the following and reload it.

(streams prn prn)

Now we send the event it should get printed twice! Simples!

Okay now let’s look at how you can have multiple processing streams working off the same event. If we add a second print stream we should get three prints of the event.

(streams prn prn)

(streams prn)

Each stream that is registered can effectively process the event in parallel so some streams can process an event and send it to another system while another can write it to the index.

Let’s change one of our prints slightly so we can see this happen.

(streams (with :state "normal" prn) prn)

(streams prn)

We should now get three prints of the event and in one we should see that the event has the state of “normal”. Okay great! Let’s break this down a bit.

Every parameter of streams is a stream and a stream can take an event and child streams. So when an event occurs it is passed to each stream, each stream might specify more streams that the transformed event should be passed to. That’s why we pass prn as the final parameter of the with statement. We’re saying add the key-value pair to the event and pass the modified event to the prn stream.

Let’s try implementing this by ourselves, there is a bit of magic left here, call-rescue is an in-built function that will send our event to other streams you can think of it as a variant of map:

(defn change-event [& children]
  (fn [event]
    (let [transformed-event (assoc event :hello :world)]
      (call-rescue transformed-event children))))

(streams (change-event prn))

If this works then we should see an event printed out that has the “hello world” key-value pair in it. change-event is a stream handler that takes a list of “children” streams and returns a function that handles an event. If the function does not pass the event onto the children streams then the event stream stops processing, which is a bit like a filter. The event is really just a map of data like all good Clojure.

At this point you actual have a good handle on how to construct your own streams. Everything else is going to be a variation on this pattern of creating a stream function that returns an event handler. The next thing to do now is go and have a look at the source code for things like with, prn and call-rescue. Peeking behind the curtain will take a certain amount Clojure experience but it really won’t be too painful, I promise, the code is reasonable and magic is minimal. Most of the functions are entirely self-contained with no magic so everything you need to know is in the function code itself.

Standard