Dataflow Programmering 13 Apr 2009

Not long after the idea of dataflow programming clicked for me while reading the excellent Concepts, Techniques, and Models of Computer Programming (affiliate link) I have been trying to figure out the best way to apply it at the library level rather than the language level. Having it at the language level is fine and dandy, but I’m happy to sacrifice a little elegance for just being easy to use for building up a page or response in a webapp from a bunch of remote services.

When rendering (heh, originally typoed that rending, kind of appropriate) a typical page in Rails, PHP, JSP, whatever. If you are nice and clean you fetch all the data you need and shove it into some kind of container which is then used to populate a template. In a complex system it is not unusual to make 20+ remote calls to render a single response. These are to caches, databases, other services, and sometimes pigeons passing by with telegrams on their legs. A couple years ago, we used a reactor style dataflow tool Tim and I wrote for javascript. I rather miss having it when wiring together backend services.

I have done a number of ad-hoc versions in services for Java, using an executor and passing around references to futures, but I don’t have anything that really matches the rather nice push and react style thing we had in javascript there. I can imagine something using Doug Lea’s jsr166y fork/join tools, but every time I start to poke into them the… well, maybe mapping it into a library really is kind of ugly. It certainly is screaming for anonymous functions, oh well guess I am not holding my breath.

So, switching to the other languages I hack in nowadays, we have Ruby (oops, no threads), C (umh, no, wrong level of abstraction), and Lua (hey, actually not bad, particularly with how LuaSocket and coroutines play together…).

Properly, I should now shut up and go hack. On that note, off to hack!