May 15, 2012
CLOS FSM with MOP Sauce

Common Lisp in the desire to be as cool as possible includes in its specification the Common Lisp Object System, or CLOS, which itself can be introspected and altered in great detail using the MetaObject Protocol, or MOP, as described in The Art of the Metaobject Protocol. Unfortunately, MOP didn’t make it into the ANSI standard, but most implementations include MOP as it is described in the book, and a compatibility package :closer-mop (available in Quicklisp) makes using the symbols described seamless between implementations.

One of the features of MOP is the ability to make an instance of a CLOS class funcallable, that is allow a class instance to be a valid first argument to funcall. This behavior in a lot of ways can resemble the traditional method model from other languages, but that’s not how I intend use it here.

The Idea

I’m going to show and tell an implementation of a generic finite state machine that uses the MOP concepts of funcallable-standard-object and funcallable-standard-class to marshal the flow of incoming events to the machine and the concept of CLOS generic method dispatch to handle the execution of transition handlers for any given state of the machine.

These features will allow us to build a structure that lets us focus solely on the problem at hand and defer features like event data binding, state-dependent method selection and unexpected state handling entirely to the language without pushing the boundaries of any specific feature.

The Design

The design we’re going for is such that we can define a class with a state slot that will hold :keyword name of a state. We’ll make instances of this class funcallable so that when we make an instance we will be able to simply (funcall fsm-instance fsm-evemt) repeatedly and have the machine dispatch to the correct event, perform any logic, and transition to the next state based on the input.

We would be able to query the state of the machine with (state fsm-instance) and receive a keyword, and we should be able to drive events into the machine until we’re in a desired or unexpected state. Any attempt to feed the machine an event while the machine is in an invalid state should result in an error.

Read More

May 9, 2012
Hinge -- v0.0.1

Hinge, my Node.js inspired API for Common Lisp wrapping libev is finally to a state where I feel okay giving it a non-zero version.

I’ve decided now would be a good milestone because the HTTP server functions, and has a set of exported APIs that let you reply to the requests it generates.

The quick set of terrible benchmarks I performed on the “Hello World!” example show me a requests/second throughput of 1500-2500 depending on the mood of ab (and the state of the number of sockets available on localhost). I intend to run it through a more serious (and HTTP/1.1 supporting) tool in the future, but for now ab has convinced me that the decisions I’ve made so far have not been abysmal.

The examples directory contains a bunch of example programs, or in the earlier cases, code snippets demonstrating various levels of functionality.

In the near future, in addition to covering the issues in the README, adding additional polish, and any bugs or blatant performance issues that come up I would like at some point to add support for:

  • HTTP Clients
  • SSL Sockets and Servers
  • SSL HTTP Servers and Clients
  • Async wrappers around FS calls
  • Websockets (RFC6455) support
  • Mongrel2 ZeroMQ handler support (through m2cl)
  • Mongrel2 ZeroMQ protocol-compatible server
  • SPDY Support (Somewhere down the list, maybe)

Any feedback is welcome however way you want to deliver it to me. This initial version stands to serve as a basic starting point for the concepts I wanted to convey, the take on the core APIs I wanted to offer, and a proof of a series of related concepts and architectural decisions. It may (hopefully) be useful as a stable base for progress, rather than chasing an otherwise potentially unstable head.

January 17, 2012
StrangeLoop 2011: We Really Don't Know How To Compute

Gerald Jay Sussman compares our computational skills with the genome, concluding that we are way behind in creating complex systems such as living organisms, and proposing a few areas of improvement.

This talk has been itching away at my brain. Both for the subject matter and the symbolic computation demonstrated.

January 6, 2012
0MQ Protips with Uncle Slava: Multipart ZMQ_SUBSCRIBE

For lack of anything more interesting to say I will tell you about something that for some reason remained not entirely obvious to me until very recently.

When using ZMQ_PUB/ZMQ_SUB sockets and ZMQ_SUBSCRIBE message filters on one side of the interaction it is possible to separate the sending of messages from the sending of subscription channel information using multipart messages.

While working on Clubot with Sean Bryant earlier it became frustrating to consume the output of the bot since the messages coming out were coming out as single strings of JSON prefixed with a series of subscription symbols and strings as in:

:PRIVMSG :CHATTER "#somechan" "Origin_nick" {..[json]..}

So to consume the message symbols and strings have to be consumed from the beginning of the string until we can start to read JSON from an offset. This was no good.

ZeroMQ promises in the documentation of zmq_send that multipart messages will be delivered either as a complete sequences or not at all. Following this logic, we should be able to send subscription information up front in a single frame then follow up the body in a second and still have the full benefits of filtering using ZMQ_SUBSCRIBE with the added benefit of being able to slurp in the entire second message on the assumption that it contains JSON data.

So putting it all together. The publisher creates a message body and subscription information as two strings. It then sends the message as a pair of multipart messages with the subscription information first as in:

(zmq:send *pub-sock* header-message ZMQ:SNDMORE)

The ZMQ_SNDMORE flag indicates that this message will be followed by additional parts and should not be consumed individually.

The body is then written as usual:

(zmq:send *pub-sock* body-message)

The message will be published to any interested peers and the the pair of messages will be filtered by the contents of the first message with the body just tagging along.

To read a multipart message stream correctly we have to use the ZMQ_RCVMORE socket option to check for additional messages in the message stream. We can skip this step if our protocol is precise and we have foreknowledge of the number of message parts arriving, but doing so would leave a potential problem in the future.

The ZMQ_RCVMORE option when queried with zmq_getsockopt from a socket will return either 0 or 1 for either completing the multipart read (0) or having additional data available (1) as message components that were delivered with ZMQ_SNDMORE.

So we can receive the orignal example message when sent as a two-part multipart message following something like the following:

;; Still filter on the first part of the entire sequence
(zmq:subscribe *sub-socket* ":PRIVMSG")
(zmq:recv *sub-socket* subscription-header)

Now subscription-header will contain a string containing just the message prefix. If we use the same message it would be :PRIVMSG :CHATTER "#somechan" "Origin_nick"

We can then check to make sure that there is additional data to read and read the data component of the message or signal an error.

(if (= (zmq:getsockopt *sub-socket* zmq:rcvmore) 1)
  (zmq:recv *sub-socket* data-message)
  (error "No message followed subscription header!"))

If that form evaluates without error then data-message should contain just the data JSON without any more required to separate it! Isn’t that exciting?

Happy cooking.

December 31, 2011
IOLib:multiplexer, 0MQ and OS X

It would appear events aren’t being signalled correctly on a DEALER->ROUTER (at least? maybe? It’s the only evidence I have) communication in in the IOLib multiplexer when the FD bound for read is one discovered from a ZeroMQ socket with ZMQ_FD then checked against ZMQ_EVENTS.

If I close the socket and terminate the context after the request the message is always delivered, much like a message is pulled from the queue when another client connects and disconnects. Events on the plain file descriptor tied to a network socket seem to fire just fine, though.

I’ll have to run some science later using the native zmq:poll to make sure they are supposed to arrive like I expect, but if they don’t I might attempt to bring a bridge over to libev and never have this problem again for as long as I parenthesize.

December 31, 2011
IRC -> 0MQ Bridge

I started working on a project to bridge IRC to IRC-based services using ZeroMQ as a transport. So far I’ve managed to convince it to connect and broadcast out messages that are easy to ZMQ_SUBSCRIBE to and handle a few commands sent as JSON over another set of 0MQ sockets. 

Hopefully it ends up more useful than just fun.

Sean Bryant started hacking together an adapter to link in Hubot as a 0MQ service called Hubot-Shaky this should make this project useful more quickly.

The project runs on SBCL and is still in the early stages. The only way to operate it is still from a REPL. I’m posting it now because it’s at a point where it’s theoretically possible to write something like a Google service, and in case any one has every been curious about digging through a small-scale but sanely packaged Common Lisp project.

Liked posts on Tumblr: More liked posts »