yarr 0.9.0

Yarr ain't RPC, Ritchie!

A simple RPC implementation using a binary serialization protocol. It supports the basic built-in types, numpy arrays work if available on both client and server. Hence, yarr is well equipped to efficiently handle large binary booty.

A friend asked me about RPC and large binary data in python. It sounded like a fun exercise so I charged forth. I choose to write my own serialization protocol to have more control and a bit more security than pickle. I also ran my benchmarks which pickle serialization and surprisingly yarr is not that slow. Lists and dict take the biggest hit, being ~ 2x slower compared to pickle.

Example

# server
yarr.yarr(('localhost', 8000), [print])
# client
yarr.call(('localhost', 8000), 'print', 'Ahoy Lads!')

Features

Compared to other RPC frameworks yarr has better support for native python types. It's speed depends on the connection. On localhost it appears to be 2-10 times faster than the other frameworks. Over LAN it's comparable to jsonrpc2-zeromq. Over internet the ZeroMQ based frameworks are twice as fast, but yarr is still on par with xmlrpc.

Here is a table comparing it to other RPC frameworks. Partial support is marked with p, usually due to type decay. The speed value is based on the benchmarks below. Feel free to add more.

Xmlrpc is part of python's stdlib. Good enough for simple use-cases.

Rpyc is glacially slow but has almost perfect type support. Surprisingly it stumbles over pep 383. It's also one of the few frameworks that can handle recursion.

Jsonrpc2-zeromq has decent speed, though it is limited by it's protocol. There is no direct support for binary data. The base64 workaround works but impacts speed.

Spyne is a big project and can do a lot more than pure RPC. For this test MessagePack and ZeroMQ were used.

Zerorpc performance is below jsonrpc2-zeromq, which was unexpected given it's choice of protocols. A dict containing tuple keys crashes the server.

Benchmarks

The values below are in calls per second. This test was run over LAN with a server ping of 0.2 ms. The most recent framework versions (May 2020) were used. When running this benchmark over internet with a ping of 25 ms, yarr performance is roughly half of jsonrpc2-zeromq.

Protocol

The main design goals were simplicity and performance. There is no clever encoding scheme like MessagePack. Data is basically just dumped into the TCP socket without any transformations. Values are transferred as little endian, there really is no point in using network byte order these days.

In some cases messages can actually be bigger than in json, for example when it contains many short strings. However if the total size stays below the MTU there is be no big performance hit. I tried adding zlib compression and a persistent connection, but that did not have a big impact and made everything more complicated.

Bugs

• Not enough pirate-speak in documentation.
• Malicious data can cause crash.
• Recursive containers cause crash.

License

Do What The Fuck You Want To Public License

Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.