Toolserver 0.4.1

Toolserver Framework for Python

This software is OSI Certified Open Source Software. OSI Certified is a

certification mark of the Open Source Initiative.

The Toolserver Framework for Python is a framework for easy webservice creation. It’s main property is that it runs as a standalone server and so can easily be deployed. It uses medusa as it’s webserver core, but sports a threading model that is much like the process model of Apache 1.3.

For the license under that this software is distributed, read LICENSE.

For an installation instruction read the file INSTALL (essentially you can just do easy_install toolserver, but read it anyway for some info on dependencies you have to satisfy manually).

For more info look at the bitbucket page: http://bitbucket.org/rfc1437/toolserver/

Some highlights why you might want to look into TooFPy:

• industry strength and fast webserver as the base. Static content is delivered directly by the high performance Medusa server, dynamic content is delivered by dynamically managed worker threads.

• namespaces for tools as the base for URL construction. Intermingled with a standard static webserver to fill in those areas where no tools are registered.

• concentrates on building the API, not on building a full web application. You can build a web application, if you want, but you don’t need to. Want to just expose some quick hacks to SOAP or XMLRPC? Piece of cake.

• all APIs are dual use: they can be accessed over the web or they can be accessed internally in other tools. If you build a full web application, it will be composed of communicating tools where the tool communication is by calling well defined APIs that are both used internally and can be used externally, as well (if you decide so).

• automatic documentation generation for tools, both in HTML format (for human consumption) and WSDL format (for SOAP clients that need those). Just go to http://localhost:4334/API/ (assuming your server runs on the default port and the localhost IP) and you will get a list of links to installed tools with their respective API documentations - all generated on-the-fly from your tools sources!

• programming by contract for Web APIs! Of course, contracts are part of the human readable documentation. Contract checking can be switched on or off.

• wrapper based REST style APIs built over your standard RPC style APIs - no need to fight over API style anymore!

• factory tools that generate transient tools that themselves can be factories. This gives you fully dynamic namespaces.

• integrated authentication scheme for REST style API usage. This enables you to build your own authentication level above the tools without forcing some prebuilt scheme on you. This changed in 0.3.6, so now there are actually two authentication schemes, the newer one is build on standard user+group definitions, can be based on IP and HTTP basic authentication and can make use of a non-standard RSA authentication.

• integrated tools for powerfull thread management. Method calls can be made synchronously and asynchronously (using the same thread pool as the dynamic web content handling does). Asynchronous calls can be immediate, queued (serialized by priority on a per-tool queue) or timed.

• powerfull life debugger for the toolserver based on the medusa monitor. You can see what your server is doing and inject python code into a running system, peek at variables, even change them.

• simple HTML rendering library to ease production of wellformed HTML and XML content.

• dynamic and static namespaces for requests: global config variables are available during lifetime of the server, context variables are available during a request, static variables are available during normal method calls.

• integrated transaction hooks that allow you to combine your methods success or failure with database transaction calls. Your method fails, the database is rolled back.

• there are base classes for special purposes. For example you can choose wether all methods should be externally accessible by default or wether none should be accessible by default. Another base class is for providing RPC style access with a challenge-response-scheme for authentication. And then there is a all-singing-all-dancing base class if you want very simple integration of authentication - just subclass from AuthenticatedTool instead of StandardTool.

• IP based access lists (hosts.allow for fixed host assignements - the given IP set’s the designated client system name and no additional authentication is needed). Just add <clientname>:<ip> lines to either hosts.allow or hosts.deny to allow or deny access. Entries in hosts.allow already set the client name. This can be used as a simple authentication check for tools that want to restrict access only to known systems. Systems in hosts.deny aren’t allowed any RPC call at all (both RPC and REST style access applies!).

• RSA key based authentication to prevent unauthorized system access when you don’t have static IPs to check or need extra protection. If you activate RSA authentication, this is mandatory - no other (minor) authentication scheme will be accepted.

• new RPC mechanisms can be easily installed, much like tools. Just write your RPC handling class like the SOAPHandler or XMLRPCHandler and put them into your local tools directory (or a global tools directory if you want to use this RPC mechanism in all instances of your toolserver). There is a new RPC mechanism based on base64 encoded binary pickles - look into PickleRPCClient.py for the client code and PickleRPCHandler.py for the handler code. This one is much faster than XML based RPC mechanisms, as only very fast standard libraries are used. PickleRPC only works with RSA authentication to enable maximum security.

• you can easily plugin WSGI compliant applications if you have wsgiref installed. This allows merging your toolserver with functionality from other sources.

• the ReactorChain gives a powerfull pattern to tools to hook into other tools processing. This mechanism is relyable: if hooked tools fail, the chain reaction will still take place, so the original tools functionality won’t be compromised. One use for this is the change system.request.rewrite that can be used to add URI preprocessing to rewrite parts of URIs before any tool has the chance to see the URI. For example this can be used to do virtual host handling.

Some random notes (yes, some day there might be a documentation):

IMPORTANT NOTE ON PICKLERPC: Pickles should be seen as insecure, as the binary encoding of data could introduce stuff that you don’t want to happen on your server. For that reason PickleRPC is disabled by default and needs to be enabled in the configuration. You should put some kind of protection in front of your server, for example checking connecting IPs for only trusted sources and you SHOULD NOT expose a server with PickleRPC to untrustworthy systems. For this, it would be a very bad idea to enable PickleRPC and open your server to the internet without any security measures! To help to prevent damage, PickleRPC is tied to RSA authentication - if you don’t set up RSA authentication, PickleRPC can’t be enabled.

HOW TO GET RSA AUTHENTICATION GOING: you first will have to generate usefull key pairs. Do this with “tsctl keygen” for the server and “tsctl keygen clientname” for the client and move the client keys to the client machine. When instantiating a RemoteToolserver instance (from the Toolserver.Client module) you can now pass pubkey, privkey and localname options. If you do, your call will have additional HTTP headers that carry a SHA256 hash, a RSA signature and your name. The server will check those (only if you enabled rsaauthenticate in the config) and provide you with the same headers in the result. Due to the RSA key stuff this will be noticeably slower but much securer, as you will only allow communication with known hosts. To enable new hosts, just put their public key into the pubkeys directory of your toolserver. They need to be named after the client. To disable a client, just remove it’s public key and restart the server.

A NOTE ON ENCRYPTION: this toolserver framework could have easily adopted SSL as the main transport security protocol. Actually it would have been much more natural to use that - every proxy should now support CONNECT and so should support HTTPS connections. The reason I didn’t do this (I might opt to do it in the future, though) is that the toolserver usually doesn’t run on well known ports. And CONNECT to servers with ports other than the default ports nowadays requires administrator intervention - every port allowed for CONNECT needs to be entered in the system configuration. This is to prevent hijacking https proxying for unauthorized outside connections. So I opted for the non standard way of encrypting requests and responses instead - the call itself is still transported by the standard HTTP and so should travel through allmost every proxy you could think of. Without administrator intervention or frontend servers. Remember, TooFPY isn’t about building web applications, it’s about building web services and distributed applications. If you want interop with other software, you should use the SOAP or XMLRPC protocols and keep off the RPC encryption stuff, as that is non-standard. You still can set up frontend servers that do the SSL stuff and put TooFPy behind them and so can still use TooFPy as a building block for web applications, relying solely on standard protocols and formats. Just see the encryption stuff and the PickleRPC stuff as additional options if you run in a pure Python environment.

A NOTE ON COMPATIBILITY: some protocols are non-standard (for example PickleRPC can only be used with Python) or have non-standard extensions (for example RSA authentication is done in some non-standard way that currently only works with Toolserver.Client). If you want to access your webservices from a wide range of clients, you shouldn’t use those non-standard features. Just don’t enable PickleRPC and don’t use RSA authentication - use CRAM authentication, instead. CRAM authentication uses a very simple scheme based on SHA1, so you can call methods with that authentication scheme from most languages (using Java or Perl is quite easy). CRAM authentication does rely on a shared secret, though. If you are forced to invent your own authentication mechanism, you can use context.request in your _validate methods and access the request data and headers directly - for example to build some IP based authentication or something using identd lookup (the IP is hooked in the request via the medusa transport - you might need to dig through the medusa source to see what I am talking about). If everything fails your needs, you can allways pass user and password fields in your calls - just be aware that they are transfered in clear over the wire, so set up some encrypted channel to communicate over. Some simple way to do this is to run the toolserver only on localhost and put an Apache in front of it and let that one handle all authentication and SSL stuff.

A NOTE ON NEW PYTHON VERSIONS: Python 2.6 deprecates the md5 and sha modules and wants you to use hashlib instead. Toolserver does so, but medusa still imports md5, so you get a deprecation warning. This is old code.