martian 2.0.post1

Embedding of configuration information in Python code.

A library to grok configuration from Python code.

Martian tutorial

Introduction

“There was so much to grok, so little to grok from.” – Stranger in a Strange Land, by Robert A. Heinlein

Martian provides infrastructure for declarative configuration of Python code. Martian is especially useful for the construction of frameworks that need to provide a flexible plugin infrastructure. Martian doesn’t actually provide infrastructure for plugin registries (except for itself). Many frameworks have their own systems for this, and if you need a generic one, you might want to consider zope.component. Martian just allows you to make the registration of plugins less verbose.

You can see Martian as doing something that you can also solve with metaclasses, with the following advantages:

• the developer of the framework doesn’t have to write a lot of ad-hoc metaclasses anymore; instead we offer an infrastructure to make life easier.

• configuration doesn’t need to happen at import time, but can happen at program startup time. This also makes configuration more tractable for a developer.

• we don’t bother the developer that uses the framework with the surprising behavior that metaclasses sometimes bring. The classes the user has to deal with are normal classes.

Why is this package named martian? In the novel “Stranger in a Strange Land”, the verb grok is introduced:

Grok means to understand so thoroughly that the observer becomes a part of the observed – to merge, blend, intermarry, lose identity in group experience.

In the context of this package, “grokking” stands for the process of deducing declarative configuration actions from Python code. In the novel, grokking is originally a concept that comes from the planet Mars. Martians grok. Since this package helps you grok code, it’s called Martian.

Martian provides a framework that allows configuration to be expressed in declarative Python code. These declarations can often be deduced from the structure of the code itself. The idea is to make these declarations so minimal and easy to read that even extensive configuration does not overly burden the programmers working with the code.

The martian package is a spin-off from the Grok project, in the context of which this codebase was first developed. While Grok uses it, the code is completely independent of Grok.

Motivation

“Deducing declarative configuration actions from Python code” - that sounds very abstract. What does it actually mean? What is configuration? What is declarative configuration? In order to explain this, we’ll first take a look at configuration.

Larger frameworks often offer a lot of points where you can modify their behavior: ways to combine its own components with components you provide yourself to build a larger application. A framework offers points where it can be configured with plugin code. When you plug some code into a plugin point, it results in the updating of some registry somewhere with the new plugin. When the framework uses a plugin, it will first look it up in the registry. The action of registering some component into a registry can be called configuration.

Let’s look at an example framework that offers a plugin point. We introduce a very simple framework for plugging in different template languages, where each template language uses its own extension. You can then supply the framework with the template body and the template extension and some data, and render the template.

Let’s look at the framework:

>>> import string
>>> class templating(FakeModule):
...
...   class InterpolationTemplate(object):
...      "Use %(foo)s for dictionary interpolation."
...      def __init__(self, text):
...          self.text = text
...      def render(self, **kw):
...          return self.text % kw
...
...   class TemplateStringTemplate(object):
...      "PEP 292 string substitutions."
...      def __init__(self, text):
...          self.template = string.Template(text)
...      def render(self, **kw):
...          return self.template.substitute(**kw)
...
...   # the registry, we plug in the two templating systems right away
...   extension_handlers = { '.txt': InterpolationTemplate,
...                          '.tmpl': TemplateStringTemplate }
...
...   def render(data, extension, **kw):
...      """Render the template at filepath with arguments.
...
...      data - the data in the file
...      extension - the extension of the file
...      keyword arguments - variables to interpolate
...
...      In a real framework you could pass in the file path instead of
...      data and extension, but we don't want to open files in our
...      example.
...
...      Returns the rendered template
...      """
...      template = extension_handlers[extension](data)
...      return template.render(**kw)

Since normally we cannot create modules in a doctest, we have emulated the templating Python module using the FakeModule class. Whenever you see FakeModule subclasses, imagine you’re looking at a module definition in a .py file. Now that we have defined a module templating, we also need to be able to import it. Fake modules are always placed automatically into the martiantest.fake namespace so you can import them from there:

>>> from martiantest.fake import templating

Now let’s try the render function for the registered template types, to demonstrate that our framework works:

>>> templating.render('Hello %(name)s!', '.txt', name="world")
'Hello world!'
>>> templating.render('Hello ${name}!', '.tmpl', name="universe")
'Hello universe!'

File extensions that we do not recognize cause a KeyError to be raised:

>>> templating.render('Hello', '.silly', name="test")
Traceback (most recent call last):
...
KeyError: '.silly'

We now want to plug into this filehandler framework and provide a handler for .silly files. Since we are writing a plugin, we cannot change the templating module directly. Let’s write an extension module instead:

>>> class sillytemplating(FakeModule):
...   class SillyTemplate(object):
...      "Replace {key} with dictionary values."
...      def __init__(self, text):
...          self.text = text
...      def render(self, **kw):
...          text = self.text
...          for key, value in kw.items():
...              text = text.replace('{%s}' % key, value)
...          return text
...
...   templating.extension_handlers['.silly'] = SillyTemplate
>>> from martiantest.fake import sillytemplating

In the extension module, we manipulate the extension_handlers dictionary of the templating module (in normal code we’d need to import it first), and plug in our own function. .silly handling works now:

>>> templating.render('Hello {name}!', '.silly', name="galaxy")
'Hello galaxy!'

Above we plug into our extension_handler registry using Python code. Using separate code to manually hook components into registries can get rather cumbersome - each time you write a plugin, you also need to remember you need to register it.

Doing template registration in Python code also poses a maintenance risk. It is tempting to start doing fancy things in Python code such as conditional configuration, making the configuration state of a program hard to understand. Another problem is that doing configuration at import time can also lead to unwanted side effects during import, as well as ordering problems, where you want to import something that really needs configuration state in another module that is imported later. Finally, it can also make code harder to test, as configuration is loaded always when you import the module, even if in your test perhaps you don’t want it to be.

Martian provides a framework that allows configuration to be expressed in declarative Python code. Martian is based on the realization that what to configure where can often be deduced from the structure of Python code itself, especially when it can be annotated with additional declarations. The idea is to make it so easy to write and register a plugin so that even extensive configuration does not overly burden the developer.

Configuration actions are executed during a separate phase (“grok time”), not at import time, which makes it easier to reason about and easier to test.

Configuration the Martian Way

Let’s now transform the above templating module and the sillytemplating module to use Martian. First we must recognize that every template language is configured to work for a particular extension. With Martian, we annotate the classes themselves with this configuration information. Annotations happen using directives, which look like function calls in the class body.

Let’s create an extension directive that can take a single string as an argument, the file extension to register the template class for:

>>> import martian
>>> class extension(martian.Directive):
...   scope = martian.CLASS
...   store = martian.ONCE
...   default = None

We also need a way to easily recognize all template classes. The normal pattern for this in Martian is to use a base class, so let’s define a Template base class:

>>> class Template(object):
...   pass

We now have enough infrastructure to allow us to change the code to use Martian style base class and annotations:

>>> class templating(FakeModule):
...
...   class InterpolationTemplate(Template):
...      "Use %(foo)s for dictionary interpolation."
...      extension('.txt')
...      def __init__(self, text):
...          self.text = text
...      def render(self, **kw):
...          return self.text % kw
...
...   class TemplateStringTemplate(Template):
...      "PEP 292 string substitutions."
...      extension('.tmpl')
...      def __init__(self, text):
...          self.template = string.Template(text)
...      def render(self, **kw):
...          return self.template.substitute(**kw)
...
...   # the registry, empty to start with
...   extension_handlers = {}
...
...   def render(data, extension, **kw):
...      # this hasn't changed
...      template = extension_handlers[extension](data)
...      return template.render(**kw)
>>> from martiantest.fake import templating

As you can see, there have been very few changes:

• we made the template classes inherit from Template.

• we use the extension directive in the template classes.

• we stopped pre-filling the extension_handlers dictionary.

So how do we fill the extension_handlers dictionary with the right template languages? Now we can use Martian. We define a grokker for Template that registers the template classes in the extension_handlers registry:

>>> class meta(FakeModule):
...   class TemplateGrokker(martian.ClassGrokker):
...     martian.component(Template)
...     martian.directive(extension)
...     def execute(self, class_, extension, **kw):
...       templating.extension_handlers[extension] = class_
...       return True
>>> from martiantest.fake import meta

What does this do? A ClassGrokker has its execute method called for subclasses of what’s indicated by the martian.component directive. You can also declare what directives a ClassGrokker expects on this component by using martian.directive() (the directive directive!) one or more times.

The execute method takes the class to be grokked as the first argument, and the values of the directives used will be passed in as additional parameters into the execute method. The framework can also pass along an arbitrary number of extra keyword arguments during the grokking process, so we need to declare **kw to make sure we can handle these.

All our grokkers will be collected in a special Martian-specific registry:

>>> reg = martian.GrokkerRegistry()

We will need to make sure the system is aware of the TemplateGrokker defined in the meta module first, so let’s register it first. We can do this by simply grokking the meta module:

>>> reg.grok('meta', meta)
True

Because TemplateGrokker is now registered, our registry now knows how to grok Template subclasses. Let’s grok the templating module:

>>> reg.grok('templating', templating)
True

Let’s try the render function of templating again, to demonstrate we have successfully grokked the template classes:

>>> templating.render('Hello %(name)s!', '.txt', name="world")
'Hello world!'
>>> templating.render('Hello ${name}!', '.tmpl', name="universe")
'Hello universe!'

.silly hasn’t been registered yet:

>>> templating.render('Hello', '.silly', name="test")
Traceback (most recent call last):
...
KeyError: '.silly'

Let’s now register .silly from an extension module:

>>> class sillytemplating(FakeModule):
...   class SillyTemplate(Template):
...      "Replace {key} with dictionary values."
...      extension('.silly')
...      def __init__(self, text):
...          self.text = text
...      def render(self, **kw):
...          text = self.text
...          for key, value in kw.items():
...              text = text.replace('{%s}' % key, value)
...          return text
>>> from martiantest.fake import sillytemplating

As you can see, the developer that uses the framework has no need anymore to know about templating.extension_handlers. Instead we can simply grok the module to have SillyTemplate be register appropriately:

>>> reg.grok('sillytemplating', sillytemplating)
True

We can now use the .silly templating engine too:

>>> templating.render('Hello {name}!', '.silly', name="galaxy")
'Hello galaxy!'

Admittedly it is hard to demonstrate Martian well with a small example like this. In the end we have actually written more code than in the basic framework, after all. But even in this small example, the templating and sillytemplating module have become more declarative in nature. The developer that uses the framework will not need to know anymore about things like templating.extension_handlers or an API to register things there. Instead the developer can registering a new template system anywhere, as long as he subclasses from Template, and as long as his code is grokked by the system.

Finally note how Martian was used to define the TemplateGrokker as well. In this way Martian can use itself to extend itself.

Grokking instances

Above we’ve seen how you can grok classes. Martian also supplies a way to grok instances. This is less common in typical frameworks, and has the drawback that no class-level directives can be used, but can still be useful.

Let’s imagine a case where we have a zoo framework with an Animal class, and we want to track instances of it:

>>> class Animal(object):
...   def __init__(self, name):
...     self.name = name
>>> class zoo(FakeModule):
...   horse = Animal('horse')
...   chicken = Animal('chicken')
...   elephant = Animal('elephant')
...   lion = Animal('lion')
...   animals = {}
>>> from martiantest.fake import zoo

We define an InstanceGrokker subclass to grok Animal instances:

>>> class meta(FakeModule):
...   class AnimalGrokker(martian.InstanceGrokker):
...     martian.component(Animal)
...     def execute(self, instance, **kw):
...       zoo.animals[instance.name] = instance
...       return True
>>> from martiantest.fake import meta

Let’s create a new registry with the AnimalGrokker in it:

>>> reg = martian.GrokkerRegistry()
>>> reg.grok('meta', meta)
True

We can now grok the zoo module:

>>> reg.grok('zoo', zoo)
True

The animals will now be in the animals dictionary:

>>> sorted(zoo.animals.items())
[('chicken', <Animal object at ...>),
 ('elephant', <Animal object at ...>),
 ('horse', <Animal object at ...>),
 ('lion', <Animal object at ...>)]

More information

For many more details and examples of more kinds of grokkers, please see src/martian/core.txt. For more information on directives see src/martian/directive.txt.

CHANGES

2.0.post1 (2023-03-23)

• Add missing python_requires in setup.py to prevent installing on too old versions.

2.0 (2023-03-23)

• Add support for Python 3.11.

• Drop support for Python 2.7, 3.5, 3.6.

• Fix test suite incompatibility with zope.interface >= 6.0.

1.5 (2022-02-11)

• Title directive now correctly handles non-ascii characters.

• Fix test suite incompatibility with zope.interface >= 5.0.

• Drop support for Python 3.4.

• Add support for Python 3.9 and 3.10.

1.4 (2020-02-23)

• Check for builtins (Python 3) everywhere that we check for __builtin__ (Python 2).

1.3.post1 (2019-03-14)

• Fix rendering of PyPI page.

1.3 (2019-03-14)

• Add support for Python 3.7 and 3.8.

1.2 (2018-05-09)

• Add a new directive martian.ignore() to explicitly not grok something in a module:

  class Example:
      pass
  
  martian.ignore('Example')

• Fix the code to be pep 8 compliant.

1.1 (2018-01-25)

• Bypass bootstrap, add coverage to tox

• Fix inspect.getargspec() deprecation in python3

1.0 (2017-10-19)

• Add support for Python 3.5, 3.6, PyPy2 and PyPy3.

• Drop support for Python 2.6 and 3.3.

0.15 (2015-04-21)

• compatibility for python 3

• adjust egg to work with newer version of setuptools

• Fix an encoding issue under Python-2.7 in the doctests.

0.14 (2010-11-03)

Feature changes

• The computation of the default value for a directive can now be defined inside the directive class definition. Whenever there is a get_default classmethod, it is used for computing the default:

  class name(Directive):
      scope = CLASS
      store = ONCE
  
      @classmethod
      def get_default(cls, component, module=None, **data):
         return component.__name__.lower()

  When binding the directive, the default-default behaviour can still be overriden by passing a get_default function:

  def another_default(component, module=None, **data):
     return component.__name__.lower()
  
  name.bind(get_default=another_default).get(some_component)

  Making the default behaviour intrinsic to the directive, prevents having to pass the get_default function over and over when getting values, for example in the grokkers.

0.13 (2010-11-01)

Feature changes

• Ignore all __main__ modules.

• List zope.testing as a test dependency.

0.12 (2009-06-29)

Feature changes

• Changes to better support various inheritance scenarios in combination with directives. Details follow.

• CLASS_OR_MODULE scope directives will be aware of inheritance of values that are defined in module-scope. Consider the following case:

  module a:
    some_directive('A')
    class Foo(object):
      pass
  
  module b:
    import a
    class Bar(a.Foo):
      pass

  As before, Foo will have the value A configured for it. Bar, since it inherits from Foo, will inherit this value.

• CLASS_OR_MODULE and CLASS scope directives will be aware of inheritance of computed default values. Consider the following case:

  module a:
    class Foo(object):
       pass
  
  module b:
    import a
    class Bar(a.Foo):
       pass
  
  def get_default(component, module, **data):
      if module.__name__ == 'a':
         return "we have a default value for module a"
      return martian.UNKNOWN

  When we now do this:

  some_directive.bind(get_default=get_default).get(b.Bar)

  We will get the value “we have a default value for module a”. This is because when trying to compute the default value for Bar we returned martian.UNKNOWN to indicate the value couldn’t be found yet. The system then looks at the base class and tries again, and in this case it succeeds (as the module-name is a).

• martian.ONCE_IFACE storage option to allow the creation of directives that store their value on zope.interface interfaces. This was originally in grokcore.view but was of wider usefulness.

Bugs fixed

• Ignore things that look like Python modules and packages but aren’t. These are sometimes created by editors, operating systems and network file systems and we don’t want to confuse them.

• Ignore .pyc and .pyo files that don’t have a matching .py file via module_info_from_dotted_name if its ignore_nonsource parameter is True. The default is True. To revert to the older behavior where .pyc files were honored, pass ignore_nonsource=False.

• Pass along exclude_filter (and the new ignore_nonsource flag) to ModuleInfo constructor when it calls itself recursively.

• Replace fake_import to import fake modules in tests with a real python import statement (from martiantest.fake import my_fake_module). This works by introducing a metaclass for FakeModule that automatically registers it as a module. The irony does not escape us. This also means that martian.scan.resolve() will now work on fake modules.

0.11 (2008-09-24)

Feature changes

• Added MULTIPLE_NOBASE option for directive store. This is like MULTIPLE but doesn’t inherit information from the base class.

0.10 (2008-06-06)

Feature changes

• Add a validateClass validate function for directives.

• Moved FakeModule and fake_import into a martian.testing module so that they can be reused by external packages.

• Introduce new tutorial text as README.txt. The text previously in README.txt was rather too detailed for a tutorial, so has been moved into core.txt.

• Introduce a GrokkerRegistry class that is a ModuleGrokker with a MetaMultiGrokker in it. This is the convenient thing to instantiate to start working with Grok and is demonstrated in the tutorial.

• Introduced three new martian-specific directives: martian.component, martian.directive and martian.priority. These replace the component_class, directives and priority class-level attributes. This way Grokkers look the same as what they grok. This breaks backwards compatibility again, but it’s an easy replace operation. Note that martian.directive takes the directive itself as an argument, and then optionally the same arguments as the bind method of directives (name, default and get_default). It may be used multiple times. Note that martian.baseclass was already a Martian-specific directive and this has been unchanged.

• For symmetry, add an execute method to InstanceGrokker.

0.9.7 (2008-05-29)

Feature changes

• Added a MethodGrokker base class for grokkers that want to grok methods of a class rather than the whole class itself. It works quite similar to the ClassGrokker regarding directive definition, except that directives evaluated not only on class (and possibly module) level but also for each method. That way, directives can also be applied to methods (as decorators) in case they support it.

0.9.6 (2008-05-14)

Feature changes

• Refactored the martian.Directive base class yet again to allow more declarative (rather than imperative) usage in grokkers. Directives themselves no longer have a get() method nor a default value factory (get_default()). Instead you will have to “bind” the directive first which is typically done in a grokker.

• Extended the ClassGrokker baseclass with a standard grok() method that allows you to simply declare a set of directives that are used on the grokked classes. Then you just have to implement an execute() method that will receive the data from those directives as keyword arguments. This simplifies the implementation of class grokkers a lot.

0.9.5 (2008-05-04)

• scan_for_classes just needs a single second argument specifying an interface. The support for scanning for subclasses directly has been removed as it became unnecessary (due to changes in grokcore.component).

0.9.4 (2008-05-04)

Features changes

• Replaced the various directive base classes with a single martian.Directive base class:

  • The directive scope is now defined with the scope class attribute using one of martian.CLASS, martian.MODULE, martian.CLASS_OR_MODULE.

  • The type of storage is defined with the store class attribute using one of martian.ONCE, martian.MULTIPLE, martian.DICT.

  • Directives have now gained the ability to read the value that they have set on a component or module using a get() method. The class_annotation and class_annotation_list helpers have been removed as a consequence.

• Moved the baseclass() directive from Grok to Martian.

• Added a martian.util.check_provides_one helper, in analogy to check_implements_one.

• The scan_for_classes helper now also accepts an interface argument which allows you to scan for classes based on interface rather than base classes.

Bug fixes

• added dummy package_dotted_name to BuiltinModuleInfo. This allows the grokking of views in test code using Grok’s grok.testing.grok_component without a failure when it sets up the static attribute.

• no longer use the convention that classes ending in -Base will be considered base classes. You must now explicitly use the grok.baseclass() directive.

• The type check of classes uses isinstance() instead of type(). This means Grok can work with Zope 2 ExtensionClasses and metaclass programming.

0.9.3 (2008-01-26)

Feature changes

• Added an OptionalValueDirective which allows the construction of directives that take either zero or one argument. If no arguments are given, the default_value method on the directive is called. Subclasses need to override this to return the default value to use.

Restructuring

• Move some util functions that were really grok-specific out of Martian back into Grok.

0.9.2 (2007-11-20)

Bug fixes

• scan.module_info_from_dotted_name() now has special behavior when it runs into __builtin__. Previously, it would crash with an error. Now it will return an instance of BuiltinModuleInfo. This is a very simple implementation which provides just enough information to make client code work. Typically this client code is test-related so that the module context will be __builtin__.

0.9.1 (2007-10-30)

Feature changes

• Grokkers now receive a module_info keyword argument. This change is completely backwards-compatible since grokkers which don’t take module_info explicitly will absorb the extra argument in **kw.

0.9 (2007-10-02)

Feature changes

• Reverted the behaviour where modules called tests or ftests were skipped by default and added an API to provides a filtering function for skipping modules to be grokked.

0.8.1 (2007-08-13)

Feature changes

• Don’t grok tests or ftests modules.

Bugs fixed

• Fix a bug where if a class had multiple base classes, this could end up in the resultant list multiple times.

0.8 (2007-07-02)

Feature changes

• Initial public release.