pmss 0.0.2

Cascading Python settings

Preference Management Style Sheets

This is intended as a language for settings. The name might be a placeholder. Or final. Who knows?

Why? WHY???

The raison d'etre is that settings cascade and override. In most cases, we would like to have (e.g. for a learning management system):

• Rosters from Google Classroom
• Except for Central Valley School District, which uses Instructure
• Except for Maplewood Elementary School, which uses Canvas
• Except for Mrs. Johnson's afterschool program, which keeps rosters in a Google Spreadsheet, by virtue of excepting the general community and not just district students.

We'd also like to have:

• System defaults
• Overridden by system settings in /etc/foo
• Overridden by user settings in ~/.foo
• Overridden by environment variables
• Overridden by settings in a database
• Overridden by command line parameters and settings set within the program

In most cases, this generates a spaghetti-like set of special cases and one-offs as systems reach maturity. Very few systems have good ways for settings to cascade. The ones which do create custom ad-hoc languages.

What does cascade? CSS. It's a well-specificed, well-documented syntax. As a result, we use a subset of CSS syntax as our native format.

Mindful of backwards-compatibility, it is possible to integrate PMSS into most systems piecemeal. We also have an opinioned, streamlined API which will hopefully make it easy to get started.

What else is needed?

• Validation and reasonable errors
  • Robust typing, with conversions and validation
  • Validation all required fields are provided
  • Above, considering contexts
  • Validation all fields are declared
  • Above, for classes and attributes
• Human-friendly (and ideally, standardized) formats. We don't want to reinvent yaml / json / XML / etc.
• Comments
• Interpolation / DRY / single source of truth
  • If system_dir is /opt/system, we'd like to be able to set tokens to e.g. ${system_dir}/tokens.rsa
  • Some systems take this further to full Turing-complete automation
  • THIS IS DANGEROUS IF SETTINGS CAN COME FROM A USER, so it should be behind a flag. Otherwise, you'll have background_color set to ${secret_id} ${access_key} ${password}.
• Multiple file support. Large config files should be able to break down into smaller ones
  • This should be configurable and safe. We don't want someone to include /etc/passwd or similar.
• Security. We'd like to be immune to things like injection attacks, which do come up if setting come from e.g. web forms.
• Some support for configuring plug-ins / modules
  • This probably means some level of successive loading, with setting available before the full system is loaded.
• Lists / aggregations. E.g. if we want to set up a list of modules to import, each with its own settings.
• Classes provided on the command line
• Dynamic reloading. If a file changes, settings update without a restart. Note this is currently more useful for development than deployment until we have more safety checks.
  • Robust validation and error handling on a reload.
• Robust test infrastructure
• Nice documentation
• pypi package (this might need a rename, since pmss is taken!)
• INI file support, to give an example of how to handle backwards-compatibility
• Better error messages
• Nicer usage()
• Selectors on the commandline
• Classes on the commandline

Note that we don't have all of this yet.

Simplified API

From Python:

import pmss

settings = pmss.init(
    prog="lo",
    description="A system for monitoring",
    epilog="For more information, see PMSS documentation."
)

pmss.register_field(
    name="server_port",
    command_line_flags=["-p", "--port"],
    type=pmss.TYPES.port,
    description="The port our system should run on.",
    default=8888
)

pmss.register_field(
    name="hostname",
    type=pmss.TYPES.hostname,
    description="The hostname",
    required=True
)

pmss.validate(settings)
normal_port = settings.server_port()
exception_port = settings.server_port(attributes={'school': 'middlesex'})

Our settings file (can be placed in the standard locations):

* {
    _biff: baz;
    hostname: testhost;
    server_port: 80;
}

[school=middlesex] {
    server_port: 8080;
}

And from the commandline:

python example.py --port=90 --hostname=testhosttwo

What's the model

We're basing this on CSS. Why?

• They cascade
• They're well-specified and libraries exist
• Many people know CSS

The downside, of course, is that many people find CSS a bit complex.

Conceptually, we plan to have series of rule sets. E.g.:

    add_ruleset(args=sys.argsv)
    add_ruleset(file="~/.settings.pmss", name="User settings file", classes=["settings_file"], interpolation=True)
    add_ruleset(environ=sys.environ, id="environ")
    delete_ruleset(id="environ")

We also plan to have series of schemas, registered in code. Modules can define schemas too. For example, a cloud service module could define that it wants API keys.

Once everything is loaded, we can run:

    validate()

Note that settings are usable (just not guaranteed correct) before validation, as e.g. command line arguments might point to a settings file and a series of modules to load, and validation might be impossible until those are loaded.

We're trying to make this batteries included, so by default, the rulesets look in the standard locations (~, /etc, etc.) and on the commandline.

We do not look at parent / child relationships, so the following are both okay:

    auth {
        google {...}
        microsoft {...}
        github {...}
    }
    rosters {...}

or:

   google {
       auth {...}
       rosters {...}
       docs {...}
   }

   microsoft {...}

For debugging, we can have fine-grained settings

if settings.verbose(types=['pmss', 'selectors']):
   print("....")

And we can decide what to make verbose from the command line whether to run --verbose=True or --pmss:verbose=True or otherwise.

Likewise, we can define classes for e.g. prod and dev:

.prod {
    database: postgresql;
}

.dev {
    database: sqlite;
}

Files

• pmsslex.py and pmssyacc.py are the lexer and parser for our variant of CSS. These are designed to be interchangeable if you'd like e.g. full CSS or an INI file or whatnot.
• pmsstypes.py handles type validation and conversion. We use other libraries where possible, to maintain compatible formats. Conversely, this is usable without the rest of the system.

Who else has thought about this?

• Build tools
  • Bazel
• Web servers
  • nginx has a similar set of constraints around routes
• Command line parsers
  • optparse
  • docopt
  • click
• Configuration languages
  • Dhall has very nice tooling which can visualize if a change to a file led to a change in output configuration, as well as diffs to see what the change was.
  • configparser is the Python default, very much based on INI files. Good ideas include interpolation.
• Schemas. There's a whole theory of types, which is out-of-scope for this list (but in-scope for a grad-level course).
  • The best schema definition language seems to be Cue which is in Go, but has basic Python integration. We should probably use this verbatim.
  • XML DTDs
  • JSON Schema
  • Traitlets thought a little bit about validation, although it seems to mix models in a way I don't like.
• CSS does most of what we want
  • tinycss2 seems like the right parser to work from.
  • SASS builds on it with useful shortcuts. Should we adopt SASS extensions?
• Configuration languages
  • pkl was recently released by Apple. It seems to have some very nice ideas, but misses a lot as well. See also HN discussion. Well worth reviewing, if not adopting.
  • gcl has nice copy about configuration. The language itself is very Turing-complete, which makes it a bit awkward. It's a little like having a code file, config.py.
  • hcl Seems nice.
  • JSonnet has nice ideas, but seems to confuse templates and inheritence. Cue, above, is a reaction to this confusion.