chrisjen

Version
Status
Documentation
Tools
Compatibility
Stats

What is chrisjen?

Named after Earth's unflappable leader in The Expanse, who knew how to get things done, chrisjen builds and runs project workflows from a plain settings file (or a Python dict). You list the parts of your project and the techniques that each part uses. chrisjen connects them into a workflow, applies it to your data, and, if you ask it to, compares alternatives and picks the best.

import dataclasses

import chrisjen


@dataclasses.dataclass
class DropNegatives(chrisjen.Technique):
    def apply(self, item, **kwargs):
        return [x for x in item if x >= 0]


@dataclasses.dataclass
class Double(chrisjen.Technique):
    def apply(self, item, **kwargs):
        return [x * 2 for x in item]


@dataclasses.dataclass
class Total(chrisjen.Technique):
    def apply(self, item, **kwargs):
        return sum(item)


settings = {
    "report_project": {"report_workers": "prepare, summarize"},
    "prepare": {"techniques": "drop_negatives, double"},
    "summarize": {"techniques": "total"},
}

project = chrisjen.Project.create(settings, item = [3, -1, 4, -1, 5])
print(project.result)
# 24

Why use chrisjen?

Intuitive

A chrisjen project is made of techniques, the actual actions, which are grouped into workers, the parts of a project (for example, "prepare" and "summarize"). A worker can also have steps, each with techniques of its own.

You describe them in an ini, toml, json, yaml, xml, or Python file (or a dict), and a Project does the rest. The same example as a file:

[report_project]
report_workers = prepare, summarize

[prepare]
techniques = drop_negatives, double

[summarize]
techniques = total

project = chrisjen.Project.create("report.ini", item = [3, -1, 4, -1, 5])
print(project.result)
# 24

chrisjen has no scripting language to learn. The settings file only names things. What those things do is plain Python.

Powerful

Each worker (and the project itself) has a design that decides how its parts are applied:

For example, to try three ways of scaling and keep whichever has the smallest spread, make the project a contest and name a criterion:

@dataclasses.dataclass
class Halve(chrisjen.Technique):
    def apply(self, item, **kwargs):
        return [x / 2 for x in item]


@dataclasses.dataclass
class SmallestSpread(chrisjen.Criteria):
    def score(self, item):
        return -(max(item) - min(item))


settings = {
    "compare_project": {
        "design": "contest",
        "criterion": "smallest_spread",
        "techniques": "double, halve, none",
    },
}
project = chrisjen.Project.create(settings, item = [1, 2, 3])
print(project.result)
# [0.5, 1.0, 1.5]
print(project.workflow.winner)
# halve

none is a built-in technique that does nothing, which is useful for testing whether a technique helps at all.

Flexible

chrisjen is built from small libraries that each do one job, and you can use any of them on its own:

A technique is an object that wraps any callable (or the import path of one), so a single interface can drive tools from many packages. Every class you define (techniques, designs, criteria, and reports) is added to a library under its snake case name as soon as it is defined, so it can be used in settings right away.

Getting started

Requirements

chrisjen requires Python 3.11 or later. It runs on Linux, macOS, and Windows. Its dependencies are installed automatically. If you want to load yaml or xml settings or use pandas file formats, also install the optional dependencies of those packages (for example, pip install pyyaml xmltodict pandas).

Installation

To install chrisjen, use pip:

pip install chrisjen

Usage

Techniques

A technique is a subclass of chrisjen.Technique. Either write an apply method, or set contents to the tool that the technique wraps: a callable, or its import path (which is only imported when the technique is used). The item is passed to the tool along with any keyword parameters that the tool accepts. Parameters come from a {name}_parameters section of your settings:

@dataclasses.dataclass
class Mean(chrisjen.Technique):
    contents: str = "statistics.fmean"


@dataclasses.dataclass
class Rounded(chrisjen.Technique):
    contents: object = round


settings = {
    "average_project": {"techniques": "mean, rounded"},
    "rounded_parameters": {"ndigits": 1},
}
print(chrisjen.Project.create(settings, item = [1, 2, 4]).result)
# 2.3

Workers and steps

A worker lists its techniques or its steps, and each step lists its techniques. In the default flow design, they are applied in order. In a contest or a survey, the techniques of each step are alternatives, and every combination of one technique from each step is tried:

settings = {
    "numbers_project": {"numbers_workers": "prepare"},
    "prepare": {
        "steps": "clean, resize",
        "clean_techniques": "drop_negatives",
        "resize_techniques": "double, halve",
    },
}
project = chrisjen.Project.create(settings, item = [1, -2, 3])
print(project.result)
# [1.0, 3.0]

Running a project

Project.create reads the settings, builds the workflow, and (unless automatic = False) applies it to item. The result is stored in result, and a report summarizes the run. apply runs the project again, on a new item if you pass one. Each run has an id, which is the name of the project and the date and time unless you pass one:

project = chrisjen.Project.create(
    "report.ini", id = "first_run", automatic = False)
print(project.apply([3, -1, 4]))
# 14
print(project.report.contents)
# project: report
# id: first_run
# paths: prepare > summarize
# result: 14

Files

Every project has a clerk, a nagata.FileManager. Pass the folder for the project's files as clerk (or pass a FileManager of your own):

import pathlib
import tempfile

root = pathlib.Path(tempfile.mkdtemp())
project = chrisjen.Project.create("report.ini", item = [1, 2], clerk = root)
project.clerk.save(project.result, file_name = "total", file_format = "pickle")
print(project.clerk.load(file_name = "total", file_format = "pickle"))
# 6

Export a workflow

Project.to_dot and Project.to_mermaid describe the workflow in Graphviz or mermaid text. Both are made by holden.

print(project.to_mermaid())
# ---
# title: report
# ---
# flowchart LR
#     prepare(prepare) --> summarize(summarize)

There is more to chrisjen, including criteria, custom designs, building workflows in code, and how the pieces fit together. See the documentation, especially the tutorial and the advanced user guide.

Contributing

Contributors are always welcome. Feel free to grab an issue to work on or make a suggested improvement. If you wish to contribute, please read the Contribution Guide and Code of Conduct.

Similar Projects

• airflow: Apache's workflow tool that is likely the market leader. It requires substantial overhead and has a learning curve but offers the greatest extensibility for non-Python workflow components and support for continuous, always-on workflows.
• jetstream: similar DAG workflow structures in pure Python with a greater emphasis on loading workflows from disk.
• luigi: Spotify's workflow tool with much greater overhead and support for controlling workflow nodes outside of Python.
• pathos: supports parallel workflow construction with heterogenuous computing framework. Among other features, it includes drop-in replacements for Python's pickle and multiprocess, called dill and multiprocessing, respectively.

License

Use of this repository is authorized under the Apache Software License 2.0.