Skip to main content

Python EXecution ENvironment, scheduler included

Project description

Python EXecution ENvironment

This is a (fairly) simple framework for building anything that requires some kind of ordered execution, such as a build system, a test suite, or similar.

How to use it

See further topics (in this order):

  1. Scheduler
  2. Task metadata
  3. Factory

Current state

The project currently sits idle until I find a use for it. It is not under very active development, but it's also not abandonware.

Feel free to fork it and make enhancements if you wish.

No APIs, names or anything is currently declared as stable, if you use this project in your projects, expect having to rename/rework things some time in the future.

Background

The idea came from limitations of various (non-Python) test suites that used static (ini) files for configuration, limiting options for dynamic execution, not having parallel execution support (or a crappy make-job-like one), etc.

Thus the goal was to create a system which would support

  1. Tree-based structure of tests, no single (flat) or double (main/sub) levels
  2. Dependency tracking and resolution between tests
  3. Parallel execution of tests with resolved dependencies
  4. Dynamic runtime configuration (selecting tests based on OS, etc.)
  5. Multiple results per one test (ie. 'pass', but with problems detected)
  6. Persistent state keeping (for re-runs of tests, for OS reboot)
  7. Some watchdog for keeping test runtime in check
  8. etc.

and the idea was to utilize python as a "configuration" language.

Generalizing this, the above formed into

  1. A "factory-time" logic, which would traverse a tree structure and generate the tests (a.k.a. execution units, python callables) based on some input
  2. An "execution-time" logic, which would use a scheduler to efficiently run the collected tests

Further generalizing, it ultimately turned into an universal execution engine info which you can feed any python callables and, given the right metadata, they get scheduled and run in the right order, with the "factory-time" being essentially just an example way of how to generate execution units.

Limitations

Needs python 3.6+, mostly due to quality-of-life features. Functionally, it could be ported to python 2.7 as it currently uses only basic threading and multiprocessing features (eg. no futures), at the expense of code readability.

  • Parallel result pipelining
    • Wish
      • A thread-safe process-safe interface to get results from running tasks
      • The user can then spawn a separate custom pool of threads/processes that query this interface and post-process results returned from tasks, in parallel
        • While other tasks are still running, not after everything finishes
    • Why Not
      • There are process-unsafe metadata returned to the user, namely the callable objects that were executed, exception objects, etc.
        • These are safe to pass only to the thread/process that added the task
      • There is a simple workaround: simply take care of the metadata in the thread that added the tasks and pass on picklable data (ie. retvals from the executed callables) to the custom process pool for further post-processing

Wishlist

  • Scheduling groups of exclusivity

    • Pass groups of callables instead of just callables to the scheduler
    • Have only one group running at a time
      • As it finishes, destroy the worker pool, create a new one for the new group
    • Allows to specify nr. of workers/spare and pool type (thread/mp) used on a per group basis!
    • Useful when a set of tasks is very different from the rest, ie. testing virtual machines (5 workers) versus testing syscalls (100 workers), and one global setting cannot cover both.
    • Also useful to logically split mutually exclusive things that need uninterrupted / unconditional access to the OS without defining "claims" to lots of tests.
    • Also useful for tasks that should logically run "together", possibly needing sequential execution due to formal requirements.
  • Adding new tasks on-the-fly

    • While processing results from the Sched.run() iterator, add new tasks
    • Useful when the overall task set depends on result(s) from some tasks
    • pexen.sched.pool support already in place
      • But proper Sched support would need heavy rewrites, removal of sanity checks, etc.
        • I did the rewrite; took 6-8 hours and the code was incomprehensible
        • Refactoring of the whole pexen.sched would be needed
  • A post-processor for factories

    • A task that requires some generic resource, ie. httpd
    • The post-processor would find (somehow) another, best suitable, task that provides it, adding it to the tasks for execution
    • Similar to how Debian packages can depend on a virtual package like mail-transport-agent and have multiple packages (exim4, postfix, etc.) provide it
  • "UseCases" that would glue Factory and Scheduler

    • Ie. "test suite UseCase" for running a test suite, collecting and formatting results, storing xUnit results in a file, supporting resume from an interrupted execution, etc.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pexen-0.0.3.tar.gz (15.6 kB view details)

Uploaded Source

Built Distribution

pexen-0.0.3-py3-none-any.whl (16.6 kB view details)

Uploaded Python 3

File details

Details for the file pexen-0.0.3.tar.gz.

File metadata

  • Download URL: pexen-0.0.3.tar.gz
  • Upload date:
  • Size: 15.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.4

File hashes

Hashes for pexen-0.0.3.tar.gz
Algorithm Hash digest
SHA256 c0d76ee8714a1098f6cc8183ffcbdccdfaf4a34d56f9f1318f5549de80b481b7
MD5 8b77975eb66813979afd3ddd62ab4b3e
BLAKE2b-256 ef1b3976217090b7cad1b46fa6220cd4f175060c16d9a934d90100f69d18279c

See more details on using hashes here.

File details

Details for the file pexen-0.0.3-py3-none-any.whl.

File metadata

  • Download URL: pexen-0.0.3-py3-none-any.whl
  • Upload date:
  • Size: 16.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.4

File hashes

Hashes for pexen-0.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 35d23ac5dcf4a1d0a40734bd10ece085af5451ff314106e1ede1b553c6787950
MD5 e912258386584fb95e7578958fcfc21f
BLAKE2b-256 d12d57919cc9575c670f4d3329afa0f633d06d13ad5b10e56f48890b6236d1b6

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page