omlish 0.0.0.dev476

omlish

Overview

Core utilities and foundational code. It's relatively large but completely self-contained, and has no required dependencies of any kind.

Notable packages

• lang - The standard library of this standard library. Usually imported as a whole (from omlish import lang), it contains an array of general purpose utilities used practically everywhere. It is kept relatively lightweight: its heaviest import is stdlib dataclasses and its transitives. Some of its contents include:

  • cached - The standard cached_function / cached_property tools, which are more capable than functools.lru_cache.
  • imports - Import tools like:
    • proxy_import - For late-loaded imports.
    • proxy_init - For late-loaded module globals.
    • auto_proxy_init - For automatic late-loaded package exports.
  • classes - Class tools and bases, such as Abstract (which checks at subclass definition not instantiation), Sealed / PackageSealed, and Final.
  • maybes - A simple, nestable formalization of the presence or absence of an object, as in many other languages.
  • maysync - A lightweight means of sharing code between sync and async contexts, eliminating the need for maintaining sync and async versions of functions.

• bootstrap - A centralized, configurable, all-in-one collection of various process-initialization minutiae like resource limiting, profiling, remote debugging, log configuration, environment variables, et cetera. Usable as a context manager or via its cli.

• collections - A handful of collection utilities and simple implementations, including:

  • cache - A configurable LRU / LFU cache with options like ttl and max size / weight.
  • hasheq - A dict taking an external __hash__ / __eq__ implementation.
  • identity - Identity-keyed collections.
  • sorted - Interfaces for value-sorted collections and key-sorted mappings, and a simple but correct skiplist-backed implementation.
  • persistent - Interfaces for persistent maps, and a simple but correct treap-backed implementation.

• dataclasses - A fully-compatible reimplementation of stdlib dataclasses with numerous enhancements and additional features. The full stdlib test suite is run against it ensuring compatibility - they are dataclasses. Current enhancements include:

  • Simple field coercion and validation.
  • Any number of @dc.init or @dc.validate methods, not just a central __post_init__.
  • Optional generic type parameter substitution in generated __init__ methods, enabling accurate reflection.
  • An optional metaclass which removes the need for re-decorating subclasses (with support for inheritance of dataclass parameters like frozen), and some basic base classes.
  • (Nearly finished) support for ahead-of-time / build-time code generation, greatly reducing import times.

  The stdlib-equivalent api is exported in such a way as to appear to be direct aliases for the stdlib api itself, simplifying tool support.

• dispatch - A beefed-up version of functools.singledispatch, most notably supporting MRO-honoring method impl dispatch.

• formats - Tools for various data formats, including:

  • json - Tools for json, including abstraction over various backends and a self-contained streaming / incremental parser.
  • json5 - A self-contained and tested Json5 parser.
  • toml - Toml tools, including a lite version of the stdlib parser (for use in older pythons).

• http - HTTP code, including:

  • clients - An abstraction over HTTP clients, with urllib and httpx implementations.
  • coro - Coroutine / sans-io style reformulation of some stdlib http machinery - namely http.server (and soon http.client). This style of code can run the same in sync, async, or any other context.

• inject - A guice-style dependency injector.

• io - IO tools, including:

  • compress - Abstraction over various compression schemes, with particular attention to incremental operation. For example it includes an incremental reformulation of stdlib's gzip.
  • coro - Utilities for coroutine / sans-io style code.
  • fdio - An implementation of classic selector-style IO dispatch, akin to the deprecated asyncore. While more modern asyncio style code is generally preferred, it nearly always involves background threads making it unsuitable for forking processes like process supervisors.

• jmespath - A vendoring of jmespath community edition, modernized and adapted to this codebase.

• marshal - A jackson-style serde system.

• manifests - A system for sharing lightweight metadata within / across codebases.

• reflect - Reflection utilities, including primarily a formalization of stdlib type annotations for use at runtime, decoupled from stdlib impl detail. Keeping this working is notoriously difficult across python versions (one of the primary reasons for only supporting 3.13+).

• sql - A collection of SQL utilities, including:

  • api - An abstracted api for SQL interaction, with support for dbapi compatible drivers (and a SQLAlchemy adapter).
  • queries - A SQL query builder with a fluent interface.
  • alchemy - SQLAlchemy utilities. The codebase has moved away from SQLAlchemy in favor of its own internal SQL api, but it will likely still remain as an optional dep for the api adapter.

• testing - Test - primarily pytest - helpers, including:

  • 'harness' - An all-in-one fixture marrying it to the codebase's dependency injector.
  • plugins/async - An in-house async-backend abstraction plugin, capable of handling all of asyncio / trio / trio-asyncio / any-future-event-loop-impl without having multiple fighting plugins (I know, I know).
  • plugins - Various other plugins.

• lite - The standard library of 'lite' code. This is the only package beneath lang, and parts of it are re-exported by it for deduplication. On top of miscellaneous utilities it contains a handful of independent, self-contained, significantly simplified 'lite' equivalents of some major core packages:

  • lite/inject.py - The lite injector, which is more conservative with features and reflection than the core injector. The codebase's MiniGuice.
  • lite/marshal.py - The lite marshalling system, which is a classic canned setup of simple type-specific 2-method classes and limited generic handling.

Lite code

A subset of this codebase is written in a 'lite' style (non-'lite' code is referred to as standard code). While standard code is written for python 3.13+, 'lite' code is written for 3.8+, and is written in a style conducive to amalgamation in which multiple python source files are stitched together into one single self-contained python script.

Code written in this style has notable differences from standard code, including (but not limited to):

• No name mangling is done in amalgamation, which means (among other things) that code must be written expecting to be all dumped into the same giant namespace. Where a standard class might be omlish.inject.keys.Key, a lite equivalent might be omlish.lite.inject.InjectorKey.
• All internal imports import each individual item out of modules rather than importing the modules and referencing their contents. Where standard code would from .. import x; x.y, lite code would from ..x import y; y. As a result there are frequently 'api' non-instantiated namespace classes serving the purpose of modules - just handy bags of stuff with shortened names.
• As lite code is tested in 3.8+ but core code requires 3.13+, packages containing lite code can't import anything standard in their (and their ancestors') __init__.py's. Furthermore, __init__.py files are omitted outright in amalgamation, so they effectively must be empty in any package containing any lite code. As a result there are frequently all.py files in mixed-lite packages which serve the purpose of __init__.py for standard usage - where importing standard packages from standard code would be done via from .. import lang, importing mixed-lite packages from standard code would be done via from ..configs import all as cfgs.

Dependencies

This library has no required dependencies of any kind, but there are some optional integrations - see __about__.py for a full list, but some specific examples are:

• asttokens / executing - For getting runtime source representations of function call arguments, an optional capability of check.
• anyio - While lite code must use only asyncio, non-trivial async standard code prefers to be written to anyio.
• pytest - What is used for all standard testing - as lite code has no dependencies of any kind its testing uses stdlib's unittest.
• wrapt - For (optionally-enabled) injector circular proxies.
• sqlalchemy - The codebase has migrated away from SQLAlchemy in favor of the internal api but it retains it as an optional dep to support adapting the internal api to it.

Additionally, some catchall dep categories include:

• compression - Various preferred compression backends like lz4, python-snappy, zstandard, and brotli.
• formats - Various preferred data format backends like orjson/ujson, pyyaml, cbor2, and cloudpickle.
• sql drivers - Various preferred and tested sql drivers.