polugins 0.2.0

Plugin system for Polars.

Polugins

Early PoC for a "plugin" system for polars.

Not production ready - barely even alpha ready. Only uploading it now for discussion and to hog the genius package name.

It's meant to solve two issues with using polars API extensions:

• You need to import the namespace to trigger the registration, even if you dont need anything from the namespace module.

• Extensions breaks static typing.

The idea is to describe a single way to expose and collect API extensions - especially for third party packages - and then used this discoverbility to also generate type stubs with the added typing from the extensions.

Users can either call register_namespaces themselves or import polars through polugins.polars instead. Lint rules can then be used to enforce that nothing is imported from polars outside of these locations.

This is still a bit annoying no matter what, unless polars does the import natively.

Package example

Packages can expose namespace through entry points called polugins.<class>, forexample polugins.lazyframe.

If building with poetry you should add this to your pyproject.toml:

[tool.poetry.plugins."polugins.<class>"]
"<accessor_name>" = "<path.to.module:NameSpace>"

# Concrete example:

[tool.poetry.plugins."polugins.lazyframe"]
"external" = "example_package:PackageNamespace"

See tests/pkgs/example_package for a example.

Usage example

Namespaces can be registered in three ways:

• By module path
• As imported module
• From entry points

Say that you have a package my_package with two LazyFrame namespaces - MyNamespace and AlsoMyNamespace and you use an external package example-package that exposes a LazyFrame namespace called external. After installing it, namespaces can be registered like so:

from polugins import register_namespaces
import polars as pl
from my_package import MyNamespace

register_namespaces(
    lazyframe_namespaces={
        'my_namespace': MyNamespace,
        'also_my_namespace': "my_package:AlsoMyNamespace" # Note the `:` to separate module path from object
    },
    entrypoints=True # Loads from example-package
  )

# All namespaces are now registered
(
  pl.LazyFrame()
  .external.some_method()
  .my_namespace.some_method()
  .also_my_namespace.some_method()
)

You need to make sure that you have called register_namespaces before trying to use any of those namespaces.

As an alternative, polars is re-exported through polugins such that entrypoint namespaces are automagically registered:

from polugins import pl

pl.LazyFrame().external.some_method(x=1)

Note that this only registers entrypoint namespaces (for now).

Generate types

Run polugins.create_types.py to create type stubs. Will create the directory .typings.

Only works for entrypoint namespaces. You can add your own namespaces to the types if you want. It's fairly straight forward, just import the namespace on top of the pyi and then add an annotation to the class with the namespace.

Will be a CLI tool at some point.

Implementation

Just a thin wrapper around polars.api.register_x_namespace and then using importlib.metadata to collect namespaces from external packages.

Types are generated by using mypy to create stubs for lazyframe, dataframe, expr and series and then adding the namespaces to these type stubs.

Notes

It's still not entirely clear how an application should register its own namespaces. Entry points can be used but

• (1) an application might just want to use its own namespaces and not expose them and
• (2) its a bit annoying, because changes to entrypoints are only registered when the package is installed, even in editable mode (I think).