red-dwarf 0.4.0

A DIMensional REDuction library for reproducing and experimenting with Polis-like data pipelines.

Red Dwarf: A Pol.is-like library

   ⚫⋆✦⋆⭑⋆⋆⋆⋆⋆         

A DIMensional REDuction library for reproducing and experimenting with Polis-like data pipelines.

Red dwarf stars are dim red stars. They are hard to see, but are the most common type -- the quiet 70% majority. They burn slow, using fuel efficiently, making them the longest-living stars in the universe. They'll be around until the last light, even supporting habitable planets for billions of years. If there's persistant life out there, it's likely in a long slow dance with a red dwarf.

Features

• Loads data from any Polis conversation on any Polis server, using only the conversation URL.
• Reproduces Polis calculation pipeline from only raw vote data.
  • "Classic" Polis pipeline = PCA dimensional reduction, KMeans clustering, and comment statistics.
• Alternative algorithms, aspiring for sensible defaults:
  • dimensional reduction: PaCMAP & LocalMAP
    • Planned: UMAP, TriMap, PHATE, ivis, LargeVis
  • clustering: HDBSCAN*
    • Planned: EVOC
• Helpful visualizations via matplotlib
  • Planned: Plotly

Goals

Code that aspires to embody and support democratic values should be...

• Legible. It should be explorable and auditable not just to researchers, but to as many curious citizens as possible.
• Re-usable. It should be easily used in contexts in which its original creators did not anticipate, nor perhaps even desire.
• Collectively stewarded. It should grow and govern itself democratically and in a plural fashion.

Roadmap

For now, see this related issue

Sponsors

Red Dwarf is generously sponsored by ZKorum SAS, creators of the Agora Citizen Network.

Are you or your organization eager to see more platforms and community built around democracy-supporting algorithms like these? Please consider getting in touch on Discord and supporting our continued work! (ping @patcon)

Usage

# OFFICIAL RELEASES

# For core, the classic polis pipeline: PCA + K-means
# (~60MB beyond scikit-learn disk space)
pip install red-dwarf

# For additional algorithms beyond classic polis: PaCMAP, LocalMAP, HDBSCAN, etc.
pip install red-dwarf[alt-algos]

# For additional packages for visualizing plots
pip install red-dwarf[plots]

# For everything (~60MB beyond core packages)
pip install red-dwarf[all]
# pip install red-dwarf[alt-algos,plots]

See docs/notebooks/polis-implementation-demo.ipynb or docs/notebooks/ for other examples.

screenshot of library-generated notebook	screenshot of Polis-generated report

• docs/notebooks/loading-data.ipynb
• docs/notebooks/heatmap.ipynb
• docs/notebooks/polis-implementation-demo.ipynb
• docs/notebooks/dump-downloaded-polis-data.ipynb
• Advanced
  • docs/notebooks/map-xids.ipynb
  • docs/notebooks/alternative-algorithms.ipynb
  • docs/notebooks/strip-pass-votes.ipynb
  • docs/notebooks/untested/tabpfn-experiment.ipynb

Architecture

This is the generalized pipeline of Polis-like processes that we're aiming to accomodate. (See issue #53 for details.)

Get Involved

Running it for local development

• Install python (preferrably virtual environment)
• Install uv (python package manager) (e.g. pip install uv)
• Install dependencies with make install-dev
• Run make command alone to see other helpful make subcommands
• Alternatively, run one of the ipynb notebooks; possibly replacing the install command at the top with %pip install -e ../../ to use the local source instead.

Contributing

• Join the Polis User Group (PUG) Discord server.
• Open a GitHub issue.
• Submit a GitHub pull request.
• Review the Awesome Polis directory to learn about related projects, academic papers, and other resources.
  • Use the "People" section to find other individuals and groups working in the field.

Changelog

See CHANGELOG.md.

License

MPL 2.0: Mozilla Public License 2.0 (See LICENSE)