Skip to main content

xDSL

Project description

Build Status for the Core backend PyPI version Downloads Downloads Code Coverage Zulip Status

xDSL: A Python-native SSA Compiler Framework

xDSL is a Python-native compiler framework built around SSA-based intermediate representations (IRs). Users of xDSL build a compiler by assembling predefined domain-specific IRs and, optionally, defining their own custom IRs. xDSL uses multi-level IRs, meaning that during the compilation process, a program will be lowered through several of these IRs. This allows the implementation of abstraction-specific optimization passes, similar to the structure of common DSL compilers (such as Devito, Psyclone, and Firedrake). To simplify the writing of these passes, xDSL uses a uniform data structure based on SSA, basic blocks, and regions, which additionally enables the writing of generic passes.

The design of xDSL is influenced by MLIR, a compiler framework developed in C++, that is part of the LLVM project. An inherent advantage of a close design is the easy interaction between the two frameworks, making it possible to translate abstractions and programs back and forth. This results in one big SSA-based abstraction ecosystem that can be worked with through Python, making analysis through simple scripting languages possible. Additionally, xDSL can leverage MLIR's code generation and low-level optimization capabilities.

Installation

To use xDSL as part of a larger project for developing your own compiler, just install xDSL via pip:

pip install xdsl

Note: This version of xDSL is validated against a specific MLIR version, interoperability with other versions may result in problems. The supported MLIR version is 20.1.7.

Subprojects With Extra Dependencies

xDSL has a number of subprojects, some of which require extra dependencies. In order to keep the set of dependencies to a minimum, these extra dependencies have to be specified explicitly. To install these, use:

pip install xdsl[gui,jax,riscv]

To install the testing/development dependencies, use:

pip install xdsl[dev]

These may be useful for projects wanting to replicate the xDSL testing setup.

Getting Started

Check out the dedicated Getting Started guide for a comprehensive tutorial.

To get familiar with xDSL, we recommend starting with our Jupyter notebooks. The notebooks consist of examples and documentation concerning the core xDSL data structures and the xDSL's Python-embedded abstraction definition language, as well as examples of implementing custom compilers, like a database compiler. There also exists a small documentation showing how to connect xDSL with MLIR for users interested in that use case.

We provide a Makefile containing a lot of common tasks, which might provide an overview of common actions.

xDSL Developer Setup

To contribute to the development of xDSL follow the subsequent steps.

Developer Installation

We use uv for dependency management of xDSL. Getting started documentation can be found here, and is also printed by the make uv-installed and make venv targets if it is not already installed on your system.

git clone https://github.com/xdslproject/xdsl.git
cd xdsl
# set up the venv and install everything
make venv

To make a custom mlir-opt available in the virtual environment, set the XDSL_MLIR_OPT_PATH variable when running make venv, like so:

XDSL_MLIR_OPT_PATH=/PATH/TO/LLVM/BUILD/bin/mlir-opt make venv

If you can't use uv

For some systems and workflows, changing to a new dependency management system may be inconvenient, impractical, or impossible. If this is the case for you, xDSL can still be installed using pip.

To create the required virtual environment (the equivalent of make venv):

python -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

The following commands can then be run using this virtual environment rather than uv by running source venv/bin/activate when starting a new shell, then eliding the uv run prefix from the commands. For example, to run the commands in the following testing section:

source venv/bin/activate
pytest
lit tests/filecheck

Testing and benchmarking

The xDSL project uses pytest unit tests and LLVM-style filecheck tests. They can be executed from the root directory:

# Executes pytests which are located in tests/
uv run pytest

# Executes filecheck tests
uv run lit tests/filecheck

# run all tests using makefile
make tests

Benchmarks for the project are tracked in the https://github.com/xdslproject/xdsl-bench repository. These run automatically every day on the main branch, reporting their results to https://xdsl.dev/xdsl-bench/. However, they can also be run manually by cloning the repository and pointing the submodule at your feature branch to benchmark.

Formatting and Typechecking

All python code used in xDSL uses ruff to format the code in a uniform manner.

To automate the formatting, we use pre-commit hooks from the pre-commit package.

# Install the pre-commit on your `.git` folder
make precommit-install
# to run the hooks:
make precommit
# alternatively, run ruff directly:
uv run ruff format

Furthermore, all python code must run through pyright without errors. Pyright can be run on all staged files through the makefile using make pyright.

[!IMPORTANT]

Experimental Pyright Features

xDSL currently relies on an experimental feature of Pyright called TypeForm TypeForm is in discussion and will likely land in some future version of Python.

For xDSL to type check correctly using Pyright, please add this to your pyproject.toml:

[tool.pyright]
enableExperimentalFeatures = true

Discussion

You can also join the discussion at our Zulip chat room, kindly supported by community hosting from Zulip.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

xdsl-0.43.0.tar.gz (1.5 MB view details)

Uploaded Source

Built Distribution

xdsl-0.43.0-py3-none-any.whl (1.7 MB view details)

Uploaded Python 3

File details

Details for the file xdsl-0.43.0.tar.gz.

File metadata

  • Download URL: xdsl-0.43.0.tar.gz
  • Upload date:
  • Size: 1.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.13

File hashes

Hashes for xdsl-0.43.0.tar.gz
Algorithm Hash digest
SHA256 639a454aa2f69f37c9a43618cd53f23086e818541dbfb203744cd261b6d7e5dc
MD5 ec52acfd832923821f11670fd34554b8
BLAKE2b-256 6b9ed0896e94dc5923a62e1a495a5dc709dc58f189aeda9faa4d8ce5d9d30497

See more details on using hashes here.

File details

Details for the file xdsl-0.43.0-py3-none-any.whl.

File metadata

  • Download URL: xdsl-0.43.0-py3-none-any.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.11.13

File hashes

Hashes for xdsl-0.43.0-py3-none-any.whl
Algorithm Hash digest
SHA256 bae25d55ed0bed789cf0868aa8567d19025cd6b2f23b95a248f603392146329d
MD5 1c3fdc72cf69fbb3ea3dcc470b54c694
BLAKE2b-256 657ef3c8263d9049fd741a228647eebe596c626a34f6288e373657f0cfb20c8f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page