python-amber

The Amber columnar time-series engine, in your Python process, with zero-copy NumPy / Arrow / pandas interop.

import amber as am

am.gentq(10_000_000)                                     # 10M synthetic trades + quotes
df = am.q("select vwap:wavg[sz;px], n:#px by sym from trades").to_pandas()

No server. No socket. No serialisation. am.eval("2+3") calls the same C function the Amber REPL calls, in the same address space, and a query result that comes back as a NumPy array is pointing at the engine's own memory — not at a copy of it.

>>> col = am.q("select from trades").column("px", raw=True)
>>> arr = numpy.asarray(col)
>>> hex(arr.ctypes.data) == hex(col.handle())            # the engine's buffer
True

Contents

• Install
• How it fits together
• Quick start
• The API
• Zero-copy, precisely
• Type mapping
• Building from source
• Building wheels
• Threading
• Troubleshooting
• Licence

Install

pip install amber-lang

The import is still import amber (the plain amber name on PyPI was already taken). Wheels exist for Linux x86_64 and aarch64 and for Apple silicon Macs, on CPython 3.10 to 3.14. Anywhere else pip builds from the sdist, which carries the engine's source, so all you need is a C compiler and bash.

A binary wheel carries libamber.so and the eight .k standard-library modules, so this is genuinely all you need — no engine checkout, no $AMBER_HOME, no build toolchain.

From source, against an engine checkout:

git clone https://github.com/BonucciAndrea/amber
git clone https://github.com/BonucciAndrea/python-amber
cd python-amber
AMBER_SRC=../amber pip install .

Optional extras, none of them required:

pip install "amber[numpy]"     # NumPy views of columns
pip install "amber[pandas]"    # .to_pandas()
pip install "amber[arrow]"     # .to_arrow(), .to_polars()
pip install "amber[all]"

They are optional because they are reached through ABI contracts, not Python imports: the buffer protocol (PEP 3118) for NumPy, the Arrow C Data Interface for PyArrow. This package pins neither and imports both lazily, at the moment you call the method that needs them.

How it fits together

   your Python process
   ┌──────────────────────────────────────────────────────────────┐
   │                                                              │
   │   import amber as am                                         │
   │        │                                                     │
   │        ▼                                                     │
   │   amber/__init__.py       dynamic dispatch, conversion       │
   │   amber/table.py          pandas / Arrow / Polars bridges    │
   │        │                                                     │
   │        ▼                                                     │
   │   amber/_amber.so         CPython extension  (src/py_amber.c)│
   │        │                  ~800 lines, no NumPy/Arrow headers │
   │        ▼                                                     │
   │   libamber.so             the engine, exporting 57 symbols   │
   │                           all named amber_* or am_ext_*      │
   │        │                                                     │
   │        ├──── column payload ────────────┐                    │
   │        │                                │  same address      │
   │        ▼                                ▼                    │
   │   numpy.ndarray                  pyarrow.Table               │
   └──────────────────────────────────────────────────────────────┘

The engine repository is not modified by this package and does not know it exists. The entire contract between them is section 6 of the engine's src/ext.h — mirrored here in include/amber_capi.h and checked at import time via amber_abi_version().

Quick start

import amber as am

# --- evaluate ------------------------------------------------------------
am.eval("2+3")                       # 5
am.eval("1 2 3 4 5")                 # array([1,2,3,4,5], dtype=int8)
am.eval('"hello"')                   # 'hello'
am.eval("`aapl`msft")                # ['aapl', 'msft']

# --- dynamic dispatch: any Amber global is an attribute -------------------
am.gentq(1_000_000)                  # runs `gentq 1000000` in the engine
am.wavg([1, 2, 3], [10.0, 20.0, 30.0])
am.u.pub                             # dotted names work: the `u.pub` global

# --- qSQL, exactly as you'd type it at the prompt -------------------------
t = am.q("select vwap:wavg[sz;px], n:#px by sym from trades where px>0")
t.to_pandas()

# --- as-of join, the thing Amber is actually for --------------------------
am.eval("taq:aj[`sym`time; trades; quotes]")
am.q("select from taq where sym=`AAPL").to_arrow()

# --- push data back in ----------------------------------------------------
import numpy as np
am.set("signal", np.random.randn(1_000_000))
am.eval("avg signal")

am.set("book", {"sym": ["AAPL", "MSFT"], "px": [187.2, 411.1]})

The API

Module level

raw=True skips conversion and hands back a Value. Use it whenever the next thing you do is give the result back to the engine — conversion is the only step in the whole path that costs anything.

am.Value

An owned handle on an engine value. Flat vectors export the buffer protocol, so numpy.asarray(v), memoryview(v) and pyarrow.py_buffer(v) are all zero-copy and all keep the engine value alive for exactly as long as they exist.

.type() .type_name() .count() .is_table() .is_dict() .is_atom() .flatten() .format(plain=True) .to_str() .to_symbols() .scalar() .ncols() .colname(i) .column(i) .keys() .values() .items() .symbol_codes() .equals(v) .to_bytes() .release() .is_released() .handle()

Values pickle (through the engine's own serialiser), so they cross multiprocessing fine. .release() hands the memory back right away instead of waiting for the GC, and it says no while a NumPy view of it is still alive. Everything else on a released value raises instead of crashing.

am.Table

A REPL

$ python -m amber
amber> gentq 1000
amber> select vwap:wavg[sz;px] by sym from trades
amber> \py df = am.q("select from trades").to_pandas()

Zero-copy, precisely

Claims about zero copy are cheap, so here is exactly what is and is not copied.

Engine → Python is zero-copy for numeric columns. amber_get_vector_ptr() returns the payload pointer; the buffer protocol wraps it with view->obj set to the owning Value, INCREF'd. NumPy stores that as the array's base, so the Amber refcount cannot reach zero while the array is alive. The test suite asserts pointer equality, not value equality:

assert arr.ctypes.data == raw.handle()
assert arrow_buffer.address == raw.handle()

Engine → Arrow is zero-copy for numeric columns. The ArrowArray's data buffer is the Amber column's payload, and its release callback drops the Amber refcount. Symbol columns are materialised to utf8 — Amber stores interned int32 ids, Arrow wants offsets and bytes, and no pointer arithmetic bridges that.

Python → engine copies. Always. Amber stores a 32-byte object header immediately before every vector payload, so it cannot adopt a foreign allocation as a native vector. That asymmetry is not a limitation anyone is working around; it matches where the volume is. Query results are large and query parameters are small.

pandas is "where pandas permits". to_pandas() builds the DataFrame from the same views. Whether pandas keeps them or consolidates them into fresh blocks is pandas' decision and it varies by version, so this package documents that honestly rather than promising something it does not control. Pass copy=True to take a deliberate snapshot and stop pinning the engine's buffers.

Type mapping

Integer width is real, not hidden. Amber stores an integer vector in the narrowest type that holds every element, so 1 2 3 is physically int8 and 1 2 3000000000 is int64 — both are "long vectors" as far as the language is concerned. The arrays you get back carry the true dtype rather than being widened behind your back; arr.astype("int64") is one call away when you want uniformity, and it should be your call.

Nulls (0Nd, 0Np, …) come back as NaT, and NaT goes in as the null.

Going the other way, Table.from_pandas maps object/string columns to symbol vectors (four bytes a row, integer comparisons — the reason Amber groups and joins fast on sym). Datetimes become real Amber timestamps, timedeltas timespans: Amber counts from 2000.01.01 and NumPy from 1970, and the move between the two happens here, so a timestamp never comes out thirty years off. datetime, date, time and timedelta objects convert the same way, and an aware datetime goes in as UTC.

Building from source

Point the build at the engine and install in editable mode. The [dev] extra pulls in pytest, NumPy, pandas and PyArrow, which the test suite needs:

# Against a prebuilt libamber.so (nothing is recompiled):
AMBER_LIB_DIR=/path/to/dir/holding/libamber.so \
AMBER_SRC=/path/to/amber \
    pip install -e ".[dev]"

# ...or against an engine checkout, letting it build the library if absent:
AMBER_SRC=/path/to/amber pip install -e ".[dev]"

AMBER_LIB_DIR supplies the prebuilt library; AMBER_SRC additionally supplies the eight .k standard-library modules, which the build copies into the package so the installed engine boots with its full vocabulary.

Run the tests. Point AMBER_HOME at a directory containing amber.k unless the build already bundled the standard library:

AMBER_HOME=/path/to/amber python -m pytest tests -q

setup.py looks for the engine in this order:

1. $AMBER_LIB_DIR — a directory containing a prebuilt libamber.so
2. $AMBER_SRC — an engine checkout; runs its own ./build.sh --shared-only
3. vendor/amber/ — what scripts/vendor_engine.sh writes, and what the sdist ships
4. ../amber/ — a sibling checkout
5. the system library path

It runs the engine's build script rather than reimplementing it, because that script already knows how to probe this toolchain for LTO, OpenMP and the right export-map flag, and a second copy of that logic here would rot.

Environment variables:

Building wheels

scripts/build_wheel.sh                    # a local wheel with the engine bundled
scripts/build_wheel.sh --manylinux        # auditwheel-repaired, redistributable

Releases are built by .github/workflows/wheels.yml with cibuildwheel: push a v* tag and it builds the sdist and every wheel, tests each one after installing it, and publishes to PyPI with trusted publishing (no token in the repo).

The plain build copies libamber.so into amber/ and links the extension with an $ORIGIN rpath, so the library is found next to the extension wherever the wheel lands. auditwheel repair then does the rest of what a redistributable wheel needs — hashing the library's name, fixing the rpath, and checking the glibc symbols the engine actually references.

Threading

The engine is single-threaded at its API boundary. Every call in this package runs with the GIL held and never releases it, which makes the GIL the engine's lock for free.

That is deliberate, and it is why you will not find Py_BEGIN_ALLOW_THREADS around long evaluations. Releasing the GIL there would let a second Python thread re-enter the interpreter mid-evaluation, and the resulting corruption would be intermittent, timing-dependent and effectively unattributable — the worst possible failure mode in exchange for parallelism you can get safely another way. If you need concurrency, use several Amber processes, which is what the engine's own peach does (fork-based, copy-on-write, no shared heap).

Two things that follow from this. Ctrl-C can't stop a long engine call halfway: Python sees the KeyboardInterrupt as soon as the call returns. And on a free-threaded Python (3.13t, 3.14t) importing the module turns the GIL back on, on purpose.

Troubleshooting

ImportError: libamber.so: cannot open shared object file — the wheel was built with AMBER_BUNDLE=0, or the library was moved after installation. Either reinstall with bundling on, or put libamber.so on LD_LIBRARY_PATH.

AmberError: 'value on a name that clearly exists — the standard library did not load. Check am.home(); if it is None you have a bare engine (primitives only, no gentq, no select). Set $AMBER_HOME to a directory containing amber.k.

ImportError: libamber ABI mismatch — the extension and the library came from different engine versions. Rebuild one of them; the message names both ABI numbers.

Underscores in Amber names — _ is a verb in Amber (drop, and floor), so my_name does not lex as one identifier. Use dotted names (my.name) or camel case. This bites everyone once.

BufferError: … has no flat buffer to export — you asked for a memoryview of a table, a general list or an atom. Use .column(i), .items() or .scalar().

Licence

GNU AGPLv3, matching the engine. See LICENSE and NOTICE.