rustychickpeas 0.3.3

High-performance in-memory graph API with RoaringBitmaps, Python bindings via PyO3

RustyChickpeas

An in-memory graph API written in Rust, using RoaringBitmaps as a fundamental data structure.

Overview

RustyChickpeas provides a high-performance, in-memory graph database API that:

• Uses RoaringBitmaps: Efficient set operations for node/relationship IDs
• Rust-First: High-performance core implementation
• Python-Friendly: PyO3 bindings for seamless Python integration
• In-Memory: Optimized for fast graph operations

Project Status

✅ Implementation Complete - Core functionality implemented with comprehensive benchmarks and Python bindings.

• ✅ Immutable GraphSnapshot with CSR adjacency and columnar properties
• ✅ GraphSnapshotBuilder for efficient bulk loading (requires u32 node IDs)
• ✅ Parallel finalization using rayon for improved performance
• ✅ Python bindings via PyO3
• ✅ Bulk loading from Parquet files

See rustychickpeas-core/benches/README.md for benchmark information.

Key Features

• ✅ Immutable GraphSnapshot - Read-optimized graph with CSR adjacency and columnar properties
• ✅ GraphSnapshotBuilder - Efficient bulk loading (requires u32 node IDs directly)
• ✅ RustyChickpeas Manager - Version management for multiple graph snapshots
• ✅ Parallel Finalization - Uses rayon to parallelize index building during finalization
• ✅ Label and property support with inverted indexes
• ✅ Efficient traversal using CSR (Compressed Sparse Row) format
• ✅ Python bindings via PyO3
• ✅ Bulk loading from Parquet files

Installation

pip install rustychickpeas

Or from source:

git clone https://github.com/freeeve/rustychickpeas.git
cd rustychickpeas/rustychickpeas-python
pip install maturin
maturin develop --release

Platform Support

RustyChickpeas is tested and supported on:

• ✅ Linux x86_64 - Full support
• ✅ macOS x86_64 (Intel) - Full support
• ✅ macOS arm64 (Apple Silicon) - Full support
• ✅ Windows x86_64 - Full support

Limitations:

• ⚠️ Linux aarch64 (ARM servers, e.g., AWS Graviton) - Not currently tested in CI. The Rust core should compile and work, but Python bindings require native runners or complex cross-compilation setup. Contributions welcome!

Python Version Support

RustyChickpeas requires Python >= 3.10 and is tested against Python 3.10, 3.11, 3.12, 3.13, and 3.14.

Supported Python Versions by Platform

Platform	Python Versions
Linux x86_64	3.10, 3.11, 3.12, 3.13, 3.14
macOS x86_64 (Intel)	3.11, 3.12, 3.13, 3.14
macOS arm64 (Apple Silicon)	3.10, 3.11, 3.12, 3.13, 3.14
Windows x86_64	3.10, 3.11, 3.12, 3.13, 3.14

Note: Python 3.10 is not available on macOS x86_64 runners due to dependency issues with the GitHub Actions environment. Python 3.10 works fine on macOS arm64 (native runners) and all other platforms.

PyO3 Compatibility

• Python 3.10-3.12: Fully supported by PyO3 0.20.3

• Python 3.13-3.14: Uses stable ABI compatibility mode (PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1) since PyO3 0.20.3 doesn't officially support these versions yet. This works correctly but uses the stable ABI for forward compatibility.

  Note for local development: When using cargo check or cargo build with Python 3.13/3.14, you may need to set the environment variable:

  export PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
  cargo check --package rustychickpeas-python
  

  maturin builds automatically handle this via the py-limited-api = "auto" setting in pyproject.toml.

Quick Start

Python

Basic Usage (with Manager)

import rustychickpeas as rcp

# Create a manager for version management
manager = rcp.RustyChickpeas()

# Create a builder with a version
builder = manager.create_builder(version="v1.0")

# Add nodes with labels
builder.add_node(1, ["Person"])
builder.add_node(2, ["Person"])
builder.add_node(3, ["Company"])

# Add relationships
builder.add_rel(1, 2, "KNOWS")
builder.add_rel(1, 3, "WORKS_FOR")

# Set properties (automatic type detection)
builder.set_prop(1, "name", "Alice")
builder.set_prop(1, "age", 30)
builder.set_prop(2, "name", "Bob")
builder.set_prop(3, "name", "Acme Corp")

# Finalize and add to manager (uses the version set when creating the builder)
builder.finalize_into(manager)

# Retrieve snapshot by version
graph = manager.get_graph_snapshot("v1.0")

# Query nodes
node = graph.get_node(1)  # Get Node object (node ID 1)
print(f"Node labels: {node.get_labels()}")  # ["Person"]
print(f"Node property 'name': {node.get_property('name')}")  # "Alice"

# Query relationships - get neighbor Node objects
neighbors = graph.get_neighbors(1, rcp.Direction.Outgoing)
print(f"Node 1 has {len(neighbors)} outgoing neighbors")
for neighbor in neighbors:
    print(f"  Neighbor ID: {neighbor.id()}, labels: {neighbor.get_labels()}")

# Or get just the neighbor IDs
neighbor_ids = graph.get_neighbor_ids(1, rcp.Direction.Outgoing)
print(f"Neighbor IDs: {neighbor_ids}")  # [2, 3]

# Get relationships as Relationship objects (includes type, start/end nodes)
relationships = graph.get_rels(1, rcp.Direction.Outgoing)
for rel in relationships:
    print(f"  {rel.get_type()}: {rel.get_start_node().id()} -> {rel.get_end_node().id()}")

Standalone Usage (without Manager)

import rustychickpeas as rcp

# Create a standalone builder
builder = rcp.GraphSnapshotBuilder(version="v1.0")

# Add nodes and relationships
builder.add_node(1, ["Person"])
builder.add_node(2, ["Person"])
builder.add_rel(1, 2, "KNOWS")

# Finalize into a snapshot
graph = builder.finalize()

# Query the graph
node = graph.get_node(1)
neighbors = graph.get_neighbors(1, rcp.Direction.Outgoing)

Loading from Parquet Files

import rustychickpeas as rcp

# Direct loading (creates a standalone snapshot)
graph = rcp.GraphSnapshot.read_from_parquet(
    nodes_path="nodes.parquet",
    relationships_path="relationships.parquet",
    node_id_column="id",
    label_columns=["label"],
    start_node_column="from",
    end_node_column="to",
    rel_type_column="type"
)

# Or load into a builder for more control
manager = rcp.RustyChickpeas()
builder = manager.create_builder(version="v1.0")
builder.load_nodes_from_parquet("nodes.parquet", node_id_column="id", label_columns=["label"])
builder.load_relationships_from_parquet("relationships.parquet", 
                                       start_node_column="from", 
                                       end_node_column="to", 
                                       rel_type_column="type")
builder.finalize_into(manager)
graph = manager.get_graph_snapshot("v1.0")

Rust

use rustychickpeas_core::RustyChickpeas;

// Create a manager (handles multiple snapshots by version)
let manager = RustyChickpeas::new();

// Create a builder from the manager with version
let mut builder = manager.create_builder(Some("v1.0"), None, None);

// Add nodes and relationships (node IDs must be u32)
builder.add_node(1, &["Person"]);
builder.add_node(2, &["Person"]);
builder.add_rel(1, 2, "KNOWS");

// Finalize the builder
let snapshot = builder.finalize(None);

// Add to manager
manager.add_snapshot(snapshot);

// Retrieve the snapshot by version
let snapshot = manager.get_graph_snapshot("v1.0").unwrap();

// Query the snapshot
let neighbors = snapshot.get_out_neighbors(1);
println!("Node 1 neighbors: {:?}", neighbors);

Version Management

RustyChickpeas supports version management at the snapshot level using the RustyChickpeas manager. Each snapshot can have a version string (e.g., "v1.0", "v2.0") that identifies it.

Python

import rustychickpeas as rcp

# Create a manager
manager = rcp.RustyChickpeas()

# Create and build version 1.0
builder1 = manager.create_builder(version="v1.0")
# ... add nodes/relationships ...
builder1.finalize_into(manager)

# Create and build version 2.0
builder2 = manager.create_builder(version="v2.0")
# ... add nodes/relationships ...
builder2.finalize_into(manager)

# Retrieve snapshots by version
v1_snapshot = manager.get_graph_snapshot("v1.0")
v2_snapshot = manager.get_graph_snapshot("v2.0")

# List all versions
versions = manager.versions()  # ["v1.0", "v2.0"]

How Version Management Works

1. Setting Version: Pass version parameter to create_builder() when creating the builder, or use GraphSnapshotBuilder.set_version(version_string) to set it later.

2. Storage: After calling builder.finalize(), add the snapshot to the manager using manager.add_snapshot(snapshot). The snapshot will be stored under its version (if set) or "latest" if no version is set.

3. Retrieval: Use manager.get_graph_snapshot(version_string) to retrieve a snapshot by version.

4. Version Storage: Versions are stored as strings and can be any identifier you choose (e.g., "v1.0", "2024-01-01", "production").

Capacity and Auto-Growing

The capacity_nodes and capacity_rels parameters are optional performance hints for pre-allocation:

• Defaults: If not specified, starts with 2^20 (1,048,576) nodes/relationships capacity
• Auto-Growing: The builder automatically grows as needed (doubling capacity each time)
• Maximum Limits:
  • Nodes: Up to 2^32 - 1 (4.3 billion) - enforced by u32 NodeId
  • Relationships: Up to 2^64 - 1 (18.4 quintillion) - limited by memory
• When to Specify Capacity: Only specify if you know the approximate size upfront to avoid reallocations. For most use cases, the default auto-growing behavior is sufficient.

Example:

# Uses default (2^20 = 1,048,576), auto-grows as needed
builder = manager.create_builder(version="v1.0")

# Large graph - specify capacity hint to avoid reallocations
builder = manager.create_builder(version="v1.0", capacity_nodes=10_000_000, capacity_rels=50_000_000)

Performance

See rustychickpeas-core/benches/README.md for benchmark information.

For Python-specific performance tests, see rustychickpeas-python/tests/benchmark_large_parquet.py.

Highlights:

• Node existence: ~7ns (constant time)
• Property lookup: ~44ns (constant time)
• Label queries: ~56ns for 100 nodes, ~348ns for 100K nodes
• BFS traversal: ~130ns per node
• Bitmap operations: Sub-microsecond for millions of elements

Limits and Scalability

Hard Limits

• Nodes: Up to 2^32 - 1 (4,294,967,295 nodes) - Limited by u32 NodeId
• Node IDs: Must be u32 (0 to 2^32 - 1) - Users should map their own IDs to u32 if needed
• Relationships: Up to 2^64 - 1 (18,446,744,073,709,551,615) - Limited by u64 counter, but practically constrained by memory
• Unique Strings (labels, relationship types, property keys): Up to 2^32 - 1 (4,294,967,295) - Limited by u32 InternedStringId
• Properties per Node: No hard limit, constrained by available memory
• Property Values: No hard limit, constrained by available memory

Note: While relationships can theoretically reach 2^64 - 1, practical limits are determined by available system memory. Each relationship requires storage in CSR arrays and indexes.

Tested Scales

✅ 1 Million Nodes + 1 Million Relationships - Tested and verified:

• Memory usage: ~2.7GB (with string interning) to ~3.4GB (without)
• Bulk load rate: ~4M entities/sec from Parquet files
• Direct builder rate: 21-31M nodes/sec, 12-19M rels/sec
• Query performance: Sub-millisecond for most operations
• See rustychickpeas-python/tests/benchmark_large_parquet.py for large-scale benchmarks

Practical Considerations

Memory Usage:

• Base overhead: ~3.5 bytes per node/relationship (structure + indexes)
• Properties: Additional memory depends on property count and size
• String interning: Reduces memory signficantly for graphs with high string duplication
• Property value interning: Optional feature saves 32-50% memory when property values have high duplication

Performance Characteristics:

• Direct Builder Operations: 21-31M nodes/sec, 12-19M rels/sec (exceeds 10M/sec target)
• Bulk Loading: ~4M entities/sec from Parquet files (sequential processing, I/O bound)
• Finalization: Parallelized using rayon for label/type indexes, property columns, and inverted indexes
• Queries: Constant-time for indexed operations (label/type lookups)
• Traversal: Linear with graph connectivity (BFS, shortest path)
• Memory: Linear growth with graph size

Recommended Usage:

• Small graphs (< 100K nodes): Excellent performance, minimal memory footprint
• Medium graphs (100K - 10M nodes): Good performance, manageable memory requirements
• Large graphs (10M+ nodes): Feasible but memory becomes the primary constraint; finalization benefits from parallelization
• Very large graphs (100M+ nodes): Possible with sufficient RAM (100GB+); parallel finalization helps reduce finalization time

Memory Estimation:

Base memory ≈ (nodes + relationships) × 3.5 bytes
+ Properties (varies by property count/size)
+ String interning overhead (minimal)
- String interning savings (~21.5% if enabled)

Example: 1M nodes + 1M relationships with properties:

• Without interning: ~3.4GB
• With basic interning: ~2.7GB
• With property value interning (50% reuse): ~1.4GB

Testing

Running Tests

Rust Tests:

cargo test --workspace

Python Tests:

cd rustychickpeas-python
pytest tests/

Test Coverage

Test coverage is set up for both Rust and Python:

Run Coverage:

./scripts/coverage.sh  # Linux/macOS
.\scripts\coverage.ps1  # Windows

Coverage Reports:

• Rust: coverage/rust/tarpaulin-report.html
• Python: coverage/python/htmlcov/index.html

See docs/COVERAGE.md for detailed coverage documentation.

License

Licensed under MIT license (LICENSE or http://opensource.org/licenses/MIT).