Skip to main content

GraFlo GraFlo logo

Describe a property graph once in YAML; load it from files, SQL, RDF, APIs or Kafka into the database of your choice.

tests pre-commit PyPI Python Downloads Docs License DOI

GraFlo is a Python library that turns records from files, SQL databases, RDF, REST APIs or Kafka topics into a labeled property graph. You describe the graph once, in a YAML file called a manifest, and GraFlo creates the schema and writes the vertices and edges into the graph database of your choice, or into a directory on disk.

It is for engineers who build a graph from several sources and want its description in one reviewable file rather than spread across load scripts.

Documentation: growgraph.github.io/graflo

What you can do with it

  • Describe a graph once and load data into it. A manifest names the vertex and edge types, says which properties identify a vertex, and says how each kind of record becomes vertices and edges. The same manifest loads into ArangoDB, Neo4j, TigerGraph, FalkorDB, Memgraph, NebulaGraph, PostgreSQL or the file backend, and records with the same identity become one vertex. GraFlo also copies an existing graph from Neo4j, ArangoDB or PostgreSQL into another database (GraphEngine.migrate_graph).
  • Change the description over time, with a recorded history. Renaming a type, combining two types or changing a property type is a typed operation. Operations are recorded as commits (graflo commit, log, checkout, verify, revert) that you can replay, check and, for most operations, undo. Two branches of changes to one manifest are reconciled with a three-way merge (graflo merge3), and two manifests written by different teams are combined into one with a union (graflo merge).
  • Check and infer descriptions. GraFlo infers a manifest from a PostgreSQL database or an OWL ontology, proposes the properties that identify a record from sample data, and checks a manifest against a conformance profile (graflo check), a set of modeling rules such as "every vertex type declares its identity".

A taste

A manifest has three blocks: schema says what the graph looks like, ingestion_model says how records map onto it, and bindings says where the records come from. This one reads CSV files with the columns person_id, person and department:

schema:
    metadata: {name: hr}
    graph:
        vertex_config:
            vertices:
            -   {name: person, properties: [id, name], identity: [id]}
            -   {name: department, properties: [name], identity: [name]}
        edge_config:
            edges: [{source: person, target: department}]
ingestion_model:
    resources:
    -   name: departments
        pipeline:
        -   {vertex: person, from: {id: person_id, name: person}}
        -   {vertex: department, from: {name: department}}
bindings:
    connectors:
    -   {regex: "^dep.*\\.csv$", sub_path: data, resource_name: departments}

This loads it into ArangoDB:

from graflo import GraphEngine, GraphManifest
from graflo.connections import ArangoConfig

manifest = GraphManifest.from_yaml("manifest.yaml")
manifest.finish_init()
engine = GraphEngine()
engine.define_and_ingest(manifest=manifest, target_db_config=ArangoConfig.from_env())

ArangoConfig.from_env() reads ARANGO_URI, ARANGO_USERNAME, ARANGO_PASSWORD and ARANGO_DATABASE; every database has such a class. See Database connections.

Documentation

Full documentation: growgraph.github.io/graflo

  • Quick start: two CSV files into a graph, step by step
  • Creating a manifest: the three blocks of a manifest
  • Examples: runnable examples, one question each, with their data under examples/
  • Concepts: schema, identity, ingestion, connectors, evolution and version control
  • Guides: database connections, graph migration, schema inference, API wiring, bulk load
  • GraFlo ontology: a manifest as RDF (graflo manifest-to-rdf, graflo rdf-to-manifest)

Installation

GraFlo needs Python 3.11 or newer. The database clients, RDF and Kafka support are part of the default install.

pip install graflo

Optional extras (see the Installation guide):

  • dev: pytest and its plugins, hypothesis, ty, pre-commit
  • docs: ProperDocs and its plugins, for building the documentation site
  • plot: draws graflo plot-manifest and the --plot figures of graflo merge and graflo merge3 (SVG, PDF, PNG); no system Graphviz or fonts needed
pip install "graflo[dev,docs,plot]"

Development

To install from a clone:

git clone git@github.com:growgraph/graflo.git && cd graflo
uv sync --extra dev --extra plot

See the Contributing Guide for the full workflow.

Tests

The database tests need the database containers. Start them from a clone with the scripts under docker/:

cd docker
./start-all.sh    # Start all services
./stop-all.sh     # Stop all services
./cleanup-all.sh  # Remove containers and volumes

Per-engine compose files and ports are documented in the docker README.

To run the tests:

uv run pytest test

TigerGraph, NebulaGraph and Kafka tests are skipped unless you pass --run-tigergraph, --run-nebula or --run-kafka.

Everything outside test/db runs without the containers, and CI runs it on every pull request:

uv run pytest test --ignore=test/db -n auto --dist loadfile

Property tests run a reduced example budget by default; HYPOTHESIS_PROFILE=full runs the full one.

License

Open source under the Apache License 2.0. Copyright and trademark notices are in NOTICE: the license grants no rights in the GraFlo and GrowGraph marks. Releases before the relicensing shipped under the Business Source License 1.1 and keep those terms; see the changelog.

Contributing

Contributions are welcome. See the Contributing Guide. Contributors accept the Contributor License Agreement once, by commenting on their first pull request.

Metadata

Release files for graflo 1.20.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for graflo 1.20.0
File Size Uploaded Provenance
graflo-1.20.0.tar.gz 2.0 MB Without provenance Details

Built distribution (wheel)

Table of built distributions (wheels) for graflo 1.20.0
File Interpreter ABI Platform
graflo-1.20.0-py3-none-any.whl Python 3 none any Details

Total release size: 4.2 MB

Release files / graflo-1.20.0.tar.gz

Download URL graflo-1.20.0.tar.gz
Size 2.0 MB
Tags Source
SHA-256 checksum
How to use checksums
ad47bb9944b3d2ed1b492946ca51e904d90841259c8bb005cb8a876736a2d6b0
BLAKE2b-256 checksum
How to use checksums
bcffb4142ba46a85a80fb69381ea1ab3fd6553e26867aaf9cfa626673434688d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.21 {"installer":{"name":"uv","version":"0.12.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / graflo-1.20.0-py3-none-any.whl

Download URL graflo-1.20.0-py3-none-any.whl
Size 2.2 MB
Tags Python 3
SHA-256 checksum
How to use checksums
47b313ae4c0735c6554f6e56c1104bdb71c0008553df85a137547b9f711a4dc9
BLAKE2b-256 checksum
How to use checksums
50b9573b242095b490e1f164925fa413dcb69592873b949e03ee068d010a3f63
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.21 {"installer":{"name":"uv","version":"0.12.21","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for graflo 1.20.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

Release history Release notifications | RSS feed

This release

1.20.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page