GraphCast

A framework for transforming tabular (CSV, SQL) and hierarchical data (JSON, XML) into property graphs and ingesting them into graph databases (ArangoDB, Neo4j).

Core Concepts

Property Graphs

GraphCast works with property graphs, which consist of:

• Vertices: Nodes with properties and optional unique identifiers
• Edges: Relationships between vertices with their own properties
• Properties: Both vertices and edges may have properties

Schema

The Schema defines how your data should be transformed into a graph and contains:

• Vertex Definitions: Specify vertex types, their properties, and unique identifiers
• Edge Definitions: Define relationships between vertices and their properties
• Resource Mapping: describe how data sources map to vertices and edges
• Transforms: Modify data during the casting process

Resources

Resources are your data sources that can be:

• Table-like: CSV files, database tables
• JSON-like: JSON files, nested data structures

Features

• Graph Transformation Meta-language: A powerful declarative language to describe how your data becomes a property graph:
  • Define vertex and edge structures
  • Set compound indexes for vertices and edges
  • Use blank vertices for complex relationships
  • Specify edge constraints and properties
  • Apply advanced filtering and transformations
• Parallel processing: Use as many cores as you have
• Database support: Ingest into ArangoDB and Neo4j using the same API (database agnostic)

Documentation

Full documentation is available at: growgraph.github.io/graphcast

Installation

pip install graphcast

Usage Examples

Simple ingest

from suthing import ConfigFactory, FileHandle

from graphcast import Schema, Caster, Patterns


schema = Schema.from_dict(FileHandle.load("schema.yaml"))

conn_conf = ConfigFactory.create_config({
        "protocol": "http",
        "hostname": "localhost",
        "port": 8535,
        "username": "root",
        "password": "123",
        "database": "_system",
}
)

patterns = Patterns.from_dict(
    {
        "patterns": {
            "work": {"regex": "\Sjson$"},
        }
    }
)

schema.fetch_resource()

caster = Caster(
    schema,
)

caster.ingest_files(
    path="./data",
    conn_conf=conn_conf,
    patterns=patterns,
)

Development

To install requirements

git clone git@github.com:growgraph/graphcast.git && cd graphcast
uv sync --dev

Tests

Test databases

Spin up Arango from arango docker folder by

docker-compose --env-file .env up arango

and Neo4j from neo4j docker folder by

docker-compose --env-file .env up neo4j

To run unit tests

pytest test

Requirements

• Python 3.11+
• python-arango

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.