roma-data-pipeline 0.1.0

Comprehensive ETL pipeline for Roman world data from Pleiades, ORBIS, Wikidata, and more

Roma Data Pipeline

Comprehensive ETL pipeline for Roman world data from Pleiades, ORBIS, Wikidata, and more.

Roma Data Pipeline generates a unified SQLite database containing 35,000+ ancient locations, 16,500+ Roman roads, 100+ historical figures, and pre-computed travel times across the Roman Empire. It aggregates, transforms, links, and exports data from multiple authoritative academic sources into a single, queryable dataset.

Quick Start

pip install roma-data-pipeline
roma-data run

This creates roma_aeterna.sqlite in your current directory (~50 MB).

What's Inside

Data	Records	Source
Ancient locations	35,378	Pleiades, ORBIS, ToposText
Province boundaries	186	AWMC
Roman roads	16,554	Itiner-e
Historical people	106	Wikidata
Travel network	2,208 edges	ORBIS
Ancient text citations	8,000+	ToposText

Features

• 8 Authoritative Sources: Aggregates data from Pleiades, AWMC, Itiner-e, Wikidata, ORBIS, and ToposText
• Full-Text Search: FTS5-powered search across locations and people
• Travel Time Calculator: ORBIS-derived travel times by foot, horse, cart, or ship
• Temporal Filtering: Filter by date range (753 BCE to 476 CE)
• Geographic Filtering: Filter by bounding box
• Multiple Export Formats: SQLite, GeoJSON, CSV
• Academic-Ready: DOI, CITATION.cff, FAIR principles compliant

Installation

From PyPI

pip install roma-data-pipeline

From Source

git clone https://github.com/romadatapipeline/roma-data-pipeline
cd roma-data-pipeline
pip install -e ".[dev]"

Docker

docker run -v $(pwd):/output ghcr.io/romadatapipeline/roma-data-pipeline

Usage

Command Line

# Run full pipeline (creates roma_aeterna.sqlite)
roma-data run

# Custom output path
roma-data run --output ./my_database.sqlite

# Select specific data sources
roma-data run --sources pleiades,orbis,wikidata

# Filter by time period (200 BCE to 200 CE)
roma-data run --start-year -200 --end-year 200

# Filter by geographic region (Western Mediterranean)
roma-data run --bbox "-10,35,20,50"

# Export to GeoJSON
roma-data export geojson locations.geojson

# Validate a database
roma-data validate ./roma_aeterna.sqlite

# Show available sources
roma-data info

Python API

from roma_data import Pipeline, Config

# Simple usage
pipeline = Pipeline()
db_path = pipeline.run()

# Custom configuration
config = Config(
    sources=["pleiades", "wikidata", "orbis"],
    output_path="./custom.sqlite",
    time_range=(-200, 200),
    bbox=(-10, 35, 20, 50),
)
Pipeline(config).run()

Query the Database

import sqlite3

conn = sqlite3.connect("roma_aeterna.sqlite")

# Find cities near Rome
cursor = conn.execute("""
    SELECT name_latin, type, latitude, longitude
    FROM locations
    WHERE type = 'city'
    AND ABS(latitude - 41.9) < 1
    AND ABS(longitude - 12.5) < 1
""")
for row in cursor:
    print(row)

# Search for temples
cursor = conn.execute("""
    SELECT name_latin FROM location_search
    WHERE location_search MATCH 'temple apollo'
""")

# Get travel time Rome to Carthage
cursor = conn.execute("""
    SELECT travel_days_ship, travel_days_foot
    FROM travel_network
    WHERE source_name LIKE '%Roma%'
    AND target_name LIKE '%Carthag%'
""")

Data Sources

Source	Description	License
Pleiades	Ancient place gazetteer with 30,000+ locations	CC-BY 3.0
AWMC	Province boundaries at 4 time periods	CC-BY 3.0
Itiner-e	Roman road network GIS data	Academic
Wikidata	Structured data for people, events, infrastructure	CC0
ORBIS	Stanford's Roman travel network model	CC-BY
ToposText	Ancient text citations by location	ODbL

Database Schema

The output SQLite database contains these tables:

• locations - Ancient places with coordinates, types, and dates
• provinces - Administrative boundaries with GeoJSON polygons
• roads - Roman road segments with GeoJSON paths
• people - Historical figures with birth/death dates and locations
• events - Battles, treaties, disasters with dates and locations
• travel_network - Pre-computed travel routes with times and costs
• ancient_sources - ToposText citations per location
• timeline_markers - Major historical milestones
• location_search - FTS5 full-text search index
• people_search - FTS5 full-text search index

See docs/schema.md for full schema documentation.

Citation

If you use this software in academic work, please cite:

@software{roma_data_pipeline,
  author = {Palaio, Thomas},
  title = {Roma Data Pipeline},
  year = {2025},
  url = {https://github.com/romadatapipeline/roma-data-pipeline},
  version = {0.1.0}
}

Also cite the underlying data sources as appropriate for your use case.

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

Adding a New Data Source

1. Create a new module in src/roma_data/sources/
2. Implement the DataSource base class
3. Add SPARQL queries or download URLs to constants.py
4. Update the CLI and pipeline to include the new source
5. Add tests and documentation

License

• Code: MIT License
• Data outputs: CC-BY 4.0

The generated data aggregates information from sources with various licenses (CC-BY, CC0, ODbL). See LICENSE-DATA for attribution requirements.

Acknowledgments

This project builds on the work of many scholars and institutions:

• Pleiades - NYU Institute for the Study of the Ancient World
• ORBIS - Stanford University
• AWMC - University of North Carolina at Chapel Hill
• Itiner-e - Roman roads research project
• ToposText - Brady Kiesling
• Wikidata - Wikimedia Foundation

Links

• Documentation
• PyPI Package
• Issue Tracker
• Changelog