Scrapy Ingest

One pipeline. Full crawl observability in SQL or Elasticsearch.

Set DB_URL, SEARCH_URL, or both — or use no DB settings and ingest writes to ingest.db (SQLite) in your project folder. Items, requests, logs, stats, parent URLs, and errors are captured automatically.

Install

pip install scrapy-ingest

This installs all runtime dependencies, including psycopg2-binary, PyMySQL, and opensearch-py.

Minimal setup (settings.py)

Only the item pipeline is required — requests, logs, stats, parent_url, and error logging are enabled automatically.

Local dev (SQLite, no server):

ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.IngestPipeline": 300,
}
CREATE_TABLES = True  # creates ingest.db in the project directory

Postgres / MySQL:

ITEM_PIPELINES = {
    "scrapy_ingest.pipelines.IngestPipeline": 300,
}

DB_URL = "postgresql://user:password@localhost:5432/database"

Run your spider:

scrapy crawl your_spider

Log level follows Scrapy LOG_LEVEL.

Choose your destination

Set DB_URL (or DB_* fields) for Postgres/MySQL/SQLite, SEARCH_URL for Elasticsearch/OpenSearch, or both. If nothing is set, ingest.db (SQLite) is used unless INGEST_SQLITE_ENABLED = False.

# Database only
DB_URL = "postgresql://user:password@localhost:5432/database"

# Elasticsearch / OpenSearch only
SEARCH_URL = "http://localhost:9200"

# Both
DB_URL = "postgresql://user:password@localhost:5432/database"
SEARCH_URL = "http://localhost:9200"

When both are configured, SQL is written first, then Elasticsearch/OpenSearch indexes. An indexing failure after a successful SQL write is logged but does not roll back database rows.

Database configuration

PostgreSQL or MySQL — pick one connection style.

Single URL:

DB_URL = "postgresql://user:password@localhost:5432/database"
# DB_URL = "mysql://user:password@localhost:3306/database"

Discrete fields (no URL encoding for special characters in passwords):

DB_TYPE = "postgres"   # or "mysql" / "mariadb"
DB_HOST = "localhost"
DB_PORT = 5432         # MySQL: 3306
DB_USER = "user"
DB_PASSWORD = "password"
DB_NAME = "database"

Tables are created automatically when CREATE_TABLES = True (default).

Elasticsearch / OpenSearch configuration

Elasticsearch and OpenSearch use the same REST bulk API. The package connects via opensearch-py, which works with both.

Minimum:

SEARCH_URL = "http://localhost:9200"

With authentication (optional):

SEARCH_URL = "https://search.example.com:9200"
SEARCH_USER = "elastic"
SEARCH_PASSWORD = "secret"
SEARCH_SSL_VERIFY = True   # set False only for self-signed certs in dev

Index names match SQL table names by default:

Override table/index names with JOBS_TABLE, ITEMS_TABLE, REQUESTS_TABLE, and LOGS_TABLE (same setting for SQL and Elasticsearch/OpenSearch).

Indexes are created on first write if they do not exist.

What is stored

Relational database (Postgres / MySQL)

Request parent_url is the page that scheduled the request (e.g. sitemap → product). Start URLs are null. The request fingerprint is a SHA1 hash of method + canonical URL for parent lookup.

Elasticsearch / OpenSearch

Same crawl data as JSON documents in the indexes listed above. Documents include the string job_id (not the integer SQL primary key). Elasticsearch/OpenSearch mode does not link parent_id; parent_url is stored on each request document.

Ingest flow and connections

One connection per destination per crawl — not connect → insert → disconnect for every product. Items, requests, and logs are buffered in memory and written in batches using the same Postgres/MySQL connection and/or Elasticsearch/OpenSearch client for the whole spider run.

Full diagrams: docs/ingest-flow.rst (Read the Docs: Ingest flow).

Crawl lifecycle

flowchart TB
    START([scrapy crawl starts]) --> OPEN[IngestPipeline open_spider]
    OPEN --> FLSTART[IngestFlusher.start]

    FLSTART --> DB{DB configured?}
    DB -->|yes| DBCONN[Open one Postgres/MySQL connection<br/>application_name = Ingest]
    DB -->|no| SKIPDB[Skip SQL]

    FLSTART --> SRCH{SEARCH_URL set?}
    SRCH -->|yes| ESCONN[Open one SearchClient<br/>X-Opaque-Id: Ingest]
    SRCH -->|no| SKIPES[Skip search]

    DBCONN --> JOB[Create job row / index doc]
    ESCONN --> JOB
    JOB --> TIMER[Start 10s periodic flush timer]
    TIMER --> CRAWL([Spider crawls])

    CRAWL --> END([Engine stops])
    END --> FINAL[Final flush + finish_job]
    FINAL --> SUMMARY[Print crawl summary]
    SUMMARY --> CLOSE[Close DB + search clients]

When data is flushed

flowchart TB
    subgraph inputs [Buffered in memory]
        I1[Items]
        I2[Requests]
        I3[Logs]
    end

    inputs --> BUF[(Collector buffer)]

    BUF --> T1{Size ≥ INGEST_BATCH_SIZE?<br/>default 50}
    BUF --> T2{Timer every 10s?}

    T1 -->|yes| FLUSH[flush batch]
    T2 -->|yes| FLUSH

    FLUSH --> SQL[SQL: INSERT + COMMIT<br/>same connection]
    FLUSH --> ES[Search: bulk index<br/>same HTTP client]

Example: 100,000 products → ~2,000 batch flushes (not 100,000 connections).

When both SQL and search are configured, SQL commits first; a search failure after SQL success is logged but does not roll back database rows.

When the spider closes, a crawl summary is printed to stderr (visible even when LOG_LEVEL is ERROR) with job id, spider, database URL, Elasticsearch/OpenSearch URL, table/index names, counts, and elapsed time. Set INGEST_SHOW_SUMMARY = False to hide it.

CLI

Run from your Scrapy project directory (loads settings.py via Scrapy).

check-config

Validate settings and ping configured destinations before running a spider:

scrapy-ingest check-config
scrapy-ingest check-config --db-url "postgresql://user:pass@localhost:5432/db"
scrapy-ingest check-config --search-url "http://localhost:9200"

Shows a spinner per step, prints a summary table to stderr, and exits 0 when all configured destinations are reachable.

jobs show

List recent jobs or print one job summary from SQL or Elasticsearch/OpenSearch (requires DB_URL and/or SEARCH_URL). When both are configured, queries use SQL; crawls still dual-write to both.

scrapy-ingest jobs show                                    # list recent jobs
scrapy-ingest jobs show --limit 20                         # list up to 20 jobs
scrapy-ingest jobs show --status running                   # filter by status (list mode)
scrapy-ingest jobs show --spider Rs_Spider                 # filter by spider (list mode)
scrapy-ingest jobs show Rs_Spider-178826754-a1b2            # one job (Field | Value table)
scrapy-ingest jobs show --search-url "http://localhost:9200"

List mode (no job id): table of recent jobs — job id, spider, status, timestamps, counts. Detail mode (with job id): Field | Value table from the jobs row. --status and --spider apply in list mode only.

The job id is the string shown in the end-of-crawl summary (or your JOB_ID setting). Exit 0 on success (including an empty list); exit 1 if a specific job id is missing or the destination is unreachable.

Typical workflow:

scrapy-ingest check-config
scrapy crawl your_spider
scrapy-ingest jobs show
scrapy-ingest jobs show --status finished --spider your_spider
scrapy-ingest jobs show <job_id>

See CLI docs for details.

Useful settings

Troubleshooting

• Password has @ or $ in DB_URL? Encode them: @ → %40, $ → %24. Or use discrete DB_* fields.
• Yield items from callbacks (not only return inside a generator).
• Elasticsearch / OpenSearch connection refused? Check SEARCH_URL, firewall, and that the cluster is running. Test with curl http://localhost:9200.
• HTTPS / self-signed cert? Set SEARCH_SSL_VERIFY = False in development only.

Standalone components

If you only want part of the collection:

# Items only
ITEM_PIPELINES = {"scrapy_ingest.pipelines.ItemsPipeline": 300}

# Requests only (parent_url + errors)
ITEM_PIPELINES = {"scrapy_ingest.pipelines.RequestsPipeline": 300}

# Logs only
EXTENSIONS = {"scrapy_ingest.extensions.LoggingExtension": 500}

Benchmarks and comparisons

How scrapy-ingest compares to other Scrapy storage packages — feature matrices, architecture notes, and a reproducible write-path benchmark.

Also in docs: scrapy-pipelines (Scrapedia/Mongo), Scrapy MongoPipeline (official example).

Synthetic benchmark (SQLite, 100k rows, batch 50): ~3.3× faster write path vs per-row COMMIT on a sample run — real crawls are usually network-bound. Run yourself:

python scripts/benchmark_ingest.py --rows 100000 --batch-size 50

Full comparison, methodology, and production tuning: docs/benchmarks.rst · Read the Docs.

Links

• Docs: https://scrapy-ingest.readthedocs.io/
• Changelog: docs/development/changelog.rst
• Issues: https://github.com/fawadss1/scrapy-ingest/issues

License

MIT License. See LICENSE.