mkpipe-extractor-sqlite

SQLite extractor plugin for MkPipe. Reads SQLite database tables via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  sqlite_source:
    variant: sqlite
    database: /full/path/to/mydb.db

Table Configuration

pipelines:
  - name: sqlite_to_pg
    source: sqlite_source
    destination: pg_target
    tables:
      - name: users
        target_name: stg_users
        replication_method: full

Incremental Replication

      - name: events
        target_name: stg_events
        replication_method: incremental
        iterate_column: updated_at
        iterate_column_type: datetime

Custom SQL

      - name: events
        target_name: stg_events
        replication_method: full
        custom_query: "SELECT id, user_id, event_type FROM events WHERE {query_filter}"

Performance Notes

• SQLite is a file-based database and does not support concurrent connections — parallel JDBC reads (partitions_count) are not meaningful and may cause lock contention.
• For large SQLite databases, keep fetchsize moderate (10,000–50,000).
• SQLite is best suited for small-to-medium datasets (< a few hundred MB). For large analytical workloads, consider migrating to a proper RDBMS.

All Table Parameters