mkpipe-loader-sqlite

SQLite loader plugin for MkPipe. Writes Spark DataFrames into SQLite database files via JDBC.

Documentation

For more detailed documentation, please visit the GitHub repository.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Connection Configuration

connections:
  sqlite_target:
    variant: sqlite
    database: /full/path/to/mydb.db

Table Configuration

pipelines:
  - name: pg_to_sqlite
    source: pg_source
    destination: sqlite_target
    tables:
      - name: public.users
        target_name: stg_users
        replication_method: full
        batchsize: 5000

Write Strategy

Control how data is written to SQLite:

      - name: public.users
        target_name: stg_users
        write_strategy: upsert       # append | replace | upsert | merge
        write_key: [id]              # required for upsert/merge

Write Throughput

      - name: public.users
        target_name: stg_users
        replication_method: full
        batchsize: 5000

Performance Notes

• SQLite uses file-level locking — write_partitions has no benefit here as concurrent writes will serialize or fail. Set write_partitions: 1 to avoid contention.
• Keep batchsize moderate (1,000–10,000) — large transactions in SQLite can cause memory pressure.
• SQLite is best suited for small-to-medium outputs (local development, testing, lightweight pipelines).

All Table Parameters