erpl-rev — query and replicate SAP through DuckDB

Run DuckDB SQL on SAP data, and bulk-replicate SAP into DuckDB — with nothing to install inside SAP.

erpl-rev is a small C++ server that registers at the SAP gateway as an RFC destination. Your ABAP calls it with CALL FUNCTION '…' DESTINATION 'ERPL_REV' and DuckDB runs behind it. Two things, in equal measure: query SAP tables, CDS views and BW calculation views with full SQL (and join them to cloud data), and replicate them into DuckDB, parquet, DuckLake, Iceberg or any DuckDB-attachable warehouse. No DuckDB extension in SAP, no DDIC structures, no SLT/SDI/BODS — payloads travel as plain JSON over RFC.

It's the inverse of erpl — where erpl makes DuckDB call into SAP, erpl-rev has SAP call out into DuckDB.

Status: research prototype, but both paths — query (SQL, federation, console, live serving) and replication (table / CDS / BW → DuckDB, parquet

• attached catalogs) — are verified end-to-end against a live SAP ABAP (A4H) system in scripts/e2e.sh.

Query — SQL on SAP data, from ABAP and beyond

• Run DuckDB SQL from ABAP. Send a query, get typed rows back — or an interactive ALV grid via the in-GUI SQL console (Z_ERPL_REV_SQL, SE38).
• Do what Open SQL can't. Joins across tables, window functions, aggregations and parquet scans over live SAP data, without staging anything.
• Federate in one statement. Join a SAP slice against a cloud parquet / Iceberg / DuckLake dataset or an attached Postgres / BigQuery table, and hand the result back to ABAP — cross-system queries SAP can't do alone.
• Query live, over the network. With --quack, remote DuckDB clients (notebooks, BI tools) query the same in-process data directly — no export step.
• Big results, fixed memory. A streamed OPEN/FETCH/CLOSE cursor decodes results page-by-page, so result size isn't bounded by ABAP memory.

Replicate — bulk-load SAP into DuckDB & the lakehouse

• Any SAP source. Replicate tables, CDS views (incl. WITH PARAMETERS) and BW / HANA calculation views into typed DuckDB — full or filtered, with source-side WHERE, column selection and idempotent UPSERT. Built for >100M-row loads.
• Delta (incremental) too. Keep a target in sync loading only what changed — watermark, insert-only, change-document (CDHDR/CDPOS), snapshot-diff and trigger-CDC (both catch deletes) methods, all merging server-side, idempotent and re-runnable. Customer-owned Open SQL only (no ODP / SAPI / RFC_READ_TABLE). See docs/delta.md.
• Real time, when you want it. A daemon at a two-second cadence, and a trigger tier that catches physical deletes a watermark cannot see. Three minutes, unedited: a million goods movements synced in ten seconds, then a material document posted, transferred and archived in SAP — each change in DuckDB a second or two later — and finally thirty seconds of continuous mixed traffic at a few dozen rows a second:

The right-hand pane is erpl-rev top, started once and never touched again: colour is the target, the glyph is the operation — ▲ insert, ◆ update, ▼ delete — and the LAST CYCLE column carries the exact per-cycle split. The clip ends by measuring itself: SAP's change time against the apply time, per row, from two independent clocks.

See docs/demo.md for what it proves and what it does not. Re-render it with bash demo/setup.sh && vhs demo/realtime.tape, or package it as a single shareable HTML file with bash demo/package-html.sh.

• Land in the open lakehouse. parquet / partitioned datasets, DuckLake or Iceberg, on local disk or cloud object storage (S3 / GCS / Azure).
• Publish into a warehouse. ATTACH Postgres / MySQL / BigQuery / MotherDuck and push a SAP slice in with one SQL statement — see docs/publishing.md.
• Fast & parallel — a live 10M-row run (50-of-400-column BSEG-shaped table, BELNR-partitioned, on the A4H trial / loopback):

Peak ~167k rows/s — 10M rows in a minute. Each worker bulk-loads a disjoint key range with a DuckDB Appender (~230× a naive per-row path); memory is bounded by batch size, and loads are restartable and idempotent (test/bench_ingest.cpp).

Why it fits the SAP data stack

• No SLT, no SDI, no Data Services, no add-on — a transport (package ZERPL_CORE) plus a registered RFC server. No core modification, no HANA license, no BTP. Runs against any NetWeaver ABAP stack (ECC, S/4HANA, BW/4HANA).
• Reads what you model — tables, CDS views (keys auto-detected), BW/HANA calc views ("_SYS_BIC"."pkg/CV") — semantics intact, not raw dumps.
• SLT semantics you know (LTRS) — field selection, source-side filter, key UPSERT — without standing up SLT.
• DDIC-typed & provably faithful — NUMC / DATS / CURR / DECIMAL / RAW map to real DuckDB types; a built-in diff harness checks the target against the source cell-by-cell (incl. a 400-column BSEG-shaped table).

Security & authorizations. Only Z* objects (no core mod), reached via a type-T destination locked down with a gateway reginfo allow-list and (off-box) SNC. FM calls run as a comm user scoped to S_RFC for function group ZERPL_REV only; the reports run under the end user's auth (CDS DCL is enforced, raw-table SELECT is not implicitly S_TABU-checked — gate program execution; the native/ADBC BW path reads cross-client). Full guide: docs/security.md.

Getting started

Five steps. The third one is a loop, and it is the only part that surprises people.

1. See it run — no SAP needed

uvx runs a published tool without installing it, and comes with uv. erpl-rev's CLI shells out to it, so install uv first — or pip install erpl-rev and drop the uvx prefix from everything below.

uvx erpl-rev --smoke

Loads the RFC stack and DuckDB and prints their versions. If that works, the binary is fine and everything from here is about SAP. (pip install erpl-rev if you would rather have it on PATH; a container image and standalone bundles are in Install options below.)

2. Collect what you need

erpl-rev makes two different connections, and mixing them up is the commonest early confusion:

  your laptop ──── ADT, HTTP :50000 ────▶  SAP   (the CLI: doctor, setup, sync, replicate)
  the server  ◀─── RFC, gateway :3300 ──▶  SAP   (registers OUTBOUND; SAP never dials in)
  your laptop ──── quack, loopback ─────▶  the server   (sql, sync ls)

For the CLI, an ABAP Development Tools login:

ADT must be reachable (/sap/bc/adt, ICF active). uvx/uv must be installed — the CLI shells out to it.

For the server, the gateway: --gwhost and --gwserv (sapgw<NN>, port 33<NN>). Terms in that paragraph you do not recognise are in the glossary.

And from your Basis team, eventually: a reginfo line allowing the program ID, an RFC user, and — on a development system — S_DEVELOP for the account that runs setup. Step 3 generates the exact request.

3. Deploy the ABAP, and get the gateway to accept you

This is a loop, not a sequence. setup finishes by making ABAP call back out through the server, which cannot happen until the gateway lets the server register, and the reginfo line that allows it is written by setup. So you go round once:

erpl-rev doctor                 # read-only: what is missing, and the fix for each
erpl-rev setup --dry-run        # the exact change set, nothing written
erpl-rev setup                  # deploy, then try the round trip

setup will deploy the ABAP and then say:

Deployed, but the round trip did NOT complete yet. That is expected if the
server is not running, or if the gateway has not been told to accept the
registration.

That is not a failure. It also writes erpl-rev-basis-handout.md. Give that to Basis — it contains the least-privilege reginfo line already filled in.

If the server will run on a different machine from the one you just typed on, pass --server-host. The reginfo line names the host allowed to register, and it defaults to this machine. Get it wrong and Basis allows your laptop while the real server is refused:

erpl-rev setup --print-runbook --server-host sapbridge01.corp

(erpl-rev setup --print-runbook prints the same handout without deploying anything, which is what you want on a system where you will never have S_DEVELOP; there, the ABAP arrives by transport instead. The binary builds that transport too — setup --package ZERPL_CORE --transport <request> on your own DEV system — and erpl-rev abap export <dir> writes the fourteen sources out as files. See INSTALL.md.)

Then start the server and go round again:

Leave the server running in one terminal:

erpl-rev --gwhost <gateway-host> --gwserv sapgw<NN>

Watch for to="running", not for "listening".

INFO [server] registration state from="starting" to="running"     <- the gateway accepted
INFO [server] listening (Ctrl-C to stop) program_id="ERPL_REV" ...

"listening" is printed as soon as the process starts serving, whether or not the gateway let it register; the state line is the one that tells you. If it says broken, the reginfo does not allow this host and program ID — that is the loop above not closed yet.

Then, in another terminal:

erpl-rev doctor                 # the round trip should now pass

doctor and setup take --gwhost/--gwserv too, and they are not the server's defaults: setup assumes the gateway is on the SAP host. Pass them explicitly if your gateway is somewhere else.

On a throwaway trial with gw/acl_mode = 0 there is no ACL to satisfy and the two halves can happen in either order — see the A4H appendix.

4. Get your first table out

erpl-rev replicate --table MARA --target mara     # full load, as a background job
erpl-rev sql "SELECT count(*) FROM mara"          # read it back
erpl-rev sql "SELECT * FROM mara LIMIT 10"

sql reads the live database through the server, so it works while the server is running. Stop the server and the DuckDB file is an ordinary DuckDB file — open it with anything.

5. Keep it in sync

A full load is a snapshot. To keep a target current, register how it should find what changed:

erpl-rev sync create mara --method WATERMARK --source MARA \
    --keys MANDT,MATNR --chg-col AEDAT --wm-kind DATE --cadence hourly --log
erpl-rev sync run mara            # one cycle now
erpl-rev sync ls                  # what is registered, and how far behind

wm-kind DATE reads whole days, and never today — so once the backfill is done, a run made the same day returns nothing. That is the complete-day rule doing its job, not a fault: a day still being written to cannot be safely marked as read. Sources with a timestamp column use NUMTS and do not wait. The table of kinds is in delta.md.

Which method to choose, and how to catch physical deletes a change column cannot see, is docs/delta.md and docs/cdc.md. To have it run continuously rather than on demand, docs/daemon.md; to watch it, erpl-rev top and docs/operations.md.

Install options

All of them carry DuckDB and nothing else: since v2026.08.30 the RFC protocol is erpl-proto, our pure-Rust implementation, linked statically — no SAP NW RFC SDK, no ICU, no LD_LIBRARY_PATH. Building from source is different and is docs/building.md.

Everything the SAP GUI reports do is also a CLI command, and --print-abap on any of them shows the ABAP instead of running it. Nothing writes to SAP or DuckDB without a confirmation or an explicit --yes.

Feedback

If erpl-rev misbehaves, please open an issue. This is a registered RFC server talking to gateways and SAP releases we cannot reproduce here, so a report with your setup is the fastest path to a fix.

If it saved you time, a star on the repo helps others find it.

Because this process usually runs under a service manager rather than on a terminal, the feedback pointer is emitted as an INFO log line at startup — where operators actually read it. On an interactive start you also get a small banner, at most once a day; silence both with DATAZOO_NO_BANNER=1.

Telemetry

The server sends anonymous usage telemetry to help us understand which bridge operations are used and where they break. Three events: server_started once at boot, rfc_call on every bridge function-module invocation (which module, success or failure, how long — sampled), and $exception when one fails.

What never leaves the machine: SAP data, SQL text, table or field names, target names, connection strings and error messages. Only bounded enumerations and numbers.

It is on by default and disabled by any one of --no-telemetry, ERPL_REV_NO_TELEMETRY, or DATAZOO_DISABLE_TELEMETRY; ERPL_REV_TELEMETRY_SAMPLE_RATE thins rfc_call without silencing it. Air-gapped hosts drop the request silently with zero impact. Full detail, event by event and property by property: TELEMETRY.md.

Docs

Getting it working

• docs/glossary.md — the SAP words, if you do not use them daily
• docs/INSTALL.md — the transport path, for a system where you will never have S_DEVELOP
• docs/security.md — what your Basis team will ask: the trust boundary, reginfo, the RFC user, SNC
• docs/docker.md — running the container image
• docs/tunnel.md — only if this host has no route to the gateway

Keeping data in sync

• docs/delta.md — the five incremental methods, and how to choose
• docs/cdc.md — the trigger tier, for physical deletes
• docs/daemon.md — continuous replication in one background job
• docs/operations.md — the operator's runbook and the monitor
• docs/upgrading.md — what changes for a system already replicating

Reading and publishing the data

• docs/sql-console.md — the DuckDB SQL console inside SAP GUI
• docs/publishing.md — parquet, DuckLake, Iceberg, or an attached warehouse
• docs/control-tables.md — the control schema as a versioned interface, and what each run records

How it behaves, and how fast

• docs/demo.md — the recorded session: what it proves and what it does not
• docs/perf-results.md — measured numbers, dated, with the box they came from
• TELEMETRY.md — every event and property that leaves the machine, and the opt-outs

Working on erpl-rev

• docs/building.md — building from source
• docs/testing.md — the test lanes and the release gate
• docs/enable-rfc-registration.md — appendix: RFC registration on an A4H trial

License

Business Source License 1.1 (BSL), Licensor DataZoo GmbH, Change License MPL 2.0 — same terms as erpl.