qvd-parser 0.1.0

Fast QVD file parser — Python bindings for qvd-parser (Rust)

qvd-parser

Fast, correct QVD (QlikView Data) file parser written in Rust.

Reads the binary format used by QlikView and Qlik Sense to store compressed data tables. Exposes a Rust API, a Python API (via PyO3), and a CLI. Export targets: JSON, CSV, Parquet.

Note: QVD is a proprietary file format associated with QlikView and Qlik Sense. This project is not affiliated with or endorsed by Qlik. See Legal & Interoperability for details.

Workspace structure

qvd-parser/
├── core/               # Core parsing library — QvdFile, QvdValue, RowIter
├── convert/            # Export backends: Parquet, JSON, CSV
├── cli/                # Command-line interface (qvd-parser-cli binary)
└── bindings-python/    # Python bindings via PyO3 + maturin

Prerequisites

• Rust toolchain (1.75+): https://rustup.rs
• For Python bindings: pip install maturin

Build

# Build all crates
cargo build --workspace

# Build release (recommended for performance measurement)
cargo build --release --workspace

# Run all Rust unit tests
cargo test --workspace

# Run integration tests (requires QVD fixtures — see below)
cargo test --workspace --features integration-tests

# Build the Python extension in-place (for development)
maturin develop

# Run Python binding tests
pytest bindings-python/tests/

CLI usage

# Schema inspection
cargo run -p qvd-parser-cli -- schema --input data.qvd

# Export to JSON
cargo run -p qvd-parser-cli -- export --input data.qvd --format json --output out.json

# Export to CSV (semicolon delimiter)
cargo run -p qvd-parser-cli -- export --input data.qvd --format csv --output out.csv --csv-delimiter ";"

# Export to Parquet (verbose timing breakdown)
cargo run --release -p qvd-parser-cli -- export --input data.qvd --format parquet --output out.parquet --verbose

Parquet export options

Flag	Default	Description
--row-group-size N	100000	Rows per Parquet row group
--encoding auto|plain|dictionary	auto	Arrow array encoding strategy (see below)
--no-statistics	off	Disable column min/max statistics (~20% faster export, less efficient predicate pushdown for consumers)
--strict	off	Fail immediately on cast errors instead of writing null

Verbose timing output

--verbose shows a per-phase breakdown of both parsing and export:

[qvd] export: data.qvd
  load header+symbols+rows        124 ms  [140 fields  119820 rows  126 B/rec]
    header                          1 ms
    symbols                        23 ms
    rows                           98 ms
  export parquet                  555 ms  [out.parquet]
    type inference                  0 ms
    index transpose                43 ms
    cast to arrow                 166 ms
    parquet encode                296 ms
    parquet close                  49 ms

Python usage

import qvd_parser

f = qvd_parser.QvdFile.open("data.qvd")
print(f.table_name, f.row_count, f.field_names())

for row in f.rows():
    print(row)  # list of Python-native values (int, float, str, tuple, None)

# Export
f.to_parquet("out.parquet")
f.to_json("out.json")
f.to_csv("out.csv", delimiter=";")

Python version compatibility

Tested against Python 3.13. CPython 3.11 and 3.12 are declared compatible for Snowflake runtime support but are not covered by CI.

Test fixtures

Integration tests require real QVD files generated by Qlik — synthetic QVD files are not possible due to the proprietary bit-packing format.

Place fixture files in fixtures/ at the workspace root:

File	Content
test_data01.qvd	2 columns (id: Int, label: Text), 2 rows: [1, "test1"], [2, "test2"]

Tests that rely on missing fixtures are either skipped or marked #[ignore].

Feature flags

Flag	Crate	Effect
integration-tests	core, convert	Enable tests that require fixture files
serde	core	Derive Serialize/Deserialize on public types

---

Release workflow

This project uses two separate repositories:

• GitLab (private) — daily development, full commit history
• GitHub (public) — published releases only, single commit per version

GitHub receives only validated, committed code from GitLab. The full development history never appears on GitHub.

Publishing a new release (Windows)

Ensure the version to publish is committed and pushed on GitLab main:

# 1. From the GitLab workspace, export the clean content
cd C:\path\to\qvd-parser
git checkout main
git pull origin main
git archive HEAD | tar -x -C C:\path\to\qvd-parser-public

# 2. Commit and push to GitHub
cd C:\path\to\qvd-parser-public
git add .
git commit -m "Release vX.Y.Z"
git push github main

git archive HEAD exports exactly the content of the last commit, respecting .gitignore. Files not committed (build artifacts, fixtures, .venv) are automatically excluded.

GitHub repository setup (once)

mkdir C:\path\to\qvd-parser-public
cd C:\path\to\qvd-parser-public
git init
git remote add github [e-mail]:nicolasstefaniuk/qvd-parser.git
git config user.email "[e-mail]"
git config user.name "Nicolas Stefaniuk"

---

Performance — Parquet export investigation

Context

QVD files use a columnar symbol table + row-major bit-packed index buffer. Parsing is inherently row-major (impossible to read one column without traversing all rows). Parquet is column-major. This impedance mismatch is the core performance challenge.

Measurements (release build, 140 fields × 119 820 rows × 16.5 MB QVD)

load header+symbols+rows    ~120 ms
  header                       1 ms
  symbols                     23 ms
  rows (bit-unpack)           95 ms

export parquet              ~650 ms
  type inference               0 ms
  index transpose             43 ms   ← column_indices_range
  cast to arrow              160 ms   ← QvdValue → Arrow arrays
  parquet encode             400 ms   ← RLE/dictionary encoding (Arrow writer)
  parquet close               48 ms

Ratio parsing:export ≈ 1:5. The bottleneck is the Arrow/Parquet writer, not our code.

Hypotheses tested

1. Cache thrashing on strided index access (fixed)

The original export collected per-column indices by striding through the row-major buffer (n_fields × 4 bytes per step), causing cache misses on wide files. Fixed by adding QvdFile::column_indices_range() in core: a single sequential pass that scatters into n_fields contiguous vectors. Measured gain: ~15% on index transpose.

2. Dictionary encoding for string columns (implemented, marginal gain)

With only 11–15 distinct values per column out of 119 820 rows, passing pre-built DictionaryArray<Int32, Utf8> to the Parquet writer should avoid per-row string copies and skip the encoder's internal deduplication.

Result: ~6% total gain in release, negligible. The Arrow writer (arrow-rs 53.x) still does substantial work on DictionaryArray. Additionally, numeric dictionary arrays (DictionaryArray<Int32, Int64> etc.) are not supported by ArrowWriter in this version and panic at runtime — restricted to Utf8 only.

Available as --encoding dictionary for experimentation. Not recommended for production use until arrow-rs improves dictionary support.

3. Column statistics (implemented, ~20% gain)

Parquet column statistics (min/max per page) cost ~130 ms on this file. They are enabled by default because they allow query engines to skip row groups during reads, which typically saves far more time than the export cost.

Disabling them with --no-statistics saves ~20% export time. Use when export speed is critical and the consumer does not rely on statistics.

Conclusion

The 5× ratio between parsing and Parquet export is inherent to the Arrow writer, not to our code. The practical levers available in our layer are:

Optimisation	Gain	Status
column_indices_range (sequential scatter)	~15% index transpose	Enabled by default
--no-statistics	~20% parquet encode	Opt-in flag
--encoding dictionary	<10% on Utf8 columns	Experimental, opt-in

A more significant improvement would require either a lower-level Parquet crate (e.g. parquet2) or parallelising column encoding — both are substantial changes outside the current scope.

---

Legal & Interoperability

Trademark notice

"Qlik", "QlikView", and "Qlik Sense" are registered trademarks of Qlik Technologies Inc. The file extension "QVD" is a technical descriptor for the file format, not a registered trademark.

This project is not affiliated with, endorsed by, or sponsored by Qlik in any way. The name qvd-parser is used solely to describe what the library does — reading files in the QVD format — and constitutes nominative descriptive use, not trademark infringement.

Reverse engineering and interoperability

This library was developed by observing the structure of QVD files produced by Qlik software. No Qlik source code was copied, decompiled, or incorporated. The parser is an independent implementation.

Reading a file format for the purpose of interoperability is explicitly permitted under applicable law:

• European Union — Directive 2009/24/EC on the legal protection of computer programs, Article 6, permits decompilation and reverse engineering for interoperability purposes, notwithstanding any contractual restriction to the contrary.
• United States — 17 U.S.C. § 1201(f) (DMCA) provides a specific exemption for reverse engineering undertaken to achieve interoperability. This is supported by established case law (Sega Enterprises Ltd. v. Accolade, Inc., 977 F.2d 1510; Sony Computer Entertainment, Inc. v. Connectix Corp., 203 F.3d 596).

File formats — as distinct from software implementations — are not protected by copyright as a matter of EU and US law (SAS Institute Inc. v. World Programming Ltd, CJEU C-406/10, 2012).

Enterprise users who require legal sign-off before adopting a dependency are encouraged to share this section with their legal team.

License

This project is licensed under the MIT License. See LICENSE for the full text.

Copyright (c) 2025 Nicolas Stefaniuk