qvd-parser 0.2.1

Fast QVD file parser — Python bindings for qvd-parser (Rust)

qvd-parser

Fast, correct QVD (QlikView Data) file parser written in Rust.

Reads the binary format used by QlikView and Qlik Sense to store compressed data tables. Exposes a Rust API, a Python API (via PyO3), and a CLI. Export targets: JSON, CSV, Parquet.

Note: QVD is a proprietary file format associated with QlikView and Qlik Sense. This project is not affiliated with or endorsed by Qlik. See Legal & Interoperability for details.

Workspace structure

qvd-parser/
├── core/               # Core parsing library — QvdFile, QvdValue, RowIter
├── convert/            # Export backends: Parquet, JSON, CSV
├── cli/                # Command-line interface (qvd-parser-cli binary)
└── bindings-python/    # Python bindings via PyO3 + maturin

Prerequisites

• Rust toolchain (1.75+): https://rustup.rs
• For Python bindings: pip install maturin

Build

# Build all crates
cargo build --workspace

# Build release (recommended for performance measurement)
cargo build --release --workspace

# Run all Rust unit tests
cargo test --workspace

# Run integration tests (requires QVD fixtures — see below)
cargo test --workspace --features integration-tests

# Build the Python extension in-place (for development)
maturin develop

# Run Python binding tests
pytest bindings-python/tests/

CLI usage

# Schema inspection
cargo run -p qvd-parser-cli -- schema --input data.qvd

# Export to JSON
cargo run -p qvd-parser-cli -- export --input data.qvd --format json --output out.json

# Export to CSV (semicolon delimiter)
cargo run -p qvd-parser-cli -- export --input data.qvd --format csv --output out.csv --csv-delimiter ";"

# Export to Parquet (verbose timing breakdown)
cargo run --release -p qvd-parser-cli -- export --input data.qvd --format parquet --output out.parquet --verbose

Parquet export options

Flag	Default	Description
--cast FIELD=TYPE	—	Force a column to a specific Arrow type. Repeat for multiple fields. Valid types: INT32, INT64, FLOAT32, FLOAT64, STRING, DATE32, TIMESTAMP_MS, BOOLEAN. Takes priority over --schema-overrides on conflict.
--schema-overrides FILE	—	Path to a JSON file containing per-field type overrides. Same {"FieldName": "TYPE"} format as --cast but file-based. Inline --cast values take priority on conflict.
--fallback-type TYPE	STRING	Type used when inference is inconclusive — i.e. a column's symbol table is all-null/empty and NumberFormat carries no recognised hint either. Same valid types as --cast. Never overrides a genuine text result from the symbol scan.
--row-group-size N	100000	Rows per Parquet row group
--strict-first	off	Fail immediately on the first cast error instead of writing null. Mutually exclusive with --strict-all.
--strict-all	off	Run the full export collecting every cast error, then fail with the complete list if any occurred. The output file is still fully written (failed values as null) even though the command exits with an error. Mutually exclusive with --strict-first.
--encoding auto|plain|dictionary	auto	Arrow array encoding strategy. CLI-only, experimental — not exposed in Python bindings (see performance notes below).
--no-statistics	off	Disable column min/max statistics (~20% faster export, less efficient predicate pushdown for consumers). CLI-only, experimental — not exposed in Python bindings.

By default (no --strict-first/--strict-all), cast failures are written as null and reported as warnings on stderr after the export completes — nothing is silently lost without a trace:

warning: 3 value(s) failed to cast and were written as null:
  field 'Amount' row 42: cannot parse 'N/A' as FLOAT64: invalid float literal
  field 'Amount' row 109: cannot parse 'N/A' as FLOAT64: invalid float literal
  field 'Date' row 7: cannot convert text 'unknown' to DATE32 — use a numeric day-offset value

# Force specific column types
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --cast DateField=DATE32 --cast Amount=FLOAT64

# Use a JSON override file
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --schema-overrides overrides.json

# Both combined — inline --cast takes priority on conflict
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --schema-overrides overrides.json --cast DateField=TIMESTAMP_MS

# Fail with a complete error report if any value fails to cast
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --strict-all

# All-null/empty columns default to STRING; force INT64 instead
cargo run --release -p qvd-parser-cli -- export \
  --input data.qvd --format parquet --output out.parquet \
  --fallback-type INT64

Verbose timing output

--verbose shows a per-phase breakdown of both parsing and export:

[qvd] export: data.qvd
  load header+symbols+rows        124 ms  [140 fields  119820 rows  126 B/rec]
    header                          1 ms
    symbols                        23 ms
    rows                           98 ms
  export parquet                  555 ms  [out.parquet]
    type inference                  0 ms
    index transpose                43 ms
    cast to arrow                 166 ms
    parquet encode                296 ms
    parquet close                  49 ms

Type inference

For each column without an explicit --cast/schema_overrides entry, the target Arrow type is resolved in this order:

1. Symbol scan — the full symbol table is scanned; the dominant non-null variant wins (Int64/Float64 only if zero text symbols are present — any text presence forces STRING, since a numeric type would silently null out non-numeric values).
2. NumberFormat.type_name hint — used only when the symbol scan is inconclusive (column is all-null or empty). NumberFormat is a Qlik display format declaration, not a data-type guarantee — a pure-text field can legally carry <Type>REAL</Type> — so it is never trusted over actual stored data.
3. --fallback-type — last resort, used only when both the symbol scan and the NumberFormat hint are inconclusive. Defaults to STRING.

Cast failures (e.g. a --cast Field=INT64 override on a column containing non-numeric text) do not abort the export by default — see --strict-first / --strict-all above to change that, and the warning output shown there for how failures are reported.

Python usage

import qvd_parser

# From a file path (Rust opens the file directly)
f = qvd_parser.QvdFile.open("data.qvd")
print(f.table_name, f.row_count, f.field_names())

for row in f.rows():
    print(row)  # list of Python-native values (int, float, str, tuple, None)

# Export
f.to_parquet("out.parquet")
f.to_parquet("out.parquet", cast_overrides={"DateField": "DATE32", "Amount": "FLOAT64"})
f.to_parquet("out.parquet", schema_overrides_file="overrides.json")
f.to_parquet("out.parquet", strict_mode="strict_first")  # raise on first cast error
f.to_parquet("out.parquet", strict_mode="strict_all")    # raise with the full error list
f.to_parquet("out.parquet", fallback_type="INT64")       # all-null columns default to INT64 instead of STRING
f.to_json("out.json")
f.to_csv("out.csv", delimiter=";")

# to_parquet() returns a CastWarnings object (default strict_mode="silent"):
# empty/falsy if every value cast successfully, otherwise inspect .entries
warnings = f.to_parquet("out.parquet", cast_overrides={"Amount": "FLOAT64"})
if warnings:
    for w in warnings.entries:
        print(w)  # "field 'Amount' row 42: cannot parse 'N/A' as FLOAT64: ..."

Parsing from in-memory bytes

When the QVD is already in memory — downloaded from a signed URL, read from object storage, or loaded via any other source that does not produce a file path — use from_bytes():

# From a signed URL (no temporary file needed)
import requests
data = requests.get(signed_url).content
f = qvd_parser.QvdFile.from_bytes(data)

# AWS S3
data = s3_client.get_object(Bucket="my-bucket", Key="data.qvd")["Body"].read()
f = qvd_parser.QvdFile.from_bytes(data)

# Python file handle (equivalent to open(), Python controls the lifecycle)
with open("data.qvd", "rb") as fh:
    f = qvd_parser.QvdFile.from_bytes(fh.read())

All methods available on open() results are available on from_bytes() results: rows(), to_dict(), to_parquet(), to_json(), to_csv(), etc.

Note on memory usage: from_bytes() requires the entire QVD to be in RAM. This is unavoidable when the source is an in-memory buffer. If you have a file path and memory is a concern, prefer open() — it lets the OS manage buffering.

Python version compatibility

Tested against Python 3.13. CPython 3.11 and 3.12 are declared compatible for Snowflake runtime support but are not covered by CI.

Test fixtures

Integration tests require real QVD files generated by Qlik — synthetic QVD files are not possible due to the proprietary bit-packing format.

Place fixture files in fixtures/ at the workspace root:

File	Content
test-data-minimal.qvd	2 columns (id: Int, label: Text), 2 rows: [1, "test1"], [2, "test2"] — primary fixture for Rust and Python tests
all-nulls.qvd	2 columns, nullable_col entirely NULL — 3 rows
all-types.qvd	6 columns covering all QvdValue variants: Int, Double, Text, DualInt, DualDouble, Null
single-row.qvd	Exactly 1 row — iterator and size_hint edge case
empty-table.qvd	0 rows, schema present — 2 columns
large-integers.qvd	Values at i32::MIN/i32::MAX and byte/word boundaries
unicode-strings.qvd	Multi-byte UTF-8 strings: accented Latin characters, CJK
high-cardinality.qvd	100 rows, unique_label with 100 distinct values
wide-record.qvd	RecordByteSize > 16 — regression fixture for the u128 overflow bug
mixed-types.qvd	Heterogeneous symbol table: Int, Double, Text, NULL in the same column
bench-symbols.qvd	1 000 rows × 3 fields, high-cardinality symbol tables — benchmark fixture
bench-rows.qvd	5 000 rows × 8 fields, mixed cardinality, all QvdValue variants — benchmark fixture

All fixtures must be generated with Qlik (QlikView or Qlik Sense) using generate-fixtures.qvs. Synthetic QVD generation is not possible due to the proprietary bit-packing format.

Tests that rely on missing fixtures are skipped at runtime when --features integration-tests is active.

Feature flags

Flag	Crate	Effect
integration-tests	core, convert	Enable tests that require fixture files
serde	core	Derive Serialize/Deserialize on public types

Python binding entry points

Method	Input	Use case
QvdFile.open(path: str)	File path	Rust opens and buffers the file — preferred when a path is available
QvdFile.from_bytes(data: bytes | bytearray | memoryview)	In-memory bytes	QVD already in RAM (signed URL, object storage, BytesIO)

---

Release workflow

This project uses two separate repositories:

• GitLab (private) — daily development, full commit history
• GitHub (public) — published releases only, single commit per version

GitHub receives only validated, committed code from GitLab. The full development history never appears on GitHub.

Publishing a new release (Windows)

Ensure the version to publish is committed and pushed on GitLab main:

# 1. From the GitLab workspace, export the clean content
cd C:\path\to\qvd-parser
git checkout main
git pull origin main
git archive HEAD | tar -x -C C:\path\to\qvd-parser-public

# 2. Commit and push to GitHub
cd C:\path\to\qvd-parser-public
git add .
git commit -m "Release vX.Y.Z"
git push github main

git archive HEAD exports exactly the content of the last commit, respecting .gitignore. Files not committed (build artifacts, fixtures, .venv) are automatically excluded.

GitHub repository setup (once)

mkdir C:\path\to\qvd-parser-public
cd C:\path\to\qvd-parser-public
git init
git remote add github [e-mail]:nicolasstefaniuk/qvd-parser.git
git config user.email "[e-mail]"
git config user.name "Nicolas Stefaniuk"

---

Performance — Parquet export investigation

Context

QVD files use a columnar symbol table + row-major bit-packed index buffer. Parsing is inherently row-major (impossible to read one column without traversing all rows). Parquet is column-major. This impedance mismatch is the core performance challenge.

Measurements (release build, 140 fields × 119 820 rows × 16.5 MB QVD)

load header+symbols+rows    ~120 ms
  header                       1 ms
  symbols                     23 ms
  rows (bit-unpack)           95 ms

export parquet              ~650 ms
  type inference               0 ms
  index transpose             43 ms   ← column_indices_range
  cast to arrow              160 ms   ← QvdValue → Arrow arrays
  parquet encode             400 ms   ← RLE/dictionary encoding (Arrow writer)
  parquet close               48 ms

Ratio parsing:export ≈ 1:5. The bottleneck is the Arrow/Parquet writer, not our code.

Hypotheses tested

1. Cache thrashing on strided index access (fixed)

The original export collected per-column indices by striding through the row-major buffer (n_fields × 4 bytes per step), causing cache misses on wide files. Fixed by adding QvdFile::column_indices_range() in core: a single sequential pass that scatters into n_fields contiguous vectors. Measured gain: ~15% on index transpose.

2. Dictionary encoding for string columns (implemented, marginal gain)

With only 11–15 distinct values per column out of 119 820 rows, passing pre-built DictionaryArray<Int32, Utf8> to the Parquet writer should avoid per-row string copies and skip the encoder's internal deduplication.

Result: ~6% total gain in release, negligible. The Arrow writer (arrow-rs 53.x) still does substantial work on DictionaryArray. Additionally, numeric dictionary arrays (DictionaryArray<Int32, Int64> etc.) are not supported by ArrowWriter in this version and panic at runtime — restricted to Utf8 only.

Available as --encoding dictionary for experimentation. Not recommended for production use until arrow-rs improves dictionary support.

3. Column statistics (implemented, ~20% gain)

Parquet column statistics (min/max per page) cost ~130 ms on this file. They are enabled by default because they allow query engines to skip row groups during reads, which typically saves far more time than the export cost.

Disabling them with --no-statistics saves ~20% export time. Use when export speed is critical and the consumer does not rely on statistics.

Conclusion

The 5× ratio between parsing and Parquet export is inherent to the Arrow writer, not to our code. The practical levers available in our layer are:

Optimisation	Gain	Status
column_indices_range (sequential scatter)	~15% index transpose	Enabled by default
--no-statistics	~20% parquet encode	Opt-in flag
--encoding dictionary	<10% on Utf8 columns	Experimental, opt-in

A more significant improvement would require either a lower-level Parquet crate (e.g. parquet2) or parallelising column encoding — both are substantial changes outside the current scope.

---

Legal & Interoperability

Trademark notice

"Qlik", "QlikView", and "Qlik Sense" are registered trademarks of Qlik Technologies Inc. The file extension "QVD" is a technical descriptor for the file format, not a registered trademark.

This project is not affiliated with, endorsed by, or sponsored by Qlik in any way. The name qvd-parser is used solely to describe what the library does — reading files in the QVD format — and constitutes nominative descriptive use, not trademark infringement.

Reverse engineering and interoperability

This library was developed by observing the structure of QVD files produced by Qlik software. No Qlik source code was copied, decompiled, or incorporated. The parser is an independent implementation.

Reading a file format for the purpose of interoperability is explicitly permitted under applicable law:

• European Union — Directive 2009/24/EC on the legal protection of computer programs, Article 6, permits decompilation and reverse engineering for interoperability purposes, notwithstanding any contractual restriction to the contrary.
• United States — 17 U.S.C. § 1201(f) (DMCA) provides a specific exemption for reverse engineering undertaken to achieve interoperability. This is supported by established case law (Sega Enterprises Ltd. v. Accolade, Inc., 977 F.2d 1510; Sony Computer Entertainment, Inc. v. Connectix Corp., 203 F.3d 596).

File formats — as distinct from software implementations — are not protected by copyright as a matter of EU and US law (SAS Institute Inc. v. World Programming Ltd, CJEU C-406/10, 2012).

Enterprise users who require legal sign-off before adopting a dependency are encouraged to share this section with their legal team.

License

This project is licensed under the MIT License. See LICENSE for the full text.

Copyright (c) 2025 Nicolas Stefaniuk