polars-marc21 0.1.0

A polars io plugin for reading MARC21 files

---

This project provides a toolkit for efficiently processing bibliographic records encoded in MARC 21, which is a popular file format used to exchange bibliographic data between libraries. In particular, the command line tool marc21 allows efficient filtering of records and extraction of data into a rectangular schema. Since the extracted data is in tabular form, it can be processed with popular frameworks such as Polars or Tidyverse. In addition, the Python package polars-marc21 provides a Polars extension that allows you to use the query syntax to create a DataFrame, without using the command line.

marc21-rs is developed by the Metadata Department of the German National Library (DNB). It is used for data analysis and for automating metadata workflows (data engineering) as part of automatic content indexing.

The marc21 tool provides the following commands:

• concat — Concatenate records from multiple inputs (alias cat)
• count — Print the number of records in the input data (alias cnt)
• dedup — Remove duplicate records from the input
• describe — Creates a frequency table of all subfield codes
• filter — Filter records that fulfill a specified condition
• frequency — Compute a frequency table of values (alias freq)
• hash — Compute SHA-256 checksum of records
• invalid — Output invalid records that cannot be decoded
• partition — Partition records by values
• print — Print records in human readable format
• sample — Select a random permutation of records
• split — Split the input into chunks of a given size

The polars-marc21 package uses the query engine to transform MARC21 records directly into a DataFrame:

>>> from polars_marc21 import scan_marc21
>>>
>>> filename = "DUMP.mrc.gz"
>>> query = "001, 075{ b | 2 == 'gndgen' }"
>>> header = "ppn,gndgen"
>>>
>>> df = scan_marc21(filename, query, header).collect()
>>> print(df)
shape: (7, 2)
┌───────────┬────────┐
│ ppn       ┆ gndgen │
│ ---       ┆ ---    │
│ str       ┆ str    │
╞═══════════╪════════╡
│ 118540238 ┆ p      │
│ 118572121 ┆ p      │
│ 118607626 ┆ p      │
│ 118632477 ┆ p      │
│ 040992020 ┆ u      │
│ 040992918 ┆ u      │
│ 040993396 ┆ u      │
└───────────┴────────┘

Check out the documentation to learn more about installing and using the tool.

Contributing

All contributors are required to "sign-off" their commits (using git commit -s) to indicate that they have agreed to the Developer Certificate of Origin.

This project uses a strict no AI / no LLM policy. Please do not use large language models (LLMs) to create issues, patches, pull requests, or comments. Although English is the preferred language, you are welcome to communicate in your native language.

License

This project is licensed under the European Union Public License 1.2.