polars-marc21 0.3.0

This project provides a toolkit for efficiently processing bibliographic records encoded in MARC 21, which is a popular file format used to exchange bibliographic data between libraries. In particular, the command line tool marc21 allows efficient filtering of records and extraction of data into a rectangular schema. Since the extracted data is in tabular form, it can be processed with popular frameworks such as Polars or Tidyverse. In addition, the Python package polars-marc21 provides a Polars extension that allows you to use the query syntax to create a DataFrame, without using the command line.

marc21-rs is developed by the Metadata Department of the German National Library (DNB). It is used for data analysis and for automating metadata workflows (data engineering) as part of automatic content indexing.

Command-Line Interface

The marc21 tool provides the following commands:

• check — Validate records against rule sets
• concat — Concatenate records from multiple inputs (alias cat)
• count — Print the number of records in the input data (alias cnt)
• dedup — Remove duplicate records from the input
• describe — Creates a frequency table of all subfield codes
• filter — Filter records that fulfill a specified condition
• frequency — Compute a frequency table of values (alias freq)
• glimpse — Print a dense preview of a data field
• grep — Search for records whose values match a pattern
• hash — Compute SHA-256 checksum of records
• invalid — Output invalid records that cannot be decoded
• partition — Partition records by values
• print — Print records in human readable format
• sample — Select a random permutation of records
• select — Transforms records into CSV or TSV format
• skosify — Convert records to SKOS/RDF
• split — Split the input into chunks of a given size

Check out the documentation to learn more about installing and using the tool.

Polars Integration

The polars-marc21 package uses the query engine to transform MARC21 records directly into a DataFrame:

from polars_marc21 import marc21_select

df = (
    marc21_select("001, 075{ b | 2 == 'gndgen' }")
    .header(["ppn", "gndgen"])
    .from_("DUMP.mrc.gz")
    .where("ldr.type == 'z'")
    .collect()
)

print(df)

shape: (7, 2)
┌───────────┬────────┐
│ ppn       ┆ gndgen │
│ ---       ┆ ---    │
│ str       ┆ str    │
╞═══════════╪════════╡
│ 118540238 ┆ p      │
│ 118572121 ┆ p      │
│ 118607626 ┆ p      │
│ 118632477 ┆ p      │
│ 040992020 ┆ u      │
│ 040992918 ┆ u      │
│ 040993396 ┆ u      │
└───────────┴────────┘

Contributing

All contributors are required to "sign-off" their commits (using git commit -s) to indicate that they have agreed to the Developer Certificate of Origin.

This project uses a strict no AI / no LLM policy. Please do not use large language models (LLMs) to create issues, patches, pull requests, or comments. Although English is the preferred language, you are welcome to communicate in your native language.

License

This project is licensed under the European Union Public License 1.2.