OAIPMH harvester
Project description
OARepo OAI-PMH harvester
An OAI-PMH harvesing library for Invenio 3.5+. The library provides initial transformation of OAI-PMH payload to an intermediary json representation which is later on transformed by a specific transformer to the format of invenio records.
Due to their generic nature, these transformers are not part of this library but have to be provided by an application.
The progress and transformation errors are captured within the database.
For now, the library does not provide error notifications, but these will be added. Sentry might be used for the logging & reporting.
Installation
pip install oarepo-oaipmh-harvester
pip install <your transformer library>
Configuration
All configuration is inside the OAIHarvesterRecord
record.
There is a command-line tool to add a new record:
invenio oarepo oai harvester add nusl \
--name "NUSL harvester" \
--url http://invenio.nusl.cz/oai2d/ \
--set global \
--prefix marcxml \
--loader sickle \
--transformer marcxml \
--transformer nusl \
--writer 'service{service=nr_documents}'
This will register an oai-pmh harvester with code "nusl", its url, oai set and metadata prefix. Records from this harvester will be loaded with the sickle loader (default loader if not specified) and at first transformed from marcxml to json format and subsequently by NUSL transformer to get nr_documents compatible json.
The json is then used by service writer to create/update the target record.
Usage
Command-line
On command line, invoke
invenio oarepo oai harvester run nusl <optional list of oai identifiers to harvest>
Options:
--all-records Re-harvest all records, not from the last timestamp
--on-background Run harvest on background (via celery task)
--identifier Harvest the passed identifier/s
You can also pass arguments for "havester add" to override the defaults from the configuration.
Harvest status
Harvester uses 3 additional types of record:
- Run represents a single run of the harvester
- Batch - the run consists of multiple batches, that is records, that are processed together.
Each such batch is represented in the
batch
record - OAI record is a link between harvested record and its original. If there are errors harvesting
the record, it is still created and its
errors
field is filled with the error
Custom parsers and transformers
Transformer
A piece of code that gets a StreamEntry (or a StreamBatch) instance, processes it and returns modified
StreamEntry. An example is a MarcXML transformer that takes the string with xml representation of the
entry and transforms it into simple json representation {abcxy: value(s)}
, where abcxy is marc field code.
See oarepo_runtime.datastream.transformers
package for StreamEntry/StreamBatch interfaces.
The transformer needs to be registered:
# mypkg.transformers
from .impl import MyTransformer
my_transformer = {"class": MyTransformer, "params": {
# default parameters that will go to the MyTransformer constructor
}}
And setup.cfg:
# setup.cfg
[options.entry_points]
oarepo.oaipmh.transformers =
my_transformer = mypkg.transformers:my_transformer
Then you can use my_transformer
when creating your harvester.
Reader
A reader is responsible for fetching records and creating a stream of StreamEntry items.
See oarepo_runtime.datastreams.readers
for details. Then register the reader into
oarepo.oaipmh.readers
entry point with the same syntax as above.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for oarepo-oai-pmh-harvester-4.0.8.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c51dcaac20781b603044683b2376fe67e7d284e89aa6d12839748887be9eb36 |
|
MD5 | d3bd70a3be576affeb3914b9fd7e23ef |
|
BLAKE2b-256 | 5534f5ea5dab7a642f127d8c8b51060a81d12bdb721b170679e0e3cc0b241d89 |
Hashes for oarepo_oai_pmh_harvester-4.0.8-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4cd90bf5192b5879aa17d8a97118c2caaa25666baa137ade23b9a626c0f6b15d |
|
MD5 | 53c8a87034fe5eb97393df5dcef855c6 |
|
BLAKE2b-256 | 9b5867b94a45d21f527e9380da004e52c34a1f5b0581e4a3bc3516e750756a76 |