No project description provided
Project description
datagov-harvesting-logic
This is a library that will be utilized for metadata extraction, validation, transformation, and loading into the data.gov catalog.
Features
The datagov-harvesting-logic offers the following features:
- Extract
- general purpose fetching and downloading of web resources.
- catered extraction to the following data formats:
- DCAT-US
- Validation
- DCAT-US
- jsonschema validation using draft 2020-12.
- DCAT-US
- Load
- DCAT-US
- conversion of dcatu-us catalog into ckan dataset schema
- create, delete, update, and patch of ckan package/dataset
- DCAT-US
Requirements
This project is using poetry to manage this project. Install here.
Once installed, poetry install
installs dependencies into a local virtual environment.
Testing
CKAN load testing
- CKAN load testing doesn't require the services provided in the
docker-compose.yml
. - catalog-dev is used for ckan load testing.
- Create an api-key by signing into catalog-dev.
- Create a
credentials.py
file at the root of the project containing the variableckan_catalog_dev_api_key
assigned to the api-key. - run tests with the command
poetry run pytest ./tests/load/ckan
Harvester testing
- These tests are found in
extract
, andvalidate
. Some of them rely on services in thedocker-compose.yml
. run using dockerdocker compose up -d
and with the commandpoetry run pytest --ignore=./tests/load/ckan
.
If you followed the instructions for CKAN load testing
and Harvester testing
you can simply run poetry run pytest
to run all tests.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for datagov_harvesting_logic-0.0.2.post3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3b4780761f223e50cac02f526a9bf55df3312eaa834c7633323d2dae756e4fd |
|
MD5 | 4db6ae77a5d0daa098e2b0056b93279f |
|
BLAKE2b-256 | cf29ad7fda7a818d1003b23ef505058fcb85a829835fb57755d3d8f6f803f5ba |
Close
Hashes for datagov_harvesting_logic-0.0.2.post3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8eea7a6a65857da4e5ea424f7e1aa4714f4c7df3569c184a7dde80cc7cbf1893 |
|
MD5 | fd26e795ed4df1515c08a6b553ab8aa8 |
|
BLAKE2b-256 | 17c40c5402c071cd615caa361b86da32c0bf6c27b7c92c7cf3fb6e30e1512438 |