No project description provided
Project description
datagov-harvesting-logic
This is a library that will be utilized for metadata extraction, validation, transformation, and loading into the data.gov catalog.
Features
- Extract
- General purpose fetching and downloading of web resources.
- Catered extraction to the following data formats:
- DCAT-US
- Validation
- DCAT-US
jsonschema
validation using draft 2020-12.
- DCAT-US
- Load
- DCAT-US
- Conversion of dcat-us catalog into ckan dataset schema
- Create, delete, update, and patch of ckan package/dataset
- DCAT-US
Requirements
This project is using poetry
to manage this project. Install here.
Once installed, poetry install
installs dependencies into a local virtual environment.
Testing
CKAN load testing
- CKAN load testing doesn't require the services provided in the
docker-compose.yml
. - catalog-dev is used for ckan load testing.
- Create an api-key by signing into catalog-dev.
- Create a
credentials.py
file at the root of the project containing the variableckan_catalog_dev_api_key
assigned to the api-key. - Run tests with the command
poetry run pytest ./tests/load/ckan
Harvester testing
- These tests are found in
extract
, andvalidate
. Some of them rely on services in thedocker-compose.yml
. Run using dockerdocker compose up -d
and with the commandpoetry run pytest --ignore=./tests/load/ckan
.
If you followed the instructions for CKAN load testing
and Harvester testing
you can simply run poetry run pytest
to run all tests.
Comparison
-
./tests/harvest_sources/ckan_datasets_resp.json
- Represents what ckan would respond with after querying for the harvest source name
-
./tests/harvest_sources/dcatus_compare.json
-
Represents a changed harvest source
-
Created:
-
datasets[0]
+ "identifier" = "cftc-dc10"
-
-
Deleted:
-
datasets[0]
- "identifier" = "cftc-dc1"
-
-
Updated:
-
datasets[1]
- "modified": "R/P1M" + "modified": "R/P1M Update"
-
datasets[2]
- "keyword": ["cotton on call", "cotton on-call"] + "keyword": ["cotton on call", "cotton on-call", "update keyword"]
-
datasets[3]
"publisher": { "name": "U.S. Commodity Futures Trading Commission", "subOrganizationOf": { - "name": "U.S. Government" + "name": "Changed Value" } }
-
-
-
./test/harvest_sources/dcatus.json
- Represents an original harvest source prior to change occuring.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for datagov_harvesting_logic-0.3.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5c095c2cd84aafd59b68b43259baec746a8cfa25aedd40c6990359332eda15e |
|
MD5 | e45a182f4210ddab3aed4788bd262d03 |
|
BLAKE2b-256 | 4b982ee96c1a218fc1d59f48f053a16e69d3052ffe53b9532cf49ce0cb32b3c4 |
Hashes for datagov_harvesting_logic-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 64ab86b03e4f8aa46f5d7ee51aaa64f786d6561a8d9e4e6a59ae39f02e3b45dc |
|
MD5 | 7ece9860e4fed75d8cc1ddba061621ce |
|
BLAKE2b-256 | 1291c297e8b43580a0a4cda76ea96206c5b9b3e3db93d5f72391632acafac235 |