A library for mapping CKAN metadata <=> Frictionless metadata.
Project description
Frictionless CKAN Mapper
A library for mapping CKAN metadata <=> Frictionless metadata.
The library has zero dependencies (not even on Data Package libs). You can use it directly or use it for inspiration. Detailed outline of the algorithm is in the docs or you can read the code.
Installation
- Python: install Python. The library is compatible with both Python 2.7+ and Python 3.3+.
pip install frictionless-ckan-mapper
Note: The package is installed as frictionless-ckan-mapper
and then imported as frictionless_ckan_mapper
.
Getting started
CKAN => Frictionless
# get a CKAN metadata item
ckan_dataset = {
"name": "my-dataset",
"title": "My awesome dataset",
"url": "http://www.example.com/data.csv"
}
# or load from an API e.g.
# ckan_dataset = json.load(urllib.urlopen(
# https://demo.ckan.org/api/3/package_show?id=my_dataset
# ))
from frictionless_ckan_mapper import ckan_to_frictionless as converter
# convert to frictionless
frictionless_package = converter.dataset(ckan_dict)
print(frictionless_package)
Frictionless => CKAN
frictionless = {
'name': "f11s-dataset",
'path': "https://datahub.io/data.csv"
}
from frictionless_ckan_mapper import frictionless_to_ckan as f2c
ckanout = f2c.dataset(frictionless)
print(ckanout)
Reference
This package contains two modules:
frictionless_to_ckan
ckan_to_frictionless
You can import them directly like so:
from frictionless_ckan_mapper import ckan_to_frictionless
from frictionless_ckan_mapper import frictionless_to_ckan
ckan_to_frictionless
resource(ckandict)
from frictionless_ckan_mapper import ckan_to_frictionless as converter
# ... Some code with a CKAN dictionary ...
output_frictionless_dict = converter.resource(ckan_dictionary)
dataset(ckandict)
from frictionless_ckan_mapper import ckan_to_frictionless as converter
# ... Some code with a CKAN dictionary ...
output_frictionless_dict = converter.dataset(ckan_dictionary)
frictionless_to_ckan
resource(fddict)
from frictionless_ckan_mapper import frictionless_to_ckan as converter
# ... Some code with a Frictionless dictionary ...
output_ckan_dict = converter.resource(frictionless_dictionary)
package(fddict)
from frictionless_ckan_mapper import frictionless_to_ckan as converter
# ... Some code with a Frictionless dictionary ...
output_ckan_dict = converter.package(frictionless_dictionary)
Design
Frictionless <=> CKAN
--------------------------------------
Data Package <=> Package (Dataset)
Data Resource <=> Resource
Table Schema <=> Data Dictionary?? (datastore resources can have schemas)
CKAN reference
Summary:
- Class diagram below of key objects (without attributes)
- Objects with their attributes in this spreadsheet: https://docs.google.com/spreadsheets/d/1XdqGTFni5Jfs8AMbcbfsP7m11h9mOHS0eDtUZtqGVSg/edit#gid=1925460244
classDiagram
class Package
class Resource
class DataDictionary
Package *-- Resource
Resource o-- DataDictionary
Source for CKAN metadata structure:
- Dataset (Package): https://docs.ckan.org/en/2.8/api/index.html#ckan.logic.action.create.package_create
Package.as_dict
method: https://github.com/ckan/ckan/blob/2.8/ckan/model/package.py#L195-L223package_show
...
Algorithm: CKAN => Frictionless
See the code in frictionless_ckan_mapper/ckan_to_frictionless.py
Algorithm: Frictionless => CKAN
See the code in frictionless_ckan_mapper/frictionless_to_ckan.py
Developers
Install the source
-
Clone the repo:
git clone https://github.com/frictionlessdata/frictionless-ckan-mapper.git
-
And install it with pip:
pip install -e .
Run the tests
Use the excellent pytest
suite as follows:
pytest tests
To test under both Python 2 and Python 3 environments, we use tox
. You can run the following command:
make test
Note: Make sure that the necessary Python versions are in your environment PATH
(Python 2.7 and Python 3.6).
Building and publishing the package
To see a list of available commands from the Makefile
, execute:
make list
Build the distribution package locally for testing purposes
If a previous build exists, make sure to also remove it before building again:
make distclean
Then:
make dist
Alternatively, this command will accomplish the same to build packages for both Python 2 and Python 3:
python setup.py sdist bdist_wheel --universal
Test the package at test.pypi.org
python -m twine upload --repository testpypi dist/*
The package will be publicly available at https://test.pypi.org/project/frictionless-ckan-mapper/ and you will be able to pip install
it as usual.
Tag a new Git release and publish to the official PyPi
Make sure to update the version of the package in the file frictionless_ckan_mapper/VERSION
. Then:
make release
You can quickly review the version to release with make version
, which will print the current version stored in VERSION
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for frictionless-ckan-mapper-1.0.9.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9972b94f59709bf83928a02f9581a42981364a69c2a150053178427432b37893 |
|
MD5 | b4d3151fc3c85ea4f493f8b822183f04 |
|
BLAKE2b-256 | a4b507ba152e84f3b48dc3a8542dd5d8a8bb2212199809a338d1ed430e20bbdd |
Hashes for frictionless_ckan_mapper-1.0.9-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d95af2d5a56bf1a300a0a20c713067e98d0b71a54cd7dfc3acb737dd4db3f968 |
|
MD5 | c09b033c4d0ace02a40d6a397df856e7 |
|
BLAKE2b-256 | b9602e01c7e6fca00f3763f24ea1f960a0830709eba36846f013740341e783d9 |