Skip to main content

Miscellaneous CKAN utility scripts

Project description

# ckanny

## Introduction

ckanny is a [command line interface](#cli) for interacting with remote and local [CKAN](http://ckan.org/) instances. Under the hood, it uses [ckanutils](https://github.com/reubano/ckanutils).

With ckanny, you can

- Download a CKAN resource
- Create a CKAN package
- Update a CKAN DataStore from data in the FileStore
- Copy a FileStore resource from one CKAN instance to another
- and much more...

ckanny performs smart updates by computing the hash of a file and will only update the datastore if the file has changed. This allows you to schedule a script to run on a frequent basis, e.g., `@hourly` via a cron job, without updating the CKAN instance unnecessarily.

## Requirements

ckanny has been tested on the following configuration:

- MacOS X 10.9.5
- Python 2.7.9

ckanny requires the following in order to run properly:

- [Python >= 2.7](http://www.python.org/download) (MacOS X comes with python preinstalled)

## Installation

(You are using a [virtualenv](http://www.virtualenv.org/en/latest/index.html), right?)

sudo pip install ckanny

## CLI

ckanny comes with a built in command line interface `ckanny`.

### Usage

ckanny [<namespace>.]<command> [<args>]

### Examples

*show help*

ckanny -h

```bash
usage: ckanny [<namespace>.]<command> [<args>]

positional arguments:
command the command to run

optional arguments:
-h, --help show this help message and exit

available commands:
ver Show ckanny version

[ds]
delete Deletes a datastore table
update Updates a datastore table based on the current filestore resource
upload Uploads a file to a datastore table

[fs]
fetch Downloads a filestore resource
migrate Copies a filestore resource from one ckan instance to another
upload Updates the filestore of an existing resource or creates a new one

[hdx]
customize Introspects custom organization values

[pk]
create Creates a package (aka dataset)
```

*show version*

ckanny ver

*fetch a resource*

ckanny fs.fetch -k <CKAN_API_KEY> -r <CKAN_URL> <resource_id>

*show fs.fetch help*

ckanny fs.fetch -h

```bash
usage: ckanny fs.fetch
[-h] [-q] [-n] [-c CHUNKSIZE_BYTES] [-u UA] [-k API_KEY] [-r REMOTE]
[-d DESTINATION]
[resource_id]

Downloads a filestore resource

positional arguments:
resource_id the resource id

optional arguments:
-h, --help show this help message and exit
-q, --quiet suppress debug statements
-n, --name-from-id Use resource id for filename
-c CHUNKSIZE_BYTES, --chunksize-bytes CHUNKSIZE_BYTES
number of bytes to read/write at a time (default:
1048576)
-u UA, --ua UA the user agent (uses `CKAN_USER_AGENT` ENV if
available) (default: None)
-k API_KEY, --api-key API_KEY
the api key (uses `CKAN_API_KEY` ENV if available)
(default: None)
-r REMOTE, --remote REMOTE
the remote ckan url (uses `CKAN_REMOTE_URL` ENV if
available) (default: None)
-d DESTINATION, --destination DESTINATION
the destination folder or file path (default:
.)
```

*create a package*

ckanny pk.create -k <CKAN_API_KEY> -r <CKAN_URL> <org_id>

*create a package with resources*

ckanny pk.create -k <CKAN_API_KEY> -r <CKAN_URL> -f 'file1.csv,file2.csv' <org_id>

*show pk.create help*

ckanny pk.create -h

```bash
usage: /Users/reubano/.virtualenvs/ckan/bin/ckanny pk.create
[-h] [-q] [-u UA] [-k API_KEY]
[-r REMOTE] [-e END] [-S START]
[-L LOCATION] [-c CAVEATS] [-y TYPE]
[-T TAGS] [-t TITLE]
[-m {observed,other,census,survey,registry}]
[-d DESCRIPTION] [-f FILES] [-s SOURCE]
[-l LICENSE_ID]
[org_id]

Creates a package (aka dataset)

positional arguments:
org_id the organization id

optional arguments:
-h, --help show this help message and exit
-q, --quiet suppress debug statements
-u UA, --ua UA
the user agent (uses `CKAN_USER_AGENT` ENV if
available) (default: None)
-k API_KEY, --api-key API_KEY
the api key (uses `CKAN_API_KEY` ENV if available)
(default: None)
-r REMOTE, --remote REMOTE
the remote ckan url (uses `CKAN_REMOTE_URL` ENV if
available) (default: None)
-e END, --end END
Data end date
-S START, --start START
Data start date (default: 09/25/2015)
-L LOCATION, --location LOCATION
Location the data represents (default: world)
-c CAVEATS, --caveats CAVEATS
Package caveats
-y TYPE, --type TYPE
Package type (default: dataset)
-T TAGS, --tags TAGS
Comma separated list of tags
-t TITLE, --title TITLE
Package title (default: Untitled 2015-09-25
12:36:14.141533)
-m {observed,other,census,survey,registry}, --methodology {observed,other,census,survey,registry}
Data collection methodology (default: observed)
-d DESCRIPTION, --description DESCRIPTION
Dataset description (default: same as `title`)
-f FILES, --files FILES
Comma separated list of file paths to add
-s SOURCE, --source SOURCE
Data source (default: Multiple sources)
-l LICENSE_ID, --license-id LICENSE_ID
Data license (default: cc-by-igo)
```
## Configuration

ckanny will use the following [Environment Variables](http://www.cyberciti.biz/faq/set-environment-variable-linux/) if set:

Environment Variable|Description
--------------------|-----------
CKAN_API_KEY|Your CKAN API Key
CKAN_REMOTE_URL|Your CKAN instance remote url
CKAN_USER_AGENT|Your user agent

## Hash Table

In order to support file hashing, ckanny creates a hash table resource called `hash_table.csv` with the following schema:

field|type
------|----
datastore_id|text
hash|text

By default the hash table resource will be placed in the package `hash_table`. ckanny will create this package if it doesn't exist. Optionally, you can set the hash table package in the command line with the `-H, --hash-table` option, or in a Python file as the `hash_table` keyword argument to `CKAN`.

Example:

ckanny ds.update -H custom_hash_table 36f33846-cb43-438e-95fd-f518104a32ed

## Scripts

ckanny comes with a built in task manager `manage.py` and a `Makefile`.

### Setup

pip install -r dev-requirements.txt

### Examples

*Run python linter and nose tests*

```bash
manage lint
manage test
```

Or if `make` is more your speed...

```bash
make lint
make test
```

## Contributing

View [CONTRIBUTING.rst](https://github.com/reubano/ckanny/blob/master/CONTRIBUTING.rst)

## License

ckanny is distributed under the [MIT License](http://opensource.org/licenses/MIT), the same as [ckanutils](https://github.com/reubano/ckanutils).


=========
Changelog
=========

Here you can find the recent changes to ckanny..

.. changelog::
:version: dev
:released: Ongoing

.. change::
:tags: docs

Updated CHANGES.

.. changelog::
:version: 0.1.0
:released: 2015-06-12

.. change::
:tags: project

First release on PyPi.

.. todo:: vim: set filetype=rst:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

ckanny-0.15.1-py27-none-any.whl (18.7 kB view hashes)

Uploaded Python 2.7

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page