onc: client library for accessing Oceans 3.0 API

onc is a Python client library that facilitates access to scientific data hosted by Ocean Networks Canada through the Oceans 3.0 API public web services. It can help you explore and download our data, by consuming our discovery, data product download, archive file download, and near real-time data access services.

Getting Started

Installation

onc can be installed from PyPI:

pip install onc

Obtaining a token

A unique Oceans 3.0 API token is required to access our data. To obtain a token, follow the steps below:

1. Register for an Oceans 3.0 account at https://data.oceannetworks.ca/Registration.

2. Log into your account at https://data.oceannetworks.ca by clicking the Log In link.

3. Click the Profile link (top right corner) to access your account profile.

4. Access the Web Services API tab and click Copy Token.

5. If you forget your token, you can always find it again in your Oceans 3.0 account profile.

Searching with discovery methods

To download ONC data, you need to specify the type of data you require and where in particular (i.e. location, device) it originates from.

In the Oceans 3.0 API, there's a unique code that identifies every location, device, property, data product type, etc. Include these codes in a group of filters (these will be used as URL parameters when making HTTP requests) that determine the data you're interested in.

Discovery methods allow you to explore the hierarchy of the ONC database to obtain the codes for your filters (they work like a "search" function).

The example below uses the getLocations method to search for locations that include "Burrard" in their name (i.e. "Burrard Inlet"):

from onc import ONC

onc = ONC("YOUR_TOKEN")
onc = ONC() # This works if the env variable "ONC_TOKEN" is set

onc.getLocations({"locationName": "Burrard"})

The previous code prints a list with locations that match the search filters provided. Each location in the list includes a dataSearchURL that points to the Data Search Tool, and a locationCode ("BIPP" and "BISS" in this example) that can be used to continue searching "inside" it, as in the following example:

onc.getDeviceCategories({"locationCode": "BIIP"})
onc.getDataProducts({"locationCode": "BIIP", "deviceCategoryCode": "CTD"})

Check more on the discovery methods guide and code examples.

Downloading data products

Once you determine the exact dictionary of filters that identifies the data you are interested in, there are multiple methods to download it.

One method is to request the ONC servers to generate a custom data product with the data. This is done through the data product download methods.

The following example downloads two PNG files with plots for 30 seconds of data from a CTD in Campbell River:

params = {
    "locationCode": "BIIP",
    "deviceCategoryCode": "CTD",
    "dataProductCode": "TSSP",
    "extension": "png",
    "dateFrom": "2019-06-20T00:00:00.000Z",
    "dateTo": "2019-06-20T00:30:00.000Z",
    "dpo_qualityControl": "1",
    "dpo_resample": "none",
}
onc.orderDataProduct(params, includeMetadataFile=False)

The filters above include codes for location, deviceCategory, and dataProduct, as well as the file extension and a time interval (in UTC). They also include a couple of filters to configure this specific data product type (starting with the "dpo_" prefix), which can be obtained from the Data Product Options documentation. You can download more than 120 different types of data products including audio & video.

Check more on the data product download methods guide and code examples.

Obtaining sensor readings in (near) real-time

Another method to obtain ONC data is by directly obtaining a time series of sensor readings (available as soon as they reach our database).

In the following example, we obtain 5 seconds of conductivity readings from the CTD at Burrard Inlet:

params = {
    "locationCode": "BIIP",
    "deviceCategoryCode": "CTD",
    "propertyCode": "conductivity",
    "dateFrom": "2019-06-20T00:00:00.000Z",
    "dateTo": "2019-06-20T00:00:05.000Z",
}
onc.getScalardata(params)

# Longer method name
# onc.getScalardataByLocation(params)

The result includes matching lists of "values" and "sampleTimes" (increases performance for long time ranges). We also use the property code "conductivity" to limit results to a specific property available in this CTD.

Check more on the near real-time data access methods guide and code examples.

Downloading archived files

ONC scripts auto-generate and archive data products of different types at set time intervals. You can directly download these data product files from our files archive, as long as you know their unique filename.

In the following example, we get a list of archived files available for a camera at Ridley Island (in a certain time span), and download one of the files:

params = {
    "locationCode": "RISS",
    "deviceCategoryCode": "VIDEOCAM",
    "dateFrom": "2016-12-01T00:00:00.000Z",
    "dateTo": "2016-12-01T00:05:00.000Z",
}
result = onc.getArchivefile(params, allPages=True)

# Longer method name
# result = onc.getArchivefileByLocation(params, allPages=True)

# Download one of the files from result["files"]
onc.downloadArchivefile("AXISQ6044PTZACCC8E334C53_20161201T000001.000Z.jpg", overwrite=True)

You can use the method downloadArchivefile() as above to download individual files or the method downloadDirectArchivefile() to download all the files that match your filters.

Alternatively, if you prefer using a download manager like aria2 or Free Download Manager, getArchivefileUrls and getArchivefileUrl can return the download URLs of the archivefile without downloading the files.

Check more on the archive file download methods guide and code examples.

Documentation

The client library documentation is hosted on GitHub Pages. For documentation and examples about Oceans 3.0 API, visit the wiki and OpenAPI page on the Oceans 3.0 Data Portal website.

Multithreading issue

We kindly ask users to not use too many threads when using threading/multiprocessing libraries on download tasks. It can cause issues for both server and client and may not appreciably increase download speeds.

Contributing

All contributions are welcome and appreciated! Please refer to the Contributing guide before submitting any issues or pull requests.