umwelt-apy 0.2.0

umwelt-apy Python package

The umwelt-apy package provides a Python-based access to the (meta-)datasets of umwelt.info, the central portal for German environmental data. The package, therefore, allows for an easy integration of those datasets into your Python-based workflows. The functionality of the package mirrors the webbased access as provided at https://umwelt.info and https://md.umwelt.info/. You can use the same queries and get the same datasets by accessing our API. The datasets are returned in either a JSON format or as Python dataframes like pandas and polars.

For more information, the API endpoint provides a SwaggerUI description of the data model where the search for multiple datasets is explained.
See also the article on our portal for further details.

Installation

The umwelt-apy package can be installed from PyPi using:

# basic version (requests only)
pip install umwelt-apy

# for using pandas
pip install "umwelt-apy[pandas]"

# for using pandas and polars
pip install "umwelt-apy[pandas,polars]"

You can also install the development version of the package from our GitLab repository using:

pip install umwelt-apy --extra-index-url https://gitlab.opencode.de/api/v4/projects/9710/packages/pypi/simple

Usage

The package provides a number of functions for querying umwelt.info and representing the resulting datasets in Python. It returns these datasets in a variety of formats, like a generator object of JSON entries, a pandas.DataFrame or a polars.DataFrame. We recommend using a polars.DataFrame, as it is the most performance-efficient way to retrieve large amounts of data due to using the column-oriented Arrow IPC format.

Core Functions

• fetch_by_url(url) — This function is ideal for reproducing an earlier search of yours on our web-based interface. You can use a button in our web interface to create a url, which fetches the same datasets you found in said search.
• fetch_by_query(query="*", ...) — This function sends a search query to umwelt.info and fetches the same datasets you would get from using this query at our web-based interface. The fetch_by_query function is easy to use for simple queries, however, reproducing all available filters through the query can be challenging. In that case, users may prefer the fetch_by_url() function instead.
• fetch_by_ids(ids) — This function fetches datasets by their unique identifier. This is ideal for when the IDs are already known; e.g., this can be useful for downloading resources. After using preview_resources you can select a subset of from the preview list using fetch_by_ids() and forward it as input to download_resources().
• download_resources(url) — This function downloads all the resources attached to the datasets of a respective query.

Examples

Example 1: Fetching datasets by a direct URL The basic function of umwelt_apy is fetch_by_url(url="*", ...). An example that queries umwelt.info for 'Luftqualität' and prints the title of all resulting datasets would be

from umwelt_apy import fetch_by_query

for dataset in fetch_by_url(url="https://md.umwelt.info/search?query=luftqualit%C3%A4t"):
    print(dataset['title'])

Example 2: Fetching by query string For background how to build a query see https://md.umwelt.info/swagger-ui/#/search/text_search If you want to know which facet values exist for a certain facet, you can use fetch_facet_values. Let's say you want to only fetch datasets from the data portal of Bavaria.

from umwelt_apy import fetch_by_query, fetch_facet_values

organisations = fetch_facet_values("organisation")
print(organisations)
results = list(fetch_by_query(query="organisation:/Land/Bayern/open.bydata AND Ozon"))

First, you fetch all organisation values. Then, you use the corresponding facet value in your query, and finally, you use the refined query to only fetch "Luftqualität" entries from the Bavarian data portal.

Example 3: Download resources attached to the datasets of a given query This is a four step process. First you retrieve the list of resources from the api (fetch_by_url or fetch_by_query), you unnest and optionally refine it further (unnest_and_filter), you screen the preview (preview_resources) and download the resources (download_resources). Note that the unnesting is a prerequisite for preview_resources() and download_resources(). Currently this workflow only works for output = "Pandas". See the tutorial for further details.

url = "https://md.umwelt.info/search/all?query=(Ozon)+AND+organisation%3A%2FLand%2FBayern%2Fopen.bydata"
from umwelt_apy import fetch_by_url, unnest_and_filter, preview_resources, download_resources
import pandas
results = (
    fetch_by_url(
        api_url = url,
        columns=["source", "id", "title", "resources"],
        filter_datasets = lambda dataset: "resources" in dataset,
        output = "Pandas",
    )
    .pipe(unnest_and_filter, formats=["MicrosoftExcelSpreadsheet", "Csv"], description_regex="Ozon")
    .pipe(preview_resources)
)

In case the list of resources in the preview should be downloaded

download_resources(results, base_dir="downloads")

More examples are provided in the docstrings of each function.

Requirements

For umwelt-apy you need at least Python 3.11 or higher as well as the request package version 2.32 or higher. If you want to use Pandas or Polars dataframes, you need versions 2.2.3 or 1.26, respectively. Full details are provided in the pyproject.toml.

Contributing

Contributions are welcome! Whether you want to fix a bug, add a new feature, or improve the documentation. You can find the source code at https://gitlab.opencode.de/umwelt-info/packages