Skip to main content

Smart Citizen Data

DOI Binder PyPI version Python application

Welcome to SmartCitizen Data. This is a data analysis framework for working with sensor data in different ways:

  • Interacting with several sensors APIs
  • Clean data, export and calculate metrics
  • Model sensor data and calibrate sensors
  • Generate data visualisations - matplotlib, plotly or uplot
  • Generate analysis reports in html or pdf and upload them to Zenodo

Installation

You can check it out in the Binder before installing if you want. Works with Python 3.10 or later (tested from 3.10 to 3.13).

You can just run:

pip install scdata

Plots (device.ts_plot(), maps, panels...) need the plotting libraries, an optional extra. Without them scdata loads and processes data, and plotting raises an error that says how to install them:

pip install "scdata[plot]"

Parquet needs the storage extra: the backups (local or S3: device.load_from_storage(), device.backup_to_storage()) and exports in parquet (device.export(..., file_format='parquet')):

pip install "scdata[storage]"

Examples (scripts and notebooks) are in examples.

Work on the source code

Simply clone the repository with:

git clone https://github.com/fablabbcn/smartcitizen-data.git
cd smartcitizen-data

Install scdata package with requirements:

python setup.py install

Or if you want to edit:

pip install --editable ".[dev]"

Tokens and config

Blueprints, calibrations and sensor names are read from BASE_POSTPROCESSING_URL, the same variable as smartcitizen-connector. By default it is Smart Citizen Flows, which manages this metadata.

A device of the Smart Citizen API is processed with a blueprint of its hardware (hardware_url in its postprocessing). A hardware lists its blueprints, at most one of each kind: process, long and backup. blueprint_kind selects the one to use, which also defines the sensors to load. Without it, the process blueprint, else the long one:

device = sc.Device(params=sc.APIParams(id=18154))
device.blueprint, device.blueprint_kind     # ('sc_air', 'process')
device.handler.blueprints                   # {'process': 'sc_air'}: the blueprints of its hardware

device = sc.Device(params=sc.APIParams(id=<device-id>), blueprint_kind='long')

The blueprint of the hardware is used before the one given (blueprint), unless override_hardware_blueprint=True. device.blueprint_kind is the kind of the blueprint the device uses. A device without a blueprint (no postprocessing, a hardware that is only backed up, or without a blueprint of the kind asked for) loads its sensors, but has no channels to process, and its blueprint and blueprint_kind are None.

Settings can be changed in ~/.config/scdata/config.yaml. The file only keeps what differs from the defaults (the defaults of new versions apply after upgrading), for instance:

data:
  reload_metadata: false
paths:
  export: /home/me/exports

The data folder is ~/.cache/scdata by default ($XDG_CACHE_HOME/scdata if it is set), with the raw, processed, export... folders inside. Each folder of paths is changed on its own, and only to a folder that exists: changing data does not move the others. To move them all, set XDG_CACHE_HOME.

Also, .env files will be picked from ~/.cache/scdata. An env.example is provided. If you want to upload data to Zenodo, you will need to fill set an environment variable called ZENODO_TOKEN in your environment. The .env file is not mandatory for scdata to work. It only serves for authentication if you want to POST or GET data without throttling. You can still do GET requests but they will be throttled if not authenticated/authorized by the platform. See user roles for more info.

Using with jupyter lab (optional)

It can also be used with jupyter lab or jupyter. For this install juypterlab.

Contribute

Issues and PR more than welcome!

Funding

This work has received funding from the European Union's Horizon 2020 research and innovation program under the grant agreement No. 689954

Metadata

Release files for scdata 1.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for scdata 1.8.0
File Size Uploaded Provenance Actions
scdata-1.8.0.tar.gz 344.1 kB Without provenance Details for scdata-1.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for scdata 1.8.0
File Interpreter ABI Platform Actions
scdata-1.8.0-py3-none-any.whl Python 3 none any Details for scdata-1.8.0-py3-none-any.whl

Total release size: 714.9 kB

Release files / scdata-1.8.0.tar.gz

Download URL scdata-1.8.0.tar.gz
Size 344.1 kB
Tags Source
SHA-256 checksum
How to use checksums
b2ac65434f1cec51734a49b1b246c9bbc2e90baa4ed027c6d93a3b3443b114b3
BLAKE2b-256 checksum
How to use checksums
8f5750e594ae1851af132ac71c9ee4b318b0aa590ae774ed7bb62b72acf445f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / scdata-1.8.0-py3-none-any.whl

Download URL scdata-1.8.0-py3-none-any.whl
Size 370.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
abdf9611089ce6bcbf1c51fa1d1655422af9ea6bd87a55537721c4093ce9dd9f
BLAKE2b-256 checksum
How to use checksums
3890df32f064368c26f0d87cec3d42af71a37bc1ae44c589ddc1e2f530b91444
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for scdata 1.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.8.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page