Gridworks Uploader

This package provides a reliable upload service using the gridworks-protocol.

The upload services communicates upstream using MQTT. Clients deliver data to the service for reliable delivery using http. For example:

    import random
    import time
    from typing import Literal
    
    import httpx
    from gwproto.messages import EventBase
    
    
    class SomeData(EventBase):
        TimestampUTC: float
        Reading: float
        TypeName: Literal["gridworks.event.some.data"] = "gridworks.event.some.data"
    
    
    if __name__ == "__main__":
        httpx.post(
            "http://127.0.0.1:8080/events",
            json=SomeData(
                TimestampUTC=round(time.time(), 3),
                Reading=round(random.random(), 3),
            ).model_dump(),
        )

Experimentation

To experiment with this package you must run the upload service, and, if you want to watch your messages delivered to a stub ingester, you must also run an MQTT broker and the stub ingester.

To set up the MQTT broker, follow the gridworks-proactor instructions.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv python install 3.12
uv tool install -p 3.12 poetry=1.8.5
git clone https://github.com/SmoothStoneComputing/gridworks-uploader.git
cd gridworks-uploader

Create a .env file at the location returned by:

gwup envfile

with these contents:

UPLOADER_APP_lONG_NAME = "test.uploader"
UPLOADER_APP_INGESTER_LONG_NAME = "test.ingester"

Create a second .env at the location returned by:

gwup stubs ingester envfile

with these contents:

STUB_INGESTER_APP_lONG_NAME = "test.ingester"
STUB_INGESTER_APP_UPLOADER_LONG_NAME = "test.uploader"

Create local test certificate authority:

poetry install --sync --with dev
poetry shell
gwcert ca create test-ca

Generate test certs for the uploader and the stub ingester:

gwup gen-test-certs
gwup stubs ingester gen-test-certs

Open 3 terminals. In each terminal, cd to the gridworks-uploader repo and run:

poetry shell

In the ingester terminal run:

gwup stubs ingester run --log-events

In the uploader terminal run:

gwup run --message-summary

In the client terminal run:

gwup stubs client run 

Or:

python src/gwupload/stubs/client/client.py 

Installation

You can install Gridworks Uploader via pip from PyPI:

uv tool install gridworks-uploader

or

pipx install gridworks-uploader

Service installation

Gridworks-Uploader can be installed as a systemd service, for example on a Raspberry Pi, using gwup service command line interface. The gwup service commands are mostly wrappers around systemctl and journalctl. To see the systemctl and journalctl command without running them simply pass --dry-run to any gwup service command.

Unit file generation

The configuration for the service is stored in a unit file. To generate a service file run:

gwup service generate

or, if the service will not run under default user, 'pi', but the user USER_NAME, run:

gwup service generate --user USER_NAME

The unit file can be also be written or edited by hand. The service file is generated at $HOME/.config/gridworks/uploader/gridworks-uploader.service. That path can be viewed any time with:

gwup service file

Once the service is installed changes to the unit file will not take effect unless the unit file is reloaded into systemd with sudo systemctl daemon-reload and the service is restarted. This can be accomplished with:

gwup service reload
gwup service restart

Installation

To install and run the service run:

gwup service install

To stop and uninstall the service run:

gwup service uninstall

The service can be re-installed without re-generating the unit file.

Starting and stopping

The service can be started, restarted and stopped:

gwup service start

gwup service restart

gwup service stop

A stopped service will restart when the device reboots. Use gwup service uninstall to prevent the service from restarting when the device reboots.

Watching

The current state of the service can be seen with:

gwup service status

The log of the service can be followed with:

gwup service log

Usage

Please see the Command-line Reference for details.

Contributing

Contributions are very welcome. To learn more, see the Contributor Guide.

License

Distributed under the terms of the MIT license, Gridworks Uploader is free and open source software.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Credits

This project was generated from @cjolowicz's Hypermodern Python Cookiecutter template.