Skip to main content

A tool to load bulk simulation data into AWS IoT Analytics for experimenting with the capabilities of AWS IoT Analytics. This tool is intended to load simulation data that can then be utilized to explore the capabilities of Datasets, integration with Amazon QuickSight, etc.

Installation:

  • Make sure you have a working Python environment
  • pip install onica-iotloader to get the latest version installed

Template

A user-provided template is invoked to create each sample message. Arbitrary Python code can be used to generate each message via this template, enabling a rich set of simulated values. The template file must set a local variable named message to the value of a single simulated message, each time it is invoked. See template.py in this repository as an example of a static, but very large (~ 127kb) message template.

Usage

onica-iotloader [--concurrency=<n>] <channel> <template> <count>

A concurrency value of 1000 seems to be ideal to maximize the ingest rate of AWS IoT Analytics from a single source (achieving approximately 3200 msgs/sec, or 400 mb/sec on an m5.24xlarge). The <channel> must be an existing AWS IoT Analytics channel, <template> is the template Pyton script described above, and <count> is the number of messages (within a margin of error) to transmit. For reference, a count of 900000 results in just over 100GB of ingest using the default template.py included herein.

To load about 100GB in 4 minutes, run the following command using the template.py included in this repository (tested on m5.24xlarge):

onica-iotloader --concurrency=1000 <channel> template.py 900000

Replace <channel> with the name of an AWS IoT Analytics Channel configured in your account.

AWS Credentials & Region

The default boto3 AWS credential & region resolution (env vars, EC2 metadata, config file, etc) is utilized.

Feedback

Contact us at opensource at onica.com, or via Github issues, with feedback!

Metadata

Release files for onica-iotloader 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for onica-iotloader 0.1.2
File Size Uploaded Provenance Actions
onica_iotloader-0.1.2.tar.gz 4.1 kB Without provenance Details for onica_iotloader-0.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for onica-iotloader 0.1.2
File Interpreter ABI Platform Actions
onica_iotloader-0.1.2-py2.py3-none-any.whl Python 2, Python 3 none any Details for onica_iotloader-0.1.2-py2.py3-none-any.whl

Total release size: 13.0 kB

Release files / onica_iotloader-0.1.2.tar.gz

Download URL onica_iotloader-0.1.2.tar.gz
Size 4.1 kB
Tags Source
SHA-256 checksum
How to use checksums
786ca40e3ff65134101b96f3081d6a8eae2ba296d702354bea5606b104c23e50
BLAKE2b-256 checksum
How to use checksums
7f245295c98082ff504a81dabd67887edf5209d770258f989ce78e96e01136a1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.10

Release files / onica_iotloader-0.1.2-py2.py3-none-any.whl

Download URL onica_iotloader-0.1.2-py2.py3-none-any.whl
Size 8.9 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
1a861348b4db9d3aa4f204081436ce3b3f8df715f3d7677dd89840e1095271fd
BLAKE2b-256 checksum
How to use checksums
c1ff8f81fafd07a10bfae41eb56ac3ee8dd913437e82c23f4f6f3877ea4769de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.10

Security for onica-iotloader 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.2 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page