A tool to load bulk simulation data into AWS IoT Analytics for experimenting with the capabilities of AWS IoT Analytics. This tool is intended to load simulation data that can then be utilized to explore the capabilities of Datasets, integration with Amazon QuickSight, etc.
Installation:
- Make sure you have a working Python environment
pip install onica-iotloaderto get the latest version installed
Template
A user-provided template is invoked to create each sample message. Arbitrary Python code can be used to generate each message via this template, enabling a rich set of simulated values. The template file must set a local variable named message to the value of a single simulated message, each time it is invoked. See template.py in this repository as an example of a static, but very large (~ 127kb) message template.
Usage
onica-iotloader [--concurrency=<n>] <channel> <template> <count>
A concurrency value of 1000 seems to be ideal to maximize the ingest rate of AWS IoT Analytics from a single source (achieving approximately 3200 msgs/sec, or 400 mb/sec on an m5.24xlarge). The <channel> must be an existing AWS IoT Analytics channel, <template> is the template Pyton script described above, and <count> is the number of messages (within a margin of error) to transmit. For reference, a count of 900000 results in just over 100GB of ingest using the default template.py included herein.
To load about 100GB in 4 minutes, run the following command using the template.py included in this repository (tested on m5.24xlarge):
onica-iotloader --concurrency=1000 <channel> template.py 900000
Replace <channel> with the name of an AWS IoT Analytics Channel configured in your account.
AWS Credentials & Region
The default boto3 AWS credential & region resolution (env vars, EC2 metadata, config file, etc) is utilized.
Feedback
Contact us at opensource at onica.com, or via Github issues, with feedback!
Metadata
Release files for onica-iotloader 0.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| onica_iotloader-0.1.2.tar.gz | 4.1 kB | Without provenance | Details for onica_iotloader-0.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| onica_iotloader-0.1.2-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details for onica_iotloader-0.1.2-py2.py3-none-any.whl |
Total release size: 13.0 kB
Release files / onica_iotloader-0.1.2.tar.gz
| Download URL | onica_iotloader-0.1.2.tar.gz |
|---|---|
| Size | 4.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
786ca40e3ff65134101b96f3081d6a8eae2ba296d702354bea5606b104c23e50
|
|
BLAKE2b-256 checksum How to use checksums |
7f245295c98082ff504a81dabd67887edf5209d770258f989ce78e96e01136a1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.10
|
Release files / onica_iotloader-0.1.2-py2.py3-none-any.whl
| Download URL | onica_iotloader-0.1.2-py2.py3-none-any.whl |
|---|---|
| Size | 8.9 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
1a861348b4db9d3aa4f204081436ce3b3f8df715f3d7677dd89840e1095271fd
|
|
BLAKE2b-256 checksum How to use checksums |
c1ff8f81fafd07a10bfae41eb56ac3ee8dd913437e82c23f4f6f3877ea4769de
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.12.1 pkginfo/1.4.2 requests/2.19.1 setuptools/40.4.3 requests-toolbelt/0.8.0 tqdm/4.27.0 CPython/2.7.10
|
Security for onica-iotloader 0.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.