Skip to main content

influxio

Tests Test coverage License Downloads

Python versions Status PyPI version

» Documentation | Changelog | PyPI | Issues | Source code | License

About

You can use influxio to import and export data into/from InfluxDB. It can be used both as a standalone program, and as a library.

influxio is, amongst others, based on the excellent dask, fsspec, influxdb-client, influx-line, line-protocol-parser, pandas, Polars, and SQLAlchemy packages.

Synopsis

# Export from API to database.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "sqlite://export.sqlite?table=demo"

# Export from data directory to line protocol format.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp"

Quickstart

If you are in a hurry, and want to run influxio without any installation, just use the OCI image on Podman or Docker.

docker run --rm --network=host ghcr.io/daq-tools/influxio \
    influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "crate://crate@localhost:4200/testdrive/demo"

Setup

Install influxio from PyPI.

pip install influxio

Usage

This section outlines some example invocations of influxio, both on the command line, and per library use. Other than the resources available from the web, testing data can be acquired from the repository's testdata folder.

Prerequisites

For properly running some of the example invocations outlined below, you will need an InfluxDB and a CrateDB server. The easiest way to spin up those instances is to use Podman or Docker.

Please visit the docs/development.rst documentation to learn about how to spin up corresponding sandbox instances on your workstation.

Command line use

Help

influxio --help
influxio info
influxio copy --help

Import

Import data from different sources into InfluxDB Server.

# From test data to API.
# Choose one of dummy, mixed, dateindex, wide.
influxio copy \
    "testdata://dateindex/" \
    "http://example:token@localhost:8086/testdrive/demo"

# With selected amount of rows.
influxio copy \
    "testdata://dateindex/?rows=42" \
    "http://example:token@localhost:8086/testdrive/demo"

# With selected amount of rows and columns (only supported by certain test data sources).
influxio copy \
    "testdata://wide/?rows=42&columns=42" \
    "http://example:token@localhost:8086/testdrive/demo"

# From line protocol file to InfluxDB API.
influxio copy \
    "file://tests/testdata/basic.lp" \
    "http://example:token@localhost:8086/testdrive/demo"

# From line protocol file to InfluxDB API.
influxio copy \
    "https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
    "http://example:token@localhost:8086/testdrive/demo"

Export from API

Export data from InfluxDB Server into different sinks.

# From API to database file.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "sqlite:///export.sqlite?table=demo"

# From API to database server.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "crate://crate@localhost:4200/testdrive/demo"

# From API to line protocol file.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "file://export.lp"

# From API to line protocol on stdout.
influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "file://-?format=lp"

Load from File

Load data from InfluxDB files into any SQL database supported by SQLAlchemy.

# From local line protocol file to SQLite.
influxio copy \
    "file://export.lp" \
    "sqlite:///export.sqlite?table=export"

# From local line protocol file to CrateDB.
influxio copy \
    "file://export.lp" \
    "crate://crate@localhost:4200/testdrive/demo"

# From remote line protocol file to SQLite.
influxio copy \
    "https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
    "sqlite:///export.sqlite?table=air-sensor-data"

# From remote line protocol file to CrateDB.
influxio copy \
    "https://github.com/influxdata/influxdb2-sample-data/raw/master/air-sensor-data/air-sensor-data.lp" \
    "crate://crate@localhost:4200/testdrive/demo"

Export from Cloud to Cloud

# From InfluxDB Cloud to CrateDB Cloud.
influxio copy \
    "https://8e9ec869a91a3517:T268DVLDHD8AJsjzOEluu...Pic4A==@eu-central-1-1.aws.cloud2.influxdata.com/testdrive/demo" \
    "crate://admin:dZ,Y18*Z...7)6LqB@green-shaak-ti.eks1.eu-west-1.aws.cratedb.net:4200/testdrive/demo?ssl=true"

crash \
    --hosts 'https://admin:dZ,Y18*Z...7)6LqB@green-shaak-ti.eks1.eu-west-1.aws.cratedb.net:4200' \
    --command 'SELECT * FROM testdrive.demo;'

Export from data directory

# From InfluxDB data directory to line protocol file.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp"

# From InfluxDB data directory to line protocol file, compressed with gzip.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://export.lp.gz"

# From InfluxDB data directory to line protocol on stdout.
influxio copy \
    "file:///path/to/influxdb/engine?bucket-id=372d1908eab801a6&measurement=demo" \
    "file://-?format=lp"

OCI

OCI images are available on the GitHub Container Registry (GHCR). In order to run them on Podman or Docker, invoke:

docker run --rm --network=host ghcr.io/daq-tools/influxio \
    influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "stdout://export.lp"

If you want to work with files on your filesystem, you will need to either mount the working directory into the container using the --volume option, or use the --interactive option to consume STDIN, like:

docker run --rm --volume=$(pwd):/data ghcr.io/daq-tools/influxio \
    influxio copy "file:///data/export.lp" "sqlite:///data/export.sqlite?table=export"

cat export.lp | \
docker run --rm --interactive --network=host ghcr.io/daq-tools/influxio \
    influxio copy "stdin://?format=lp" "crate://crate@localhost:4200/testdrive/export"

In order to always run the latest nightly development version, and to use a shortcut for that, this section outlines how to use an alias for influxio, and a variable for storing the input URL. It may be useful to save a few keystrokes on subsequent invocations.

docker pull ghcr.io/daq-tools/influxio:nightly
alias influxio="docker run --rm --interactive ghcr.io/daq-tools/influxio:nightly influxio"
SOURCE=https://github.com/daq-tools/influxio/raw/main/tests/testdata/basic.lp
TARGET=crate://crate@localhost:4200/testdrive/basic

influxio copy "${SOURCE}" "${TARGET}"

InfluxDB parameters

timeout

The network timeout value is specified in seconds, the default value is 60 seconds. Both details deviate from the standard default setting of the underlying InfluxDB client library, which uses milliseconds, and a default value of 10_000 milliseconds.

If you need to adjust this setting, add the parameter timeout to the InfluxDB URL like this:

influxio copy \
    "http://example:token@localhost:8086/testdrive/demo?timeout=300" \
    "crate://crate@localhost:4200/testdrive/demo"

CrateDB parameters

if-exists

When targeting the SQLAlchemy database interface, the target table will be created automatically, if it does not exist. The if-exists URL query parameter can be used to configure this behaviour. The default value is fail.

  • fail: Raise a ValueError.
  • replace: Drop the table before inserting new values.
  • append: Insert new values to the existing table.

Example usage:

influxio copy \
    "http://example:token@localhost:8086/testdrive/demo" \
    "crate://crate@localhost:4200/testdrive/demo?if-exists=replace"

Project information

Contribute

Contributions of all kinds are much very welcome, in order to make the software more solid.

For installing the project from source, please follow the development documentation.

Status

Breaking changes should be expected until a 1.0 release, so version pinning is recommended, especially when you use it as a library.

Prior art

There are a few other projects which are aiming at similar goals.

Supported by

JetBrains logo.

Special thanks to the people at JetBrains s.r.o. for supporting us with excellent development tooling.

Metadata

Release files for influxio 0.7.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for influxio 0.7.4
File Size Uploaded Provenance Actions
influxio-0.7.4.tar.gz 25.9 kB Without provenance Details for influxio-0.7.4.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for influxio 0.7.4
File Interpreter ABI Platform Actions
influxio-0.7.4-py3-none-any.whl Python 3 none any Details for influxio-0.7.4-py3-none-any.whl

Total release size: 49.5 kB

Release files / influxio-0.7.4.tar.gz

Download URL influxio-0.7.4.tar.gz
Size 25.9 kB
Tags Source
SHA-256 checksum
How to use checksums
8b33886468d56632bdb1959fcb57c6ccbceba8e072c52403702e3489bcc43819
BLAKE2b-256 checksum
How to use checksums
572811330bc2399290f732a3b616afa9072b1869230a5bf6b0e7c9056d6a4029
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / influxio-0.7.4-py3-none-any.whl

Download URL influxio-0.7.4-py3-none-any.whl
Size 23.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5a3f249fd8f55b95c1fd905545470295fb4961f06590ae00bc600003fb895bf6
BLAKE2b-256 checksum
How to use checksums
62123726ea3435c5aa4d7edc17c918f00512f931d4826d44be2ca2c27a8a2d42
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for influxio 0.7.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page