Skip to main content

bit.io Python SDK and CLI

Project description

bit.io Python SDK & Command Line Tool

bitdotio module

The bit.io Python SDK supports two major versions of bit.io: v1 and v2.

The v1 SDK is instantiated using a v1 bit.io API token and supports listings repos, running queries, and importing data using methods of the SDK object returned by the bitdotio function. The v1 SDK object can also generate pre-configured psycopg2 connections to v1 bit.io.

The v2 SDK currently only supports generation of pre-configured psycopg2 connections to v2 bit.io. We plan to add further SDK features for v2 based on user feedback.

Both versions of the SDK are instantiated by calling the bitdotio function with either a v1 or v2 API token. The version of SDK object returned is based on the version of the provided token.

Example v1 usage:

#!/usr/bin/env python3
import bitdotio
from pprint import pprint

# Connect to bit.io
b = bitdotio.bitdotio(<YOUR_API_KEY>)

# List repos
pprint(b.list_repos())

# How about some database queries?
conn = b.get_connection()
cur = conn.cursor()
cur.execute("SELECT 1")
pprint(cur.fetchone())

Example v2 usage:

#!/usr/bin/env python3
import bitdotio
from pprint import pprint

# Connect to bit.io
b = bitdotio.bitdotio(<YOUR_API_KEY>)

# How about some database queries?
# NOTE: v2 connections require specification of the database name, because
# v2 bit.io is built around independent user databases, not repos (schemas) on a single database.
conn = b.get_connection(<YOUR_DATABASE_NAME>)
cur = conn.cursor()
cur.execute("SELECT 1")
pprint(cur.fetchone())

Requirements

In order to support different environments, we have a few ways to install the bitdotio package with or without the psycopg2 dependency.

  1. If you already have psycopg2 installed, you can install the default bitdotio package:
pip install bitdotio
  1. If you already have Postgres installed, you can install with the psycopg2 dependency:
pip install 'bitdotio[psycopg2]'
  1. If you do not have or cannot install Postgres, you can install with the psycopg2-binary dependency:
pip install 'bitdotio[psycopg2-binary]'

Install Postgres

To install Postgres on Windows, go to https://www.postgresql.org/download/ and download the version that is correct for your computer, or use your operating system's preferred package manager.

After you have Postgres installed you can install this library with pip install bitdotio[psycopg2].

Usage

Once you have bitdotio installed all you need is your API key to start working with bit.io.

You can get your API key by logging into bit.io and scrolling down to the "Connect any data tool" box, clicking on "API/SDK", and copying the key from there.

See https://docs.bit.io/docs/connecting-to-bitio screenshots and examples.

Python DB-API usage

bitdotio provides easy Python access to querying your data with just a bit.io API key:

#!/usr/bin/env python3
import bitdotio

# Connect to bit.io
b = bitdotio.bitdotio(<YOUR_API_KEY>)

conn = b.get_connection()
# for v2: conn = b.get_connection(<YOUR_DATABASE_NAME>)
cur = conn.cursor()
cur.execute("SELECT 1")
print(cur.fetchone())

The connection and cursor provided by bitdotio are fully Python DB-API compatible, are in fact Pyscopg2 connections and cursors.

Full documentation on Psycopg2 can be found on https://www.psycopg.org/docs/usage.html.

v1 bit.io concepts

bit.io is a database, with extra features like easy sharing and collaboration. We have a few concepts that the SDK works with:

  • Repostories ("repos") - a schema, in Postgres terms, that contains tables and columns. You can have public and private repositories, and you can write SQL that joins any repo you have access to with another repo. A repo contains tables, and tables contain columns.

Detailed documentation on interacting with each concept with the SDK:

v1 bit.io CLI usage

Installing the bitdotio module also installs the command line tool bit which lets you interact with v1 bit.io from scripts or the command line. This is installed next to your python binary.

You'll want to grab your API key from your bit.io account - to get the key, log into to bit.io and click on the green "Connect" button, and copy the API key.

Note: v2 bit.io doesn't have a CLI yet. The CLI will not work with v2 bit.io tokens.

Usage: bit [OPTIONS] COMMAND [ARGS]...

Options:
  -k, --key TEXT  Your bit.io API key, available when you click the connect
                  button in bit.io  [required]

  -v, --verbose   Verbose output  [default: False]
  --help          Show this message and exit.

Commands:
  import
  query
  repo

All of the commands return JSON.

Note that while the examples below use the key on the command line, you can set the key via the enviornment variable BITIO_KEY (this keeps the key from showing up in, eg, a ps command, and allows secret injection for systems like Kubernetes):

BITIO_KEY=<your key> bit repo list

This is the same as:

bit -k <your_key> repo list

Importing Data

With the command line tool you can upload data from a file, from a url, or just use JSON directly. All the importing commands (aside from status) take an optional table_name to indicate the destination table name; if that table doesn't exist the import will create it. If you don't provide a table_name bit.io creates a table based on the filename.

Importing from a file

bit -k <your_key> import file -r my_new_repo -f foo.csv -t from_file

You'll get:

{
    "result": "success",
    "message": "Uploaded file addresses.csv saved successfully from sender adam@bit.io and an ingestion job has been created with the status RECEIVED",
    "table_name": "from_file",
    "repo_name": "my_new_repo",
    "job_id": "34a0539f-39f8-483c-84b5-c858c7d26e10",
    "file_name": "addresses.csv"
}

You can use that job_id in the status command.

You can also pipe your file in; if you do this, you must provide the table_name option:

cat foo.csv | bit -k <your_key> import file -r my_new_repo -f - --table_name foo_table

And you'll get:

{
    "result": "success",
    "message": "Uploaded file <stdin> saved successfully from sender adam@bit.io and an ingestion job has been created with the status RECEIVED",
    "table_name": "foo_table",
    "repo_name": "my_new_repo",
    "job_id": "c9a551b2-3149-4439-9527-952e7a3fe23",
    "file_name": "<stdin>"
}

Import status

bit -k <your_key> import status -i 34a0539f-39f8-483c-84b5-c858c7d26e10

And you'd get:

{
    "job_id": "34a0539f-39f8-483c-84b5-c858c7d26e10",
    "state": "DONE",
    "status_message": "",
    "retries": 0,
    "file_type": null,
    "original_filename": "addresses.csv",
    "repo_name": "my_new_repo",
    "table_name": "from_file"
}

Import from url

bit -k <your_key> import url -r my_new_repo -u https://storage.googleapis.com/bitdotio-demo-datasets/atl_2020_home_sales.csv

And you'll get:

{
    "result": "success",
    "message": "https://storage.googleapis.com/bitdotio-demo-datasets/atl_2020_home_sales.csv queued for processing successfully, and an ingestion job has been created with the status RECEIVED",
    "table_name": "atl_2020_home_sales.csv",
    "repo_name": "my_new_repo",
    "job_id": "a837e2ed-2bd2-44ae-a3f0-e18f52274061",
    "file_name": "atl_2020_home_sales.csv",
    "url": "https://storage.googleapis.com/bitdotio-demo-datasets/atl_2020_home_sales.csv"
}

Import JSON

echo "[[1,2], [3,4]]" | bit -k <your_key> import json-data -r my_new_repo -t json_test

And you'll get:

{
    "result": "success",
    "message": "Uploaded file adam_e6d3aaf4-0469-4bf7-a06a-eb6683c1e563.json saved successfully from sender adam@bit.io and an ingestion job has been created with the status RECEIVED",
    "table_name": "json_test",
    "repo_name": "my_new_repo",
    "job_id": "436d0335-184c-42f1-98b3-d24be2ef7701",
    "file_name": "adam_e6d3aaf4-0469-4bf7-a06a-eb6683c1e563.json"
}

Querying data

You can also query your data with the command line tool:

bit -k <your_key> query -q "SELECT *  FROM \"a/demo_repo\".\"atl_home_sales\""

Or from stdin:

echo  "SELECT *  FROM \"a/demo_repo\".\"atl_home_sales\"" | bit -k <your_key> query -qf -

Repo Management

List repos

bit -k <your_key> repo list

You'll get something like:

[
    {
        "name": "demo_repo",
        "description": "",
        "documentation": "",
        "bytes": 622592,
        "is_private": true
    },
    {
        "name": "my_new_repo",
        "description": "",
        "documentation": "",
        "bytes": 13099008,
        "is_private": true
    },
    {
        "name": "my-repo",
        "description": "my-repo",
        "documentation": "",
        "bytes": 16384,
        "is_private": true
    },
    {
        "name": "power-usage",
        "description": "",
        "documentation": "",
        "bytes": 16384,
        "is_private": true
    }
]

Create a repo

bit -k <your_key> repo create -r my_new_repo

And you'll get:

{
    "name": "my_new_repo",
    "description": "",
    "documentation": "",
    "bytes": 0,
    "is_private": true
}

Get a specific repo's information

bit -k <your_key> repo retrieve -r my_new_repo

And you'll get:

{
    "name": "my_new_repo",
    "description": "",
    "documentation": "",
    "bytes": 0,
    "is_private": true
}

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bitdotio-1.1.0b1.tar.gz (60.0 kB view hashes)

Uploaded source

Built Distribution

bitdotio-1.1.0b1-py3-none-any.whl (78.2 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page