Skip to main content

Useful towel-related stuff.

Project description

pySpark-connector-kairosdb provides you an easy way to get data from KairosDB and make it available on Spark as a DataFrame. It’s simple as that:

#!/usr/bin/env python

# sconnk means "py(S)park-(CONN)ector-(K)airosdb"
from sconnk import Connection, Dataframe

query_data = {
                    "start_relative": { "value": "5", "unit": "years" },
                    "metrics": [{ "name": "test_", "limit": 5 },
                                            { "name": "DP_058424", "limit": 10 },
                                            { "name": "teste_gzip", "limit": 5 },
                                            { "name": "DP_063321", "limit": 10 }]
    }

    # Creating a connection with KairosDB database (in the given address).
    conn = Connection("http://localhost:8080/")

    # Performing our query on KairosDB.
    json_data = conn.query(query_data)

    # Creating a new Dataframe object passing the JSON returned by KairosDB API.
    df = Dataframe(json_data).df

    # Print the dataframe.
    df.show()

Remember this’s a ALPHA module without good documentation, examples and even well implemented features. We’ve a long highway to cross.

Future

This module is in development and we’ve the following plannings for the future of this module:

  • Write good documentation

  • Write tests - a lot of them.

  • Add support to RDD

  • Don’t write to JSON file in order to parse it in Spark.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pySpark-connector-kairosdb-0.1.0.tar.gz (2.7 kB view details)

Uploaded Source

File details

Details for the file pySpark-connector-kairosdb-0.1.0.tar.gz.

File metadata

File hashes

Hashes for pySpark-connector-kairosdb-0.1.0.tar.gz
Algorithm Hash digest
SHA256 062a97f39546265aa4c9160b37b785e53b91d6deff7bd21adcd0bb7c0b05b9d5
MD5 ad38d0e8f3fbe1be6e05bddcfea5b09f
BLAKE2b-256 d5d81c0b46f8bf206e31a225788e6ac1601f565d3278f5623da052267048a522

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page