Skip to main content

A Python datalake client.

Project description

Ness

A Python datalake client.

Test Coverage Package version

Requirements

Installation

pip install pyarrow ness

Quickstart

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake")
df = dl.read("mytable")

Sync

# Sync all tables
dl.sync()

# Sync a single table
dl.sync("mytable")

# Sync and read a single table
df = dl.read("mytable", sync=True)

Format

Specify the input data source format, the default format is parquet:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", format="csv")

AWS Profile

Files are synced using default AWS profile, you can configure another one:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", profile="myprofile")

Command Line

Usage: ness sync [OPTIONS] S3_URI

Options:
  --format TEXT   Data lake source format.
  --profile TEXT  AWS profile.
  --table TEXT    Table name to sync.
  --help          Show this message and exit.
ness sync bucket/key --table mytable

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ness-0.1.5.tar.gz (4.0 kB view details)

Uploaded Source

Built Distribution

ness-0.1.5-py3-none-any.whl (5.8 kB view details)

Uploaded Python 3

File details

Details for the file ness-0.1.5.tar.gz.

File metadata

  • Download URL: ness-0.1.5.tar.gz
  • Upload date:
  • Size: 4.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.6 CPython/3.10.4 Linux/5.13.0-1017-azure

File hashes

Hashes for ness-0.1.5.tar.gz
Algorithm Hash digest
SHA256 594e51398993d473472ef023b71b893ba54b94d5bc4da77d33669c878a92cdb4
MD5 caf61435b554edeabcd99df13ddf7cbd
BLAKE2b-256 5046553614eae9025f85136dfde16ee5d0c8bf8fc2005cb10a7387562df27eed

See more details on using hashes here.

File details

Details for the file ness-0.1.5-py3-none-any.whl.

File metadata

  • Download URL: ness-0.1.5-py3-none-any.whl
  • Upload date:
  • Size: 5.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.6 CPython/3.10.4 Linux/5.13.0-1017-azure

File hashes

Hashes for ness-0.1.5-py3-none-any.whl
Algorithm Hash digest
SHA256 2c408889803f3268974e2d91a04f29cca799812988493ad4c36836c5aff9286e
MD5 3ce12135fb62dbc499c7510b6deea88a
BLAKE2b-256 f9776fae072ee9a6adc1ed9e469acd088d4d353951d7734327b768e58f7e5182

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page