Ness

A Python datalake client.

Requirements

AWS CLI

Installation

pip install pyarrow ness

Quickstart

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake")
df = dl.read("mytable")

Sync

# Sync all tables
dl.sync()

# Sync a single table
dl.sync("mytable")

# Sync and read a single table
df = dl.read("mytable", sync=True)

Format

Specify the input data source format, the default format is parquet:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", format="csv")

AWS Profile

Files are synced using default AWS profile, you can configure another one:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", profile="myprofile")

Command Line

Usage: ness sync [OPTIONS] S3_URI

Options:
  --format TEXT   Data lake source format.
  --profile TEXT  AWS profile.
  --table TEXT    Table name to sync.
  --help          Show this message and exit.

ness sync bucket/key --table mytable

Release files for ness 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ness 0.1.5
File	Size	Uploaded
ness-0.1.5.tar.gz	4.0 kB	Apr 4, 2022	Details

Built distribution (wheel)

Table of built distributions (wheels) for ness 0.1.5
File	Interpreter	ABI	Platform	Reset
ness-0.1.5-py3-none-any.whl 5.8 kB Apr 4, 2022	Python 3	none	any	Details

Total release size: 9.8 kB

Release files / ness-0.1.5.tar.gz

Download URL	ness-0.1.5.tar.gz
Size	4.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`594e51398993d473472ef023b71b893ba54b94d5bc4da77d33669c878a92cdb4`
BLAKE2b-256 checksum How to use checksums	`5046553614eae9025f85136dfde16ee5d0c8bf8fc2005cb10a7387562df27eed`
Upload date	Apr 4, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.1.6 CPython/3.10.4 Linux/5.13.0-1017-azure`

Release files / ness-0.1.5-py3-none-any.whl

Download URL	ness-0.1.5-py3-none-any.whl
Size	5.8 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`2c408889803f3268974e2d91a04f29cca799812988493ad4c36836c5aff9286e`
BLAKE2b-256 checksum How to use checksums	`f9776fae072ee9a6adc1ed9e469acd088d4d353951d7734327b768e58f7e5182`
Upload date	Apr 4, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.1.6 CPython/3.10.4 Linux/5.13.0-1017-azure`