ness 0.1.1

A Python datalake client.

Ness

A Python datalake client.

Requirements

• AWS CLI

Installation

Using Conda:

conda install -c conda-forge pyarrow ness

Using Pip:

pip install pyarrow ness

Quickstart

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake")
df = dl.read("mytable")

Sync

# Sync all tables
df = dl.sync()

# Sync a single table
df = dl.sync("mytable")

Format

Specify the input data source format, the default format is parquet:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", format="csv")

AWS Profile

Files are synced using default AWS profile, you can configure another one:

import ness

dl = ness.dl(bucket="mybucket", key="mydatalake", profile="myprofile")