Skip to main content

Polystores is an abstraction and a collection of clients to interact with cloud storages.

Project description

[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Build Status](https://travis-ci.com/polyaxon/polystores.svg?branch=master)](https://travis-ci.com/polyaxon/polystores)
[![PyPI version](https://badge.fury.io/py/polystores.svg)](https://badge.fury.io/py/polystores)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/a33947d729f94f5da7f7390dfeef7f94)](https://www.codacy.com/app/polyaxon/polystores?utm_source=github.com&utm_medium=referral&utm_content=polyaxon/polystores&utm_campaign=Badge_Grade)
[![Slack](https://img.shields.io/badge/chat-on%20slack-aadada.svg?logo=slack&longCache=true)](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)


# polystores

Polystores is an abstraction and a collection of clients to interact with cloud storages.


## Install

```bash
$ pip install -U polystores
```

N.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use with `polystores`.

### Install S3

```bash
pip install boto3 botocore
```

### Install GCS


```bash
pip install google-cloud-storage
```

### Install Azure Storage


```bash
pip install azure-storage
```

## Stores

This module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.


## S3

### Normal instantiation

```python
from polystores.stores.s3_store import S3Store

s3_store = S3Store(endpoint_url=...,
access_key_id=...,
secret_access_key=...,
session_token=...,
region=...)
```


### Using env vars

```bash
export AWS_ENDPOINT_URL=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SECURITY_TOKEN=...
exprot AWS_REGION=...
```

And then you can instantiate the store
```python
from polystores.stores.s3_store import S3Store

s3_store = S3Store()
```

### Using a client

```python
from polystores.stores.s3_store import S3Store

s3_store = S3Store(client=client)
```

### Important methods

```python
s3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)
s3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.check_key(key, bucket_name=None)
s3_store.get_key(key, bucket_name=None)
s3_store.read_key(key, bucket_name=None)
s3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)
s3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')
s3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_file(key, local_path, bucket_name=None, use_basename=True)
s3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)
```


## GCS

### Normal instantiation

```python
from polystores.stores.gcs_store import GCSStore

gcs_store = GCSStore(project_id=...,
credentials=...,
key_path=...,
key_path=...,
scopes=...)
```

### Using a client

```python
from polystores.stores.gcs_store import GCSStore

gcs_store = GCSStore(client=client)
```

### Important methods

```python
gcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
gcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)
gcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)
gcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)
gcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)
```

## Azure Storage

### Normal instantiation

```python
from polystores.stores.azure_store import AzureStore

az_store = AzureStore(account_name=...,
account_key=...,
connection_string=...)
```

### Using env vars

```bash
export AZURE_ACCOUNT_NAME=...
export AZURE_ACCOUNT_KEY=...
export AZURE_CONNECTION_STRING=...
```

And then you can instantiate the store
```python
from polystores.stores.azure_store import AzureStore

az_store = AzureStore()
```

### Using a client

```python
from polystores.stores.azure_store import AzureStore

az_store = AzureStore(client=client)
```

### Important methods

```python
az_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
az_store.upload_file(filename, blob, container_name=None, use_basename=True)
az_store.download_file(blob, local_path, container_name=None, use_basename=True)
az_store.upload_dir(dirname, blob, container_name=None, use_basename=True)
az_store.download_dir(blob, local_path, container_name=None, use_basename=True)
```

## Running tests

```
pytest
```


## License

[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores?ref=badge_large)


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

polystores-0.1.5.tar.gz (25.7 kB view details)

Uploaded Source

Built Distribution

polystores-0.1.5-py2.py3-none-any.whl (31.7 kB view details)

Uploaded Python 2Python 3

File details

Details for the file polystores-0.1.5.tar.gz.

File metadata

  • Download URL: polystores-0.1.5.tar.gz
  • Upload date:
  • Size: 25.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.6.4

File hashes

Hashes for polystores-0.1.5.tar.gz
Algorithm Hash digest
SHA256 8a210105258a895f35e90826890c947b7879b28e23b15f8de7b3b13dbc45652c
MD5 ef2d4053bd3ef5d19fcbb677e7ac24e4
BLAKE2b-256 7181e305195c05f57cecc52e47c7f6018cd0b4373a075054822245b9038efd41

See more details on using hashes here.

File details

Details for the file polystores-0.1.5-py2.py3-none-any.whl.

File metadata

  • Download URL: polystores-0.1.5-py2.py3-none-any.whl
  • Upload date:
  • Size: 31.7 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.6.4

File hashes

Hashes for polystores-0.1.5-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 d6e78588b86b33cf89a415d57fbdc8663fd6ccbbcf684bcb5ea7b2fb6a7cc718
MD5 5027391959e0e79722af2ba0a9b77d40
BLAKE2b-256 d31f070887c0cb3bc07b4dbe1c61461f5d6f39b1254a4d0a02d62a24222e3a7b

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page