Polystores is an abstraction and a collection of clients to interact with cloud storages.
Project description
[](LICENSE)
[](https://travis-ci.com/polyaxon/polystores)
[](https://badge.fury.io/py/polystores)
[](https://www.codacy.com/app/polyaxon/polystores?utm_source=github.com&utm_medium=referral&utm_content=polyaxon/polystores&utm_campaign=Badge_Grade)
[](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)
# polystores
Polystores is an abstraction and a collection of clients to interact with cloud storages.
## Install
```bash
$ pip install -U polystores
```
N.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use with `polystores`.
### Install S3
```bash
pip install boto3 botocore
```
### Install GCS
```bash
pip install google-cloud-storage
```
### Install Azure Storage
```bash
pip install azure-storage
```
## Stores
This module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.
## S3
### Normal instantiation
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(endpoint_url=...,
access_key_id=...,
secret_access_key=...,
session_token=...,
region=...)
```
### Using env vars
```bash
export AWS_ENDPOINT_URL=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SECURITY_TOKEN=...
exprot AWS_REGION=...
```
And then you can instantiate the store
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store()
```
### Using a client
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(client=client)
```
### Important methods
```python
s3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)
s3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.check_key(key, bucket_name=None)
s3_store.get_key(key, bucket_name=None)
s3_store.read_key(key, bucket_name=None)
s3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)
s3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')
s3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_file(key, local_path, bucket_name=None, use_basename=True)
s3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)
```
## GCS
### Normal instantiation
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(project_id=...,
credentials=...,
key_path=...,
key_path=...,
scopes=...)
```
### Using a client
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(client=client)
```
### Important methods
```python
gcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
gcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)
gcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)
gcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)
gcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)
```
## Azure Storage
### Normal instantiation
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(account_name=...,
account_key=...,
connection_string=...)
```
### Using env vars
```bash
export AZURE_ACCOUNT_NAME=...
export AZURE_ACCOUNT_KEY=...
export AZURE_CONNECTION_STRING=...
```
And then you can instantiate the store
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore()
```
### Using a client
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(client=client)
```
### Important methods
```python
az_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
az_store.upload_file(filename, blob, container_name=None, use_basename=True)
az_store.download_file(blob, local_path, container_name=None, use_basename=True)
az_store.upload_dir(dirname, blob, container_name=None, use_basename=True)
az_store.download_dir(blob, local_path, container_name=None, use_basename=True)
```
## Running tests
```
pytest
```
## License
[](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores?ref=badge_large)
[](https://travis-ci.com/polyaxon/polystores)
[](https://badge.fury.io/py/polystores)
[](https://www.codacy.com/app/polyaxon/polystores?utm_source=github.com&utm_medium=referral&utm_content=polyaxon/polystores&utm_campaign=Badge_Grade)
[](https://join.slack.com/t/polyaxon/shared_invite/enQtMzQ0ODc2MDg1ODc0LWY2ZTdkMTNmZjBlZmRmNjQxYmYwMTBiMDZiMWJhODI2ZTk0MDU4Mjg5YzA5M2NhYzc5ZjhiMjczMDllYmQ2MDg)
# polystores
Polystores is an abstraction and a collection of clients to interact with cloud storages.
## Install
```bash
$ pip install -U polystores
```
N.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use with `polystores`.
### Install S3
```bash
pip install boto3 botocore
```
### Install GCS
```bash
pip install google-cloud-storage
```
### Install Azure Storage
```bash
pip install azure-storage
```
## Stores
This module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.
## S3
### Normal instantiation
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(endpoint_url=...,
access_key_id=...,
secret_access_key=...,
session_token=...,
region=...)
```
### Using env vars
```bash
export AWS_ENDPOINT_URL=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SECURITY_TOKEN=...
exprot AWS_REGION=...
```
And then you can instantiate the store
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store()
```
### Using a client
```python
from polystores.stores.s3_store import S3Store
s3_store = S3Store(client=client)
```
### Important methods
```python
s3_store.list(bucket_name, prefix='', delimiter='', page_size=None, max_items=None, keys=True, prefixes=True)
s3_store.list_prefixes(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.list_keys(bucket_name, prefix='', delimiter='', page_size=None, max_items=None)
s3_store.check_key(key, bucket_name=None)
s3_store.get_key(key, bucket_name=None)
s3_store.read_key(key, bucket_name=None)
s3_store.upload_bytes(bytes_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None)
s3_store.upload_string(string_data, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, encoding='utf-8')
s3_store.upload_file(filename, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_file(key, local_path, bucket_name=None, use_basename=True)
s3_store.upload_dir(dirname, key, bucket_name=None, overwrite=False, encrypt=False, acl=None, use_basename=True)
s3_store.download_dir(key, local_path, bucket_name=None, use_basename=True)
```
## GCS
### Normal instantiation
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(project_id=...,
credentials=...,
key_path=...,
key_path=...,
scopes=...)
```
### Using a client
```python
from polystores.stores.gcs_store import GCSStore
gcs_store = GCSStore(client=client)
```
### Important methods
```python
gcs_store.list(key, bucket_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
gcs_store.upload_file(filename, blob, bucket_name=None, use_basename=True)
gcs_store.download_file(blob, local_path, bucket_name=None, use_basename=True)
gcs_store.upload_dir(dirname, blob, bucket_name=None, use_basename=True)
gcs_store.download_dir(blob, local_path, bucket_name=None, use_basename=True)
```
## Azure Storage
### Normal instantiation
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(account_name=...,
account_key=...,
connection_string=...)
```
### Using env vars
```bash
export AZURE_ACCOUNT_NAME=...
export AZURE_ACCOUNT_KEY=...
export AZURE_CONNECTION_STRING=...
```
And then you can instantiate the store
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore()
```
### Using a client
```python
from polystores.stores.azure_store import AzureStore
az_store = AzureStore(client=client)
```
### Important methods
```python
az_store.list(key, container_name=None, path=None, delimiter='/', blobs=True, prefixes=True)
az_store.upload_file(filename, blob, container_name=None, use_basename=True)
az_store.download_file(blob, local_path, container_name=None, use_basename=True)
az_store.upload_dir(dirname, blob, container_name=None, use_basename=True)
az_store.download_dir(blob, local_path, container_name=None, use_basename=True)
```
## Running tests
```
pytest
```
## License
[](https://app.fossa.io/projects/git%2Bgithub.com%2Fpolyaxon%2Fpolystores?ref=badge_large)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
polystores-0.1.5.tar.gz
(25.7 kB
view details)
Built Distribution
File details
Details for the file polystores-0.1.5.tar.gz
.
File metadata
- Download URL: polystores-0.1.5.tar.gz
- Upload date:
- Size: 25.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.6.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
8a210105258a895f35e90826890c947b7879b28e23b15f8de7b3b13dbc45652c
|
|
MD5 |
ef2d4053bd3ef5d19fcbb677e7ac24e4
|
|
BLAKE2b-256 |
7181e305195c05f57cecc52e47c7f6018cd0b4373a075054822245b9038efd41
|
File details
Details for the file polystores-0.1.5-py2.py3-none-any.whl
.
File metadata
- Download URL: polystores-0.1.5-py2.py3-none-any.whl
- Upload date:
- Size: 31.7 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.4.2 requests/2.20.1 setuptools/40.6.2 requests-toolbelt/0.8.0 tqdm/4.28.1 CPython/3.6.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
d6e78588b86b33cf89a415d57fbdc8663fd6ccbbcf684bcb5ea7b2fb6a7cc718
|
|
MD5 |
5027391959e0e79722af2ba0a9b77d40
|
|
BLAKE2b-256 |
d31f070887c0cb3bc07b4dbe1c61461f5d6f39b1254a4d0a02d62a24222e3a7b
|