(Asyncio OR Threadsafe) Python Client for Google Cloud Storage
This is a shared codebase for gcloud-aio-storage and gcloud-rest-storage
Installation
$ pip install --upgrade gcloud-{aio,rest}-storageUsage
To upload a file, you might do something like the following:
import aiofiles
import aiohttp
from gcloud.aio.storage import Storage
async with aiohttp.ClientSession() as session:
client = Storage(session=session)
async with aiofiles.open('/path/to/my/file', mode="r") as f:
contents = await f.read()
status = await client.upload(
'my-bucket-name',
'path/to/gcs/folder',
output,
)
print(status)Note that there are multiple ways to accomplish the above, ie,. by making use of the Bucket and Blob convenience classes if that better fits your use-case.
Of course, the major benefit of using an async library is being able to parallelize operations like this. Since gcloud-aio-storage is fully asyncio-compatible, you can use any of the builtin asyncio method to perform more complicated operations:
my_files = {
'/local/path/to/file.1': 'path/in/gcs.1',
'/local/path/to/file.2': 'path/in/gcs.2',
'/local/path/to/file.3': 'different/gcs/path/filename.3',
}
async with Storage() as client:
# Prepare all our upload data
uploads = []
for local_name, gcs_name in my_files.items():
async with aiofiles.open(local_name, mode="r") as f:
contents = await f.read()
uploads.append((gcs_name, contents))
# Simultaneously upload all files
await asyncio.gather(
*[
client.upload('my-bucket-name', path, file_) for path, file_ in uploads
]
)You can also refer smoke test for more info and examples.
Note that you can also let gcloud-aio-storage do its own session management, so long as you give us a hint when to close that session:
async with Storage() as client:
# closes the client.session on leaving the context manager
# OR
client = Storage()
# do stuff
await client.close() # close the session explicitlyFile Encodings
In some cases, aiohttp needs to transform the objects returned from GCS into strings, eg. for debug logging and other such issues. The built-in await response.text() operation relies on chardet for guessing the character encoding in any cases where it can not be determined based on the file metadata.
Unfortunately, this operation can be extremely slow, especially in cases where you might be working with particularly large files. If you notice odd latency issues when reading your results, you may want to set your character encoding more explicitly within GCS, eg. by ensuring you set the contentType of the relevant objects to something suffixed with ; charset=utf-8. For example, in the case of contentType='application/x-netcdf' files exhibiting latency, you could instead set contentType='application/x-netcdf; charset=utf-8. See #172 for more info!
Emulators
For testing purposes, you may want to use gcloud-aio-storage along with a local GCS emulator. Setting the $STORAGE_EMULATOR_HOST environment variable to the address of your emulator should be enough to do the trick.
For example, using fsouza/fake-gcs-server, you can do:
docker run -d -p 4443:4443 -v $PWD/my-sample-data:/data fsouza/fake-gcs-server
export STORAGE_EMULATOR_HOST='0.0.0.0:4443'Any gcloud-aio-storage requests made with that environment variable set will query fake-gcs-server instead of the official GCS API.
Note that some emulation systems require disabling SSL – if you’re using a custom http session, you may need to disable SSL verification.
Contributing
Please see our contributing guide.
2 maintainers
Metadata
2 maintainers
Release files for gcloud-aio-storage 6.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| gcloud-aio-storage-6.1.1.tar.gz | 18.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| gcloud_aio_storage-6.1.1-py2.py3-none-any.whl | Python 2, Python 3 | none | any | Details |
Total release size: 41.5 kB
Release files / gcloud-aio-storage-6.1.1.tar.gz
| Download URL | gcloud-aio-storage-6.1.1.tar.gz |
|---|---|
| Size | 18.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a7b74074083280d9b42e7890e696b459b40b03c3b6e5fb43e573f2c2f5e955c3
|
|
BLAKE2b-256 checksum How to use checksums |
863a760f8d638aceeb15eb3476895aff496e89e91449872de4fbbeb70f6a88fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.14.0 pkginfo/1.7.1 requests/2.26.0 setuptools/57.0.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.5
|
Release files / gcloud_aio_storage-6.1.1-py2.py3-none-any.whl
| Download URL | gcloud_aio_storage-6.1.1-py2.py3-none-any.whl |
|---|---|
| Size | 23.3 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
e0f84ee027501a5505ebddaa5831ae4243e2f651dd560f852cfaa0377de3ca57
|
|
BLAKE2b-256 checksum How to use checksums |
5b50e2ee771b226bd95fe5446b8177a9aea902bc8cc47920024a6eab90f8202e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.14.0 pkginfo/1.7.1 requests/2.26.0 setuptools/57.0.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.5
|
