Virtual filesystem for SQLite to read from and write to S3
Project description
sqlite-s3vfs
Virtual filesystem for SQLite to read from and write to S3
Installation
sqlite-s3vfs depends on APSW, which is not officially available on PyPI, but can be installed directly from GitHub.
pip install sqlite-s3vfs
pip install https://github.com/rogerbinns/apsw/releases/download/3.36.0-r1/apsw-3.36.0-r1.zip --global-option=fetch --global-option=--version --global-option=3.36.0 --global-option=--all --global-option=build --global-option=--enable-all-extensions
Usage
sqlite-s3vfs is an APSW virtual filesystem that requires boto3 for its communication with S3.
import apsw
import boto3
import sqlite_s3vfs
# A boto3 bucket resource
bucket = boto3.Session().resource('s3').Bucket('my-bucket')
# An S3VFS for that bucket
s3vfs = sqlite_s3vfs.S3VFS(bucket=bucket)
# sqlite-s3vfs stores many objects under this prefix
# Note that it's not typical to start a key prefix with '/'
key_prefix = 'my/path/cool.sqlite'
# Connect, insert data, and query
with apsw.Connection(key_prefix, vfs=s3vfs.name) as db:
cursor = db.cursor()
cursor.execute(f'''
CREATE TABLE foo(x,y);
INSERT INTO foo VALUES(1,2);
''')
cursor.execute('SELECT * FROM foo;')
print(cursor.fetchall())
# Get the serialized form of the sqlite file, say to upload to S3 as a single object
for chunk in s3vfs.serialize(key_prefix=key_prefix):
print(chunk)
See the APSW documentation for more examples.
Tests
The tests require the dev dependencies and APSW to installed, and MinIO started
pip install -r requirements-dev.txt
pip install https://github.com/rogerbinns/apsw/releases/download/3.36.0-r1/apsw-3.36.0-r1.zip --global-option=fetch --global-option=--version --global-option=3.36.0 --global-option=--all --global-option=build --global-option=--enable-all-extensions
./start-minio.sh
can be run with pytest
pytest
and finally Minio stopped
./stop-minio.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
sqlite-s3vfs-0.0.10.tar.gz
(3.4 kB
view hashes)
Built Distribution
Close
Hashes for sqlite_s3vfs-0.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 213a4305434b189f36eeaf3479b054debc68bc89b324800893c7692c1472a42d |
|
MD5 | 6c35f8a7fbc50a718210a70b52e873a9 |
|
BLAKE2b-256 | ce3791c72d7e366994e70e86d4b35642bb2b15a1bd68eb077af9e60529a2423e |