datablob

Client for Updating a Simple Data Warehouse on Blob Storage

design philosophy

• optimize for simplicity and user friendliness
• storage is cheap (compared to compute)
• pre-compute as much as possible
• should work out of the box
• advanced configuration should be opt-in
• explicit is better than implicit
• straightforwardness over magic

install

pip install datablob

supported formats

• csv
• geojson (points only)
• json
• json lines
• parquet, including geoparquet
• shapefile (points only)
• tsv
• xlsx (Microsoft Excel)

basic usage

from datablob import DataBlobClient

client = DataBlobClient(
    bucket_name="example-test-bucket-123", bucket_path="prefix/to/dataportal"
)

client.update_dataset(name="fleet", version="2", data=rows, xlsx=True)
# automatically creates the following files
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/meta.json
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.csv
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.points.geojson
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.json
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.jsonl
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.parquet
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.points.shp.zip
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.tsv
# s3://example-test-bucket-123/prefix/to/dataportal/fleet/v2/data.xlsx

advanced usage

client.update_dataset(
    name="fleet",
    version="1",
    data=rows,
    column_names=["ID", "Make", "Model", "Year"]
    description="List of Vehicles in Fleet",
    xlsx=True,
    xlsx_data_types=["Number", "Text", "Text", "Number"]
)

examples

• Clever Vehicle Locations
• Simple Bus Routes
• Simple Bus Stops