gcp-storage-emulator 2022.1.20

A stub emulator for the Google Cloud Storage API

Local Emulator for Google Cloud Storage

Google doesn't (yet) ship an emulator for the Cloud Storage API like they do for Cloud Datastore.

This is a stub emulator so you can run your tests and do local development without having to connect to the production Storage APIs.

THIS IS A WORK IN PROGRESS AND ONLY SUPPORTS A LIMITED SUBSET OF THE API

---

Installation

pip install gcp-storage-emulator

CLI Usage

Starting the emulator

Start the emulator with:

gcp-storage-emulator start

By default, the server will listen on http://localhost:9023 and data is stored under ./.cloudstorage. You can configure the folder using the env variables STORAGE_BASE (default ./) and STORAGE_DIR (default .cloudstorage).

If you wish to run the emulator in a testing environment or if you don't want to persist any data, you can use the --in-memory parameter. For tests, you might want to consider starting up the server from your code (see the Python APIs)

If you're using the Google client library (e.g. google-cloud-storage for Python) then you can set the STORAGE_EMULATOR_HOST environment variable to tell the library to connect to your emulator endpoint rather than the standard https://storage.googleapis.com, e.g.:

export STORAGE_EMULATOR_HOST=http://localhost:9023

Wiping data

You can wipe the data by running

gcp-storage-emulator wipe

You can pass --keep-buckets to wipe the data while keeping the buckets.

Example

Use in-memory storage and automatically create default storage bucket my-bucket.

gcp-storage-emulator start --host=localhost --port=9023 --in-memory --default-bucket=my-bucket

Python APIs

To start a server from your code you can do

from gcp_storage_emulator.server import create_server

server = create_server("localhost", 9023, in_memory=False)

server.start()
# ........
server.stop()

You can wipe the data by calling server.wipe()

This can also be achieved (e.g. during tests) by hitting the /wipe HTTP endpoint

Example

import os

from google.cloud import storage
from gcp_storage_emulator.server import create_server

HOST = "localhost"
PORT = 9023
BUCKET = "test-bucket"

# default_bucket parameter creates the bucket automatically
server = create_server(HOST, PORT, in_memory=True, default_bucket=BUCKET)
server.start()

# Avoid authentication when connecting to a storage emulator
# https://github.com/googleapis/python-storage/issues/324
os.environ["STORAGE_EMULATOR_HOST"] = f"http://{HOST}:{PORT}"
client = storage.Client.create_anonymous_client()
client.project = "test"

bucket = client.bucket(BUCKET)
blob = bucket.blob("blob1")
blob.upload_from_string("test1")
blob = bucket.blob("blob2")
blob.upload_from_string("test2")
for blob in bucket.list_blobs():
    content = blob.download_as_bytes()
    print(f"Blob [{blob.name}]: {content}")

server.stop()

Docker

Pull the Docker image.

docker pull oittaa/gcp-storage-emulator

Inside the container instance, the value of the PORT environment variable always reflects the port to which requests are sent. It defaults to 8080. The directory used for the emulated storage is located under /storage in the container. In the following example the host's directory $(pwd)/cloudstorage will be bound to the emulated storage.

docker run -d \
  -e PORT=9023 \
  -p 9023:9023 \
  --name gcp-storage-emulator \
  -v "$(pwd)/cloudstorage":/storage \
  oittaa/gcp-storage-emulator

import os

from google.cloud import exceptions, storage

HOST = "localhost"
PORT = 9023
BUCKET = "test-bucket"

# Avoid authentication when connecting to a storage emulator
# https://github.com/googleapis/python-storage/issues/324
os.environ["STORAGE_EMULATOR_HOST"] = f"http://{HOST}:{PORT}"
client = storage.Client.create_anonymous_client()
client.project = "test"

try:
    bucket = client.create_bucket(BUCKET)
except exceptions.Conflict:
    bucket = client.bucket(BUCKET)

blob = bucket.blob("blob1")
blob.upload_from_string("test1")
print(blob.download_as_bytes())