Read and write neuroglancer Precomputed formats to cloud storage
Project description
[![Build Status](https://travis-ci.org/seung-lab/cloud-volume.svg?branch=master)](https://travis-ci.org/seung-lab/cloud-volume)
# cloud-volume
Python client for reading and writing to Neuroglancer Precomputed volumes on cloud services. (https://github.com/google/neuroglancer/tree/master/src/neuroglancer/datasource/precomputed)
When working with a particular dataset, say an EM scan of a mouse, fish, or fly brain, you’ll typically store that as a grayscale data layer accessible to neuroglanger. You may store additional labellings and processing results as other layers.
## Setup
You’ll need to set up your cloud credentials as well as the main install.
### Credentials
` mkdir -p ~/.neuroglancer/secrets/ echo $GOOGLE_STORAGE_PROJECT > ~/.neuroglancer/project_name # needed for Google mv aws-secret.json ~/.neuroglancer/secrets/ # needed for Google mv google-secret.json ~/.neuroglancer/secrets/ # needed for Amazon mv boss-secret.json ~/.neuroglancer/secrets/ # needed for the BOSS `
### pip
` pip install cloud-volume `
### Manual ` git clone git@github.com:seung-lab/cloud-volume.git cd cloud-volume mkvirtualenv cloud-volume workon cloud-volume pip install -e . `
## Other Languages
Julia - https://github.com/seung-lab/CloudVolume.jl
## Usage
Supports reading and writing to neuroglancer data layers on Amazon S3, Google Storage, and the local file system.
Supported URLs are of the forms:
$PROTOCOL://$BUCKET/$DATASET/$LAYER
### Supported Protocols * gs: Google Storage * s3: Amazon S3 * boss: The BOSS (https://docs.theboss.io/docs) * file: Local File System (absolute path)
### Examples
` vol = CloudVolume('gs://mybucket/retina/image') # Basic Example vol = CloudVolume('gs://buck/ds/chan', mip=0, bounded=True, fill_missing=False) # Using multiple initialization options vol = CloudVolume('gs://buck/ds/chan', info=info) # Creating a new volume's info file from scratch image = vol[:,:,:] # Download the entire image stack into a numpy array vol[64:128, 64:128, 64:128] = image # Write a 64^3 image to the volume vol.save_mesh(12345) # save 12345 as ./12345.obj vol.save_mesh([12345, 12346, 12347]) # merge three segments into one obj `
### CloudVolume Constructor
CloudVolume(cloudpath, mip=0, bounded=True, fill_missing=False, info=None)
mip - Which mip level to access
bounded - Whether access is allowed outside the bounds defined in the info file
fill_missing - If a chunk is missing, should it be zero filled or throw an EmptyVolumeException?
info - Use this info object rather than pulling from the cloud (useful for creating new layers).
### CloudVolume Methods
Better documentation coming later, but for now, here’s a summary of the most useful method calls. Use help(cloudvolume.CloudVolume.$method) for more info.
create_new_info (class method) - Helper function for creating info files for creating new data layers.
refresh_info - Repull the info file.
refresh_provenance - Repull the provenance file.
slices_from_global_coords - Find the CloudVolume slice from MIP 0 coordinates if you’re on a different MIP. Often used in combination with neuroglancer.
reset_scales - Delete mips other than 0 in the info file. Does not autocommit.
add_scale - Generate a new mip level in the info property. Does not autocommit.
commit_info - Push the current info property into the cloud as a JSON file.
commit_provenance - Push the current provenance property into the cloud as a JSON file.
get_mesh - Download an object and save it in .obj format. You can combine equivialences into a single object too.
### CloudVolume Properties
Accessed as vol.$PROPERTY like vol.mip. Parens next to each property mean (data type:default, writability). (r) means read only, (w) means write only, (rw) means read/write.
mip (uint:0, rw) - Read from and write to this mip level (0 is highest res). Each additional increment in the number is typically a 2x reduction in resolution.
bounded (bool:True, rw) - If a region outside of volume bounds is accessed throw an error if True or Fill the region with black (useful for e.g. marching cubes’s 1px boundary) if False.
fill_missing (bool:False, rw) - If a file inside volume bounds is unable to be fetched use a block of zeros if True, else throw an error
info (dict, rw) - Python dict representation of Neuroglancer info JSON file. You must call vol.commit_info() to save your changes to storage.
provenance (dict-like, rw) - Data layer provenance file representation. You must call vol.commit_provenance() to save your changes to storage.
available_mips (list of ints, r) - Query which mip levels are defined for reading and writing.
dataset_name (str, rw) - Which dataset (e.g. test_v0, snemi3d_v0) on S3, GS, or FS you’re reading and writing to. Known as an “experiment” in BOSS terminology. Writing to this property triggers an info refresh.
layer (str, rw) - Which data layer (e.g. image, segmentation) on S3, GS, or FS you’re reading and writing to. Known as a “channel” in BOSS terminology. Writing to this property triggers an info refresh.
base_cloudpath (str, r) - The cloud path to the dataset e.g. s3://bucket/dataset/
layer_cloudpath (str, r) - The cloud path to the data layer e.g. gs://bucket/dataset/image
info_cloudpath (str, r) - Generate the cloud path to this data layer’s info file.
scales (dict, r) - Shortcut to the ‘scales’ property of the info object
scale (dict, r)† - Shortcut to the working scale of the current mip level
shape (Vec4, r)† - Like numpy.ndarray.shape for the entire data layer.
volume_size (Vec3, r)† - Like shape, but omits channel (x,y,z only).
num_channels (int, r) - The number of channels, the last element of shape.
layer_type (str, r) - The neuroglancer info type, ‘image’ or ‘segmentation’.
dtype (str, r) - The info data_type of the volume, e.g. uint8, uint32, etc. Similar to numpy.ndarray.dtype.
encoding (str, r) - The neuroglancer info encoding. e.g. ‘raw’, ‘jpeg’, ‘npz’
resolution (Vec3, r)† - The 3D physical resolution of a voxel in nanometers at the working mip level.
downsample_ratio (Vec3, r) - Ratio of the current resolution to the highest resolution mip available.
underlying (Vec3, r)† - Size of the underlying chunks that constitute the volume in storage. e.g. Vec(64, 64, 64)
key (str, r)† - The ‘directory’ we’re accessing the current working mip level from within the data layer. e.g. ‘6_6_30’
bounds (Bbox, r)† - A Bbox object that represents the bounds of the entire volume.
† These properties can also be accessed with a function named like vol.mip_$PROPERTY($MIP). By default they return the current mip level assigned to the CloudVolume, but any mip level can be accessed via the corresponding mip_ function. Example: vol.mip_resolution(2) would return the resolution of mip 2.
### VolumeCutout Functions
When you download an image using CloudVolume it gives you a VolumeCutout. These are numpy.ndarray subclasses that support a few extra properties to help make book keeping easier. The major advantage is save_images() which can help you debug your dataset.
dataset_name - The dataset this image came from.
layer - Which layer it came from.
mip - Which mip it came from
layer_type - “image” or “segmentation”
bounds - The bounding box of the cutout
num_channels - Alias for vol.shape[3]
save_images() - Save Z slice PNGs of the current image to ./saved_images for manual inspection
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloud_volume-0.2.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba74eac27c43af507ea52d0db276f411d1d41dd11d3b7e429c61e7b28e3229f6 |
|
MD5 | eca07b74412708ebfe5ffd8796353787 |
|
BLAKE2b-256 | 11a3b1fb92e4b4f025b7400572832650ffc36631dbdda67bdca80b8bbbd86acd |