spatialstudio 1.1.1.35

Utilities for creating Spatials (4D videos)

spatialstudio

SpatialStudio is the fundamental package for creating and editing Spatials (4D videos). For more info check out https://true3d.com/.

Installation + Getting Started

Here's how to install spatialstudio:

pip install spatialstudio

Here's a barebones example to create your first spatial:

from spatialstudio import splv

# create encoder:
width, height, depth = (128, 128, 128)
encoder = splv.Encoder(
    width, height, depth,
    framerate=30.0, outputPath='my_spatial.splv'
)

# generate frames (moving red cube):
for i in range(100):
    frame = splv.Frame(width, height, depth)

    frame.fill(
        minPos=(i     , i     , i     ),
        maxPos=(i + 20, i + 20, i + 20),
        voxel=(255, 0, 0)
    )

    encoder.encode(frame)

# finish the encoding:
encoder.finish()

Classes

Frame

Represents a 3D volume - a structured grid of voxels. This is a single frame of an splv file.

Constructors

Create an empty frame:

frame = splv.Frame(width, height, depth)

• width (int): Frame width in voxels.
• height (int): Frame height in voxels.
• depth (int): Frame depth in voxels.
  
  

Create a frame from a NumPy array:

frame = splv.Frame(array, leftRight, upDown, frontBack)

• array (buffer): NumPy array containing voxel data. Must have shape (w, h, d, 4) and be of type float32 or uint8.
• leftRight, upDown, frontBack (str): Axis mapping for left-right, up-down, front-back. Must each be one of ("x", "y", and "z") and must be distinct. (defalt: "x", "y", and "z", respectively).

Accessors

Get the voxel at a given position:

voxel   = frame[x, y, z] # None
r, g, b = frame[x, y, z] # Tuple[int, int, int]

Set the voxel at a given position:

frame[x, y, z] = (r, g, b)
frame[x, y, z] = None

Voxels have type Optional[Tuple[int, int, int]]. None represents aj empty voxel, otherwise it is interpreted as a tuple of color coponents, (r, g, b).

Methods

Create an identical copy of a frame:

clonedFrame = frame.clone() # splv.Frame

Load a frame from a file:

frame = splv.Frame.load(path) # splv.Frame

• path (str): The path to load from.
  
  

Load a frame from a NanoVDB file:

frame = splv.Frame.load_from_nvdb(path, minPos, maxPos, leftRight, upDown, frontBack) # splv.Frame

• path (str): Path to the NanoVDB file to load.
• minPos (Tuple[int, int, int]): The minimum voxel coordinates that will be written into the frame.
• maxPos (Tuple[int, int, int]): The maximum voxel coordinates that will be written into the frame.
• leftRight, upDown, frontBack: Axis mappings, define which axes correspond to which dimension. Must each be one of ("x", "y", and "z") and must be distinct. (defalt: "x", "y", and "z", respectively).
  
  

Create a frame from a color + depth image:

frame = splv.Frame.from_rgbd_simple(colorImg, depthImg, focalLength, width, height, depth) # splv.Frame

• colorImg (buffer): The color image from which to generate the frame. Must have shape (w, h, 3) and contain uint8 color values in the format [r, g, b].
• depthImg (buffer): The corresponding depth image from which to generate the frame. The depth. Must have shape (w, h) and contain float32 depth values. The depth values are in camera space.
• focalLength the focal length of the camera used to capture the images, in pixels. This function assumes a pinhole camera.
• width, height, depth: The dimensions of the returned frame.
  
  

Create a frame from a color + depth image, providing full intrinsics and extrinsics matrices:

frame, minWorldPos, maxWorldPos = splv.Frame.from_rgbd(colorImg, depthImg, intrinsics, extrinsics, minPos, maxPos, width, height, depth) # Tuple[splv.Frame, Tuple[float, float, float], Tuple[float, float, float]]

• colorImg (buffer): The color image from which to generate the frame. Must have shape (w, h, 3) and contain uint8 color values in the format [r, g, b].
• depthImg (buffer): The corresponding depth image from which to generate the frame. The depth. Must have shape (w, h) and contain float32 depth values. The depth values are in camera space.
• intrinsics (buffer): The camera intrinsics matrix. Must be a 3x3 matrix.
• extrinsics (buffer): The camera extrinsics matrix, must be a 3x4 matrix (affine transform).
• minPos (Tuple[float, float, float]): The minimum world-space position to be included in the frame (default: (-1, -1, -1)).
• maxPos (Tuple[float, float, float]): The maximum world-space position to be included in the frame (default: (1, 1, 1)).
• width, height, depth: The dimensions of the returned frame.

Returns:

• The frame populated with voxels from the images.
• The minimum world-space position of any voxel, regardless of whether or not it was clipped by minPos/maxPos. This is helpful if you're not yet sure what the world bounds should be.
• The maximum world-space position of any voxel, regardless of whether or not it was clipped by minPos/maxPos. This is helpful if you're not yet sure what the world bounds should be.
  
  

Save a frame to a file:

frame.save(path)

• path (str): Path to save to.
  
  

Save a frame as a NanoVDB file (.nvdb):

frame.save_to_nvdb(path)

• path (str): Path to save to.
  
  

Get the dimensions of a frame:

width, height, depth = frame.get_dims() # Tuple[int, int, int]

Get the number of nonempty "bricks" in the frame (BRICK_SIZE^3 regions of voxels):

numBricks = frame.get_num_bricks() # int

Get the number of voxels in a frame:

numVoxels = frame.get_num_voxels() # int

Fill a region of a frame with a given voxel:

frame.fill(minPos, maxPos, (r, g, b))
frame.fill(minPos, maxPos, None)

• minPos (Tuple[int, int, int]): The minimum voxel coordinates to be filled.
• maxPos (Tuple[int, int, int]): The maximum voxel coordinates to be filled.
• voxel (Optional[Tuple[int, int, int]]): The voxel to fill with. None indicates an empty voxel, a tuple (r, g, b) of uint8s represents a filled voxel of the specified color.
  
  

Add a given frame into a frame:

frame.add(otherFrame, offset, leftRight, upDown, frontBack, flipLeftRight, flipUpDown, flipFrontBack)

• src (splv.Frame): The frame to add.
• offset (Tuple[int, int, int]): Position within the frame to add src. The origin (bottom-left-front) of src will end up at (x, y, z) in the frame (default: (0, 0, 0)).
• leftRight, upDown, frontBack (str): Axis mappings, define which axes correspond to which dimension. Must each be one of ("x", "y", and "z") and must be distinct. (defalt: "x", "y", and "z", respectively).
• flipLeftRight, flipUpDown, flipFrontBack (bool): Whether to reflect src over each of the axes, applied AFTER the axis mapping (default: False).
  
  

Subtract a given frame from a frame:

frame.subtract(otherFrame, offset, leftRight, upDown, frontBack, flipLeftRight, flipUpDown, flipFrontBack)

• src (splv.Frame): The frame to subtract.
• offset (Tuple[int, int, int]): Position within the frame to subtract src. The origin (bottom-left-front) of src will be subtracted from (x, y, z) in the frame (default: (0, 0, 0)).
• leftRight, upDown, frontBack (str): Axis mappings, define which axes correspond to which dimension. Must each be one of ("x", "y", and "z") and must be distinct. (defalt: "x", "y", and "z", respectively).
• flipLeftRight, flipUpDown, flipFrontBack (bool): Whether to reflect src over each of the axes, applied AFTER the axis mapping (default: False).
  
  

Clips a frame to given bounds:

frame.clip(minPos, maxPos)

• minPos (Tuple[int, int, int]): The minimum voxel coordinates that will remain in the frame after clipping.
• maxPos (Tuple[int, int, int]): The maximum voxel coordinates that will remain in the frame after clipping.
  
  

Scale (resample) a frame to given dimensions:

resampled = frame.resampled(width, height, depth, alphaCutoff) # splv.Frame

• width, height, depth (int): New dimensions.
• alphaCutoff (float): Alpha threshold for resampling (default: 0.25). Note that lower values (near 0) work well for downscaling, whereas higher values work better for upscaling.
  
  

Downscale a frame by an integer factor:

coarsened = frame.coarsened(scale, alphaCutoff) # splv.Frame

• scale (int): The factor by which to downscale. The frame returned will have dimensions of the source frame multiplied by 1/scale (default: 2).
• alphaCutoff (float): Alpha threshold for coarsening (default: 0.0).
  
  

Extract a subregion of a frame:

subregion = frame.subregion(minPos, maxPos) # splv.Frame

• minPos (Tuple[int, int, int]): The minimum voxel coordinates to be contained in the subregion.
• maxPos (Tuple[int, int, int]): The maximum voxel coordinates to be contained in the subregion.
  
  

Create a clone of a frame without voxels that are completely hidden by other voxels:

newFrame = frame.without_occluded() # splv.Frame

Create a clone of a frame without isolated voxels with no neighboring voxels:

newFrame = frame.without_orphaned() # splv.Frame

Write debug text into a frame:

frame.write_string(text, startPos, voxel, outlineVoxel, axis, flip, scale, maxWidth)

• text (str): The text to write. Currently the only characters supported are A-Z, 0-9, +-*=/, and !?.:
• startPos (Tuple[int, int, int]): The position at which to start writing the text.
• voxel (Optional[Tuple[int, int, int]]): The voxel to write the text with (default: (0, 0, 0)).
• outlineVoxel (Optional[Tuple[int, int, int]]): The voxel to outline the text with (default: (255, 255, 255)).
• axis (str): The axis along which to advance the cursor after each character. Must be either "z" or "x" (default: "z").
• flip (bool): Whether to increase or decrease the cursor position along axis after each character. True means decrease, False is increase (default: False).
• scale (int): The size of the text (default: 1).
• maxWidth (Optional[int]): The maximum number of voxels the text can span before line wrapping. If None, this is just the frame's dimension (default: None ).

Render a frame to an image:

img, depthImg = frame.render(width, height, fov, camPos, camTarget, stepScale) # Tuple[buffer, buffer]
img, depthImg = frame.render(width, height, intrinsics, extrinsics, stepScale) # Tuple[buffer, buffer]

• width, height (int): The output image dimensions, output image will be am RGBA uint8 array of shape (height, width, 4), and the output depth image will be a float32 array of shape (height, width).
• fov (float): The cameras vertical field-of-view in degrees (default: 60).
• camPos (Tuple[float, float, float]): The camera position to render from. Note that the frame will always be centered at (0, 0, 0), with its maximum extent spanning [-1, 1], and all other extents sized proportionally. (default: (1, 1, 1)).
• camTarget (Tuple[float, float, float]): The position for the camera to look at when rendering (default: (0, 0, 0)).
• intrinsics (buffer): The camera intrinsics matrix. Must be a 3x3 matrix.
• extrinsics (buffer): The camera extrinsics matrix, must be a 3x4 matrix (affine transform).
• stepScale (float): The simulated size of each ray step, controls how transparent the voxels appear overall. If holes appear in the render, you should increase this. If the render is not smooth enough, you should decrease this. (default: 3.0).

Iterator

Iterate over all nonempty voxels in a frame:

for pos, color in frame:
    x, y, z = pos
    r, g, b = color
    
    print(f"voxel at ({x}, {y}, {z}) has color ({r}, {g}, {b})")

Encoder

Compresses and encodes frames into splv files. Performs all spatial encoding-related tasks.

Constructors

Create an encoder:

encoder = splv.Encoder(
    width, height, depth, 
    framerate,
    audioParams,
    gopSize,
    motionVectors,
    vqRangeCutoff,
    outputPath
)

• width, height, depth (int): Spatial dimensions.
• framerate (float): Target framerate.
• audioParams (tuple, optional): Audio parameters as (channels, sampleRate, bit_depth), or None for no audio (default: None).
• gopSize (int): Group of Pictures size (default: 30).
• motionVectors (str): Motion vector algorithm to use. Must be one of "off", "fast", or "full" (default: "fast").
• vqRangeCutoff (float): Vector quantization range cutoff (default: 0.025). Must be in the range [0.0, 1.0].
• outputPath (str): Output file path.

Methods

Encodes a frame:

encoder.encode(frame)

• frame (splv.Frame): The frame to encode.
  
  

Encode audio data:

encoder.encode_audio(buf)

• buf (buffer): The raw PCM samples to encode.
  
  

Finalize encoding, flush to file:

encoder.finish()

• Note that this function MUST be called before the splv file is valid.

Decoder

Decompresses and decodes splv files into their constituent framess. Performs all spatial decoding-related tasks.

Constructors

Create a decoder from an splv file:

decoder = splv.Decoder(path)

• path(str): The path to the splv to decode from.

Methods

Get the dimensions of a decoder's splv:

width, height, depth = decoder.get_dims() # Tuple[int, int, int]

Get the framerate of the decoder's splv:

framerate = decoder.get_framerate() # float

Get the number of frames in the decoder's splv:

frameCount = decoder.get_frame_count() # int

Get the number of audio channels in the decoder's splv:

audioChannelCount = decoder.get_audio_channel_count() # int

Get the sample rate of any audio in the decoder's splv:

audioChannelCount = decoder.get_audio_sample_rate() # int

Get the total number of audio frames in the decoder's splv:

audioFrameCount = decoder.get_audio_frame_count() # int

Decode a frame:

frame = decoder.decode(idx) # splv.Frame

• idx (int): The index of the frame to decode, 0 representing the first frame.
  
  

Decode audio frames:

audioFames = decoder.decode_audio(startFrame, numFrames) # buffer

• startFrame (int): The first audio frame to decode.
• numFrames (int): The number of audio frames to decode.

The returned buffer will be of type float32 and have shape (decoder.get_audio_channel_count(), numFrames).

Iterator

Iterate over all frames in the decoder's splv:

for frame in decoder:
    print(frame.get_num_voxels())

Utility Functions

Concatenate multiple splv files:

splv.concat(paths, outPath)

• paths (array): Paths to splvs to concatenate.
• outPath (str): Path to write the concatenated splv.
  
  

Split an splv file into chunks of a specified duration:

splv.split(path, splitLength, outDir)

• path (str): Path to input splv file.
• splitLength (float): Duration of each chunk in seconds
• outDir (str): Output directory for split files
  
  

Re-encode an splv file with new compression parameters:

splv.transcode(path, gopSize, motionVectors, vqRangeCutoff, outPath)

• path (str): Path to input splv file
• gopSize (int): Group of Pictures size (default: 30)
• motionVectors (str): Motion vector algorithm (default: "fast")
• vqRangeCutoff (float): Vector quantization range cutoff (default: 0.025)
• outPath (str): Path for transcoded output file.
  
  

Scale (resample) an splv file to new dimensions:

splv.resample(path, width, height, depth, alphaCutoff, outPath)

• path (str): Path to input splv file.
• width, height, depth (int): New dimensions in voxels.
• alphaCutoff (float): Alpha threshold for resampling (default: 0.25).
• outPath (str): Path for resampled output file.
  
  

Downscale an splv file by an integer factor:

splv.coarsen(path, scale, alphaCutoff, outPath)

• path (str): Path to input splv file.
• scale (int): The factor by which to downscale. The new splv will have dimensions of the source splv multiplied by 1/scale (default: 2).
• alphaCutoff (float): Alpha threshold for coarsening (default: 0.0).
• outPath (str): Path for coarsened output file.
  
  

Add audio to an splv file post-export:

splv.add_audio(path, audioParams, buf)

• path (str): Path to splv file.
• audioParams (tuple): Audio parameters as (channels, sampleRate, bit_depth).
• buf(buffer): Buffer containing the raw audio PCM samples to insert.
  
  

Get the metadata of an splv file:

metadata = splv.get_metadata(path) # dict

• path (str): Path to splv file.
  
  

Dump each frame in an splv file to disk using splv.Frame.save:

splv.dump(path, outDir)

• path (str): The path to the splv file to dump.
• outDir (str): The directory where each .vv will be saved.