grid-cortex-client 0.2.52

python client for grid cortex

Grid Cortex Client

This Python client library provides an interface for interacting with cortex models.

Design Overview

The client is designed :

1. CortexClient (src/grid_cortex_client/cortex_client.py):

   • The primary user-facing class.
   • Orchestrates model execution by dispatching to the appropriate model handler.
   • The main method is run(model_id: str, **kwargs: Any) -> Any;:
     • Identifies the correct model handler (e.g., DepthModel, DetectionModel) using an internal registry that maps keywords in the model_id to handler classes.
     • Instantiates the handler with the given model_id.
     • Calls the internal run_model() method, passing the handler instance and all **kwargs as input_data to the handler's preprocess method.
   • The run_model(model: BaseModel, input_data: Dict[str, Any], ...) method handles the detailed steps:
     • Calls the handler's preprocess method with input_data.
     • Makes the HTTP POST request to the appropriate model endpoint (e.g., /model_id/run).
     • Calls the handler's postprocess method with the server's response.

2. HTTPClient (src/grid_cortex_client/client.py):

   • Manages low-level HTTP communication, API key authentication, and request/response cycles.

3. WebSocketSession (src/grid_cortex_client/ws.py):

   • Manages WebSocket connections for models that require streaming data or real-time and continuous interactions.

4. Model Abstraction (src/grid_cortex_client/models/):

   • BaseModel (models/base_model.py): An abstract base class defining the contract for all model handlers, requiring preprocess(self, input_data: Dict[str, Any], **kwargs: Any) and postprocess(self, response_data: Dict[str, Any], **kwargs: Any) methods.
   • Task-Specific Model Handlers (e.g., DepthModel, DetectionModel, SegmentationModel in models/depth.py, models/detection.py, models/segmentation.py):
     • Concrete implementations of BaseModel.
     • preprocess: Converts the input_data dictionary (from **kwargs in client.run()) into the JSON payload expected by the specific model server.
     • postprocess: Transforms the server's JSON response into a user-friendly Python object (e.g., PIL Image, list of detections, etc.).

5. Reusable Utilities (src/grid_cortex_client/preprocessing.py, src/grid_cortex_client/postprocessing.py):

   • Contain stateless helper functions for common tasks like image loading/encoding and parsing specific response formats.
   • Uses msgpack for binary transport (image bytes, NumPy arrays).

Project Structure

• pyproject.toml: Project metadata and dependencies.
• src/grid_cortex_client/: Main library source code.
  • cortex_client.py: CortexClient.
  • client.py: HTTPClient, custom errors.
  • ws.py: WebSocketSession.
  • preprocessing.py & postprocessing.py: Utility functions.
  • models/: Model-specific handlers.
    • base_model.py: BaseModel ABC.
    • depth.py: DepthModel.
    • detection.py: DetectionModel.
    • segmentation.py: SegmentationModel.
    • stereo.py: FoundationStereoModel.
    • grasp.py: GraspModel.
    • graspgen.py: GraspGenModel.
• examples/: Example usage scripts.
  • example.py: General usage examples.
  • grasp.py: Grasp generation with M2T2 model.
  • foundationstereo.py: Stereo depth estimation with FoundationStereo.
  • graspgen.py: Grasp generation with GraspGen from point clouds.
• README.md: This file.

Installation

Prerequisites

• Python 3.8+ (Recommended: 3.10+)

Steps

1. Navigate to the grid-cortex-client directory.
2. Install using pip:
   • For development (editable install):

     BUILD_VERSION=0.0.1 uv pip install -e .
     

     This installs the package in editable mode and includes all dependencies from pyproject.toml.
   • To install from a built wheel (if available in dist/):

     uv pip install dist/grid_cortex_client-*.whl
     

Configuration

The CortexClient requires an API key and the base URL for the Cortex API.

1. Environment Variables (Recommended):

   • GRID_CORTEX_API_KEY: Your API key.
   • GRID_CORTEX_BASE_URL: Base URL (e.g., https://cortex-stage.generalrobotics.dev). Defaults are provided if not set. The client automatically uses these if api_key or base_url are not passed during instantiation.

2. Direct Instantiation:

   from grid_cortex_client import CortexClient
   
   client = CortexClient(api_key="your_api_key_here", base_url="your_custom_base_url_here")
   

Available Models & Handlers

The client uses a keyword-based system to map a given model_id to its appropriate handler. The model_id you use should typically contain one of the keywords associated with a handler.

Model Task	Handler Class	Keywords for model_id	Key Input Parameters (**kwargs for client.run)	Expected Output Type (from client.run)
Depth Estimation	DepthModel	midas, marigold, depthpro, metric3d, depthanything2, zoedepth	image_input (path, URL, or PIL.Image)	PIL.Image.Image (depth map)
Stereo Depth Estimation	FoundationStereoModel	foundationstereo	left_image, right_image, K (camera intrinsics), baseline (stereo baseline)	numpy.ndarray (depth map)
Object Detection	DetectionModel	owlv2	image_input (path, URL, or PIL.Image), prompt (str or List[str]), box_threshold (float, optional)	List[Dict] (list of detections)
Image Segmentation	SegmentationModel	clipseg, lseg, gsam2	image_input (path, URL, or PIL.Image), prompt (str)	PIL.Image.Image (mask)
Grasp Generation	GraspModel	m2t2	xyz (point cloud), rgb (RGB values), seg (segmentation labels)	Dict[str, numpy.ndarray] (grasps and confidence)
Grasp Generation (Depth Images)	GraspGenModel	graspgen	depth_image, seg_image, camera_intrinsics, aux_args (num_grasps, gripper_config, camera_extrinsics)	Dict[str, numpy.ndarray] (grasps, confidence, latency_ms)

Logging

The client uses the standard Python logging module. You can configure the logging level to control verbosity:

import logging

# For verbose output from the client and underlying HTTP library:
# logging.basicConfig(level=logging.INFO)

# To reduce verbosity (show only warnings and errors):
logging.getLogger("grid_cortex_client").setLevel(logging.WARNING)
logging.getLogger("httpx").setLevel(logging.WARNING) # For the HTTPX library

Contributing a model

Please refer to the main repository guidelines on adding a new model to cortex (server side). Then add a model and corresponding test in client:

1. Create a new handler class in src/grid_cortex_client/models/ inheriting from BaseModel.
2. Implement the preprocess and postprocess methods.
3. Add relevant keywords and the new handler class to the _MODEL_ID_TO_HANDLER_CLASS dictionary in src/grid_cortex_client/cortex_client.py.
4. Add tests and update documentation (including the "Available Models" table in README.md).