Skip to main content

Google Cloud Storage Plugin for Search Toolkit

Google Cloud Storage backend for mistralai-search-toolkit.

This plugin implements the Search Toolkit's ObjectStorage interface, enabling the ingestion pipeline to load files directly from Google Cloud Storage.

Installation

pip install mistralai-search-toolkit-storage-gcs

Or as an optional dependency of the core package:

pip install mistralai-search-toolkit[storage-gcs]

Quick Start: Load Files from GCS in Ingestion Pipeline

1. Upload a File to GCS

import asyncio
from mistralai.search.toolkit.plugins.storage.gcs import GCSBlobStorage


async def upload_file():
    storage = GCSBlobStorage(
        bucket_name="your-bucket",
        project_id="your-project",
    )

    # Upload a file
    with open("document.pdf", "rb") as f:
        data = f.read()

    await storage.put(key="documents/document.pdf", data=data)


asyncio.run(upload_file())

2. Load Files from GCS in Ingestion Pipeline

import asyncio
import os
from mistralai.search.toolkit.ingestion.loaders import FileLoader
from mistralai.search.toolkit.ingestion.pipelines import Pipeline
from mistralai.search.toolkit.ingestion.text_splitters import CharacterTextSplitter
from mistralai.search.toolkit.embedding import MistralEmbedder, MODEL_1024_EMBEDDING
from mistralai.client import Mistral
from mistralai.search.toolkit.plugins.storage.gcs import GCSBlobStorage
from mistralai.search.toolkit.plugins.vespa import VespaClientConfig
from vespa_app import app


async def ingest_from_gcs():
    # Create GCS storage factory
    def gcs_storage_factory():
        return GCSBlobStorage(
            bucket_name="your-bucket",
            project_id="your-project",
        )

    # Create FileLoader backed by GCS
    file_loader = FileLoader(storage_factory=gcs_storage_factory)

    # Create ingestion pipeline
    mistral_client = Mistral(api_key=os.environ.get("MISTRAL_API_KEY"))
    vespa_config = VespaClientConfig(
        endpoint=os.environ.get("VESPA_ENDPOINT", "http://localhost:8080"),
    )
    vector_store = app.get_search_index(vespa_config, collection_name="articles")

    pipeline = Pipeline(
        loader=file_loader,
        text_splitter=CharacterTextSplitter(chunk_size=512),
        embedder=MistralEmbedder(client=mistral_client, model_name=MODEL_1024_EMBEDDING),
        stores=vector_store,
    )

    # Ingest documents from GCS
    num_chunks = await pipeline.run(
        documents=[
            "documents/document1.pdf",
            "documents/document2.pdf",
        ]
    )

    print(f"Indexed {num_chunks} chunks")


asyncio.run(ingest_from_gcs())

Configuration

Basic Setup

storage = GCSBlobStorage(
    bucket_name="your-bucket",
    project_id="your-project",
)

Using Service Account

from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file("/path/to/service-account-key.json")

storage = GCSBlobStorage(
    bucket_name="your-bucket",
    project_id="your-project",
    credentials=credentials,
)

Authentication

Environment Variables

Set credentials using environment variables:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

Or authenticate with gcloud CLI:

gcloud auth application-default login

The plugin will automatically use credentials from:

  • GOOGLE_APPLICATION_CREDENTIALS environment variable
  • Application Default Credentials (if running in GCP)

License

This plugin is licensed under the Apache License 2.0.

Support

For Search Toolkit issues, refer to the Search Toolkit documentation.

For Google Cloud Storage documentation, visit GCS Docs.

Metadata

Release files for mistralai-search-toolkit-storage-gcs 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mistralai-search-toolkit-storage-gcs 0.1.0
File Size Uploaded Provenance Actions
mistralai_search_toolkit_storage_gcs-0.1.0.tar.gz 16.3 kB With provenance Details for mistralai_search_toolkit_storage_gcs-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mistralai-search-toolkit-storage-gcs 0.1.0
File Interpreter ABI Platform Actions
mistralai_search_toolkit_storage_gcs-0.1.0-py3-none-any.whl Python 3 none any Details for mistralai_search_toolkit_storage_gcs-0.1.0-py3-none-any.whl

Total release size: 27.6 kB

Release files / mistralai_search_toolkit_storage_gcs-0.1.0.tar.gz

Download URL mistralai_search_toolkit_storage_gcs-0.1.0.tar.gz
Size 16.3 kB
Tags Source
SHA-256 checksum
How to use checksums
ab4e0618b44da891956b4700a5c67db9f80fd06c2205eeb7066362126c3d9efb
BLAKE2b-256 checksum
How to use checksums
94085345f2ad4194c987d4f1c90c357f275203e366d060e170e5eb68d03c9f0b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Release files / mistralai_search_toolkit_storage_gcs-0.1.0-py3-none-any.whl

Download URL mistralai_search_toolkit_storage_gcs-0.1.0-py3-none-any.whl
Size 11.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f37a29775f4d9f89654e9cf9f862481335631a74951621ce825ec0aaec64532e
BLAKE2b-256 checksum
How to use checksums
4e6f76ee28a7e48dc28c64c15a372aa831dde4a6a3bf69d42c5f09a0be9e0902
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Security for mistralai-search-toolkit-storage-gcs 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for mistralai_search_toolkit_storage_gcs-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page