Skip to main content

Google Cloud Storage Plugin for Search Toolkit

Google Cloud Storage backend for mistralai-search-toolkit.

This plugin implements the Search Toolkit's ObjectStorage interface, enabling the ingestion pipeline to load files directly from Google Cloud Storage.

Installation

pip install mistralai-search-toolkit-storage-gcs

Or as an optional dependency of the core package:

pip install mistralai-search-toolkit[storage-gcs]

Quick Start: Load Files from GCS in Ingestion Pipeline

1. Upload a File to GCS

import asyncio
from mistralai.search.toolkit.plugins.storage.gcs import GCSBlobStorage


async def upload_file():
    storage = GCSBlobStorage(
        bucket_name="your-bucket",
        project_id="your-project",
    )

    # Upload a file
    with open("document.pdf", "rb") as f:
        data = f.read()

    await storage.put(key="documents/document.pdf", data=data)


asyncio.run(upload_file())

2. Load Files from GCS in Ingestion Pipeline

import asyncio
import os
from mistralai.search.toolkit.ingestion.loaders import FileLoader
from mistralai.search.toolkit.ingestion.pipelines import Pipeline
from mistralai.search.toolkit.ingestion.text_splitters import CharacterTextSplitter
from mistralai.search.toolkit.embedding import MistralEmbedder, MODEL_1024_EMBEDDING
from mistralai.client import Mistral
from mistralai.search.toolkit.plugins.storage.gcs import GCSBlobStorage
from mistralai.search.toolkit.plugins.vespa import VespaClientConfig
from vespa_app import app


async def ingest_from_gcs():
    # Create GCS storage factory
    def gcs_storage_factory():
        return GCSBlobStorage(
            bucket_name="your-bucket",
            project_id="your-project",
        )

    # Create FileLoader backed by GCS
    file_loader = FileLoader(storage_factory=gcs_storage_factory)

    # Create ingestion pipeline
    mistral_client = Mistral(api_key=os.environ.get("MISTRAL_API_KEY"))
    vespa_config = VespaClientConfig(
        endpoint=os.environ.get("VESPA_ENDPOINT", "http://localhost:8080"),
    )
    vector_store = app.get_search_index(vespa_config, collection_name="articles")

    pipeline = Pipeline(
        loader=file_loader,
        text_splitter=CharacterTextSplitter(chunk_size=512),
        embedder=MistralEmbedder(client=mistral_client, model_name=MODEL_1024_EMBEDDING),
        stores=vector_store,
    )

    # Ingest documents from GCS
    num_chunks = await pipeline.run(
        documents=[
            "documents/document1.pdf",
            "documents/document2.pdf",
        ]
    )

    print(f"Indexed {num_chunks} chunks")


asyncio.run(ingest_from_gcs())

Configuration

Basic Setup

storage = GCSBlobStorage(
    bucket_name="your-bucket",
    project_id="your-project",
)

Using Service Account

from google.oauth2 import service_account

credentials = service_account.Credentials.from_service_account_file("/path/to/service-account-key.json")

storage = GCSBlobStorage(
    bucket_name="your-bucket",
    project_id="your-project",
    credentials=credentials,
)

Authentication

Environment Variables

Set credentials using environment variables:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

Or authenticate with gcloud CLI:

gcloud auth application-default login

The plugin will automatically use credentials from:

  • GOOGLE_APPLICATION_CREDENTIALS environment variable
  • Application Default Credentials (if running in GCP)

License

This plugin is licensed under the Apache License 2.0.

Support

For Search Toolkit issues, refer to the Search Toolkit documentation.

For Google Cloud Storage documentation, visit GCS Docs.

Metadata

Release files for mistralai-search-toolkit-storage-gcs 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mistralai-search-toolkit-storage-gcs 0.2.0
File Size Uploaded Provenance Actions
mistralai_search_toolkit_storage_gcs-0.2.0.tar.gz 16.3 kB With provenance Details for mistralai_search_toolkit_storage_gcs-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mistralai-search-toolkit-storage-gcs 0.2.0
File Interpreter ABI Platform Actions
mistralai_search_toolkit_storage_gcs-0.2.0-py3-none-any.whl Python 3 none any Details for mistralai_search_toolkit_storage_gcs-0.2.0-py3-none-any.whl

Total release size: 27.6 kB

Release files / mistralai_search_toolkit_storage_gcs-0.2.0.tar.gz

Download URL mistralai_search_toolkit_storage_gcs-0.2.0.tar.gz
Size 16.3 kB
Tags Source
SHA-256 checksum
How to use checksums
4ff351bff26c18cfea72031bee47951c941574cc260a1659902f1808551e6b50
BLAKE2b-256 checksum
How to use checksums
89e4b93df47b0e19d057fdd0b62e1f24ae6be5d3bda4cd8476cfef0c53bd95b4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Release files / mistralai_search_toolkit_storage_gcs-0.2.0-py3-none-any.whl

Download URL mistralai_search_toolkit_storage_gcs-0.2.0-py3-none-any.whl
Size 11.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1e7c65bcdf8328dae75f58688338ede3dd478e4d743a9a379a3250b701003353
BLAKE2b-256 checksum
How to use checksums
84e0847cacedfd5ae99314c82735adefda56d97d804da86c7b645f2f452501f6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

Security for mistralai-search-toolkit-storage-gcs 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for mistralai_search_toolkit_storage_gcs-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 3, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page