🐝 broodminder-data

Make BroodMinder beehive data easily accessible across diverse consumption use cases — bulk export, periodic delta sync, OpenAPI 3.1, and AI agents via MCP.

broodminder-data (formerly broodminder-export) provides a unified developer platform for BroodMinder beehive sensor telemetry (internal/ambient temperature, relative humidity, scale weight, swarm indicators, acoustics, radar, and inspection notes). Similar to empower-personal-dashboard, this project provides a published OpenAPI 3.1 specification, a Python CLI and client SDK, and a Model Context Protocol (MCP) server for AI agents.

5 Core Consumption Modalities

                     ┌──────────────────────────────────────────────┐
                     │         BroodMinder Cloud API               │
                     └──────────────────────┬───────────────────────┘
                                            │
                                ┌───────────▼───────────┐
                                │   broodminder-data    │
                                └───────────┬───────────┘
                                            │
         ┌──────────────────┬───────────────┼───────────────┬──────────────────┐
         ▼                  ▼               ▼               ▼                  ▼
  📦 Bulk Export      🔄 Delta Sync   📜 OpenAPI 3.1   🤖 MCP Server    🐍 Python SDK / CLI
  Full history       Incremental     Published spec   Claude, Cursor,   Typed models &
  JSON / CSV / NDJSON catch-up cron   & Redocly docs   Antigravity CLI   scriptable client

1. 📦 Bulk Historical Export: Walks your entire apiary → hive → device hierarchy with rate-limit-aware, resumable 180-day windowing to extract complete multi-year histories into compressed JSON, NDJSON, and CSV.
2. 🔄 Periodic Delta Sync: Lightweight, incremental polling engine designed for regular cron or background services, fetching only new readings since the last checkpoint while buffering for late-arriving BLE uploads.
3. 📜 Published OpenAPI 3.1 Specification: Formal, validated OpenAPI 3.1 contract covering all observed endpoints, query parameters, error responses (including HTTP 412 auth responses), and canonical telemetry schemas.
4. 🤖 Model Context Protocol (MCP) Server: Native MCP integration (broodminder-mcp) connecting AI agents (Claude Desktop, Antigravity CLI, Cursor, Windsurf, Claude Code) directly to hive metrics, temperature trends, weight deltas, and notes.
5. 🐍 Unified Python Client Library & CLI: Strongly-typed domain models, offline sandbox mode, and a standalone broodminder CLI.

Architecture Discussions & Roadmap

We are tracking each expanded capability in GitHub Discussions. Join the conversation:

• 💬 Discussion #148: Periodic Delta Sync Engine for Incremental Telemetry & Continuous Ingestion
• 💬 Discussion #149: Published OpenAPI 3.1 Specification & Interactive Documentation (Redocly/Swagger)
• 💬 Discussion #150: Model Context Protocol (MCP) Server for Hive Monitoring & AI Agent Integration
• 💬 Discussion #151: Unified Python Client SDK and Standalone CLI Library

Table of Contents

• Features
• What You Get
• Get an API Key
• Installation
• Configuration
• Quickstart Usage
  • 1. Discover Account Topology
  • 2. Bulk History Export
  • 3. Incremental Catch-up Sync
  • 4. Build Analysis-Ready Datasets
• OpenAPI 3.1 Specification & Interactive Docs
• Model Context Protocol (MCP) Server
• Python SDK Usage
• PyPI Packaging & Automated Publishing
• Output Files & Schema
• API Behavior & Rate Limits
• Testing & Quality Gates
• Project Structure
• Privacy & Security
• Contributing
• License

Features

• 📦 Complete export — walks every apiary → hive → device and pulls all readings and notes across your entire history.
• 🔁 Resumable — checkpoints each time window; stop and re-run anytime and it skips what's already fetched.
• 🚦 Rate-limit-aware — respects the ~1000 calls/day cap, self-throttles, and resumes cleanly after a 429.
• 🧹 Idempotent outputs — de-duplicates overlapping windows, so re-runs never double-count.
• 🗜️ Compact — raw and flattened outputs are gzipped (a multi-year, 90-hive account is tens of MB).
• 📜 OpenAPI 3.1 spec — formal machine-readable API definition with Redocly validation.
• 🤖 Agent-ready — MCP server architecture for conversational hive analysis and automated inspections.
• 🧪 Contract-tested — live contract test suite pins the API's real behavior and acts as a canary when upstream changes.
• 🔌 Reusable client — bm/client.py is transport-clean and easy to lift into a notebook, script, or MCP server.

What You Get

A flattened, analysis-ready row per reading:

Metric presence varies by device type — temperature is near-universal; weight appears only on hives with a scale; audio and radar appear on specialized monitors.

Get an API Key

The External User API is in alpha. Request a key from BroodMinder (support@broodminder.com). The key is tied to your account and only authorizes access to your own data.

Installation

From PyPI

# Core package
pip install broodminder-data

# With Model Context Protocol (MCP) agent support:
pip install "broodminder-data[mcp]"

From Source (Local Development)

git clone https://github.com/petry-projects/broodminder-data.git
cd broodminder-data

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/pip install -e ".[mcp,dev]"

Requires Python 3.10+.

Configuration

cp .env.example .env

Edit .env and paste your key:

BROODMINDER_API_KEY=your-api-key-here
BROODMINDER_BASE_URL=https://external-api.mybroodminder.com

.env is git-ignored and never leaves your machine.

Quickstart Usage

1. Discover Account Topology

Confirm auth and see your apiaries, hives, and a sensor data sample:

.venv/bin/python scripts/discover.py

2. Bulk History Export

Pull historical telemetry (resumable; respects the daily quota):

.venv/bin/python scripts/extract_all.py --start 2025-01-01

Options:

3. Incremental Catch-up Sync

Resume forward from each hive's latest extracted window:

.venv/bin/python scripts/extract_all.py --catchup

4. Build Analysis-Ready Datasets

Convert raw windows into clean, de-duplicated NDJSON and CSV:

.venv/bin/python scripts/flatten.py --merge

OpenAPI 3.1 Specification & Interactive Docs

broodminder-data maintains a formal OpenAPI 3.1 specification (also accessible via the root symlink openapi.yaml).

Local Linting & Preview

Using Redocly CLI:

# Lint specification against OpenAPI 3.1 rules
npx @redocly/cli lint openapi.yaml

# Launch interactive documentation preview server
npx @redocly/cli preview-docs openapi.yaml

Model Context Protocol (MCP) Server

Connect your hive data directly to AI agents (Claude Desktop, Antigravity CLI, Cursor, Windsurf, Claude Code):

Agent Configuration (claude_desktop_config.json or mcp.json)

{
  "mcpServers": {
    "broodminder": {
      "command": "python3",
      "args": ["-m", "bm.mcp_server"],
      "env": {
        "BROODMINDER_API_KEY": "your-api-key-here"
      }
    }
  }
}

Core MCP Tools

• get_apiary_summary: High-level inventory of apiaries, hives, and device counts.
• get_hive_status: Latest sensor telemetry (brood temperature, ambient temperature, humidity, weight).
• get_telemetry_trends: Time-series rollups (min, max, mean, delta) over specified lookback windows.
• get_hive_notes: Recent inspection notes, treatments, and queen observations.
• get_device_health: Battery levels and sync freshness across sensors.

Python SDK Usage

from bm.client import BroodMinderClient

# Automatically reads BROODMINDER_API_KEY from environment or .env
client = BroodMinderClient()

# List apiaries and hives
apiaries = client.get_apiaries()
for apiary in apiaries:
    print(f"Apiary: {apiary['name']} (ID: {apiary['apiary_id']})")
    hives = client.get_hives(apiary_id=apiary['apiary_id'])
    for hive in hives:
        print(f"  - Hive: {hive['name']}")

# Fetch time-series readings for a device (epoch seconds)
readings = client.get_device_readings(
    device_id="42:11:22:33:44:55",
    start=1704067200,  # 2024-01-01T00:00:00Z
    end=1706745600,    # 2024-02-01T00:00:00Z
)
print(f"Fetched {len(readings)} readings")

PyPI Packaging & Automated Publishing

broodminder-data adopts the automated, tokenless PyPI Trusted Publishing (OIDC) approach pioneered in don-petry/brand-ops.

1. Tokenless Trusted Publishing Architecture

Releases publish directly from GitHub Actions without storing long-lived, sensitive API tokens:

• GitHub Actions exchanges its cryptographic OIDC ID token with PyPI for a short-lived upload token.
• PyPI validates the repository (petry-projects/broodminder-data), workflow (publish.yml), and environment (pypi).

2. Onboarding Steps (First Release Setup)

Before publishing the first release, the account owner registers a Pending Publisher on PyPI:

1. Log in to pypi.org/manage/account/publishing/.
2. Under "Add a pending publisher", enter:
   • PyPI Project Name: broodminder-data
   • Owner: petry-projects
   • Repository name: broodminder-data
   • Workflow name: publish.yml
   • Environment name: pypi
3. Click "Add publisher".

3. Local Onboarding & Verification

Run the onboarding tool to inspect registry availability, build the sdist and wheel, and verify package metadata:

# Probe PyPI status, build sdist/wheel, and run twine verification
python scripts/pypi_onboard.py

4. Automated Publishing Workflow

• Automatic: Creating a GitHub Release automatically builds and publishes packages to PyPI via .github/workflows/publish.yml.
• Manual Trigger (with Dry Run): You can also run the workflow manually via workflow_dispatch with dry_run: true (default) to test artifact generation without releasing.

Output Files & Schema

Extracted data is saved under data/extract/ (git-ignored):

API Behavior & Rate Limits

The machine-readable description is in openapi.yaml. Notable quirks handled automatically:

• Authentication: X-Api-Key header. Missing or invalid keys return HTTP 412 (not 401/403).
• Time Windows: Maximum ~6 months per request — auto-chunked.
• No Pagination: Each window is a single JSON array payload.
• Rate Limit: ~1,000 calls per UTC day with no Retry-After header — the client tracks calls, self-throttles, and catches 429 responses cleanly.

Testing & Quality Gates

# Run unit & offline tests
.venv/bin/python -m pytest

# Byte-compile verification
python3 -m compileall bm scripts tests

• Offline tests (tests/test_offline.py, tests/test_scripts_refactor.py): Run fast and hermetically without network access.
• Live contract tests (tests/test_contract.py): Automatically run when BROODMINDER_API_KEY is present to verify live API compatibility; skip gracefully otherwise.
• OpenAPI validation: npx @redocly/cli lint openapi.yaml.

Project Structure

broodminder-data/
├── bm/
│   ├── __init__.py
│   └── client.py            # Reusable BroodMinderClient (auth, retry, windowing)
├── scripts/
│   ├── discover.py          # Auth check + account topology/schema sample
│   ├── extract_all.py       # Resumable, budget-aware extraction (--catchup)
│   ├── flatten.py           # Raw → NDJSON/CSV/coverage (--merge)
│   ├── cron_sync.sh         # Routine unattended forward catch-up sync
│   └── cron_backfill.sh     # Initial unattended multi-day backfill
├── tests/
│   ├── conftest.py
│   ├── test_offline.py      # Fast deterministic unit tests
│   ├── test_scripts_refactor.py # Script unit test coverage
│   └── test_contract.py     # Live contract tests (skip without key)
├── openapi/
│   └── broodminder-openapi.yaml # OpenAPI 3.1 specification
├── openapi.yaml -> openapi/broodminder-openapi.yaml # Root symlink
├── redocly.yaml             # Redocly linting & preview configuration
├── requirements.txt         # Runtime dependencies
└── pyproject.toml           # Build configuration & metadata

Privacy & Security

.env (your API key) and data/ (your extracted hive data) are git-ignored and never leave your machine. All test fixtures use synthetic or anonymized values. Please never paste an API key or raw hive telemetry into an issue, PR, or discussion.

Contributing

See CONTRIBUTING.md, SECURITY.md, and the Code of Conduct. Check out open Discussions to weigh in on upcoming features and architectural decisions.

License

MIT © Petry Projects.