cascache-server 0.2.0

Flexible ContentAddressableStorage written in python

cascache_server

⚠️ PROOF OF CONCEPT - NOT PRODUCTION READY

This is a conceptual implementation for research and development purposes. While it demonstrates core CAS functionality and includes comprehensive testing, it is not intended for production use. Use at your own risk.

See docs/concept.md for more details about the concept and technologies used.

Conten Adressable Storage (CAS) server solution in python. Focus was to make a caching server that is compliant to bazel-remote and by that, implementing the Remote Execution API (REAPI) via gRPC. This is a companion app, to the cascade workflow tool.

Warning: Large parts of this tool were generated with the help of AI. Special thanks to Claude Sonnet for the excellent support!

Features

• ✅ REAPI v2 Compliant - ContentAddressableStorage, ActionCache, and Capabilities services
• ✅ Comprehensive Testing - 218 tests, Docker support, comprehensive error handling
• ✅ High Performance - Optimized with protobuf serialization, streaming, and LRU caching
• ✅ Thread-Safe - Atomic file writes, thread-safe metrics, validated under concurrent load
• ✅ Secure - Token-based authentication with SHA256 hashing and CLI management
• ✅ Rate Limiting - Token bucket algorithm protects against overload
• ✅ Multiple Backends - Filesystem, Memory, S3 (AWS S3, MinIO, Cloudflare R2, Backblaze B2)
• ✅ Eviction Policies - TTL, LRU, and Size Quota eviction with real timestamp tracking
• ✅ Bazel Integration - 11x faster builds with ActionCache and CAS
• ✅ Monitoring Dashboard - Real-time web UI for metrics and performance
• ✅ Metrics Tracking - Cache hits, misses, storage size, operations

Quick Start

Installation

From PyPI (Recommended):

# Install cascache_server
pip install cascache_server

# Verify installation
cascache_server --version

# With optional dashboard support
pip install cascache_server[dashboard]

# With all optional features
pip install cascache_server[all]

From Source (Development):

Install from the current directory:

uv sync
uv run cascache_server --help

Or via pip:

pip install -e ".[dev]"
cascache_server --help

Start the Server

# Start with default settings (filesystem storage, port 50051)
just serve

# Start with debug logging
just serve-debug

# Start with dashboard enabled (requires uv pip install -e ".[dashboard]")
just serve-dashboard

# Or directly:
uv run python -m cascache_server

Configure Bazel

Add to your .bazelrc:

build --remote_cache=grpc://localhost:50051

Test with a build:

# First build - cold cache
bazel clean
bazel build //...

# Second build - should be ~10x faster
bazel build //...

Configuration

Configure the server using environment variables:

Variable	Default	Description
CAS_STORAGE_TYPE	filesystem	Storage backend: filesystem, memory, s3
CAS_STORAGE_PATH	/tmp/cas	Storage path (filesystem) or bucket (S3)
CAS_PORT	50051	gRPC server port
CAS_HOST	[::]	Server host (IPv6 all interfaces)
CAS_WORKERS	10	Thread pool size
CAS_MAX_BLOB_SIZE	1000000000	Max blob size (1GB)
CAS_MAX_AGE_DAYS	30	Blob TTL in days (deprecated, use CAS_EVICTION_MAX_AGE_DAYS)
CAS_LOG_LEVEL	INFO	Logging level: DEBUG, INFO, WARNING
CAS_LOG_FORMAT	json	Log format: json, text
CAS_REAPI_VERSIONS	simple	REAPI versions (comma-separated)
CAS_ENABLE_CACHE	true	Enable LRU caching layer
CAS_CACHE_SIZE	1000	LRU cache size (entries)
CAS_CACHE_MAX_BLOB_SIZE	65536	Max blob size to cache (64KB)
CAS_AUTH_ENABLED	false	Enable token-based authentication
CAS_AUTH_TOKENS_FILE	/etc/cas/tokens.json	Path to tokens file
CAS_RATE_LIMIT_ENABLED	true	Enable rate limiting
CAS_RATE_LIMIT_RPS	1000.0	Rate limit (requests/second)
CAS_RATE_LIMIT_BURST	100	Rate limit burst capacity

S3 Backend Configuration:

Variable	Default	Description
AWS_ACCESS_KEY_ID	-	AWS access key (required for S3)
AWS_SECRET_ACCESS_KEY	-	AWS secret key (required for S3)
AWS_REGION	us-east-1	AWS region
CAS_S3_ENDPOINT_URL	-	Custom S3 endpoint (for MinIO, R2, B2)
CAS_S3_PREFIX		S3 object prefix for blob isolation

Eviction Configuration:

Variable	Default	Description
CAS_EVICTION_ENABLED	true	Enable automatic eviction
CAS_EVICTION_POLICY	ttl	Eviction policy: ttl, lru, size_quota
CAS_EVICTION_MAX_AGE_DAYS	30	TTL eviction: max age in days
CAS_EVICTION_MAX_SIZE_GB	100	LRU/Size Quota: max storage size in GB
CAS_EVICTION_TARGET_SIZE_GB	80	LRU/Size Quota: target size after eviction
CAS_EVICTION_INTERVAL_HOURS	24	Eviction check interval in hours

Dashboard Configuration:

Variable	Default	Description
CAS_DASHBOARD_ENABLED	false	Enable web dashboard
CAS_DASHBOARD_PORT	8080	Dashboard HTTP port
CAS_DASHBOARD_HOST	0.0.0.0	Dashboard bind address

Performance Optimizations

The server includes several performance optimizations:

1. Protobuf Binary Serialization - ActionCache uses binary protobuf instead of JSON (5-10x faster)
2. Streaming Support - Large blobs (>1MB) are streamed to prevent memory bloat
3. LRU Caching - Frequently accessed small blobs are cached in memory
4. Optimized I/O - Uses stat() for size checks, direct file streaming
5. Zero-Copy Proto Handling - Eliminated redundant serialization/deserialization

These optimizations enable Python to achieve performance comparable to compiled languages for I/O-bound workloads.

Results: 11x Bazel build speedup (cold: 11s, warm: 1s)

Thread Safety & Concurrency

The server is production-hardened for concurrent workloads:

1. Atomic File Writes - temp + rename pattern prevents corruption
2. Thread-Safe Metrics - Locks protect all counter updates
3. Rate Limiting - Token bucket algorithm per-client limits
4. Validated Under Load - 99 tests including 8 concurrency stress tests

For detailed architecture and implementation, see:

• spec/design.md - Section 11: Performance Architecture
• spec/design.md - Section 12: Concurrency Architecture
• spec/design.md - Section 13: Security Architecture

Authentication

Enable Authentication

export CAS_AUTH_ENABLED=true
export CAS_AUTH_TOKENS_FILE=/etc/cas/tokens.json
just serve

Token Management

# Create a token
cascache_server admin create-token --name "team-ci" --expires-days 30
# Output: cas-token-abc123...

# List all tokens
cascache_server admin list-tokens

# Revoke a token
cascache_server admin revoke-token cas-token-abc123

Client Configuration (Bazel)

# .bazelrc
build --remote_cache=grpc://localhost:50051
build --remote_header=authorization=Bearer cas-token-abc123...

Security Features

• ✅ SHA256 Hashing - Only token hashes stored, never plaintext
• ✅ File Permissions - tokens.json restricted to owner (chmod 600)
• ✅ Token Expiration - Configurable TTL (default: 30 days)
• ✅ Token Revocation - Instant invalidation of compromised tokens
• ✅ Cryptographic Tokens - Generated with secrets.token_urlsafe(32)

For detailed setup guide, see spec/operations.md - Authentication.

Rate Limiting

Configuration

export CAS_RATE_LIMIT_ENABLED=true
export CAS_RATE_LIMIT_RPS=1000.0    # Requests per second
export CAS_RATE_LIMIT_BURST=100      # Burst capacity

The server uses a token bucket algorithm for rate limiting:

• Per-client rate limits (identified by metadata or method)
• Allows bursts while maintaining average rate
• Returns RESOURCE_EXHAUSTED when limit exceeded
• Thread-safe for concurrent requests

Monitoring

Access cache statistics programmatically:

from cascache_server.storage import CachedStorage

storage = CachedStorage(backend, cache_size=1000)
stats = storage.get_cache_stats()

# Returns:
# {
#   "get_cache": {"hits": 234, "misses": 12, "maxsize": 1000, "currsize": 156},
#   "exists_cache": {"hits": 567, "misses": 23, "maxsize": 1000, "currsize": 234},
#   "size_cache": {"hits": 123, "misses": 8, "maxsize": 1000, "currsize": 89}
# }

Dashboard

Real-time web dashboard for monitoring server metrics and performance.

Enable Dashboard

# Install dashboard dependencies
uv pip install -e ".[dashboard]"

# Start server with dashboard (easiest)
just serve-dashboard

# Or manually with environment variables
export CAS_DASHBOARD_ENABLED=true
export CAS_DASHBOARD_PORT=8080
export CAS_DASHBOARD_HOST=0.0.0.0
just serve

The dashboard will be available at http://localhost:8080

Features

• Real-time Metrics - Auto-refreshes every 2 seconds
• Cache Performance - Hit rate, hits/misses, total requests
• Storage Stats - Blob count, total size (human-readable)
• Operations - Read/write/delete counters
• Eviction Tracking - Eviction count, last/next run times
• Server Info - Uptime, status

API Endpoints

# Get JSON stats
curl http://localhost:8080/api/stats

# Health check
curl http://localhost:8080/health

Response Example:

{
  "cache": {
    "hits": 1234,
    "misses": 56,
    "total_requests": 1290,
    "hit_rate": "95.66%"
  },
  "storage": {
    "blobs_count": 456,
    "total_bytes": 1073741824,
    "total_bytes_human": "1.00 GB"
  },
  "operations": {
    "reads": 1500,
    "writes": 500,
    "deletes": 10
  },
  "eviction": {
    "evictions": 25,
    "last_run": 1707648000.0,
    "next_run": 1707734400.0
  },
  "server": {
    "uptime_seconds": 86400,
    "uptime_human": "1d 0h 0m 0s"
  }
}

Configuration

Variable	Default	Description
CAS_DASHBOARD_ENABLED	false	Enable web dashboard
CAS_DASHBOARD_PORT	8080	Dashboard HTTP port
CAS_DASHBOARD_HOST	0.0.0.0	Dashboard bind address

Note: Dashboard requires Python 3.13 or lower due to FastAPI/Pydantic compatibility. Python 3.14 support will be added when upstream dependencies are updated.

Eviction Policies

Automatic garbage collection with configurable eviction policies.

TTL Eviction (Time-To-Live)

Evict blobs older than a specified age:

export CAS_EVICTION_ENABLED=true
export CAS_EVICTION_POLICY=ttl
export CAS_EVICTION_MAX_AGE_DAYS=30      # Evict blobs >30 days old
export CAS_EVICTION_INTERVAL_HOURS=24    # Check every 24 hours

uv run cascache_server serve

LRU Eviction (Least Recently Used)

Evict least recently accessed blobs when storage quota is exceeded:

export CAS_EVICTION_ENABLED=true
export CAS_EVICTION_POLICY=lru
export CAS_EVICTION_MAX_SIZE_GB=100      # Max 100GB storage
export CAS_EVICTION_TARGET_SIZE_GB=80    # Evict down to 80GB
export CAS_EVICTION_INTERVAL_HOURS=1     # Check every hour

uv run cascache_server serve

Size Quota Eviction (Oldest First)

Evict oldest blobs (by creation time) when storage quota is exceeded:

export CAS_EVICTION_ENABLED=true
export CAS_EVICTION_POLICY=size_quota
export CAS_EVICTION_MAX_SIZE_GB=100      # Max 100GB storage
export CAS_EVICTION_TARGET_SIZE_GB=80    # Evict down to 80GB
export CAS_EVICTION_INTERVAL_HOURS=1     # Check every hour

uv run cascache_server serve

Features:

• Background worker runs at configurable intervals
• Non-blocking eviction (separate thread)
• Graceful shutdown with server
• Manual triggering via run_now() method
• Stats tracking (runs, total evicted, bytes freed)
• Pluggable policy architecture (TTL, LRU, custom)

Manual Eviction:

from cascache_server.eviction import (
    EvictionWorker,
    TTLEvictionPolicy,
    LRUEvictionPolicy,
    SizeQuotaEvictionPolicy,
)
from cascache_server.storage.filesystem import FilesystemStorage

storage = FilesystemStorage("/tmp/cas")

# TTL policy: evict blobs older than 7 days
ttl_policy = TTLEvictionPolicy(max_age_days=7)

# LRU policy: keep storage under 10GB (evict least recently used)
lru_policy = LRUEvictionPolicy(
    max_size_bytes=10 * 1024 * 1024 * 1024,
    target_size_bytes=8 * 1024 * 1024 * 1024,
)

# Size Quota policy: keep storage under 10GB (evict oldest first)
size_quota_policy = SizeQuotaEvictionPolicy(
    max_size_bytes=10 * 1024 * 1024 * 1024,
    target_size_bytes=8 * 1024 * 1024 * 1024,
)

# Create worker with chosen policy
worker = EvictionWorker(storage, size_quota_policy)
worker.run_now()

stats = worker.get_stats()
print(f"Evicted {stats['total_evicted']} blobs ({stats['total_freed_bytes']} bytes)")

Metadata Tracking:

All storage backends now track real timestamps:

• FilesystemStorage: Timestamps stored in .meta JSON files alongside blobs
• MemoryStorage: Timestamps tracked in-memory dictionaries
• S3Storage: Timestamps stored in S3 object metadata

Eviction policies use real created_at and accessed_at timestamps for production-ready eviction decisions.

Development

Setup

uv sync

Testing

# Run all tests (218 tests)
just test

# Run unit tests only
just test-unit

# Run integration tests only
just test-integration

# Run with coverage
just test-cov

# Run Bazel integration tests (Docker)
just test-bazel-docker

# Run Bazel integration tests with authentication
just test-bazel-docker-auth

# Run Bazel integration tests with S3 backend
just test-bazel-docker-s3

# Run Bazel tests locally (requires Bazel installed)
just test-bazel-local

Note on Bazel Integration: When running Bazel tests, you may see this warning:

ERROR: Failed to query remote execution capabilities: The client supported API versions, 2.0 to 2.0, is not supported by the server
WARNING: Remote Cache: The client supported API versions, 2.0 to 2.0, is not supported by the server, 2.0 to 0.0

This is expected and harmless - it's Bazel detecting that we don't support remote execution (only caching). The cache works correctly despite this message.

How to verify caching is working:

• Primary indicator: Build time speedup (cold: ~10s, warm: ~1s = 10x faster)
• Secondary indicators: Bazel shows checking cached actions and remote-cache in action logs
• The speedup proves the cache is functioning correctly, even if cache hit counts aren't explicitly shown

Linting & Formatting

# Check for linting issues
just lint

# Auto-fix lint issues
just lint-fix

# Format code
just format

# Type-check
just typecheck

# Run all CI checks
just ci

Docker Deployment

Build and Run

# Build the Docker image
docker build -t cascache_server:latest .

# Run with filesystem storage
docker run -p 50051:50051 \
  -v $(pwd)/data:/data/cas \
  -e CAS_STORAGE_PATH=/data/cas \
  -e CAS_LOG_LEVEL=INFO \
  cascache_server:latest

# Run with S3 storage (MinIO/AWS S3/R2/B2)
docker run -p 50051:50051 \
  -e CAS_STORAGE_TYPE=s3 \
  -e CAS_STORAGE_PATH=my-cas-bucket \
  -e AWS_ACCESS_KEY_ID=minioadmin \
  -e AWS_SECRET_ACCESS_KEY=minioadmin \
  -e CAS_S3_ENDPOINT_URL=http://minio:9000 \
  cascache_server:latest

# Run with eviction enabled
docker run -p 50051:50051 \
  -v $(pwd)/data:/data/cas \
  -e CAS_STORAGE_PATH=/data/cas \
  -e CAS_EVICTION_ENABLED=true \
  -e CAS_EVICTION_POLICY=lru \
  -e CAS_EVICTION_MAX_SIZE_GB=50 \
  -e CAS_EVICTION_TARGET_SIZE_GB=40 \
  cascache_server:latest

Docker Compose

See tests/bazel/docker-compose.yml for filesystem example and tests/bazel/docker-compose.s3.yml for S3/MinIO example.

Implementation Status

Component	Status	Tests
Storage Layer	✅ Complete	72 tests
CAS Service	✅ Complete	12 tests
ActionCache Service	✅ Complete	11 tests
Capabilities Service	✅ Complete	-
gRPC Integration	✅ Complete	10 tests
Bazel Integration	✅ Validated	2 tests
LRU Caching	✅ Complete	11 tests
Thread Safety	✅ Complete	8 tests
Authentication	✅ Complete	7 tests
Rate Limiting	✅ Complete	14 tests
S3 Backend	✅ Complete	29 tests
TTL Eviction	✅ Complete	13 tests
LRU Eviction	✅ Complete	17 tests
Size Quota Eviction	✅ Complete	15 tests
Eviction Worker	✅ Complete	12 tests
Metadata Tracking	✅ Complete	10 tests

Total: 185 passing tests (118 baseline + 57 eviction + 10 metadata)

Performance Benchmarks

Bazel Build Performance:

• Cold build: 11 seconds
• Warm build: 1 second
• Speedup: 11x (cache hit rate >90%)

Storage Backend Performance:

Operation	Filesystem	S3 (MinIO)	Memory
Small blob (1KB) get	~1ms	~5ms	<1µs
Large blob (10MB) get	~100ms	~200ms	~50ms
Cached get	<10µs	<10µs	<1µs

Memory Usage:

• Baseline: ~50MB
• With LRU cache (1000 entries): ~80MB
• With LRU cache (10000 entries): ~200MB

Documentation

Architecture & Design

• spec/design.md - Complete architecture documentation including:
  • Component architecture and layered design
  • Performance architecture and optimizations (Section 11)
  • Concurrency architecture and thread safety (Section 12)
  • Security architecture (Section 13)
  • Design decisions and trade-offs
• spec/roadmap.md - Development roadmap and milestones
• spec/operations.md - Deployment and operations guide
• spec/api.md - API reference
• spec/testing.md - Testing strategy and guidelines

Development

• .github/copilot-instructions.md - AI pair programming context

Architecture

The project follows a strict layered architecture:

1. gRPC API Layer - Protocol buffer handlers
2. Service Layer - Business logic (CAS, ActionCache, Capabilities)
3. Storage Abstraction - StorageBackend interface
4. Storage Implementations - Filesystem, Memory, S3, Cached wrapper
5. Eviction Layer - Pluggable eviction policies (TTL, LRU, Size Quota)
6. Models - Digest, Blob, ActionResult, BlobMetadata dataclasses

Storage Backends

• FilesystemStorage - Production-ready persistent storage with metadata tracking
• MemoryStorage - In-memory storage for testing
• S3Storage - AWS S3, MinIO, Cloudflare R2, Backblaze B2 support
• CachedStorage - LRU cache wrapper for any backend

Development & Contributing

Setup Development Environment

# Clone repository
git clone https://gitlab.com/cascascade/cascache_server.git
cd cascache_server

# Install dependencies
uv sync

# Run tests
just test

# Run linter
just lint

# Run type checker
just typecheck

Running Tests

# All tests
just test

# Unit tests only
just test-unit

# Integration tests
just test-integration

# Bazel integration (Docker)
just test-bazel-docker

# With coverage
just test-cov

Making Changes

1. Create a branch: git checkout -b feature/my-feature
2. Make changes and add tests
3. Run checks: just ci
4. Commit with conventional commits: git commit -m "feat: add new feature"
5. Push and create merge request

Deployment

See docs/deployment.md for comprehensive deployment guide.

Quick deployment:

# First-time setup
./scripts/first_deploy.sh

# Bump version and deploy
just release-patch  # 0.1.0 → 0.1.1
just release-minor  # 0.1.0 → 0.2.0
just release-major  # 0.1.0 → 1.0.0

Automated deployment via GitLab CI:

# Bump version
just bump-patch

# Commit and tag
git add src/cascache_server/_version.py
git commit -m "chore: bump version to 0.1.1"
git tag v0.1.1

# Push (CI will deploy to PyPI)
git push && git push --tags

Resources

• REAPI Specification
• Bazel Remote Caching
• gRPC Python Documentation

License

MIT