robocorp-adapters-custom 0.1.4

Custom Work Item Adapters for Robocorp Producer-Consumer Automation

robocorp_adapters_custom

Custom Work Item Adapters for Robocorp Producer-Consumer Automation

---

Overview

This repository provides custom adapters for Robocorp's workitems library, enabling scalable producer-consumer automation workflows with pluggable backend support (SQLite, Redis, Amazon DocumentDB/MongoDB, Yorko Control Room, etc.). The architecture is designed for easy backend switching via environment variables, supporting both local development and distributed cloud deployments.

Features

• Pluggable Adapter Pattern: Easily switch between SQLite, Redis, Amazon DocumentDB/MongoDB, Yorko Control Room, and other backends by changing environment variables.
• Producer-Consumer Workflows: Modular tasks for producing, consuming, and reporting on work items.
• Control Room Integration: Connect robots to self-hosted Yorko Control Room via REST API.
• Orphan Recovery: Built-in scripts and adapter logic for recovering orphaned work items.
• File Attachments: Hybrid storage (inline for small files, GridFS for large files in DocumentDB, filesystem for other adapters).
• Automatic Schema Migration: SQLite adapter supports seamless schema upgrades.
• Distributed Processing: Redis and DocumentDB adapters enable high-throughput, multi-worker scaling.
• Cloud-Native Support: DocumentDB adapter optimized for AWS environments with TLS/SSL encryption and replica set support.

Key Components

• _sqlite.py, _redis.py, _docdb.py, _yorko_control_room.py: Custom adapters implementing the BaseAdapter interface.
• workitems_integration.py: Dynamic adapter loader for seamless backend switching.
• scripts/config.py: Loads and validates environment-based configuration.
• scripts/seed_sqlite_db.py, scripts/seed_redis_db.py, scripts/seed_docdb_db.py: Seed scripts for populating test data.
• yamls/robot.yaml, yamls/conda.yaml: Task and environment definitions for RCC workflows.
• devdata/: Environment configs, input/output data, and test artifacts.
• docs/: Implementation guides and architecture documentation.

Installation

This project is packaged as a standard Python package using pyproject.toml and can be installed via pip.

Install from PyPI (when published)

pip install robocorp-adapters-custom

Install from Source

# Clone the repository
git clone https://github.com/joshyorko/robocorp_adapters_custom.git
cd robocorp_adapters_custom

# Install in development mode
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

Requirements

• Python 3.10+
• Dependencies are automatically installed: robocorp-workitems, requests, redis, pymongo

Getting Started

Quick Integration

To use these adapters in your own Robocorp project:

1. Install the package using pip (see Installation above).
2. Set the RC_WORKITEM_ADAPTER environment variable to select your adapter:
   • SQLite: robocorp_adapters_custom._sqlite.SQLiteAdapter
   • Redis: robocorp_adapters_custom._redis.RedisAdapter
   • DocumentDB/MongoDB: robocorp_adapters_custom._docdb.DocumentDBAdapter
   • Yorko Control Room: robocorp_adapters_custom._yorko_control_room.YorkoControlRoomAdapter
3. Alternatively, use one of the pre-configured environment JSON files in devdata/ to set all required variables for your chosen backend when running RCC or your robot tasks.

No code changes are required—just install the package, update your environment configuration, and you're ready to go!

1. Environment Setup

• Install the package via pip or clone the repository.
• Configure environment variables for your chosen adapter (see below).

2. Adapter Selection

Set the RC_WORKITEM_ADAPTER environment variable to select your backend:

• SQLite: robocorp_adapters_custom._sqlite.SQLiteAdapter
• Redis: robocorp_adapters_custom._redis.RedisAdapter
• DocumentDB/MongoDB: robocorp_adapters_custom._docdb.DocumentDBAdapter
• Yorko Control Room: robocorp_adapters_custom._yorko_control_room.YorkoControlRoomAdapter

Other required variables:

• SQLite: RC_WORKITEM_DB_PATH=devdata/work_items.db
• Redis: REDIS_HOST=localhost
• DocumentDB: DOCDB_HOSTNAME=localhost, DOCDB_PORT=27017, DOCDB_USERNAME=<user>, DOCDB_PASSWORD=<pass>, DOCDB_DATABASE=<dbname>
  • For AWS DocumentDB: Also set DOCDB_TLS_CERT=<path/to/rds-combined-ca-bundle.pem>
  • Alternatively, use: DOCDB_URI=mongodb://<user>:<pass>@<host>:<port>/?ssl=true
• Yorko Control Room: YORKO_API_URL=http://localhost:8000, YORKO_API_TOKEN=<token>, YORKO_WORKSPACE_ID=<uuid>, YORKO_WORKER_ID=<worker-id>

3. Running Tasks

Use RCC or the robot.yaml tasks:

SQLite:

rcc run -t Producer -e devdata/env-sqlite-producer.json
rcc run -t Consumer -e devdata/env-sqlite-consumer.json
rcc run -t Reporter -e devdata/env-sqlite-for-reporter.json

Redis:

rcc run -t Producer -e devdata/env-redis-producer.json
rcc run -t Consumer -e devdata/env-redis-consumer.json
rcc run -t Reporter -e devdata/env-redis-reporter.json

DocumentDB/MongoDB:

rcc run -t Producer -e devdata/env-docdb-local-producer.json
rcc run -t Consumer -e devdata/env-docdb-local-consumer.json
rcc run -t Reporter -e devdata/env-docdb-local-reporter.json

Yorko Control Room:

rcc run -t Producer -e devdata/env-yorko-control-room-producer.json
rcc run -t Consumer -e devdata/env-yorko-control-room-consumer.json

See Yorko Control Room Adapter Guide for detailed setup.

4. Seeding and Debugging

• Seed SQLite: python scripts/seed_sqlite_db.py
• Seed Redis: python scripts/seed_redis_db.py
• Seed DocumentDB: python scripts/seed_docdb_db.py (or with custom env: python scripts/seed_docdb_db.py --env devdata/env-docdb-local-producer.json)
• Check DB: python scripts/check_sqlite_db.py
• Recover Orphans: python scripts/recover_orphaned_items.py
• Diagnose Reporter: python scripts/diagnose_reporter_issue.py

Project Conventions

• All configuration is via environment variables (see scripts/config.py).
• Queue names are set by RC_WORKITEM_QUEUE_NAME.
• Output queue names can be customized via RC_WORKITEM_OUTPUT_QUEUE_NAME (optional, defaults to {queue_name}_output).
• File attachments:
  • SQLite/Redis: Large files stored on disk, small files inline
  • DocumentDB: Large files stored in GridFS (>1MB), small files inline (base64)
• Adapters must implement 9 methods (see docs/ADAPTER_RESEARCH_SUMMARY.md).
• Switching backends requires only env var changes—no code changes.

Adapter Comparison

Feature	SQLite	Redis	DocumentDB/MongoDB
Best For	Local development, single-worker	High-throughput, multi-worker	AWS-native, distributed processing
Scalability	Single process	Horizontal scaling	Horizontal scaling with replica sets
Persistence	File-based	In-memory (optional persistence)	Durable, replicated storage
File Storage	Filesystem	Filesystem	GridFS (integrated)
Cloud Integration	N/A	ElastiCache support	Native AWS DocumentDB
TLS/SSL	N/A	Supported	Required for AWS DocumentDB
Setup Complexity	Low	Medium	Medium-High
Dependencies	None (stdlib)	redis-py	pymongo

When to Use DocumentDB/MongoDB Adapter

• AWS Environments: Native integration with Amazon DocumentDB clusters
• Multi-Region Deployments: Replica set support for high availability
• Large File Handling: Built-in GridFS for efficient large file storage (>1MB)
• Enterprise Features: TLS/SSL encryption, connection pooling, and automatic failover
• MongoDB Compatibility: Drop-in replacement for existing MongoDB-based workflows

Output Queue Configuration

By default, adapters automatically append _output to the input queue name when creating output work items. For example:

• Input queue: qa_forms → Output queue: qa_forms_output

In multi-stage workflows, this can lead to confusing cascading names:

• Producer: qa_forms → qa_forms_output
• Consumer: qa_forms_output → qa_forms_output_output (confusing!)
• Reporter: qa_forms_output_output → qa_forms_output_output_output (even worse!)

Solution: Custom Output Queue Names

Use the RC_WORKITEM_OUTPUT_QUEUE_NAME environment variable to explicitly set the output queue name:

Example: Clean multi-stage workflow

// Producer
{
  "RC_WORKITEM_QUEUE_NAME": "qa_forms",
  // Output defaults to "qa_forms_output"
}

// Consumer
{
  "RC_WORKITEM_QUEUE_NAME": "qa_forms_output",
  "RC_WORKITEM_OUTPUT_QUEUE_NAME": "qa_forms_processed"  // Explicit name!
}

// Reporter
{
  "RC_WORKITEM_QUEUE_NAME": "qa_forms_processed"
  // No output needed for final stage
}

This prevents confusing cascading names and makes database management much clearer. The feature is backward compatible—if you don't set RC_WORKITEM_OUTPUT_QUEUE_NAME, the adapter will use the default {queue_name}_output behavior.

References & Documentation

• Adapter implementation: docs/CUSTOM_WORKITEM_ADAPTER_GUIDE.md
• Adapter interface: docs/ADAPTER_RESEARCH_SUMMARY.md
• Producer-consumer architecture: docs/# Producer-Consumer Architecture Migrati.md
• Task definitions: yamls/robot.yaml
• Environment setup: yamls/conda.yaml, devdata/

License

MIT (or project-specific license)

---

Tip: Always check the relevant YAML and devdata files for environment setup and test data before running tasks or debugging issues.