Rhylthyme Specification

Standards and schema for a real-time program logistics description markup language.

Version Information

• Current Version: 0.1.0
• Program Schema Version: 0.3.0-alpha (0.2.0-alpha still shipped; see CHANGELOG)

Overview

Rhylthyme is a JSON-based markup language for describing real-time programs that coordinate multiple parallel tracks of work with resource constraints, timing dependencies, and flexible execution patterns. It's designed for scenarios like restaurant kitchens, laboratory workflows, manufacturing processes, and any situation requiring coordinated real-time execution.

Features

• Basic program structure with tracks and steps
• Simplified "on" trigger syntax
• Resource constraints
• Variable and fixed duration types
• Batch processing with staggering
• Environment integration
• Manual triggers and indefinite durations
• Step replicates with per-instance triggers (instances: "each" | "all" | "any") and in-flight limits (replicates.maxInFlight), schema 0.3.0-alpha
• Step alerts (alerts): reminders anchored to a step's start or end, package 0.2.3-alpha, both program schemas
• Video records (run records timed in a video's media timeline) and video suggestions (proposed step times for a person to confirm)

Schema Structure

The Rhylthyme schema defines programs as JSON objects with the following key components:

Program-Level Properties

• programId (required): Unique identifier for the program
• name (required): Human-readable name
• description: Detailed description
• version: Program version number (should match schema version)
• environmentType: Type of environment (e.g., 'kitchen', 'laboratory')
• actors: Number of available workers/operators
• tracks (required): Array of parallel execution tracks
• resourceConstraints (required): Resource usage limits
• startTrigger: How the program begins execution

Example: Restaurant Breakfast Service

{
  "programId": "restaurant-breakfast",
  "name": "Restaurant Breakfast Service",
  "description": "Professional restaurant breakfast service workflow with coordinated cooking and plating",
  "version": "0.1.0",
  "environmentType": "kitchen",
  "actors": 2,
  "startTrigger": {
    "type": "manual"
  },
  "tracks": [
    {
      "trackId": "scrambled-eggs",
      "name": "Scrambled Eggs",
      "description": "Multiple orders of scrambled eggs",
      "batch_size": 3,
      "steps": [
        {
          "stepId": "eggs-crack-whisk",
          "name": "Crack and Whisk Eggs",
          "description": "Crack eggs into a bowl and whisk with salt and pepper",
          "task": "prep-work",
          "trigger": {
            "type": "programStart"
          },
          "duration": {
            "type": "fixed",
            "seconds": 60
          }
        },
        {
          "stepId": "eggs-heat-pan",
          "name": "Heat Pan",
          "description": "Place pan on stove and heat to medium",
          "trigger": {
            "on": "eggs-crack-whisk"
          },
          "duration": {
            "type": "variable",
            "minSeconds": 60,
            "maxSeconds": 120,
            "defaultSeconds": 90,
            "triggerName": "pan-ready"
          },
          "task": "stove-burner"
        }
      ]
    }
  ],
  "resourceConstraints": [
    {
      "task": "stove-burner",
      "maxConcurrent": 4,
      "description": "Maximum number of stove burners that can be used simultaneously"
    }
  ]
}

Track Structure

Tracks represent parallel workflows within a program. Each track can have multiple steps and can be executed multiple times (batch processing).

Track Properties

• trackId (required): Unique identifier
• name (required): Human-readable name
• description: Detailed description
• batch_size: Number of iterations (default: 1)
• stagger: Time delay between batch instances
• priority: Execution priority (lower = higher priority)
• steps (required): Array of sequential steps

Example: Bacon Cooking Track

{
  "trackId": "bacon",
  "name": "Bacon",
  "description": "Multiple orders of bacon",
  "batch_size": 2,
  "steps": [
    {
      "stepId": "bacon-prep",
      "name": "Prepare Bacon",
      "description": "Place bacon strips in cold pan",
      "task": "prep-work",
      "trigger": {
        "type": "programStart"
      },
      "duration": {
        "type": "fixed",
        "seconds": 60
      }
    },
    {
      "stepId": "bacon-cook",
      "name": "Cook Bacon",
      "description": "Cook bacon, flipping occasionally until crispy",
      "trigger": {
        "on": "bacon-prep"
      },
      "duration": {
        "type": "variable",
        "minSeconds": 480,
        "maxSeconds": 720,
        "defaultSeconds": 600,
        "triggerName": "bacon-done"
      },
      "task": "stove-burner"
    }
  ]
}

Step Structure

Steps are the individual work units within a track. They define what work is done, how long it takes, and when it starts.

Step Properties

• stepId (required): Unique identifier
• name (required): Human-readable name
• description: Detailed description
• task: Resource/tool required for this step
• trigger (required): When this step begins
• duration (required): How long the step takes
• resources: Array of required resources
• flex: Whether this step can expand to fill available time
• alerts: Reminders shown to the person running the program (see Step alerts)

Trigger Types

1. Program Start

{
  "type": "programStart"
}

Step begins immediately when the program starts.

2. On Another Step (Simplified Syntax)

{
  "on": "eggs-crack-whisk"
}

Step begins when the specified step completes.

3. On Step with Offset

{
  "on": "eggs-crack-whisk",
  "offsetSeconds": 30
}

Step begins 30 seconds after the specified step completes.

4. On Step Start Event

{
  "on": "eggs-crack-whisk",
  "event": "start"
}

Step begins when the specified step starts (not when it completes).

5. Manual Trigger

{
  "type": "manual",
  "triggerName": "start-cooking"
}

Step begins when manually triggered.

Duration Types

1. Fixed Duration

{
  "type": "fixed",
  "seconds": 60
}

Step takes exactly 60 seconds.

2. Variable Duration

{
  "type": "variable",
  "minSeconds": 480,
  "maxSeconds": 720,
  "defaultSeconds": 600,
  "triggerName": "bacon-done"
}

Step duration varies based on conditions, with manual completion trigger.

3. Time String Format

{
  "type": "fixed",
  "timeString": "2m30s"
}

Duration specified as a human-readable time string.

Resource Constraints

Resource constraints limit how many instances of a task can run simultaneously.

{
  "resourceConstraints": [
    {
      "task": "stove-burner",
      "maxConcurrent": 4,
      "description": "Maximum number of stove burners that can be used simultaneously"
    },
    {
      "task": "toaster",
      "maxConcurrent": 2,
      "description": "Maximum number of toasters that can be used simultaneously"
    }
  ]
}

Advanced Features

Batch Processing with Staggering

{
  "trackId": "toast",
  "name": "Toast",
  "description": "Multiple orders of toast",
  "batch_size": 4,
  "stagger": "30s",
  "steps": [
    {
      "stepId": "toast-cook",
      "name": "Make Toast",
      "description": "Place bread in toaster and toast until golden brown",
      "trigger": {
        "type": "programStartOffset",
        "offsetSeconds": 120
      },
      "duration": {
        "type": "fixed",
        "seconds": 210
      },
      "task": "toaster"
    }
  ]
}

This creates 4 toast orders, each starting 30 seconds after the previous one.

Flexible Steps

{
  "stepId": "wait-for-eggs",
  "name": "Wait for Eggs",
  "description": "Flexible waiting period while eggs cook",
  "task": "waiting",
  "trigger": {
    "on": "eggs-start-cooking"
  },
  "duration": {
    "type": "variable",
    "minSeconds": 0,
    "maxSeconds": 300,
    "defaultSeconds": 180
  },
  "flex": true
}

Flexible steps can expand or contract to fill available time gaps.

Environment Integration

Programs can reference environment definitions that specify available resources and their capabilities.

{
  "programId": "restaurant-breakfast",
  "name": "Restaurant Breakfast Service",
  "version": "0.1.0",
  "environmentType": "kitchen",
  "environment": "restaurant-standard.json"
}

Validation

The schema enforces:

• Required fields are present
• Step dependencies are valid
• Resource constraints are reasonable
• Duration values are positive
• Track and step IDs are unique within their scope

Usage Examples

Simple Sequential Workflow

{
  "programId": "simple-cooking",
  "name": "Simple Cooking",
  "version": "0.1.0",
  "tracks": [
    {
      "trackId": "main",
      "name": "Main Track",
      "steps": [
        {
          "stepId": "prep",
          "name": "Preparation",
          "task": "prep-work",
          "trigger": {"type": "programStart"},
          "duration": {"type": "fixed", "seconds": 60}
        },
        {
          "stepId": "cook",
          "name": "Cooking",
          "task": "stove-burner",
          "trigger": {"on": "prep"},
          "duration": {"type": "fixed", "seconds": 300}
        }
      ]
    }
  ],
  "resourceConstraints": [
    {"task": "stove-burner", "maxConcurrent": 2}
  ]
}

Complex Parallel Workflow

{
  "programId": "breakfast-service",
  "name": "Breakfast Service",
  "version": "0.1.0",
  "actors": 3,
  "tracks": [
    {
      "trackId": "eggs",
      "name": "Eggs Station",
      "batch_size": 5,
      "stagger": "45s",
      "steps": [
        {
          "stepId": "eggs-prep",
          "name": "Prepare Eggs",
          "task": "prep-work",
          "trigger": {"type": "programStart"},
          "duration": {"type": "fixed", "seconds": 30}
        },
        {
          "stepId": "eggs-cook",
          "name": "Cook Eggs",
          "task": "stove-burner",
          "trigger": {"on": "eggs-prep"},
          "duration": {"type": "variable", "minSeconds": 120, "maxSeconds": 180, "defaultSeconds": 150}
        }
      ]
    },
    {
      "trackId": "bacon",
      "name": "Bacon Station",
      "batch_size": 3,
      "stagger": "60s",
      "steps": [
        {
          "stepId": "bacon-cook",
          "name": "Cook Bacon",
          "task": "stove-burner",
          "trigger": {"type": "programStartOffset", "offsetSeconds": 30},
          "duration": {"type": "fixed", "seconds": 600}
        }
      ]
    }
  ],
  "resourceConstraints": [
    {"task": "stove-burner", "maxConcurrent": 4},
    {"task": "prep-work", "maxConcurrent": 2}
  ]
}

Advanced Trigger Examples

Cross-Track Dependencies

{
  "stepId": "plate-breakfast",
  "name": "Plate Breakfast",
  "description": "Plate the complete breakfast when all components are ready",
  "task": "plating",
  "trigger": {
    "on": "eggs-cook",
    "on": "bacon-cook",
    "on": "toast-cook"
  },
  "duration": {"type": "fixed", "seconds": 30}
}

Offset from Step Completion

{
  "stepId": "serve-breakfast",
  "name": "Serve Breakfast",
  "description": "Serve the plated breakfast after a brief rest period",
  "task": "service",
  "trigger": {
    "on": "plate-breakfast",
    "offsetSeconds": 15
  },
  "duration": {"type": "fixed", "seconds": 20}
}

Step Start Event

{
  "stepId": "preheat-oven",
  "name": "Preheat Oven",
  "description": "Start preheating when cooking begins",
  "task": "oven",
  "trigger": {
    "on": "eggs-cook",
    "event": "start"
  },
  "duration": {"type": "fixed", "seconds": 300}
}

Schema Files

• schemas/program_schema_0.3.0-alpha.json - Program schema (current; adds instances on step-referencing triggers and maxInFlight on step-level replicates)
• schemas/program_schema_0.2.0-alpha.json - Program schema (previous; still accepted)
• schemas/environment_schema_0.1.0-alpha.json - Environment schema
• schemas/runs_schema_0.1.0-alpha.json - Run record schema (execution history; get_runs_schema_path())
• schemas/video_suggestions_schema_0.1.0-alpha.json - Suggested step times in a video (get_video_suggestions_schema_path(); see Video suggestions)

From Python:

from rhylthyme_spec import get_program_schema_path, PROGRAM_SCHEMA_VERSIONS

get_program_schema_path()               # 0.2.0-alpha (default, unchanged)
get_program_schema_path("0.3.0-alpha")  # current
PROGRAM_SCHEMA_VERSIONS                 # ("0.2.0-alpha", "0.3.0-alpha")

Per-instance triggers (0.3.0-alpha)

When a trigger references a replicated step it may say how it fans in:

{"type": "afterStep", "stepId": "bake", "instances": "each"}

"all" (default) waits for every instance — the barrier that 0.2.0 programs already get implicitly; "each" replicates the referencing step once per instance and pairs instance i with instance i, transitively; "any" starts after the first instance ends. Each value carries an OWL-Time $comment in the schema. Expansion rewrites all three into 0.2.0 constructs, so 0.2.0 programs validate unchanged and resolve to identical times.

In-flight limits (0.3.0-alpha)

Step-level replicates may cap how many instances are between the step and its rejoin point:

"replicates": {"count": 3, "mode": "serial", "maxInFlight": 2}

An instance is in flight from its own start until it has ended in every instances: "each" descendant. maxInFlight: k holds instance i + k until instance i leaves flight, so three trays through one oven onto a rack that holds two never strand a hot tray. It is not resourceConstraints[].maxConcurrent: that bounds one task at one instant, while maxInFlight bounds a chain of tasks across time. Expansion rewrites it into an ordinary compound{all} trigger tagged _synthetic: "inFlight".

Step alerts (0.2.3-alpha)

A step may carry alerts, each anchored to the step's start or end and shifted by a signed offsetSeconds (a number or a time string such as "-2m"; default 0):

"alerts": [
  {"event": "end", "offsetSeconds": "-2m", "message": "Get the tray ready"},
  {"event": "end", "level": "alarm"}
]

message (1-200 characters) is optional; runners show a default such as "Ends in 2 min". level is notice (default) or alarm, which native apps may ring as a device alarm. In OWL-Time terms an alert is a time:Instant at the time:hasBeginning or time:hasEnd of the step's time:ProperInterval, shifted by a time:Duration. Alerts never reference another step, and replicate expansion copies them onto every instance. The amendment is additive to both 0.2.0-alpha and 0.3.0-alpha: every previously valid program stays valid.

Some alerts cannot be placed in advance and never fire; the validators report W_ALERT_BEFORE_MANUAL_START, W_ALERT_BEFORE_INDEFINITE_END, W_ALERT_BEFORE_PROGRAM_START, and the error E_ALERT_BAD_OFFSET for an unreadable offset. User documentation: Alerts and Notifications.

Run Records

A run record is written by a runtime after a program has been executed (schemas/runs_schema_0.1.0-alpha.json, get_runs_schema_path()). It is a separate document from the program so history never rewrites the plan: runId, programId, programVersion (sha256: of the program's canonical JSON), runtime {kind, version, clockMode, speed}, environmentId, startedAt, endedAt, outcome (completed | aborted | abandoned), context (program metadata plus userTags), and one steps[] entry per expanded step with planned {start, end, durationType, ...} frozen at run start, actual {start, end}, endedBy (executor | timer | trigger | abort | instrument | skipped | annotation | observer), triggerFiredAt, waitedOn, pausedSeconds and optional notes. All step times are seconds from startedAt, except in media records.

Video records

A run record can also describe a video: runtime.kind is video (made by a dataset converter or the marking screen) and runtime.clockMode is media. Every time is then seconds of the video's own media timeline, and program time 0 is the video's first frame. startedAt is when recording began if known, otherwise when the timeline was made. A step that never happened has no actual; outcome: completed means the video shows the procedure to its end. Steps timed from a dataset's labels end with endedBy: annotation, and the dataset's error tags go in notes; steps a person marked while watching, including confirming a suggestion, end with endedBy: observer. Duration predictions and calibration read only wall records, so media records stay out of them.

{
  "runId": "2026-11-05T00:00:00Z-1c7e",
  "programId": "video-recipe",
  "programVersion": "sha256:…",
  "runtime": {"kind": "video", "version": "0.1.0", "clockMode": "media", "speed": 1},
  "startedAt": "2026-11-05T00:00:00Z",
  "outcome": "completed",
  "steps": [
    {"stepId": "n1", "planned": {"start": 0, "end": 60, "durationType": "fixed", "seconds": 60},
     "actual": {"start": 3.37, "end": 71.04}, "endedBy": "annotation"},
    {"stepId": "n2", "planned": {"start": 60, "end": 90, "durationType": "fixed", "seconds": 30}}
  ]
}

Video suggestions

A video suggestions document (schemas/video_suggestions_schema_0.1.0-alpha.json, get_video_suggestions_schema_path()) holds step times in one video proposed by a model or a baseline (a proposer) for a person to confirm. It is never shown on a public page; confirmed times become a video record.

{
  "schemaVersion": "0.1.0-alpha",
  "programVersion": "sha256:…",
  "video": {"sha256": "…", "durationSeconds": 1234.5},
  "proposer": {"name": "timer-reanchored", "version": "0.1.0", "settings": {}},
  "createdAt": "2026-11-05T00:00:00Z",
  "steps": [
    {"stepId": "n3", "present": true, "start": 12.0, "end": 80.5, "confidence": 0.4},
    {"stepId": "n4", "present": false}
  ]
}

• programVersion, video, proposer (name, version), createdAt and steps are required; schemaVersion, proposer.settings (free-form) and confidence are optional.
• programVersion is sha256: and 64 lowercase hex digits, as in a run record. video.sha256 is the bare 64-digit lowercase hex SHA-256 of the video file, with no prefix.
• start and end are seconds of the video's media timeline (0 is its first frame). They are required when present is true and forbidden when it is false (the proposer thinks the step did not happen). The schema does not compare start with end or with durationSeconds; consumers check those.
• confidence is a number from 0 to 1.
• A step left out has no suggestion; a stepId may appear more than once when the proposer thinks the step happened more than once. Repeats are occurrences ordered by start: for a step with replicates they fill its instances in order, and occurrences beyond the step's instance count (one for a step without replicates) are extra intervals.

Tests

python -m pytest -q tests/

Contributing

We welcome feedback and contributions. Please:

1. Test thoroughly with your use cases
2. Report issues with detailed examples
3. Suggest improvements with concrete proposals

License

Apache License 2.0