cist-kernel 0.0.68

CLI for cist kernel telemetry and compilation

CIST Monorepo

Polyglot monorepo for the Contract Instruction Set (.cis) language.

This repository ships three user-facing pieces together:

• cist-kernel Python package — the CLI, the compiler, and the cloud compilation client.
• A VS Code extension bundled inside the package — cist init installs it into your local VS Code extensions folder. Recognises both .cis source files and .mex compiled binaries.
• A build orchestrator (scripts/build.py) for cutting releases.

Language Snapshot

use math
use compute
use risk

market_data := from data (source: "binance", symbols: ["BTC-USD"], resolution: "1m")

sma_20 := math.sma(market_data.close, 20)
signal := compute.crossover(market_data.price, sma_20)

if signal and position.size == 0:
    execute(market_buy(size: risk.calculate_size(portfolio_value: 10000)))
elif position.pnl < -100:
    execute(close_position())

Repository Layout

cist/
├── pyproject.toml
├── cist.toml                   ← workspace config (created by cist init)
├── README.md
├── .gitignore                  ← *.mex and .env excluded by default
├── scripts/
│   └── build.py
├── cist_cli/
│   ├── __init__.py
│   ├── main.py
│   ├── network.py
│   ├── compiler/               ← NEW: compilation pipeline
│   │   ├── __init__.py
│   │   ├── lexer.py            ← CISTCompiler: syntax + semantic validation → IR
│   │   ├── bundle.py           ← compile bundle + harness generation
│   │   └── mex.py              ← MexWriter / MexReader: .mex binary format
│   ├── cloud/
│   │   ├── service.py          ← local /compile + /telemetry service
│   │   └── modal_stub.py       ← optional Modal deployment entrypoint
│   ├── workspace/              ← NEW: workspace scaffold
│   │   ├── __init__.py
│   │   ├── init.py             ← init_workspace(): creates directory structure
│   │   └── templates.py        ← all generated-file templates
│   └── assets/
│       └── *.vsix              ← bundled VS Code extension
├── tests/
│   ├── test_compiler.py
│   ├── test_mex.py
│   ├── test_workspace.py
│   ├── test_entrypoint.py
│   ├── test_build_script.py
│   └── test_vscode_grammar.py
└── extensions/
    ├── package.json            ← declares .cis and .mex language contributions
    ├── language-configuration.json
    └── syntaxes/
        └── cis.tmLanguage.json

Each contract lives in its own sub-directory under contracts/:

contracts/
└── {contract_name}/
    ├── {contract_name}.cis     ← main source file
    ├── program.mex             ← compiled model-executable bundle
    └── bench/
        ├── example.cis         ← test/bench scenarios (extend freely)
        └── manifest.md         ← safe-data instructions for the bench harness

Getting Started

1. Install or upgrade

python -m pip install --upgrade cist-kernel

The --upgrade flag matters. A plain pip install cist-kernel keeps an existing older version and will not pull in new features.

If cist is not on your PATH after install, use the fallback:

python -m cist_cli

On Windows, the installer will print the exact PowerShell one-liner to add the scripts folder to your PATH permanently the first time you run via python -m cist_cli.

2. Initialise a workspace (if you haven't already)

cist init              # in the current directory
cist init --path ~/my-workspace

cist init creates the following structure from scratch (safe to re-run — existing files are never overwritten):

<workspace>/
├── cist.toml                    project name, kernel version, compiler settings
├── .env                         CIST_CLOUD_API_KEY placeholder
├── .gitignore                   *.mex and .env pre-seeded
└── contracts/
    └── example/
        ├── example.cis          starter contract (edit freely)
        └── bench/
            ├── example.cis      bench / test scenarios (extend freely)
            └── manifest.md      safe-data instructions for the bench harness

It also silently extracts the bundled VS Code extension into every detected editor extensions folder (.vscode, .vscode-insiders, .antigravity), removing any stale legacy versions. Reload the window once with Ctrl+Shift+P → Developer: Reload Window to activate syntax highlighting for .cis and .mex files.

3. Add your API key

Open the .env file and fill in your key:

CIST_CLOUD_API_KEY=sk_live_your_key_here

4. Write your contract

Edit contracts/example/example.cis or create a new sub-directory:

use math
use hyper

contract OilPriceMonitor:
    which defines:
        while True:
            oil_price = hyper.get_price("OIL")

            then:
                if oil_price > 80.0:
                    print("Consider selling.")

Add test scenarios in contracts/example/bench/:

contract BenchHighPrice:
    which defines:
        oil_price = 90.0
        then:
            if oil_price > 80.0:
                print("bench: high price path OK")

5. Compile

cist compile contracts/example/example.cis

If you only want local validation plus a local .mex artifact, use the build command instead of the cloud stream:

cist build contracts/example/example.cis

What happens:

1. Syntax check — the compiler validates brackets, strings, use imports, from data(…) expressions, selective imports, block markers, indentation, and scans contract and write fn declarations.
2. Bundle emit — if the file is valid, program.mex is written next to the source file. This binary now stores the model-facing compile bundle: source, IR, bench manifests, harness instructions, and the generated task input.
3. Cloud stream — the .mex bundle is streamed to the CIST Cloud compiler. The cloud worker materialises or refines Rust contracts, runs the Composer bundler, and streams logs, artifacts, and errors back in real time.

✓ example.cis  42 lines · modules: math, hyper · 0 data stream(s)
✓ Bundle written → contracts/example/program.mex
Connecting to CIST Cloud...

Add --deploy to mark the build for live execution:

cist compile contracts/example/example.cis --deploy

Add --verbose to see the full module and data-stream breakdown:

cist compile contracts/example/example.cis --verbose

---

CLI Reference

cist init [--path <dir>]

Initialises a CIST workspace.

• Creates contracts/{name}/{name}.cis and contracts/{name}/bench/example.cis.
• Creates contracts/{name}/bench/manifest.md so the safe layer has explicit data-generation instructions.
• Writes cist.toml (project name, kernel version, compiler settings).
• Writes .env stub with CIST_CLOUD_API_KEY placeholder.
• Seeds .gitignore with *.mex and .env (appends if the file already exists; never duplicates).
• Removes stale legacy extension versions and silently extracts the bundled .cis + .mex VS Code extension.

cist compile <file.cis> [--deploy] [--verbose]

Compiles a .cis contract.

• Step 1 — local validation: bracket matching, quote closure, use imports, from data(…) expressions, mixed tab/space indentation, UTF-8 encoding.
• Step 2 — bundle emit: writes a program.mex binary alongside the source. The binary is zlib-compressed, integrity-checked with CRC-32, and contains the model-executable compile bundle.
• Step 3 — cloud stream: streams the compile bundle to the CIST Cloud. Logs, generated file artifacts, bench output, and errors are printed in real time.
• --deploy flags the build as live.
• --verbose prints the full module list and data-stream count.

cist build <file.cis> [--debug]

Builds the local .mex artifact without calling the cloud compiler.

• Runs the same local validation path as cist compile.
• Writes program.mex next to the source file.
• --debug writes the .mex payload as readable JSON instead of the compressed binary container.

cist check <file.cis>

Runs the validator only and prints any syntax or structural diagnostics without writing a .mex file.

cist serve [--host <addr>] [--port <port>]

Runs the local cloud compiler service with two routes:

• /compile — accepts a compile bundle and streams SSE logs
• /telemetry — returns the latest compile or rig state as JSON

Install the optional cloud extra first if you want to run this locally:

python -m pip install "cist-kernel[cloud]"

cist telemetry [--session-id <id>]

Fetches the latest compiler telemetry from the configured cloud endpoint.

cist sync-vscode-extension

Re-installs or updates the bundled VS Code extension without running a full init.

---

The .mex Binary Format

When cist compile succeeds it writes a program.mex file alongside the source. The format is:

Offset	Size	Content
0	4 B	Magic: MEX\x00
4	2 B	Format version (big-endian uint16, currently 1)
6	4 B	CRC-32 of the compressed payload (big-endian uint32)
10	N B	zlib-compressed UTF-8 JSON payload

The payload holds the model-executable compile bundle: source text, compiler IR, bench and manifest documents, nursery schema hints, harness instructions, and the model task input.

.mex files are recognised by the VS Code extension (file-tree icon, language ID mex) but are intentionally non-editable binary blobs. They are excluded from version control via the workspace .gitignore.

---

Configuration

cist.toml

Created by cist init at the workspace root. Controls project metadata and compiler behaviour:

[project]
name = "example"
version = "0.1.0"
description = ""

[kernel]
version = "0.0.38"   # kernel version this workspace was initialised with

[compiler]
strict = false        # set true to treat warnings as errors
debug = false         # set true to write program.mex as readable JSON during development

[cloud]
backend = "auto"     # auto | nineth | template
model = "1984-m3-0404"

[runtime]
composer_root = "composer"
telemetry_poll_seconds = 5

Environment Variables

Loaded from .env (if present) and from the shell environment.

Variable	Required	Default	Purpose
CIST_CLOUD_API_KEY	Yes (for compile)	—	Bearer token for the cloud compiler
CIST_CLOUD_ENDPOINT	No	https://api.cist.io/v1	Override the cloud endpoint
CIST_CLOUD_MODEL	No	1984-m3-0404	Model name used by the Nineth-backed compiler worker
CIST_CLOUD_BACKEND	No	auto	auto, nineth, or deterministic template fallback
CIST_COMPOSER_ROOT	No	—	Override the Composer runtime root on the cloud worker
CIST_ENABLE_LIVE_RUN	No	0	Set to 1 to let deploy builds continue from bench into composer live
COMPOSER_EXECUTE_LIVE_TRADES	No	0	Manual composer live runs only submit venue orders when this is 1; the cloud service injects it for live deploy subprocesses
CIST_SKIP_VSCODE_INSTALL	No	—	Set to 1 to skip extension install (useful in CI)
VSCODE_EXTENSIONS	No	—	Override the VS Code extensions root directory

---

Validation Artifacts

This revision includes permanent run logs under logs/2026-04-11:

• logs/2026-04-11/python-pytest.log — 46 passed
• logs/2026-04-11/composer-lifecycle.log — 22 passed, 0 failed, 3 ignored
• logs/2026-04-11/composer-bench.log
• logs/2026-04-11/composer-live.log
• logs/2026-04-11/cist-init.log
• logs/2026-04-11/cist-build.log

For the fuller architecture walkthroughs, see report.md and composer/report.md.

---

Build Artifacts

Build both deliverables with:

python scripts/build.py

Expected outputs:

• dist/cist_kernel-*.whl — Python wheel containing the CLI and the bundled .vsix.

The build script verifies that the VSIX version inside the wheel matches the project version in pyproject.toml before declaring success.

---

Notes

• Local validation is lightweight and non-executing. It never runs contract code.
• Cloud compilation requires CIST_CLOUD_API_KEY unless you are targeting a local anonymous endpoint such as http://127.0.0.1:8000.
• The VS Code grammar uses Python-style scope categories with CIST-specific handling for use, contract, write fn, which defines:, then:, and from data(…).
• The cist_cli/compiler/ and cist_cli/workspace/ packages are independently importable for use in tooling or test fixtures.