PyneCore - Pine Script to Python Transpiler

Backlog & known bugs: Projects board #10 — open work is in Backlog (prioritised P0/P1/P2), shipped work in Done. Confirmed unfixed transpiler bugs live there too; several are pinned by strict xfail tests (see Running Tests).

Installation (New Computer)

Fastest: the sandbox-setup skill

From the repo root, run the bundled setup script (idempotent — safe to re-run):

bash .claude/skills/sandbox-setup/setup.sh

It builds .venv, installs everything below, detects PINE_CPU_COUNT / PINE_MAX_WORKERS, seeds a demo dataset, and runs a smoke test that ends in smoke test : PASS / READY. In Claude Code you can also invoke it as /sandbox-setup.

Manual (equivalent steps)

# From the repo root:
python3 -m venv .venv                          # 1. virtualenv
.venv/bin/pip install -e .                      # 2. this transpiler (editable)
.venv/bin/pip install 'opencode-pyneruntime[cli]'  # 3. PyneCore runtime + `pyne` CLI
.venv/bin/pip install optuna numpy              # 4. optimizer deps (not pulled in by the runtime)
source .venv/bin/activate                       # 5. activate (Linux/macOS)
# .venv\Scripts\activate                        #    (Windows)

Verify the install:

.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"

Note: All commands below assume you are at the repo root with the venv activated.

How to Convert Pine Script to Python

Single File Conversion

python -m pine2pyne path/to/script.pine -o path/to/output.py

Batch Conversion

python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/

Running Tests

There are three independent test systems. They answer different questions, so a green run of one says nothing about the others.

System	Question it answers	Runtime
1. Unit tests (tests/, pytest)	Are the transpiler's internals correct?	~5 s
2. Sample suites (tools/test_all_*.py)	Does every corpus sample transpile and run?	~35 s
3. Ground truth (tools/*_ground_truth*)	Does it produce the same numbers as before?	minutes

1. Unit tests — pytest

2,000+ tests over pine2pyne/ (93% line / 86% branch coverage).

.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py -q

# with coverage
.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py \
    --cov=pine2pyne --cov-report=term-missing

--ignore is required: tests/test_screener_regression.py imports a screener module from a sibling repo that is absent here, and fails at collection — which aborts the whole run. Tracked as a P0 card on board #10.

Some tests are @pytest.mark.xfail(strict=True) and pin confirmed unfixed bugs, each with a card on board #10. Strict means fixing a bug turns its marker into a failure until the marker is removed, so nothing regresses silently in either direction.

The highest-priority one is P0: codegen emits arithmetically wrong Python — 10 - (5 - 3) transpiles to 10 - 5 - 3, which evaluates to 2 rather than 8. See docs/test_plan.md for the full bug inventory and the measured coverage/mutation results.

2. Sample suites — end-to-end transpile + run

Transpiles every .pine in the corpus and runs it through pyne.

python tools/test_all_samples.py          # sample/pinescript  (~411 files)
python tools/test_all_sample_pds.py       # sample_pds         (~328 files)

python tools/test_all_samples.py "ex_001*"      # filter by pattern
python tools/test_all_samples.py --timeout 30   # default 15s
python tools/test_all_samples.py --verbose      # stderr on failure
python tools/test_all_samples.py -j 0           # all CPU cores

• Source: sample/pinescript/, sample_pds/ · Output: workdir/scripts/
• Data: workdir/data/demo.ohlcv · Results: test_results{,_pds}.{json,txt}

Per-sample expectations live in tools/sample_expectations.toml:

[expds_156_Out_of_bounds_index]
expect_error = "index_error"                    # MUST raise; running clean = FAIL

[ex_300_ticker_modify]
requires_feeds = { D = "demo", "2D" = "demo_2D" }   # -> --security D=demo …

expect_error marks Pine documentation samples that exist to demonstrate an error — they are reported as XFAIL and count as passes. requires_feeds supplies extra OHLCV feeds for samples calling request.security() at a timeframe other than the chart's; pynecore deliberately will not resample the chart feed, so the feed must be explicit. Generate one with:

cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D

Shared logic for both suites lives in tools/_suite_common.py — edit it there, not in each harness.

3. Ground truth — numeric regression

Diffs freshly generated output CSVs against ~1,985 stored baselines in ground_truth/ (779 MB). This is the only system that catches silently wrong numbers — output that runs fine but computes something different.

python tools/generate_ground_truth.py     # (re)build baselines
python tools/compare_ground_truth.py      # diff current output vs baselines
python tools/compare_ground_truth.py --suite sample_pds -j 12
./tools/verify_ground_truth.sh            # end-to-end wrapper

Regenerate baselines only when you have confirmed the new output is correct — otherwise a bug gets frozen in as the expected result.

Test data

workdir/data/ is gitignored, so a fresh clone has none and the sample suites abort with Data file not found. Restore the exact dataset the recorded results were measured against:

git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv
git show 227f3d2^:workdir/data/demo.toml  > workdir/data/demo.toml

(CCXT BTCUSDT 1D. /sandbox-setup instead seeds a small synthetic fixture — fine for a smoke test, but results will not match the recorded rates.)

For Claude Code / LLM Development

See CLAUDE.md in this directory for transpiler architecture, output format specs, optimizer documentation, and CSV rounding rules. See pine2pyne/README.md for transpiler internals and transformation rules.

Optimizing Strategies

Quick start: pyne optimize script.py data.ohlcv params.json -n 20

See Optimizer.md for full documentation (parameter JSON format, parallel execution, output files).

Distributed Optimization (multi-machine)

Two tools distribute pyne optimize across SSH clusters:

Tool	Model	Best for
pyne-dynamic.sh	Flat-queue (on-demand dispatch)	Multi-variant runs, heterogeneous clusters, long jobs
pyne-parallel.sh	Static pre-assignment	Quick jobs on similar-speed machines

# Dynamic (recommended): flat-queue, pre-syncs once, auto-adapts to machine speed
# Run from workdir/ directory:
../tools/pyne-dynamic.sh scripts/strategy.py data/data.ohlcv optimize_variants/ \
  -H ../tools/machines.txt -C 24 --name my_run --output-dir runs/output/

# Static: pre-assigns chunks by core count
./tools/pyne-parallel.sh scripts/strategy.py data/data.ohlcv optimize.json \
  -H tools/machines.txt --sync

# Check progress / collect results
../tools/pyne-dynamic.sh --status
../tools/pyne-dynamic.sh --collect

Both use the same machines.txt format. See CLAUDE.md for cluster setup, machine file format, and troubleshooting.