pytest-embedded-arduino-cli

日本語版 README

A pytest plugin to test Arduino projects using pytest-embedded and arduino-cli.

Overview

pytest-embedded-arduino-cli is a small plugin that keeps pytest-embedded's generic DUT / serial / expect flow and replaces Arduino-specific build and upload with arduino-cli.

This package does not depend on pytest-embedded-arduino. It is intended to stay generic enough to work well for Arduino projects beyond ESP32-specific assumptions.

Guides

• Your First Test: create a first test with one physical board and understand each file, pytest pass/fail, startup waits, and timeout design.
• Testing Basics: what a test is, how a run works, what only real hardware can tell you, how many boards to use, and how to keep tests independent. Start here if you are new to pytest.
• Advanced Testing: collection rules, traps in expect, where logs and artifacts live, cleaning up after an early exit, when a conftest.py is justified, and principles for peer tests.
• Local Examples: choose a runnable example in this repository by purpose and required hardware.
• Real Projects: external projects that use the plugin, from the smallest host-core setup to a large multi-board one.
• FAQ: a symptom-first index into the two guides above, for when something has already gone wrong.

Design

• Build with arduino-cli compile
• Upload with arduino-cli upload
• Use pytest-embedded as the runtime foundation
• Avoid EspSerial and ESP-specific flashing services
• Resolve sketch settings from sketch.yaml and --profile
• Treat the test file directory as the sketch directory

Quick Start

For an Arduino project, use tests/ as the workspace to keep the Python environment separate from the project root.

mkdir tests
cd tests
uv init
uv add pytest-embedded-arduino-cli
arduino-cli sketch new hello
cd hello
touch sketch.yaml test_hello.py

# After editing sketch.yaml, hello.ino, and test_hello.py
cd ..
uv run pytest hello --profile=<PROFILE> --port=<PORT>

Your First Test provides the file contents and explains every part.

Runtime dependencies include:

• pytest
• pytest-embedded
• pytest-embedded-serial
• PyYAML

Requirements

• arduino-cli available in PATH
• Every platform in sketch.yaml pins a core version that Arduino CLI can resolve from its indexes; the workflow must not depend on an unversioned core preinstalled in the environment
• A serial port accessible from the host when running hardware tests

When sketch.yaml declares platform or library versions, Arduino CLI resolves them through its local package and library indexes. Without a platform version, Arduino CLI uses a preinstalled core, so the build version cannot be identified from the configuration and an environment without that core fails. Always pin the version. Add platform_index_url to the profile when the platform needs an additional Board Manager URL. The indexes do not need to be refreshed on every test run, but they should be updated periodically or before CI/release verification. If a build fails because a declared platform or library version cannot be found, try:

arduino-cli core update-index
arduino-cli lib update-index

Project Layout

The expected layout is one sketch directory per test app.

tests/
  my_app/
    sketch.yaml
    my_app.ino
    test_my_app.py

When pytest runs a specific .py file, this plugin treats that file's directory as the sketch directory. Build settings are resolved from the nearest sketch.yaml.

Usage

Build, upload, and run tests:

uv run pytest tests/my_app --port=/dev/ttyACM0

Select an Arduino CLI profile from sketch.yaml:

uv run pytest tests/my_app --profile esp32s3 --port=/dev/ttyACM0

Build only:

uv run pytest tests/my_app --run-mode=build

Force a clean Arduino CLI compile:

uv run pytest tests/my_app --clean

Upload and test against an already-built image:

uv run pytest tests/my_app --run-mode=test --port=/dev/ttyACM0

--run-mode=test skips compile, reuses the existing build output, uploads it, and then runs the test.

Run this package's own tests:

uv run pytest

Main Options

• --run-mode=all|build|test
• --profile
• --peer-profile=NAME:PROFILE
• --peer-port=NAME:PORT
• --clean
• --device-lock=auto|off|required
• --device-lock-timeout=SECONDS
• --device-lock-dir=PATH
• --device-lock-key=KEY
• --save-state
• --save-state-dir=PATH
• --arduino-cli-no-log-summary
• --arduino-test-timeout=SECONDS
• --arduino-test-artifact-dir=PATH
• --arduino-test-missing-config=skip|error

--clean passes --clean to arduino-cli compile. It is useful when Arduino CLI's incremental build cache should be ignored. It also removes the ArduTest artifact directory before running. It does not touch Arduino CLI's shared caches; see Build Output and Cache Cleanup (Linux).

--save-state enables local test verification state caching to state.json for development convenience. By default, this is disabled. When enabled, test results are recorded per profile in .pytest-results/state.json (or a custom directory specified with --save-state-dir). This feature is useful during development to track which tests are passing/failing without relying on external CI systems.

Example:

uv run pytest tests/my_app --profile esp32 --port=/dev/ttyACM0 --save-state

State file structure (.pytest-results/state.json):

{
  "schema_version": 1,
  "updated_at": "2026-05-11T12:00:00.123456+09:00",
  "profiles": {
    "esp32": {
      "tests": {
        "tests/my_app/test_my_app.py::test_something": {
          "last_result": "passed",
          "last_run_at": "2026-05-11T12:00:00.123456+09:00",
          "last_success_at": "2026-05-11T12:00:00.123456+09:00"
        }
      }
    }
  }
}

For peer tests (multi-DUT), only the primary DUT state is recorded.

--save-state-dir specifies the directory for state.json storage. The default is .pytest-results (relative to pytest rootdir unless absolute).

Build Output and Cache Cleanup (Linux)

--clean only covers the build path of the run it is given to. Arduino CLI's shared caches under ~/.cache/arduino and ~/.arduino15 are never touched by this plugin, by pytest, or by --clean. Cleaning those is outside the scope of a pytest plugin, so the commands below are provided as a reference for writing your own make clean target or maintenance script.

Paths below are the Linux defaults. Confirm the actual values with:

arduino-cli config get directories

What lives where

The plugin's own outputs are separate and all live under the pytest rootdir: the ArduTest artifact directory (ardutest/ by default, already removed by --clean), .pytest-results/ from --save-state, and the pytest-embedded log directory under $TMPDIR.

Case 1: a stale cache is suspected

Work outwards from the narrowest scope; stop as soon as the problem goes away.

# 1. This run only. Rebuilds the build path, keeps the shared caches.
uv run pytest tests/my_app --clean

# 2. Every build directory in the repository.
#    The build path is always <sketch_dir>/build/<profile>, so locating sketch.yaml
#    finds them exactly, without catching Python's own build/ directory.
find . -name sketch.yaml -not -path './.venv/*' -printf '%h/build\n' | xargs -r rm -rf

# 3. The shared compile caches as well. Everything here is regenerated, nothing is re-downloaded.
rm -rf ~/.cache/arduino/cores ~/.cache/arduino/sketches

To review the list before deleting, run the find without the xargs pipe first.

Case 2: reclaiming disk space

Check the actual sizes first. The order of magnitude differs greatly between these directories:

du -sh ~/.arduino15/internal ~/.arduino15/staging ~/.arduino15/packages ~/.cache/arduino/*

# Download cache. Pure cache, nothing needs reinstalling afterwards.
arduino-cli cache clean          # equivalent to: rm -rf ~/.arduino15/staging

# Build caches.
rm -rf ~/.cache/arduino/cores ~/.cache/arduino/sketches

# Profile-scoped installs. Usually by far the largest directory.
rm -rf ~/.arduino15/internal

~/.arduino15/internal deserves particular attention in a profile-based workflow like this plugin's. Each distinct set of sketch.yaml profile dependencies is installed in isolation under its own hashed directory, so every core version, every toolchain version and every library version you have ever referenced from a profile is kept side by side. A single ESP32 toolchain is roughly 2 GB, and the directory reaching tens of GB is normal. Deleting it is safe, but the next build of each profile re-downloads what it needs, so do it when you can afford the download time.

Note that arduino-cli cache clean only clears the download staging directory. It does not touch the build caches under ~/.cache/arduino, which is why they are listed separately above.

Log Directory Summary

pytest-embedded writes serial logs to <tmpdir>/pytest-embedded/<UTC timestamp>/<test name>/dut.log. This plugin adds result files to the same directory so the outcome is visible from tree or ls alone:

/tmp/pytest-embedded/2026-07-30_08-03-16-959795/
├── ERROR-1_FAILED-1_PASSED-1_SKIPPED-1   # empty file; the name carries the counts
├── SUMMARY.txt                           # whole run, human readable
├── summary.json                          # same data, machine readable
├── test_diff/
│   ├── PASSED.txt                         # test id, timings, profile, log file names
│   └── dut.log
└── test_memory/
    ├── FAILED.txt                         # the above plus the failure text
    └── dut.log

Each test directory holds exactly one of PASSED.txt, FAILED.txt, ERROR.txt (setup or teardown failure, which is where build and upload errors land), SKIPPED.txt, or XFAILED.txt/XPASSED.txt. Failure text is truncated to the last 32 KB; dut.log in the same directory always has the full serial log. Nothing is written when no DUT was created, because pytest-embedded does not create a log directory in that case.

Use --arduino-cli-no-log-summary to disable these files.

--arduino-test-artifact-dir selects the ArduTest artifact root. The default is ardutest, resolved relative to pytest's rootdir. The directory is created only when an artifact is saved.

--arduino-test-missing-config controls required ArduTest config that is not provided. The default is skip; use error when missing config should fail the pytest run.

--device-lock controls cross-process exclusion for physical DUTs. The default is auto. In auto mode, this plugin lets compile finish first, then waits for a device lock immediately before upload when a physical serial port such as /dev/ttyUSB0, /dev/ttyACM0, or COM3 is resolved. The lock is held until the DUT is no longer in use, so another pytest process cannot upload new firmware while a test is still running on that board.

Device locks are keyed by the resolved physical serial port, not by profile. Port paths are resolved through symlinks before hashing, so /dev/serial/by-id/... and /dev/serial/by-path/... aliases lock the same device as the underlying /dev/ttyUSB0 node. The primary DUT lock is acquired before the primary upload and held until module teardown. Peer DUTs use their own resolved ports and, when the peers fixture is requested, their locks are acquired in a deterministic order before peer upload and held until module teardown. socket://... targets are not locked by default because they usually represent host-process or TCP/IP DUTs. Use --device-lock=off to disable locking, --device-lock=required to fail when no lock key can be resolved, and --device-lock-key=KEY for unusual environments where the port string is not a stable physical-device identity. The lock files live in a user runtime/cache directory by default and use OS file locking, so a leftover lock file after a forced termination does not by itself keep the device locked. Default lock directories are:

• Windows: %LOCALAPPDATA%\pytest-embedded-arduino-cli\locks (or %APPDATA%, then ~/AppData/Local, as fallbacks)
• Linux/macOS with XDG_RUNTIME_DIR: $XDG_RUNTIME_DIR/pytest-embedded-arduino-cli/locks
• Linux/macOS with XDG_CACHE_HOME: $XDG_CACHE_HOME/pytest-embedded-arduino-cli/locks
• Linux/macOS fallback: ~/.cache/pytest-embedded-arduino-cli/locks

Use --device-lock-dir=PATH to override this location.

Use pytest-embedded standard options for runtime control, such as:

• --port
• --flash-port
• --baud
• --embedded-services

pytest-embedded-serial is installed as a normal dependency so hardware tests can use the serial service without extra package installation. If --embedded-services is not specified, this plugin enables serial by default.

When upload and runtime serial communication use the same port, pass only --port. On a board whose bootloader and running sketch use different ports, pass the upload destination as --flash-port and the runtime connection as --port.

The port passed to arduino-cli upload is resolved in this order:

1. --flash-port
2. --port
3. TEST_SERIAL_PORT_<PROFILE>
4. TEST_SERIAL_PORT
5. profiles.<PROFILE>.port in sketch.yaml, only when it is a socket://... URL

When both --flash-port and --port are present, the priority above selects --flash-port for upload while preserving --port for the pytest-embedded serial connection.

Because of how pytest parses arguments, options that take path-like values such as --port and --flash-port are safer when written with =, for example --port=/dev/ttyUSB0. Depending on the environment, uv run pytest --port /dev/ttyUSB0 may cause that path to be interpreted as another base path. If needed, uv run pytest --rootdir . --port /dev/ttyUSB0 is also a valid workaround.

For targets that run on the host machine and expose the DUT over TCP/IP, use the URL format supported by pytest-embedded-serial / pyserial. If the selected sketch.yaml profile defines port: socket://localhost, --port=socket://localhost can be omitted.

uv run pytest tests/my_app --profile host

When the port number is specified, such as socket://localhost:56789, the DUT connects to that socket directly. When the port number is omitted, such as socket://localhost, the plugin is expected to read port from *.host-arduino.json generated under the build output directory and then connect to socket://localhost:<port>. This resolution should prefer the host-arduino information file instead of capturing upload stdout.

{
  "pid": 21228,
  "port": 56789
}

Host execution is an early, lightweight test path for pure logic and serial protocol checks without physical hardware. Results may differ depending on the host OS, gcc or other toolchain versions, and the Serial class implementation provided by the host Arduino core, so this does not guarantee behavior on real hardware. Use real hardware for peripherals, timing, interrupts, memory layout, Flash/NVS, and board-specific APIs. Build success can also differ by board core and platform, so running build tests with the production board profile is still recommended. For socket://... ports, this plugin batches serial reads to avoid the very slow one-byte-at-a-time redirect behavior that can otherwise appear with host Arduino cores.

Some platforms deliver the runtime console through their own pluggable monitor instead of a plain serial port. For those, the plugin reads the runtime port through arduino-cli monitor -m <profile> -l <protocol> --quiet instead of opening it with pyserial. The choice is automatic: the plugin checks arduino-cli compile --show-properties for a pluggable_monitor.pattern.<protocol> of the platform's own (builtin:serial-monitor does not count). The protocol is serial for a plain path, and the scheme for a port such as wchlink://FBC18F0680B0 reported by the platform's discovery; socket:// stays with pyserial. Tests keep using dut and peers unchanged. Monitor settings come from the board defaults and the profile's port_config in sketch.yaml, not from --baud. Boards that use the built-in serial monitor, such as ESP32 and AVR, are opened with pyserial as before. See SPEC.md 13.9 for details.

The arduino_cli_build_properties fixture returns the same expanded build properties as a dict, for conftests and plugins that need values such as runtime.tools.<tool>.path.

Example:

export TEST_SERIAL_PORT_ESP32S3=/dev/ttyUSB1
uv run pytest tests/my_app --profile esp32s3

Profile resolution works as follows:

1. If --profile is specified, that profile is used
2. Otherwise, if sketch.yaml defines default_profile, that profile is used
3. Otherwise, if sketch.yaml has exactly one profile, it is selected automatically
4. Otherwise, pytest exits with an error because the profile is ambiguous

In practice, explicitly specifying --profile is recommended. If you do not want to pass --profile, define default_profile in sketch.yaml. The single-profile auto-selection is supported as a fallback, but it is better not to rely on it for regular project configuration.

Peer DUTs

Tests that need additional DUTs can place peer_<name>/ sketch directories next to the primary sketch. The primary sketch remains available as dut; peer sketches are available through the peers fixture.

tests/
  my_app/
    sketch.yaml
    my_app.ino
    test_my_app.py
    peer_echo/
      sketch.yaml
      peer_echo.ino

def test_with_peer(dut, peers):
    echo = peers["echo"]

    dut.expect_exact("MAIN_READY")
    echo.expect_exact("ECHO_READY")

Peer DUTs are uploaded and connected only for tests that request the peers fixture. If a peer_* directory exists, its sketch is compiled before primary upload so all compile work finishes before any upload begins. Requesting peers enables upload / connect for all detected peer DUTs; peers["<name>"] is the mapping API for accessing the connected peer.

Build and upload order is fixed:

1. the primary DUT is built first
2. peer DUTs are built in peer name order
3. the primary DUT is uploaded

After that, dut and peers are set up in the order the test function lists them:

• dut connects to the primary DUT
• peers uploads peer DUTs in peer name order, then connects them

With def test_x(dut, peers), the primary is connected before the peers are uploaded. With def test_x(peers, dut), the peers are uploaded and connected first.

On real serial hardware, short boot-time messages can be missed if a sketch prints them immediately after reset or upload. Host Arduino core socket runs often keep enough output for this not to matter, but hardware tests should use a startup delay, repeated READY message, or an explicit handshake from Python before relying on early output.

sketch.yaml is not extended for peer configuration. Each peer directory is a normal sketch directory with its own .ino and sketch.yaml.

Peer profiles are resolved in this order:

1. --peer-profile <name>:<profile>
2. default_profile in peer_<name>/sketch.yaml
3. Skip the peer test if no profile is resolved

--profile is for the primary DUT only and is not inherited by peer DUTs. Peer DUTs also do not use single-profile auto-selection. If a peer should run without --peer-profile, define default_profile in that peer's sketch.yaml.

Peer ports are resolved in this order:

1. --peer-port <name>:<port>
2. TEST_SERIAL_PORT_PEER_<NAME>_<PROFILE>
3. TEST_SERIAL_PORT_PEER_<NAME>
4. profiles.<PROFILE>.port in the peer sketch.yaml, only when it is a socket://... URL
5. Skip the peer test if no port is resolved

--peer-profile and --peer-port may be specified multiple times. Use one option per peer instead of comma-separated values.

uv run pytest tests/my_app \
  --profile esp32 \
  --peer-profile echo:host \
  --peer-port echo:socket://localhost

For compile-time defines, place a build_config.toml in the sketch directory:

[defines]
TEST_WIFI_SSID = "WIFI_SSID"
TEST_WIFI_PASSWORD = "WIFI_PASSWORD"

[flags]
PYTEST_BUILD = true
ENABLE_TEST_HOOKS = true

In [defines], the left side is the environment variable name and the right side is the C/C++ define name. For example, TEST_WIFI_SSID becomes -DWIFI_SSID="..." at compile time. [flags] is for value-less defines. Only true entries are passed, such as -DPYTEST_BUILD; false entries are omitted.

Set values before running pytest:

export TEST_WIFI_SSID=my-ssid
export TEST_WIFI_PASSWORD=my-password
uv run pytest tests/my_app --port=/dev/ttyACM0

You can also load these values from a dotenv file through uv run. --env-file is a uv option, so put it before pytest:

uv run --env-file .env pytest tests/my_app --port=/dev/ttyACM0

If an environment variable is missing, the plugin still passes the define with an empty string value. This allows the test or sketch code to decide how to handle missing settings. The plugin does not add test flags such as PYTEST_BUILD automatically. Projects that need them should declare them explicitly under [flags].

The plugin injects these defines/flags through arduino-cli compile --build-property <property>=... and auto-selects an empty property so board-defined flags are not overwritten. It prefers build.extra_flags, then build.defines; if ESP32 board options occupy both, it falls back to empty C/C++ compile flag properties such as compiler.cpp.extra_flags and compiler.c.extra_flags. You can pin the property explicitly to skip the detection probe (about one second faster):

build_property = "build.defines"        # all profiles

[profiles.esp32]
build_property = "build.defines"        # per-profile override

For command visibility, follow pytest's standard verbosity:

• -v shows the arduino-cli compile / arduino-cli upload command line
• -vv also shows execution context such as cwd, sketch_dir, build_path, profile, and port

Capturing the full serial log

dut.expect(...) stops reading as soon as the pattern matches, so bytes the device sends after the matched line are not guaranteed to reach the dut.log file — it can end mid-line. This is a capture-timing effect, not a device fault: the log file and the live -s console are fed from the same stream, and the tail in flight can be lost when the DUT is closed.

The plugin already drains the receive buffer on a best-effort basis when the serial connection closes, which usually brings dut.log up to roughly what -s shows. It is not a full guarantee. When you need the complete trailing output deterministically:

• Have the device print an end-of-output marker as its last line and expect it, so reading naturally extends to the true end:

  dut.expect_exact("=== END ===")
  

• Or drain explicitly before the test finishes:

  import pexpect
  
  dut.expect_exact("last line I care about")
  dut.expect(pexpect.TIMEOUT, timeout=2)  # read whatever still arrives
  

Note that arduino_test.run() stops reading at the ArduTest RESULT event, so any LOG / tick lines the device emits after RESULT are not collected unless you drain afterwards.

ArduTest Fixture

This package includes an experimental arduino_test fixture for sketches that use the separate Arduino-side ArduTest library. ArduTest is expected to be declared by the sketch's sketch.yaml, with the library version pinned there for reproducible tests. Detailed usage examples will be added under examples/ after the API and protocol settle.

def test_board(arduino_test):
    arduino_test.run()

arduino_test.run() fails the pytest test automatically when ArduTest reports a failed or error result. Use additional assertions only when you want to check collected logs, metrics, artifacts, or metadata.

The current fixture speaks ArduTest protocol version 1.

Use fixture methods for fixed test-local ArduTest values:

def test_sample_rate(arduino_test):
    arduino_test.set_capability("measurement.current")
    arduino_test.set_config("sample_rate", 1000)
    arduino_test.run("test_sample_rate")

Use environment variables, .env, or CI variables for values that depend on the machine, board, lab setup, or secrets.

Example

Your First Test creates the minimal sketch, sketch.yaml, and pytest file in your own project. This repository includes the same handshake in examples/01_basic:

uv run pytest examples/01_basic --profile=esp32 --port=/dev/ttyUSB0

See examples/README.md for the purpose and hardware requirements of every example. Main samples:

• examples/01_basic
  • Minimal serial round trip with a handshake that tolerates startup timing
  • Uses esp32 as the default profile and also supports uno
  • Includes port resolution from TEST_SERIAL_PORT and TEST_SERIAL_PORT_<PROFILE>
• examples/02_env_define
  • Demonstrates compile-time defines from environment variables
  • Uses Wi-Fi on ESP32-class targets to explain build_config.toml
• examples/03_dut_input
  • Demonstrates runtime input over serial through dut.write(...)
  • Works on both esp32 and uno
• examples/04_unity_basic
  • Demonstrates a minimal Unity-based test sketch for ESP32
• examples/05_nvs_persistent
  • Demonstrates that ESP32 Preferences / NVS data remains by default
  • Unsupported profiles are skipped before build because the example is specifically about ESP32 persistence
• examples/06_erase_flash
  • Demonstrates EraseFlash=all for resetting ESP32 persistent data before upload
  • Pairs with 05_nvs_persistent
• examples/07_arduino_library_project
  • Demonstrates a practical Arduino library project with tests/ as the uv root
  • Includes run_wsl.sh as a practical test workspace example
• examples/08_arduino_ide_project
  • Demonstrates an Arduino IDE style sketch project with tests/ as the uv root
  • Uses thin wrapper #include files so the runner can reference sketch-side code that is not separated as a library
• examples/09_host_arduino_core
  • Demonstrates a board core that builds and runs the Arduino sketch on the host machine
  • Uses port: socket://localhost in sketch.yaml to connect to the TCP/IP endpoint opened by the host executable
  • Useful for simple pure-logic and serial-protocol checks, not a replacement for real hardware tests or build tests with the real board profile
• examples/10_build_flags
  • Demonstrates value-less compile-time defines with [flags] in build_config.toml
  • Shows how a project can explicitly enable test flags such as PYTEST_BUILD
• examples/11_ardutest
  • Minimal examples for using the experimental arduino_test fixture with the ArduTest Arduino library
  • Detailed protocol/API and artifact-saving tests live in the ArduTest test suite: https://github.com/tanakamasayuki/ArduTest/tree/main/tests

Execution guidance for examples/ is described in examples/README.md.

Warnings

You may see PytestExperimentalApiWarning: record_xml_attribute is an experimental feature.

This warning comes from pytest-embedded, not from this plugin. It is usually safe to ignore. If you want to suppress it in your project, add a warning filter in pytest.ini, pyproject.toml, or a local config such as examples/pytest.ini.

What This Plugin Does Not Try To Be

• A drop-in replacement for pytest-embedded-arduino
• An ESP-specific flashing layer
• A board auto-discovery tool

Future Extensions

• Board-family-specific upload strategies
• Smarter artifact discovery
• Serial reset / monitor helpers
• TCP/IP connection helpers for host Arduino cores
• Device farm scheduling beyond same-device exclusion
• Optional fqbn or sketch path overrides

Release

This repository uses GitHub Actions for releases.

Before triggering a release:

• Update the ## Unreleased section in CHANGELOG.md
• Make sure uv run pytest tests passes locally if needed

Release flow:

1. Open GitHub Actions
2. Run the Release workflow manually
3. Enter the release version such as 0.1.0
4. Choose whether to publish to PyPI

The workflow will:

• Update versions in pyproject.toml and src/pytest_embedded_arduino_cli/__init__.py
• Move CHANGELOG.md unreleased entries into ## <version>
• Run tests and build the package
• Commit the release changes and create tag v<version>
• Create a GitHub Release
• Publish to PyPI when enabled

PyPI publishing is configured for Trusted Publishing via GitHub Actions.