pynuttx 0.0.6a6

Comprehensive Python debugging and analysis toolkit for NuttX RTOS - includes GDB extensions, memory analysis, crash diagnostics, trace parsing, and remote debugging capabilities

PyNuttX - NuttX/Vela GDB Extension Toolkit

A comprehensive Python-based GDB debugging toolkit for NuttX RTOS, providing powerful extensions for thread analysis, memory debugging, crash diagnostics, filesystem inspection, and much more.

Production Ready: Extensively used in Xiaomi Vela OS for embedded systems debugging, crash analysis, and performance profiling.

---

📋 Table of Contents

• Introduction
• Installation & Setup
• Loading GDB Extensions
• Usage Scenarios
• Command Reference
• Quick Start Examples
• Advanced Features
• FAQ
• Contributing

---

🎯 Introduction

PyNuttX implements numerous custom GDB commands under nuttx/tools/pynuttx/, primarily focused on:

• Thread Management: Thread inspection, backtrace, context switching
• Memory Analysis: Memory dumps, leak detection, heap validation
• System Inspection: Process status, filesystem, network, sensors, and more

Key Features

✅ 100+ Custom GDB Commands covering all aspects of embedded debugging ✅ Post-Mortem Analysis via coredump and memory dump files ✅ Multi-Architecture Support: ARM, RISC-V, Xtensa, Tricore, and more ✅ Production Tested in commercial IoT products (Xiaomi Vela OS) ✅ Extensible Framework for custom command development

---

🚀 Installation & Setup

Prerequisites

GDB Version Requirement:

• GDB 11 or higher with python enabled is required

# Use prebuilt gdb-multiarch (recommended)
trunk/prebuilts/gcc/linux/gdb-multiarch/bin/gdb-multiarch

# Special architectures require specific GDB:
# - Tricore: prebuilts/gcc/linux/tricore/bin/tricore-gdb
# - Xtensa: xtensa-esp32s3-elf-gdb (xt-gdb does NOT support Python extensions)

⚠️ Note: Some vendor toolchains may have compilation issues causing missing libpython support, preventing Python module usage.

Install Dependencies

Install required Python packages:

# Install dependencies (Linux)
/usr/bin/python3 -m pip install -r nuttx/tools/pynuttx/requirements.txt

# Or via pip if pynuttx is installed
pip install pynuttx

Important Notes

1. Code-Tool Compatibility: ⚠️ Always match your code version with tool version

   • Don't use stable tools to analyze dev branch code
   • Keep tool and firmware in sync

2. Architecture-Specific Tools: Use the correct GDB for your target architecture

   • Tricore: tricore-gdb
   • Xtensa: xtensa-esp32s3-elf-gdb (see reference)
   • ARM: gdb-multiarch or arm-none-eabi-gdb

---

📥 Loading GDB Extensions

Method 1: Load via -ex Parameter (Recommended)

# Start GDB with automatic loading
gdb-multiarch nuttx/vela_ap.elf -ex "source nuttx/tools/pynuttx/gdbinit.py"

# Or load manually in GDB shell
(gdb) source nuttx/tools/pynuttx/gdbinit.py

Method 2: Import as Python Module

# Set PYTHONPATH so Python can find the module
export PYTHONPATH=/myworkspace/nuttx/tools/pynuttx:$PYTHONPATH

# Option 1: Load during GDB startup
gdb-multiarch nuttx/nuttx.elf -ex "py import nxgdb"

# Option 2: Import after GDB starts
(gdb) py import nxgdb

Verify Installation

# Check available custom commands
(gdb) help user-defined

# Get help for specific command
(gdb) memdump --help
(gdb) info nxthread --help

---

🎪 Usage Scenarios

PyNuttX works in multiple debugging contexts:

Scenario	Description	Use Case
Online Debugging	Live device via JTAG/SWD	JLink, OpenOCD, TRACE32
GDB Stub	Serial/USB debugging	Embedded GDB stub on target
Post-Mortem	Crash dump analysis	Load memory dumps via gdbserver.py
Coredump	ELF coredump files	Analyze with gdb <elf> <coredump>

Example: Connect to Live Target

# Via OpenOCD (JTAG/SWD)
openocd -f interface/jlink.cfg -f target/stm32f4x.cfg &
gdb-multiarch nuttx -ex "target remote :3333" -ex "source tools/pynuttx/gdbinit.py"

# Via Serial GDB Stub
gdb-multiarch nuttx -ex "target remote /dev/ttyUSB0" -ex "source tools/pynuttx/gdbinit.py"

Example: Post-Mortem Analysis

# Start gdbserver with memory dumps
python3 tools/pynuttx/gdbserver.py \
    -a arm \
    -e nuttx.elf \
    -r ram.bin:0x20000000 psram.bin:0x60000000 \
    -p 6667

# Connect and analyze
gdb-multiarch nuttx.elf -ex "target remote :6667" -ex "source tools/pynuttx/gdbinit.py"

---

📚 Command Reference

🧵 Thread & Scheduling

Command	Description	Documentation	Status
info nxthread	Display NuttX threads with Vela-specific data (replaces info threads for better NuttX integration)	thread	✅ Done
nxthread <id>	Switch to specific thread context	thread	✅ Done
setregs <addr>	Restore register context from saved state	-	✅ Done
deadlock	Detect deadlock conditions	-	✅ Done
critmon	Critical section monitor (same as device critmon)	shell	✅ Done
ps	Process status (same as device ps)	-	✅ Done
crash busyloop	Detect busy-loop threads	Crash Analysis	✅ Done

Example:

(gdb) info nxthread
  PID GROUP PRI POLICY   TYPE    NPX STATE   EVENT      SIGMASK  STACKBASE  STACKSIZE   USED  FILLED  COMMAND
    0     0   0 FIFO     Kthread -   Running            00000000 0x00000000         0      0    0.0%!  Idle_Task
    1     1 100 RR       Task    -   Waiting Semaphore  00000000 0x20600000      2048    824   40.2%   nsh_main

(gdb) nxthread 1
(gdb) bt

💾 Memory Management

Command	Description	Documentation	Status
memdump	Dump memory allocation info (same as device memdump)	memdump	✅ Done
memleak	Detect memory leaks by analyzing unreachable allocations	memleak	✅ Done
memcheck	Verify heap integrity and core data structures	memcheck	✅ Done
memfind <pattern>	Search for pattern in heap/bss/data (auto address range)	memfind	✅ Done
kasan	KASAN helper to check address tag accessibility	kasan	✅ Done
memclassify	Classify memory by backtrace to identify module usage	memclassify	✅ Done
mm visualize	Visualize memory state with size distribution charts	memmap & visualize	✅ Done
dump ram	Export memory data to file	dump ram	✅ Done
mm range	Show memory regions in current system	memrange	✅ Done
crash stackoverflow	Detect stack overflow conditions	Crash Analysis	✅ Done

Example:

# Show top 20 memory consumers without backtrace (faster)
(gdb) memdump --top 20 --no-backtrace

# Detect memory leaks
(gdb) memleak
Searching for leaked memory, please wait a moment...
Leak catch!, use '*' mark pid is not exist:
   CNT   PID        Size    Sequence    Address Callstack
    34    30         256        8810 0x3d52f080  [0x00c3210e0] <malloc+12>
...

# Check heap integrity
(gdb) mm check

# Visualize memory fragmentation
(gdb) mm visualize

🖥️ System & Diagnostics

Command	Description	Documentation	Status
foreach list	Traverse NuttX linked lists	list	✅ Done
tlsdump	Dump and verify TLS info / task info integrity	tlsdump	✅ Done
nxgcore	Pull device coredump (wraps GDB's gcore)	nxgcore	✅ Done
free	Show memory info (same as device free)	shell	✅ Done
uname	Display firmware version	-	✅ Done
irqinfo	Show IRQ statistics (same as device irqinfo)	-	✅ Done
dmesg	Display kernel log (ramlog)	-	✅ Done
resetcause	Show last reset reason	-	🚧 Not started
wdog	Show registered watchdog timer info	-	✅ Done
worker	Show registered worker info	-	✅ Done
noteram	View note trace data, perform notedump operations	noteram	✅ Done
notesnap	View task switching state before crash	notesnap	✅ Done
target nxstub	Connect to gdbserver via Vela proxy (native GDB thread support)	target nxstub	✅ Done
diagnose	Run all diagnostic commands, generate JSON report	diagnose	✅ Done
watchdog	View/control watchdog status during debugging	-	🚧 Ongoing
circbuf	View circular buffer state (libc/misc/lib_circbuf.c)	-	✅ Done
pmconfig	View PM configuration info	pmconfig	✅ Done
elfimport	Import ELF symbols	elfimport	✅ Done
crash blame	Analyze crash probability by module (for initial JIRA assignment)	Crash Analysis	🚧 Ongoing
crash thread	Identify which thread caused the crash	Crash Analysis	✅ Done
crash memory	Determine if crash was caused by memory issues	Crash Analysis	🚧 Ongoing

Example:

# View kernel log
(gdb) dmesg

# Generate comprehensive diagnostic report
(gdb) diagnose

# Check system info
(gdb) uname
(gdb) free

🗂️ Filesystem

Command	Description	Documentation	Status
fdinfo	Show open file descriptors per task	fdinfo	✅ Done
mount	Display mount information	mount	✅ Done
foreach inode	Traverse inode tree	foreach inode	✅ Done
info shm	Show shared memory usage	info shm	✅ Done
info yaffs	View YAFFS partition context	info yaffs	✅ Done
info romfs	View ROMFS partition context	info romfs	✅ Done
info fatfs	View FAT filesystem partition context	info fatfs	✅ Done
info lrofs	View LROFS partition context	info lrofs	✅ Done
unqlite tools	View unqlite memory usage	-	📅 Q1

Example:

# Check open files
(gdb) fdinfo

# View mount points
(gdb) mount

# Inspect filesystem partition
(gdb) info fatfs /dev/mmcsd0

🌐 Network & Connectivity

Command	Description	Documentation	Status
netstats	Print IOB and protocol/socket info	net	✅ Done
netcheck	Run network diagnostics	net	✅ Done
btsocket	Print Bluetooth IPC server/client cache queue	btdiag	✅ Done
btdev	Print Bluetooth adapter BLE/BREDR device info	btdiag	✅ Done
bttimeval	Print Bluetooth thread peak timestamp	btdiag	✅ Done
btstack	Print protocol stack list (e.g., GATT pending list)	btdiag	✅ Done
btsnoop	Convert Bluetooth /dev/tty circular buffer to HCI log	btdiag	✅ Done
lyrainfo	Display Lyra Lite connection, mesh, and transmission info	lyrainfo	✅ Done

Example:

# Network statistics
(gdb) netstats

# Bluetooth diagnostics
(gdb) btdev
(gdb) btsnoop /dev/ttyBT0 output.hci

🎨 Graphics (LVGL)

Command	Description	Documentation	Status
imagecache	View image cache memory with allocation backtrace	-	🚧 Not started
info draw_unit	View current draw unit info, verify rendering context	lvgl	✅ Done
info style	View LVGL object style and type information	lvgl	✅ Done
dump obj	Display LVGL object tree	lvgl	✅ Done
show fb	Display framebuffer content	-	🚧 Not started

📡 RPC & IPC

Command	Description	Documentation	Status
rpmsgservice	View RPMSG service information	rpmsgdump	✅ Done
rptun	View RPMSG virtio shared memory buffer	-	🚧 Ongoing
rpmsgport	View RPMSG port memory and debug info	-	📅 Mar 31
rpmsg trace	RPMSG trace enhancement with visualization	-	📅 Mar 31

📊 Sensors

Command	Description	Documentation	Status
uorb	View /dev/uorb/ node subscriptions and publications	uorb	✅ Done

🎬 Media

Command	Description	Documentation	Status
mediadump	View pipeline filter EOF state and internal dump data	mediadump	✅ Done

🪟 XMS (Window Manager)

Command	Description	Documentation	Status
wm dump	Dump application window state and shared memory	-	🚧 Ongoing

📱 Application Frameworks

Command	Description	Documentation	Status
dump aiotmemory	Print Quick App JS memory usage	dump aiotmemory	✅ Done

🔧 Development Tools

Command	Description	Documentation	Status
profile	Analyze Python function call time (performance profiling)	profile	✅ Done
viztracer	Capture Python trace data, view with Perfetto	-	✅ Done
time	Measure Python tool execution time (like shell time)	-	✅ Done
addr2line	Convert addresses/variables/expressions to backtrace	utils	✅ Done
hexdump	Dump memory at address with printable characters	-	✅ Done
debugpy	Debug Python code in IDE	debugpy	✅ Done

---

🚀 Quick Start Examples

Example 1: Memory Leak Detection

# Connect to target or load memory dump
gdb-multiarch nuttx.elf -ex "target remote :3333" -ex "source tools/pynuttx/gdbinit.py"

# Run memory leak detection
(gdb) memleak
Searching for leaked memory, please wait a moment
Searching global symbol in: /path/to/nuttx.elf
Search all memory use 28.98 seconds

Leak catch!, use '*' mark pid is not exist:
   CNT   PID        Size    Sequence    Address Callstack
    34    30         256        8810 0x3d52f080
        [0x00c3210e0] <malloc+12>
        [0x00c47c904] <lv_malloc+6>
        ...

Alloc 44 count, have 8 some backtrace leak, total leak memory is 10112 bytes

Example 2: Crash Analysis

# After a crash, connect to device
gdb-multiarch nuttx.elf -ex "target remote /dev/ttyUSB0" -ex "source tools/pynuttx/gdbinit.py"

# View kernel log
(gdb) dmesg
[  123.456] Assertion failed at file:line

# Check thread status
(gdb) info nxthread

# Identify crashing thread
(gdb) crash thread

# Run comprehensive diagnostics
(gdb) diagnose

# Check for memory corruption
(gdb) memcheck

# Detect stack overflow
(gdb) crash stackoverflow

Example 3: Post-Mortem Analysis

# 1. Collect memory dumps from crashed device
(gdb) dump memory ram.bin 0x20000000 0x20100000
(gdb) dump memory psram.bin 0x60000000 0x64000000

# 2. Start gdbserver with memory files
python3 tools/pynuttx/gdbserver.py \
    -a arm \
    -e nuttx.elf \
    -r ram.bin:0x20000000 psram.bin:0x60000000 \
    -p 6667

# 3. Connect and analyze
gdb-multiarch nuttx.elf \
    -ex "target remote :6667" \
    -ex "source tools/pynuttx/gdbinit.py"

# 4. Analyze the crash
(gdb) info nxthread
(gdb) bt
(gdb) memdump --top 20

Example 4: Filesystem Debugging

# Check open file descriptors
(gdb) fdinfo

# View mount points
(gdb) mount

# Inspect filesystem details
(gdb) info fatfs /dev/mmcsd0

# Traverse inode tree
(gdb) foreach inode

Example 5: Network Debugging

# View network statistics
(gdb) netstats

# Run network diagnostics
(gdb) netcheck

# Check Bluetooth status
(gdb) btdev
(gdb) btsocket

---

🎓 Advanced Features

Save Command Output to File

# Use pipe to save output
(gdb) pipe info nxthread | tee output.log
(gdb) pipe memdump | tee memdump.txt

View All Custom Commands

# List all user-defined commands
(gdb) help user-defined

# Get help for specific command
(gdb) <command> --help

GDB Proxy for Native Thread Support

# Connect via nxstub for better thread awareness
(gdb) target nxstub --proxy :1234 # For example, connect to qemu
(gdb) info threads  # Now shows NuttX threads correctly

Comprehensive Diagnostics

# Run all diagnostic commands and generate JSON report
(gdb) diagnose

# This automatically runs:
# - Thread analysis
# - Memory checks
# - System state inspection
# - Network diagnostics
# - And more...

---

❓ FAQ

Q: How do I add my own custom commands?

A: Follow the GDB Python API to create custom commands.

Example:

import gdb

class MyCommand(gdb.Command):
    """My custom command description"""

    def __init__(self):
        super().__init__("mycommand", gdb.COMMAND_USER)

    def invoke(self, arg, from_tty):
        print("Hello from my command!")

MyCommand()

Save as mycommand.py and load:

(gdb) source mycommand.py
(gdb) mycommand

Q: Can thread functionality display in VSCode or other frontends?

A: Not yet perfectly. Current implementation replaces same-named commands which doesn't perfectly adapt to frontend tools. Frontend adaptation is under development.

Q: GDB version error or Python module not found?

A:

1. Ensure GDB 11+ is installed
2. Check GDB Python support: gdb --batch -ex "python import sys; print(sys.version)"
3. Use prebuilt GDB from Vela repository
4. Install dependencies: pip install -r requirements.txt

Q: Commands are slow on embedded target?

A: Use post-mortem analysis instead:

1. Dump memory from target
2. Use gdbserver.py to load dumps
3. Analyze offline (much faster!)

Q: How to debug multi-core systems?

A:

1. Dump memory from each core separately
2. Load each core's memory with corresponding ELF file
3. Use separate GDB sessions or load all in one gdbserver instance

Q: Code and tool version mismatch?

A: ⚠️ Always keep tool and firmware versions synchronized!

• Use same branch for both code and tools
• Don't mix stable branch tools with dev branch code

---

🤝 Contributing

Contributions are welcome! PyNuttX is part of the Apache NuttX project.

Development Setup

# Clone repository
git clone https://github.com/apache/nuttx.git
cd nuttx/tools/pynuttx

# Install in development mode
pip install -e .

# Install development dependencies
pip install -r requirements.txt

Adding New Commands

1. Create your command in appropriate module under nxgdb/
2. Add documentation in docs/
3. Add tests in tests/
4. Submit pull request

See Contributing Guide for details.

---

📖 Resources

• Documentation: See docs/ directory for detailed command documentation
• NuttX Website: https://nuttx.apache.org/
• Issue Tracker: https://github.com/apache/nuttx/issues
• Mailing List: https://nuttx.apache.org/community/

---

📄 License

Licensed under the Apache License, Version 2.0. See LICENSE for details.

---

🙏 Acknowledgments

Maintained by the Apache NuttX community and extensively developed by the Xiaomi Vela team.

Special Thanks To:

• All contributors and maintainers
• Xiaomi Vela OS team for production testing and enhancements
• The embedded systems community

---

For support and questions, please refer to the NuttX Documentation or contact the community via mailing lists.