pylow 0.1.0

One CLI. Zero code changes. Full system flow visibility for any Python service or distributed system.

pytrace

One CLI. Zero code changes. Full system flow visibility for any Python service or distributed system.

How it works under the hood

Three layers working together:

• Layer 1 → OTel auto-instrumentation (HTTP, DB, gRPC, queues — zero code changes)
• Layer 2 → bpftrace USDT (Python function call tree, syscalls)
• Layer 3 → pytrace CLI (stitches both, renders the flow)

---

Complete CLI Command Reference & Outputs

Here is the usage documentation and sample outputs for every command in the pytrace tool:

1. pytrace attach <pid>

Attach to any running Python process and start collecting trace logs immediately.

pytrace attach 4821

Output:

✓ Attached to process 4821. Monitoring execution events...

2. pytrace flow

Renders the complete execution flow tree (local/distributed spans & events) from the trace repository.

pytrace flow --last

Output:

handle_request 450ms
├── authenticate_user 20ms
│   └── [redis GET session_id] 15ms
└── call_llm_chain 410ms
    └── [POST api.openai.com/v1/chat/completions] 400ms
        └── waiting (epoll_wait) 390ms   ← bottleneck

3. pytrace stitch

Stitch distributed traces together across service boundaries using traceparent headers.

pytrace stitch --services api,worker,ml-service

Output:

REQUEST trace-id: t_demo_flow_123

  api-gateway          450ms  handle_request
  └── api-gateway          20ms  authenticate_user
      └── api-gateway          15ms  redis GET session_id
  └── api-gateway          410ms  call_llm_chain
      └── api-gateway          400ms  POST api.openai.com/v1/chat/completions
          └── api-gateway          390ms  waiting (epoll_wait)

4. pytrace slow

Continuously daemonize/monitor and surface slow execution paths exceeding a latency threshold.

pytrace slow --threshold 200ms --watch

Output:

Continuous monitoring daemon started. Threshold: 200ms, Watch: False
SLOW PATHS detected (last 5 min):

  #1  handle_request → call_llm_chain → POST api.openai.com/v1/chat/completions → [waiting (epoll_wait)]
      avg: 390ms  occurrences: 1
      root cause: epoll_wait 310ms — network latency to openai

5. pytrace diff

Compare execution flow metrics between versions or releases to detect regressions.

pytrace diff --before deploy-v1.2 --after deploy-v1.3

Output:

Comparing before v1.2 vs after v1.3...

REGRESSIONS:

  handle_request     +150ms avg  (was 200ms, now 350ms)
  call_llm           +140ms avg  (was 180ms, now 320ms)
  serialize          +12ms avg  (was 3ms, now 15ms)

NEW CALLS in v1.3:
  validate_schema    8ms  (added input validation)

REMOVED in v1.3:
  legacy_cache_check (removed)

6. pytrace syscall <pid>

Trace syscall counts and histogram latency patterns for the target process.

pytrace syscall 4821

Output:

Attaching syscall counter to PID 4821...
✓ Attached. Monitoring syscall events... Ctrl+C to stop.

--- BPF Map: @sys_counts ---
  sys_enter_read: 231
  sys_enter_write: 184
  sys_enter_epoll_wait: 42

7. pytrace malloc <pid>

Profile allocations and heap sizing metrics.

pytrace malloc 4821

Output:

Attaching allocator profile to PID 4821...
✓ Attached. Monitoring memory allocations... Ctrl+C to stop.

--- BPF Map: @alloc_sizes (bytes allocated) ---
[64, 127]              45 |@@@@@@@@@@@                         |
[512, 1023]           120 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[4096, 8191]           18 |@@@@                                |

8. pytrace tcp <pid>

Trace outbound TCP latency.

pytrace tcp 4821

Output:

Attaching TCP latency tracer to PID 4821...
✓ Attached. Monitoring TCP sendmsg... Ctrl+C to stop.

--- BPF Map: @tcp_send_us (ns delay) ---
[100000, 200000]       21 |@@@@@@@@@@                          |

9. pytrace io <pid>

Trace Block and File I/O read/write latencies.

pytrace io 4821

Output:

Attaching File I/O latency tracer to PID 4821...
✓ Attached. Collecting block I/O events... Ctrl+C to stop.

--- BPF Map: @read_lat (ns) ---
[4096, 8191]          150 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16384, 32767]        23 |@@@@@                               |

10. pytrace flame <pid>

Generate sampling-based user/kernel stack flame graphs.

pytrace flame 4821 --duration 5

Output:

Attaching kernel profile sampler to PID 4821 for 5s...
✓ Attached. Sampling for 5s...
✓ Saved flame graph to flamegraph.svg

11. pytrace sched <pid>

Monitor runqueue latency and scheduling delays.

pytrace sched 4821

Output:

Attaching scheduler delay tracer to PID 4821...
✓ Attached. Collecting scheduler runqueue events... Ctrl+C to stop.

--- BPF Map: @runq_latency_us (us delay) ---
[1, 2]                 98 |@@@@@@@@@@@@@@@@@@@@@@@@            |
[4, 8]                142 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

12. pytrace pycall <pid>

Profile Python PyObject_Call execution timings.

pytrace pycall 4821

Output:

Attaching Python function call timer to PID 4821...
✓ Attached. Collecting PyObject_Call events... Ctrl+C to stop.

--- BPF Map: @latency (time spent per function in us) ---
SLOW execute_query took 12400us
SLOW process_job took 1850us

13. pytrace pyframe <pid>

Log Python execution contexts at the frame level (file, function, line).

pytrace pyframe 4821

Output:

Attaching Python frame USDT tracer to PID 4821...
✓ Attached. Collecting USDT frame events... Ctrl+C to stop.

ENTER execute_query() @ db/models.py:142
slow_query() @ db/models.py:142 — p99: 340ms

14. pytrace pycpu <pid>

Identify CPU hotspots in Python runtime execution stacks.

pytrace pycpu 4821

Output:

Attaching CPU hotspot sampler to PID 4821...
✓ Attached. Sampling CPU stacks... Ctrl+C to stop.

--- BPF Map: @stacks ---
  [0x7f3b821034bc, 0x7f3b821035dc]: 145
  -> Resolved: call_llm_chain @ gateway/orchestrator.py:120

15. pytrace pyexcept <pid>

Trace raised and caught exceptions within Python virtual machine.

pytrace pyexcept 4821

Output:

Attaching Python exception tracer to PID 4821...
✓ Attached. Monitoring exceptions... Ctrl+C to stop.

EXCEPTION KeyError @ tid=10234
  [ustack]:
    get_user_context @ db/client.py:48

16. pytrace pyiowait <pid>

Trace Python code blocked waiting on blocking I/O calls.

pytrace pyiowait 4821

Output:

Attaching I/O Wait blocking call tracer to PID 4821...
✓ Attached. Monitoring blocking sys_read calls... Ctrl+C to stop.

BLOCKING READ 12ms
  [ustack]:
    fetch_metadata @ db/client.py:54

17. pytrace pygil <pid>

Profile GIL lock acquisition delays and thread contention.

pytrace pygil 4821

Output:

Attaching GIL lock contention tracer to PID 4821...
✓ Attached. Monitoring GIL wait states... Ctrl+C to stop.

GIL WAIT 1250us tid=10234 stack:
  [ustack]:
    calculate_features @ ml/engine.py:89

18. pytrace pyleak <pid>

Profile heap allocations to detect memory leak patterns.

pytrace pyleak 4821

Output:

Attaching memory leak tracer to PID 4821...
✓ Attached. Collecting memory allocation metrics... Ctrl+C to stop.

=== TOP ALLOCATORS ===
@allocs[0x7f3b821034bc]: 10485760 bytes
  -> allocating callsite: load_dataset @ ml/data.py:12

19. pytrace pyreq <pid>

Measure end-to-end request lifecycle breakdown.

pytrace pyreq 4821

Output:

Attaching Request Lifecycle timer to PID 4821...
✓ Attached. Collecting Request Latency counts... Ctrl+C to stop.

REQ START tid=10234
REQ DONE total=340ms db=310ms other=30ms

20. pytrace timeline <pid>

Trace absolute chronological timeline call graph.

pytrace timeline 4821 --duration 5.0 --threshold 2.0

Output:

[     0.000ms] → handle_request()  server.py:45
[     0.040ms]   → parse_headers()  http.py:12
[     0.051ms]   ← parse_headers()  [0.011ms]
[     0.055ms]   → execute_query()  db.py:88
[    91.230ms]   ← execute_query()  [91.175ms]  ⚠️ SLOW

21. pytrace pythread <pid>

Trace thread-aware function call timelines with self-time.

pytrace pythread 4821

Output:

Attaching thread-aware tracer to PID 4821...
✓ Attached. Collecting threaded events... Ctrl+C to stop.

--- Thread ID: 10001 ---
  parse_headers() spent 0.00ms (Self time: 0.00ms)

22. pytrace pyasync <pid>

Trace async await coroutine metrics and yields.

pytrace pyasync 4821

Output:

Attaching async/coroutine tracer to PID 4821...
✓ Attached. Monitoring coroutine suspends/resumes... Ctrl+C to stop.

--- Coroutine: 0x7f3b821034bc ---
  Suspended counts: 2
  Total CPU Time: 80us

23. pytrace pyargs <pid>

Profile Python function call argument types and layout.

pytrace pyargs 4821

Output:

Attaching argument Layout layout-tracer to PID 4821...
✓ Attached. Dereferencing Python structs... Ctrl+C to stop.

1000 CALL obj=0x7f3b821034bc args=0x7f3b821051fa

24. pytrace pysyscall <pid>

Profile syscalls attributed directly to Python frames.

pytrace pysyscall 4821

Output:

Attaching syscall-to-Python attribution tracer to PID 4821...
✓ Attached. Monitoring slow read and futex syscalls... Ctrl+C to stop.

=== SLOW READ fd=4 dur=12ms ===
  [ustack]:
    fetch_metadata @ db/client.py:54

25. pytrace pynplus1 <pid>

Detect potential ORM loop-driven N+1 query patterns.

pytrace pynplus1 4821

Output:

Attaching N+1 query loop detector to PID 4821...
✓ Attached. Monitoring ORM execute loops... Ctrl+C to stop.

⚠️  N+1 CANDIDATE: db/models.py:142
   Called 15x in 5s (3.0/s)

26. pytrace pygraph <pid>

Trace hierarchical call relationships.

pytrace pygraph 4821

Output:

handle_request()  calls=1  avg=0.00ms  (server.py:45)
  execute_query()  calls=1  avg=0.00ms  (db.py:88)

27. pytrace pyanomaly <pid>

Identify slow function calls using statistical baselines.

pytrace pyanomaly 4821

Output:

[BASELINE] execute_query(): mean=10.33ms stddev=0.76ms
🚨 ANOMALY execute_query(): 45.00ms vs baseline 10.33ms

28. pytrace pydash <pid>

Stream traces directly to live curses dashboard.

pytrace pydash 4821

Output:

Attaching curses dashboard to PID 4821...
=== LIVE FUNCTION TRACER ===
RECENT CALLS:
  handle_request() 120.40ms

29. pytrace pysingle <pid> <target_func>

Trace single request / execution call tree with self time.

pytrace pysingle 4821 handle_request

Output:

[     0.000ms] → handle_request()  server.py:45
[     0.011ms]   → validate_token()  auth.py:12
[     0.015ms]   ← validate_token()  total=0.004ms  self=0.003ms