tlabel 0.4.0

TouchLabel AI - Tactile Data Annotation Toolkit

TouchLabel AI 🦞

Sensor-Agnostic Tactile Data Annotation Toolkit

load → review → export · Three steps to close the loop

中文文档

---

🔥 What's New in v0.2.0b1

Major improvements for downstream compatibility and user experience:

• ✨ Enhanced metadata: Added sensor_id, calibration_params, feature_names, and episode boundary markers (is_first/is_last)
• 🔄 LeRobot integration: Bidirectional converters for seamless data exchange with Hugging Face LeRobot framework
• 💾 HDF5 export: Scientific computing standard format for research workflows
• 📚 Comprehensive tutorials: Step-by-step guides for GelSight, PaXini, and Daimon sensors
• 🎯 Better error messages: Clear guidance with exact pip install commands and format descriptions
• 📖 Improved docs: 5-Minute Quick Start, sensor-specific loading instructions, troubleshooting table

See CHANGELOG.md for full details.

---

🎯 Got tactile data from different sensors that refuse to talk to each other? TLabel makes them speak the same language — load any format, annotate in a visual panel, export a unified schema.

---

🎯 5-Minute Quick Start

New to TLabel? Start here. This tutorial gets you from zero to annotated data in 5 minutes — no sensor data needed.

Step 1: Install (30 seconds)

pip install tlabel

That's it. The core package installs in seconds with just numpy as a dependency.

Step 2: Try the Built-in Demo (1 minute)

Open a Python terminal or Jupyter notebook:

import tlabel

# Load built-in GelSight demo data (no files needed!)
data = tlabel.demo()

# Open the interactive annotation panel
data.review()

A colorful panel pops up right in your notebook:

• Timeline at the top shows contact (green), slip (red), and idle (gray) frames
• Radar chart displays all 22 dimensions for the current frame
• Detail panel lets you inspect and edit individual values

Try this: Click on different frames in the timeline. Notice how the radar chart updates. Spot any frames that look wrong (e.g., contact=0 but force_magnitude > 0).

Step 3: Make Your First Correction (2 minutes)

Found a mislabeled frame? Fix it:

# In the panel, use the batch correction tool:
# 1. Select a range of frames (drag on timeline)
# 2. Set contact=0 in the correction panel
# 3. Click "Apply" — cascade rules auto-zero 7 related fields

# Or do it programmatically:
data.batch_patch(10, 20, "contact", 0)  # Frames 10-20: no contact

The cascade system ensures physical consistency: setting contact=0 automatically zeroes force_magnitude, slip_event, and resets manipulation_phase to "idle".

Step 4: Export Your Annotations (30 seconds)

# Export to JSON (full TLabel Format v2 schema)
data.export("my_annotations.json")

# Or export to CSV for quick analysis in Excel/pandas
data.export("my_annotations.csv")

Done! You've just completed the full annotation loop: load → review → correct → export.

What's Next?

• Have real sensor data? See the step-by-step tutorials:
  • GelSight / DIGIT Tutorial
  • PaXini PXCap Tutorial
  • Daimon DM-TacClaw Tutorial
• Want to understand the 22 dimensions? Check out TLabel Format v2
• Need a web-based tool for your team? Try tlabel-web
• Try the interactive demo in your browser: Live Demo

---

📥 Loading Your Own Data

Once you're comfortable with the demo, load your real sensor data:

GelSight / DIGIT (vision-based tactile sensors)

# Install extra dependencies
pip install tlabel[gelsight]

# Load your .pkl file (from gelsight-force-estimation or similar)
data = tlabel.load("my_gelsight_episode.pkl")
data.review()

What happens: The adapter extracts 22 dimensions from background-subtracted tactile images, including force magnitude, slip detection, optical flow, and manipulation phase inference.

PaXini PXCap (distributed force array)

pip install tlabel[paxini]

# Load your .h5 or .hdf5 file
data = tlabel.load("my_paxini_episode.h5")
data.review()

What happens: The adapter reads 6D force/torque vectors from each taxel region, detects contact and slip events, and maps them to 20 TLabel dimensions (no optical flow for force-only sensors).

Daimon DM-TacClaw (multimodal robot arm + tactile)

pip install tlabel[daimon]

# Load from LeRobot-style directory structure
data = tlabel.load("path/to/daimon_episode/")
data.review()

Requirements: The directory should contain:

• meta/info.json — episode metadata
• data/chunk-*.parquet — observation data (114-dim state vector)
• videos/ — FFV1-encoded tactile video files (optional, degrades gracefully if missing)

What happens: The adapter decodes FFV1 video frames, extracts tactile features (deformation, contact area, texture), and fuses them with robot state data.

Troubleshooting Common Issues

Error	Solution
ImportError: No module named 'cv2'	Run pip install tlabel[gelsight] or pip install opencv-python
ImportError: No module named 'h5py'	Run pip install tlabel[paxini] or pip install h5py
ImportError: No module named 'pyarrow'	Run pip install tlabel[daimon] or pip install pyarrow
ValueError: Unknown format	Check file extension (.pkl, .h5, .hdf5, .parquet) or pass format="gelsight" explicitly
FileNotFoundError	Double-check the path; use absolute paths if unsure

---

⚡ 30-Second Demo

👉 Try it live in your browser — no install needed, see the panel in action right now.

Or fire it up in Jupyter:

import tlabel
data = tlabel.demo()    # ← built-in GelSight demo
data.review()           # ← interactive panel pops up

Or try other sensors:

tlabel.demo('digit').review()    # DIGIT sensor
tlabel.demo('paxini').review()   # PaXini force sensor
tlabel.demo('daimon').review()   # Daimon DM-TacClaw

What you'll see: a color-coded timeline (🟢 contact / 🔴 slip / ⬜ idle), 22-dim radar chart, frame detail editor, and batch patching — all in one panel.

---

🚀 Quick Start

pip install tlabel

import tlabel

# Load — auto-detect sensor format, no config needed
data = tlabel.load("gelsight_force.pkl")     # GelSight / DIGIT
data = tlabel.load("paxini_episode.h5")      # PaXini
data = tlabel.load("daimon_data/")           # Daimon (directory or .parquet)

# Annotate — interactive Jupyter panel
data.review()          # Chinese UI
data.review(lang="en") # English UI

# Export — unified TLabel Format
data.export("output.json")   # TLabel Format v2 JSON
data.export("output.csv")    # flat CSV

Pre-annotate — AI-assisted, review & correct

from tlabel.predict import PredictEngine engine = PredictEngine() results = engine.predict(data) # predict contact, slip, phase... engine.apply(data, results, min_confidence=0.7) # apply only high-confidence data.review() # review & correct in panel

That's it. Full loop. 🔁

---

## 🤔 Why TLabel?

| Pain | TLabel's Answer |
|------|-----------------|
| Every sensor spits out a different format | **One `load()` call, auto-detected** |
| Raw tactile data is unreadable numbers | **Visual panel: timeline + radar + details** |
| Fixing labels frame-by-frame is soul-crushing | **Batch patch + cascade rules, fix ranges in one click** |
| Your lab mate exported... something... | **Unified TLabel Format v2, 22 dimensions, no ambiguity** |
| "But we use DIGIT and they use PaXini" | **Sensor-agnostic. Load both, same schema, same tool.** |
| "Manually labeling thousands of frames is tedious" | **AI pre-annotation: predict contact, slip, phase — review & correct** |

---

## 📡 Supported Sensors

| Sensor | Format | Dimensions | Optical Flow | Status |
|--------|--------|:----------:|:------------:|:------:|
| **GelSight Mini** | `.pkl` | 22 | ✅ | ✅ Stable |
| **DIGIT** | `.pkl` | 22 | ✅ | ✅ Stable |
| **Daimon DM-TacClaw** | `.parquet` / dir | 22 (video) / 20 (no video) | ✅ / — | ✅ Stable |
| **PaXini PXCap** | `.h5` / `.hdf5` | 20 | — | ✅ Stable |

> Force-type sensors (PaXini) lack optical images → 20 dims. Image-type → full 22. Daimon gracefully degrades when no video is present. No errors, no surprises.

---

## 📦 Installation

```bash
# Just the core (numpy only, ~2s install)
pip install tlabel

# Per-sensor extras
pip install tlabel[gelsight]   # GelSight / DIGIT → opencv-python
pip install tlabel[paxini]     # PaXini → h5py
pip install tlabel[daimon]     # Daimon → pyarrow + opencv-python

# I want it all
pip install tlabel[all]

---

🎨 Panel Features

• 🎨 Color-coded timeline: green = contact · red = slip · gray = idle — patterns jump out instantly
• 🕸 22-dim radar chart: see the full feature vector at a glance, bilingual labels
• ✏️ Frame & batch patching: fix one frame or a range, your call
• 🔗 Cascade rules: set contact=0 → 7 related fields auto-zero + phase resets to idle
• 🌐 Bilingual toggle: 中文 / English, one click top-right
• 📤 Export: JSON / CSV, auto-detected by file extension

---

TLabel Format v2 — 22 Dimensions

Static Features (18-dim)

#	Key	Description
1	contact	Binary contact flag
2	deformation_magnitude	Surface deformation intensity
3	force_magnitude	Normal force magnitude
4	force_peak	Peak force in episode window
5	force_direction	Force vector angle (°)
6	slip_entropy	Uncertainty of slip detection
7	slip_event	Binary slip event flag
8	texture_energy	Surface texture frequency energy
9	edge_density	Contact edge pixel ratio
10	contact_area	Contact region area ratio
11	centroid_x	Contact centroid x-position
12	normal_field_magnitude	Normal pressure field magnitude
13	normal_field_variance	Normal field spatial variance
14	shear_field_magnitude	Shear stress magnitude
15	shear_field_direction	Shear direction angle (°)
16	delta_force_normal	Frame-to-frame ΔF_normal
17	delta_force_shear	Frame-to-frame ΔF_shear
18	friction_cone_ratio	Tangential/normal force ratio

Temporal Features (4-dim, v0.2.0)

#	Key	Image-type	Force-type	Description
19	optical_flow_magnitude	✅	—	Inter-frame motion magnitude (Farneback)
20	optical_flow_direction	✅	—	Optical flow angle (°)
21	temporal_deformation_rate	✅	✅	Rate of deformation change
22	contact_transition	✅	✅	Contact state transition probability

---

📖 API Quick Reference

import tlabel

# ── Loading ──
data = tlabel.load(path)                     # Auto-detect sensor format
data = tlabel.load(path, format="gelsight")  # Force specific adapter

# ── Demo ──
data = tlabel.demo()                         # Built-in demo data
tlabel.list_demos()                          # See available sensors

# ── Properties ──
data.num_frames        # int — total frame count
data.duration_s        # float — episode duration
data.sensor_type       # str — sensor identifier
data.dimension_keys    # list — all dimension keys
data.modified_count    # int — frames with manual patches

# ── Frame Access ──
frame = data[0]                          # Index access
frame = data.get_frame(42)               # By frame_idx
frame.contact                            # Contact value
frame.slip_event                         # Slip event value
frame.is_modified                        # Has patches?

# ── Patching ──
frame.patch("contact", 0)                         # Single frame (cascade=True)
frame.patch("contact", 0, cascade=False)           # No cascade
data.batch_patch(10, 50, "contact", 0)             # Range patch

# ── Review & Export ──
data.review()                    # Jupyter panel (Chinese)
data.review(lang="en")           # English
data.export("output.json")       # JSON (TLabel Format v2)
data.export("output.csv")        # CSV

Cascade Rules (contact → 0)

When contact is set to 0, these fields are automatically zeroed:

Auto-zeroed Field	Condition
force_magnitude	always
force_peak	always
slip_event	always
delta_force_normal	always
delta_force_shear	always
contact_area	always
contact_transition	only if value > 0.5
manipulation_phase	→ "idle" (if not already)

🤖 AI-Assisted Pre-Annotation

Let the engine suggest labels, then you review and correct — human-in-the-loop, not black-box.

from tlabel.predict import PredictEngine

engine = PredictEngine()

# Option 1: cold start — no prior labels
results = engine.predict(data)

# Option 2: warm start — learn from partially labeled data first
engine.fit(data)        # extract statistics from existing annotations
results = engine.predict(data)

# Apply only high-confidence predictions (≥0.7)
applied = engine.apply(data, results, min_confidence=0.7)
print(f"Auto-filled {applied} fields")

# Review the summary
stats = engine.summary(results)
print(stats)
# → {total_frames, predicted_fields, avg_confidence, method_distribution, ...}

data.review()  # check predictions in the panel, correct as needed

What it predicts:

Dimension	Method	Confidence Range
contact	Rule (force + deformation + area)	0.4 – 0.9
slip_event	Rule (shear + delta + entropy)	0.55 – 0.8
manipulation_phase	Rule (contact + slip + force cascade)	0.55 – 0.65
Missing dims (with fit())	Statistics (mean from labeled frames)	~0.4

💡 Tip: Use fit() on partially labeled data first — even 10–20% labeled frames significantly improve statistical predictions. Predictions below your confidence threshold are simply skipped, so you stay in control.

---

🗂 Project Structure

tlabel/
├── core/
│   ├── types.py          # TLabelFrame / TLabelData containers
│   ├── loader.py         # Auto-detect & dispatch loading
│   └── registry.py       # Adapter registry
├── adapters/
│   ├── base.py           # BaseAdapter interface
│   ├── gelsight.py       # GelSight Mini / DIGIT
│   ├── paxini.py         # PaXini PXCap
│   └── daimon.py         # Daimon DM-TacClaw (+ video decoding)
├── viewer/
│   ├── panel.py          # Jupyter _repr_html_ renderer
│   └── templates.py      # HTML + JS + CSS template engine
├── predict/
│   └── engine.py         # AI-assisted pre-annotation engine
├── demo.py               # Built-in demo data loader
└── export/
    └── writer.py         # JSON / CSV export + NumpyEncoder

---

💬 Feedback

Found a bug? Have an idea? Just want to say hi?

• 🐛 Bug report → Open an Issue
• 💡 Feature request → GitHub Discussions
• 🌟 Using tlabel in your research? → We'd love to hear about it! Drop us a star ⭐

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Good first issues:

• 🔌 Add a new sensor adapter (SynTouch? XELA? Your call.)
• 📊 Improve radar chart UI (dark mode, interactive hover)
• 🌐 Add more language support (日本語, 한국어)
• 🧪 Add integration tests for edge cases

---

📄 License

MIT © Niuzu Tech

---

If this saved you from manually labeling tactile data, a ⭐ would make our day!