KinHEC: Kinematic Trajectory Benchmark for Hand-Eye and Robot-World Calibration

Dataset on the Hugging Face Hub | pip install kinhec | Examples | MIT license

KinHEC is a large-scale, fully synthetic, ground-truth-complete benchmark for hand-eye calibration (A X = X B), simultaneous robot-world / hand-eye calibration (A X = Y B) and their spatio-temporal extension (unknown clock offset between the robot and the camera), together with the Python toolkit that generates it, loads it, validates it, benchmarks reference solvers on it and visualises it.

Instead of images, a sequence consists of two time-stamped pose streams: the robot flange trajectory, obtained by forward kinematics of a published manipulator model, and the pose of the observed reference in the camera frame, which is what a marker detector, a PnP solver or a visual-odometry front end actually reports. Both streams are stored twice, as exact ground truth and as measurements corrupted by physically motivated noise. This abstracts the image processing away and leaves the numerical calibration problem, which can then be studied at scale under precisely controlled conditions.

The published full tier holds 77,628 sequences (171 million robot-stream frames, 40 million camera-stream frames, 50.1 GB of Parquet) across eight tracks that vary sensor noise, motion degeneracy, temporal misalignment, outliers, monocular scale, systematic robot-model errors and image-formation realism. It is reproducible from this toolkit with a single command.

1. Installation

Python 3.10 or newer with numpy, scipy, pandas, pyarrow, pyyaml and matplotlib.

pip install kinhec
kinhec --version
kinhec info                      # robots, camera and target presets, tracks, solvers

For the test suite and the OpenCV cross-checks, and for the Hugging Face examples:

pip install "kinhec[dev]"        # pytest, opencv-python-headless
pip install "kinhec[hf]"         # datasets, huggingface_hub

Working on the toolkit itself:

git clone https://github.com/Ezharjan/kinhec.git
cd kinhec
pip install -e ".[dev]"
python -m pytest -q              # 115 tests, about a minute

The package uses the src layout, so an editable install is what puts kinhec on the path; no PYTHONPATH changes are needed. On Windows, run the commands in an Anaconda Prompt or in a PowerShell with conda initialised.

2. Quick start

Work on the published dataset:

pip install kinhec huggingface_hub
hf download Ezharjan/KinHEC --repo-type dataset --local-dir kinhec_data --include "sequences.parquet" "robot/noise-*.parquet" "camera/noise-*.parquet" "robots.json" "cameras_targets.json" "generation_info.json"

from kinhec import load_manifest, load_sequence, solvers, se3

man = load_manifest("kinhec_data")                         # one row per sequence, every factor as a column
seq = load_sequence("kinhec_data", "noise-000000", man)    # both streams and the ground truth

A, B = seq.axxb_motions()                                  # relative motions between keyframes
X = solvers.daniilidis(A, B)                               # solve A X = X B
print(se3.pose_error(X, seq.X))                            # (rotation error [deg], translation error [mm])

A_abs, B_abs = seq.axyb_poses()                            # absolute pose pairs
X, Y = solvers.shah(A_abs, B_abs)                          # solve A X = Y B

Or generate a copy locally, which takes about a minute and needs no download:

kinhec generate --tier smoke --out data      # 46 sequences covering every track and noise model
kinhec validate data
kinhec benchmark data
kinhec visualize data

The scripts in scripts/ wrap these chains for Windows (.bat) and Unix (.sh); scripts/smoke_test runs all four steps and the unit tests.

3. What is in a sequence

Shards hold one row group per sequence, so a single sequence is read without touching the rest of the file. The full column documentation is in the dataset card.

Eight tracks vary one group of factors at a time; the mixed track samples the whole factor space at random and is meant as training data for learning-based solvers.

Thirteen manipulator models with published kinematics are used (UR3/UR5/UR10, UR3e/UR5e/UR10e/UR16e/UR20, Franka Emika Panda, KUKA LBR iiwa 7 and 14, ABB IRB 120, PUMA 560) plus a free-floating 6-DoF generator; two configurations (eye-in-hand, eye-to-hand); five trajectory families; four camera and five target presets. Forward kinematics is checked against reference poses computed with the Robotics Toolbox for Python and against a chain built from the iiwa_stack URDF, and the PnP refinement against OpenCV's iterative solver.

4. Generating a dataset

kinhec plan --tier full                       # 77,628 sequences: noise 5120, motion 3580, temporal 2592,
                                              # outliers 1760, monocular 640, robot_error 1536, pnp 2400,
                                              # mixed 60000
kinhec generate --tier full --out data_full --workers 8
kinhec validate data_full                     # every sequence is checked; --max-sequences N for a sample
kinhec benchmark data_full --workers 8        # baselines into data_full/benchmarks
kinhec visualize data_full                    # figures into data_full/figures
kinhec card data_full                         # refresh the dataset card from the index

scripts/generate_full runs plan, generate, validate (on a 2000-sequence sample), benchmark and visualize, stopping at the first step that fails; scripts/generate_tier does the same for any tier.

• Cost. Measured on the machine that produced the published copy (a desktop CPU, --workers 8): the full tier takes about 30 h to generate, which is roughly 11 s of single-core time per sequence on average, and the eleven baselines over all 77,628 sequences take about 7 h. The smaller tiers range from about a minute (smoke, 46 sequences) through minutes (small, 562) to a few hours (medium, 9,996). Cost per sequence differs by more than an order of magnitude between tracks: the tracks that solve inverse kinematics for every capture pose and the long mixed sequences with per-frame PnP dominate.
• Disk. 50.1 GB for the full tier: 40.1 GB of robot streams, 9.6 GB of camera streams, 134 MB of index and metadata and 333 MB of baseline results (Parquet, zstd-compressed).
• Resume. Shards are the unit of work, and a shard counts as complete only when both of its Parquet files exist; they are written atomically, so an interrupted run never leaves a truncated shard. Re-running the same command skips completed shards and rebuilds the index. Ctrl+C cancels the shards that have not started, lets the running ones finish, writes the index and exits. kinhec index <folder> rebuilds sequences.parquet from the shard metadata at any time.
• Failures. A shard that fails does not abort the run: the error is reported with the recipe of the offending sequence, its traceback goes to the log, every other shard is still generated and indexed, and the command ends with a summary and exit status 1. Re-running retries exactly those shards. --fail-fast stops submitting new shards after the first failure instead.
• Determinism. Each sequence is seeded from the tier's global seed and its own id, so the output depends on neither the number of workers nor the order in which shards are produced.
• Scaling the design. A tier is a YAML file (src/kinhec/configs/*.yaml) listing the factor levels of every track. Copy one and pass --tier my_tier.yaml, or raise repeats_multiplier to enlarge every track proportionally. --tracks noise pnp generates a subset; --limit N keeps the first N sequences per track for debugging.
• Logs go to _to_delete/kinhec_work/generate_<tier>.log together with the expanded plan (plan_<tier>.json, about 90 MB for the full tier). The index is refreshed every ten completed shards and at the end of the run.

5. Cleaning up and restarting

kinhec clean (also scripts/clean) removes what a run leaves behind, so that the next one starts from a known state. It operates on the repository root, which it detects from the current folder or from the installed package (--root DIR overrides), and it never touches anything outside that root unless the path is given explicitly.

kinhec clean                                  # caches (__pycache__/, .pytest_cache/) and the scratch folder _to_delete/
kinhec clean --dry-run                        # list only (also -n)
kinhec clean --partial data_full              # leftovers of an interrupted run inside the dataset: *.tmp files
                                              #   and half-written shards; complete shards are kept for resuming
kinhec clean --data data_full                 # the whole dataset folder (asks y/N; --yes or -y skips the question)
kinhec clean --build                          # build/, dist/, src/*.egg-info (recreated by pip install -e .)
kinhec clean --all --yes                      # all of the above and every data_* folder in the root

--all never removes a folder named data/; name it explicitly if that is what you want. --data and --partial accept only folders that look like KinHEC datasets (generation_info.json, sequences.parquet, or robot/ and camera/ shard folders), which --force overrides. Read-only files are removed as well, version-control folders are left alone, and the command exits with status 1 if something could not be removed.

6. Using the data

from kinhec import load_manifest, load_sequence

man = load_manifest("data_full")
deg = man[(man.track == "motion") & (man.axis_scatter_lambda2 < 1e-6)]   # degenerate rotation axes
seq = load_sequence("data_full", deg.sequence_id.iloc[0], man)

seq.robot.t, seq.robot.q_meas, seq.robot.q_gt                # robot clock, encoder and true joints
seq.robot.T_base_ee_gt, seq.robot.T_base_ee_meas             # flange pose, true and as reported
seq.camera.t, seq.camera.t_gt                                # camera clock and true capture instants
seq.camera.T_cam_target_meas, seq.camera.visible, seq.camera.outlier
seq.X, seq.Y, seq.time_offset, seq.scale                     # ground truth of the calibration itself

A, B = seq.axxb_motions(min_rot_deg=5.0, max_keyframes=200)  # keyframe selection is tunable
A_abs, B_abs = seq.axyb_poses(time_offset=0.0)               # pair the streams with a chosen offset

Solvers in kinhec.solvers: tsai_lenz, park_martin, horaud_dornaika, daniilidis, andreff (optionally with unknown scale), refine_lm and ransac_axxb for A X = X B; shah, li_kronecker and refine_lm_axyb for A X = Y B; estimate_time_offset for the clock offset. On noise-free sequences every closed-form solver recovers the ground truth to about 1e-6 deg and 1e-6 mm, which the test suite asserts.

The twelve scripts in example/ are runnable use cases: loading and inspection, A X = X B and A X = Y B calibration, noise-robustness curves, degeneracy analysis, clock-offset estimation, outlier-robust estimation, monocular scale, robot-model errors, PnP image formation, custom generation, and export including the Hugging Face datasets library. The plotting scripts live in visualization/.

Command-line utilities: kinhec export <data> <sequence_id> --format csv|tum, kinhec validate, kinhec benchmark --solvers ... --tracks ... --max-sequences N, kinhec visualize, kinhec card, kinhec index, kinhec plan, kinhec clean, kinhec info. Every command has --help.

7. Benchmark protocol

kinhec benchmark applies one protocol to every solver: pair the streams (row-wise for synchronous sequences; for asynchronous sequences interpolate the measured robot stream at t - offset, with the ground-truth offset as oracle and with the offset estimated by angular-speed cross-correlation as estimated), keep the frames that carry a measurement, select keyframes at least 5 deg or 2 cm apart (at most 200; if fewer than three survive that filter, every paired frame is used), feed consecutive keyframe motions to the A X = X B solvers and absolute keyframe poses to the A X = Y B solvers, and report the geodesic rotation error [deg] and the Euclidean translation error [mm] of X (and of Y), the solver runtime, the clock-offset error and, for the scale-aware solver, the relative scale error. Outputs land in benchmarks/: results.parquet and .csv (one row per sequence, solver and synchronisation mode), leaderboard.md (medians per track and per factor level), summary.json and protocol.json.

Because every factor is swept, the results are curves rather than single numbers. The clearest example is the rotation-axis diversity of the motion track, where the median translation error of X moves by more than three orders of magnitude between fully degenerate and general motion:

8. Validation

kinhec validate <folder> checks the presence of every required file; the index against the schema, and every row group of both stream files against the index (each must be referenced by exactly one row, so both an orphaned row group and a duplicated reference are reported); and, per sequence, the frame counts, strictly increasing time stamps, identical clocks for synchronous sequences, time stamps consistent with the ground-truth offset and jitter, finite and orthonormal ground-truth poses, quaternions that are unit and have w >= 0 as stored, measurement availability consistent with the noise model, visibility and outlier counts, forward kinematics of the true joints (with the true DH parameters of dh_error sequences) reproducing the ground-truth flange poses, and the identity A_i X = Y B_i on every camera frame, with a tolerance of 1e-7 deg and 1e-6 mm. A sequence that cannot be read at all is reported as a finding rather than raising. The command exits with status 1 if anything fails, and the published tiers ship validated.

python -m pytest -q runs the 115 unit tests: forward kinematics against Robotics Toolbox reference poses and a URDF chain, Jacobians and inverse kinematics, PnP against OpenCV, all solvers, the noise models, the generator, I/O, validation, the failure handling and resumption of the shard runner, the clean command, the dataset card, and regression tests for the trajectory schedules.

9. Repository layout

pyproject.toml              package metadata
requirements.txt            runtime dependencies
src/kinhec/                 the toolkit (generator, loaders, solvers, benchmark, validation, visualisation, CLI)
src/kinhec/configs/         tier definitions: smoke.yaml, small.yaml, medium.yaml, full.yaml
example/                    12 runnable use-case scripts
visualization/              plotting scripts
tests/                      pytest suite
scripts/                    launchers for Windows and Unix (smoke test, generation, clean-up)
docs/figures/               figures used by this README
CHANGELOG.md                version history

Generated datasets (data/, data_*/) and the scratch folder _to_delete/ are not tracked; both are produced by the commands above and removed by kinhec clean.

10. Extending the toolkit

• Robots: add a RobotModel to src/kinhec/robots.py (standard or Craig DH, limits, speed caps, control rate, optional tool transform and joint signs) and list it in a tier YAML. tests/test_kinematics.py shows how to cross-check a new model against an independent implementation.
• Noise models: add a function to src/kinhec/noise.py, dispatch it in generator.py and register the name in config.py (validate).
• Trajectory families: return a Trajectory (a continuous-time pose or joint function plus keyframe times) from src/kinhec/trajectories.py.
• Solvers: implement f(A, B) -> X or f(A_abs, B_abs) -> (X, Y) and register it in benchmark.SOLVERS to have it evaluated by the same protocol and listed in the leaderboard.
• Tiers: any YAML with the structure of src/kinhec/configs/full.yaml.

11. Publishing a copy

A generated folder is self-contained and ready for the Hugging Face Hub: README.md is a dataset card with YAML front matter (license, tags, size category and the configs that expose the index and every track's streams to the dataset viewer), LICENSE and CITATION.cff are included, and every table is Parquet, with CSV copies of the index and of the benchmark results beside them.

pip install huggingface_hub
hf auth login
hf repos create <name> --repo-type dataset
kinhec card data_full --hf-repo <user>/<name>    # absolute figure URLs, repository named in the examples
hf upload <user>/<name> data_full . --repo-type dataset

The three positional arguments of hf upload are the repository id, the local folder and the path inside the repository. The upload is resumable: running the same command again transfers only what is missing, which matters for a copy the size of the full tier.

12. Design summary

• Frames: T_a_b is the pose of b in a; positions in metres, unit quaternions (w, x, y, z) with w >= 0; camera frame in the OpenCV convention; target frame with z towards the viewer.
• Eye-in-hand: X = T_ee_cam, Y = T_base_world (the board, or the visual-odometry origin), A_i = T_base_ee(i), B_i = inv(T_cam_target(i)). Eye-to-hand: X = T_ee_marker, Y = T_base_cam, B_i = T_cam_target(i). In both cases A_i X = Y B_i, and A = inv(A_i) A_j, B = inv(B_i) B_j satisfy A X = X B.
• Camera time stamps are t = t_gt + time_offset_s + jitter; the robot clock is the reference.
• Robot measurement models: flange-frame twist noise, joint-encoder noise through forward kinematics, and DH model error (the true geometry is perturbed, poses are reported with the nominal model, so the error is smooth, pose dependent and systematic). Camera measurement models: anisotropic camera-frame Gaussian noise (depth dominated), pixel noise followed by Levenberg-Marquardt PnP, and visual-odometry random walk; Student-t tails; three outlier models; monocular scale with drift.
• Motion-observability descriptors per sequence: second and third eigenvalues of the rotation-axis scatter matrix, mean and maximum rotation angle, mean translation, and the condition number of the stacked [R_A - I] blocks.

13. References

• Y. C. Shiu and S. Ahmad, "Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX = XB," IEEE Trans. Robotics and Automation 5(1), 1989.
• R. Y. Tsai and R. K. Lenz, "A new technique for fully autonomous and efficient 3D robotics hand/eye calibration," IEEE Trans. Robotics and Automation 5(3), 1989.
• J. C. K. Chou and M. Kamel, "Finding the position and orientation of a sensor on a robot manipulator using quaternions," Int. J. Robotics Research 10(3), 1991.
• F. C. Park and B. J. Martin, "Robot sensor calibration: solving AX = XB on the Euclidean group," IEEE Trans. Robotics and Automation 10(5), 1994.
• H. Zhuang, Z. S. Roth and R. Sudhakar, "Simultaneous robot/world and tool/flange calibration by solving homogeneous transformation equations of the form AX = YB," IEEE Trans. Robotics and Automation 10(4), 1994.
• R. Horaud and F. Dornaika, "Hand-eye calibration," Int. J. Robotics Research 14(3), 1995.
• F. Dornaika and R. Horaud, "Simultaneous robot-world and hand-eye calibration," IEEE Trans. Robotics and Automation 14(4), 1998.
• K. Daniilidis, "Hand-eye calibration using dual quaternions," Int. J. Robotics Research 18(3), 1999. doi:10.1177/02783649922066213
• N. Andreff, R. Horaud and B. Espiau, "Robot hand-eye calibration using structure-from-motion," Int. J. Robotics Research 20(3), 2001. doi:10.1177/02783640122067372
• K. H. Strobl and G. Hirzinger, "Optimal hand-eye calibration," IEEE/RSJ IROS, 2006.
• G. Schweighofer and A. Pinz, "Robust pose estimation from a planar target," IEEE Trans. Pattern Analysis and Machine Intelligence 28(12), 2006.
• A. Li, L. Wang and D. Wu, "Simultaneous robot-world and hand-eye calibration using dual-quaternions and Kronecker product," Int. J. Physical Sciences 5(10), 2010.
• M. Shah, "Solving the robot-world/hand-eye calibration problem using the Kronecker product," ASME J. Mechanisms and Robotics 5(3), 2013.
• T. Collins and A. Bartoli, "Infinitesimal plane-based pose estimation," Int. J. Computer Vision 109(3), 2014.
• J. Heller, M. Havlena and T. Pajdla, "Globally optimal hand-eye calibration using branch-and-bound," IEEE Trans. Pattern Analysis and Machine Intelligence 38(5), 2016.
• A. Tabb and K. M. Ahmad Yousef, "Solving the robot-world hand-eye(s) calibration problem with iterative methods," Machine Vision and Applications 28, 2017.
• F. Furrer, M. Fehr, T. Novkovic, H. Sommer, I. Gilitschenski and R. Siegwart, "Evaluation of combined time-offset estimation and hand-eye calibration on robotic datasets," Field and Service Robotics (FSR 2017), Springer Proceedings in Advanced Robotics 5, 2018. doi:10.1007/978-3-319-67361-5_10
• I. Ali, O. Suominen, A. Gotchev and E. R. Morales, "Methods for simultaneous robot-world-hand-eye calibration: a comparative study," Sensors 19(12), 2019.
• K. Koide and E. Menegatti, "General hand-eye calibration based on reprojection error minimization," IEEE Robotics and Automation Letters 4(2), 2019; dataset: "A dataset for hand-eye calibration evaluation," University of Padova Research Data, 2019.
• J. Wu, Y. Sun, M. Wang and M. Liu, "Hand-eye calibration: 4-D Procrustes analysis approach," IEEE Trans. Instrumentation and Measurement 69(6), 2020.
• I. Enebuse, M. Foo, B. S. K. K. Ibrahim, H. Ahmed, F. Supmak and O. S. Eyobu, "A comparative review of hand-eye calibration techniques for vision guided robots," IEEE Access 9, 2021.
• Kinematic parameters: Universal Robots, "DH parameters for calculations of kinematics and dynamics" (support article); Franka Emika, Franka Control Interface documentation, "Robot and interface specifications"; KUKA, LBR iiwa 7 R800 / 14 R820 specification and the iiwa_stack URDF (IFL-CAMP); ABB, IRB 120 product specification; P. Corke, Robotics Toolbox for Python (PUMA 560 model).

14. Citation

@misc{kinhec2026,
  title  = {KinHEC: A Kinematic Trajectory Benchmark for Hand-Eye and Robot-World Calibration},
  author = {Aizierjiang Aiersilan},
  year   = {2026},
  note   = {Dataset and toolkit},
  url    = {https://huggingface.co/datasets/Ezharjan/KinHEC}
}

Released under the MIT license. Version history: CHANGELOG.md.