ICMV 2025 · Proceedings of SPIE

WholeBodyPose

A Unified End-to-End Framework for Sign Language Recognition and Pose-Based Training Data

Cristian Lazo-Quispe1, Renato Castro-Cruz1, Mauricio Salazar-Espinosa2, Gissella Bejarano3

1Universidad Nacional de Ingeniería, Lima, Peru 2Pontificia Universidad Católica del Perú 3Marist University, New York, USA

WholeBodyPose pipeline Five stages from left to right: input video, image or webcam frames; a pose backend (RTMPose, ViTPose+ or MediaPipe); the unified COCO-WholeBody output of 133 keypoints per person (17 body, 6 feet, 68 face, 42 hands); temporal clean-up with NLMS smoothing and a sequence filter; and training-ready pose sequences of shape T by 133 by 2, saved as npz or json, which feed ISLR models such as SPOTER, SILT, GCN-BERT and Pose-TGCN. Input Video, image or webcam RGB frames, any FPS optional enhancement Pose backend RTMPose RTMW-l, multi-person ViTPose+ huge … small MediaPipe pose + hands + face Unified output COCO-WholeBody 133 keypoints per person (N, 133, 2) + scores pixels · RGB input Temporal clean-up NLMS smoothing online: uses past frames only Sequence filter drops empty, static or no-hands frames optional Training-ready Pose sequences (T, 133, 2) .npz / .json feeds ISLR models SPOTER · SILT GCN-BERT · Pose-TGCN WholeBodyPose pipeline: input frames, pose backend, COCO-WholeBody 133 keypoints, temporal clean-up, training-ready pose sequences. 1 · Input Video, image or webcam RGB frames, any FPS · optional enhancement 2 · Pose backend RTMPose RTMW-l ViTPose+ huge … small MediaPipe pose+hands+face 3 · Unified output COCO-WholeBody, 133 keypoints body 17 · feet 6 · face 68 · hands 42 (N, 133, 2) 4 · Temporal clean-up (optional) NLMS smoothing · sequence filter online, past frames only · drop empty/static frames 5 · Training-ready Pose sequences (T, 133, 2) .npz / .json → SPOTER, SILT, GCN-BERT, Pose-TGCN
WholeBodyPose pipeline. Every backend takes an RGB image and returns float32 pixel coordinates in COCO-WholeBody order; undetected keypoints are (0, 0) with score 0. The video pipeline runs at a chosen processing FPS, writes a browser-playable overlay and saves the sequence with its scores and FPS.

TL;DR Every pose estimator speaks its own dialect. WholeBodyPose puts RTMPose (RTMW), ViTPose+ and MediaPipe behind one interface that always returns the 133 COCO-WholeBody keypoints (body, feet, face, both hands), plus the video pipeline, temporal filtering (NLMS) and visualization needed to turn raw sign language videos into training-ready pose sequences. We release the pose sequences of 13 ISLR subsets built with it.

Abstract

Sign language recognition systems require high-quality, temporally consistent pose estimation data for effective training. However, existing approaches suffer from inconsistent output formats across different pose estimation models, inadequate temporal filtering, and a lack of standardized preprocessing pipelines. We present WholeBodyPose, an end-to-end framework that unifies multiple state-of-the-art pose estimation models (MediaPipe, RTMPose, ViTPose) under a standardized COCO-133 format. Our framework addresses the critical gaps identified in recent work, providing researchers with a consistent interface to generate high-quality training data.

We provide a modular architecture that supports the easy integration of new pose estimation models, including preprocessing pipelines from raw video input to training-ready keypoint sequences, quality control mechanisms, and visualization tools. Additionally, our library is optimized for real-time use cases, enabling seamless integration into live inference pipelines and interactive sign language systems. This capability makes it suitable not only for research and training, but also for deployment in production-level applications. Our open-source library democratizes access to high-quality pose-based sign language recognition research and establishes a new standard for preprocessing pipelines in this domain.

Keywords: Sign Language Recognition, Pose Estimation, Framework, Keypoint Filtering, Real Time Processing

Key contributions

  • One API, three backends

    MediaPipe, RTMPose and ViTPose behind the same call: predict(frame) returns keypoints and confidence scores, so switching backend needs no code changes.

  • COCO-133 standardization

    Heterogeneous outputs aligned to the 133-keypoint COCO-WholeBody indexing (17 body, 6 feet, 68 face, 42 hands), optionally extended with virtual chest and mid-hip points (135).

  • Video to training-ready sequences

    A complete preprocessing pipeline from raw video to keypoint sequences, with image enhancement for poor lighting, quality control, real-time NLMS smoothing and visualization.

  • Validated on ISLR benchmarks

    Pose versions of five public ISLR datasets, and consistent accuracy gains for Pose-TGCN, GCN-BERT, SILT and SPOTER trained from scratch on WholeBodyPose keypoints.

Backends and keypoint layouts

Default settings reproduce the paper; any backend argument can be overridden, e.g. PoseEstimator("rtmpose", mode="lightweight").

BackendModelPeopleRuns onInstall extra
rtmposeYOLOX-m + RTMW-l 384×288 (OpenMMLab)multiONNX Runtime CPU / CUDAcore (+ [gpu])
vitposeViTPose+ huge / large / base / small, full-frame top-downsingleONNX Runtime CPU / CUDAcore (+ [gpu])
mediapipeBlazePose heavy + Hands + Face MeshsingleCPU[mediapipe]
A person with the 133 COCO-WholeBody keypoints drawn: body, feet, face and both hands, colour-coded by part.
(a) COCO-133 keypoints
The same person drawn in the OpenPose-style 135-keypoint layout, with virtual chest and mid-hip points added.
(b) OpenPose-135 keypoints

Paper, Figure 1. The DrawerPose renderer draws every backend with the same colour coding and skeletal links; the OpenPose-135 mode adds virtual chest and mid-hip points.

Datasets

Pose sequences of 13 isolated sign language recognition subsets from five public datasets, extracted with WholeBodyPose (RTMPose, 135 keypoints) and hosted on the Hugging Face Hub: CristianLazoQuispe/pose-action-recognition.

SubsetLanguageClassesTrainValTestSizeFiles (HDF5 per split, label map)
WLASL American Sign Language · source: WLASL v0.3 (Li et al., WACV 2020)
WLASL100ASL1001,4423382580.28 GBTrain Val Test Labels
WLASL300ASL3003,5499006680.69 GBTrain Val Test Labels
WLASL2000ASL2,00014,2893,9162,8782.81 GBTrain Val Test Labels
MS-ASL American Sign Language · source: MS-ASL (Vaezi Joze & Koller, BMVC 2019)
MSASL100ASL1003,7741,1907571.09 GBTrain Val Test Labels
MSASL200ASL2006,2962,0411,3591.87 GBTrain Val Test Labels
MSASL500ASL50011,3593,7022,7203.38 GBTrain Val Test Labels
MSASL1000ASL1,00016,0025,2874,1724.84 GBTrain Val Test Labels
ASL Citizen American Sign Language · source: ASL Citizen (Desai et al., 2023)
ASLCitizen100ASL1001,7073931,4440.62 GBTrain Val Test Labels
ASLCitizen300ASL3004,6991,1284,0581.73 GBTrain Val Test Labels
ASLCitizen1000ASL1,00015,2563,71412,6155.58 GBTrain Val Test Labels
ASLCitizen2731ASL2,73140,15410,30432,94115.13 GBTrain Val Test Labels
LSA64 Argentinian Sign Language · source: LSA64 (Ronchetti et al., CACIC 2016)
LSA64LSA642,5603203200.58 GBTrain Val Test Labels
PUCP-DGI Peruvian Sign Language · source: PUCP-DGI (Pontificia Universidad Católica del Perú)
DGI_PUCP30LSP305001081070.03 GBTrain Val Test Labels
DGI_PUCP38LSP386511401400.03 GBTrain Val Test Labels
Clip counts per split, read from the HDF5 files (one group per clip). Subset names open the folder on Hugging Face; the file links open each split file and its *_maplabels.json. ASL: American, LSA: Argentinian, LSP: Peruvian Sign Language. WLASL2000-test covers 1,999 of the 2,000 glosses, as in the official split. DGI_PUCP38 covers all 931 PUCP-DGI305 videos (38 classes).

File format

One file per split, <Subset>_135-{Train,Val,Test}.hdf5, with one group per clip: data of shape (T, 2, 135) (x, y normalized by the frame size, every frame, undetected points at 0), label, video_name, width, height. Keypoints 0–132 are COCO-WholeBody; 133 is the mid-chest and 134 the mid-hip point. No videos are redistributed.

License

License: CC BY-NC-SA 4.0; the terms of each source dataset also apply to its subset. Citation required: cite WholeBodyPose and the source dataset of every subset you use (BibTeX in the dataset card).

Rebuild from the videos

wholebodypose-build-islr rebuilds any subset from the original videos; schema, loading code and per-dataset notes are in docs/datasets.

96-keypoint SignERT subset

The 96-point layout used by SignERT (11 upper-body points, both hands, mid-chest, 42 face points) is an exact column subset of these files:

from wholebodypose.utils.keypoint_subsets import SIGNERT_96
data96 = f[key]["data"][()][:, :, SIGNERT_96]   # (T, 2, 135) -> (T, 2, 96)

Data processing

What was checked before and after extraction. Full provenance, including the open questions, is documented in docs/datasets.

Annotation checks (WLASL)

  • 12 video ids listed in the nslt_*.json subset files have no gloss in WLASL_v0.3.json; their 19 entries were dropped, leaving exactly the 21,083 instances of v0.3.
  • The official class list misspells lose as losear; fixed, so the class list and v0.3 share the same 2,000 glosses.
  • For every subset and split, every labelled id has a video and every video has a label (checked in both directions).
  • Split ids cross-checked against the independent Uni-Sign label files: identical for every split of WLASL100, 300 and 2000.

WLASL provenance

Subsets and splits

  • WLASL-N: the first N glosses of WLASL v0.3, official splits.
  • MS-ASL-N: the official MS-ASL subsets and splits.
  • ASL Citizen-N: the N most frequent glosses of the whole dataset, ties broken alphabetically; every clip keeps its official split.
  • LSA64: signer-independent (8 signers train, 9th val, 10th test).
  • PUCP-DGI30: stratified random split (70 / 15 / 15).

Extraction

  • RTMPose-WholeBody (YOLOX-m + RTMW-l 384×288) on every frame at the original resolution, no temporal subsampling.
  • Keypoints with score below 2.5 are set to (0, 0), and an incomplete hand or foot chain is zeroed rather than guessed.
  • Coordinates are normalized by the frame size; the original width and height are stored with each clip.
  • Clip length T equals the number of decoded video frames (checked for every WLASL clip).

Reproducibility

  • wholebodypose-build-islr --preset icmv2025 reproduces the published keypoints: on 10 test videos per dataset, median absolute difference below 1e-5 (normalized) and 99.96–100 % detection agreement.
  • The sample lists rebuilt from the original metadata match the published split files: counts, video names, labels and label maps.
  • The older 96-keypoint files are reproduced bit for bit from the 135-keypoint files with SIGNERT_96.
  • Known caveat: the release fed RGB frames to RTMPose, which expects BGR; the preset replays this, the library default is the correct BGR input.

Reproduction table

ISLR models

Four ISLR models trained from scratch on WholeBodyPose keypoints (RTMPose, 135 points) instead of their original pose inputs: Top-1 accuracy on WLASL100 improves for every model.

MethodPer-instancePer-classΔ Top-1Original code
Top-1Top-5Top-1Top-5
Pose-TGCN55.4378.68––
+ WholeBodyPose58.1482.5859.2283.58+2.71
GCN-BERT60.1583.98––
+ WholeBodyPose66.6791.4768.1592.08+6.52
SILT62.79–––
+ WholeBodyPose66.6784.8868.1585.72+3.88
SPOTER63.18–––
+ WholeBodyPose67.0586.4367.4787.42+3.87
Paper, Table 3: accuracy (%) on WLASL100. Models trained from scratch using only pose inputs, with simple geometric augmentations (rotation, zoom, perspective) on the keypoints. Baseline numbers are those reported by each method; “–” not reported. The trained ISLR checkpoints are not released; the links point to each model's original code.

Quick start

Install
pip install "wholebodypose[mediapipe] @ git+https://github.com/CristianLazoQuispe/WholebodyPoseEstimation.git"
Python
from wholebodypose import PoseEstimator

est = PoseEstimator("rtmpose")                         # or "vitpose", "mediapipe"
keypoints, scores = est.predict_image("person.jpg")    # (N, 133, 2), (N, 133)
res = est.process_video("sign.mp4", "outputs/")        # res["total_keypoints"]: (T, 133, 2)
Command line
wholebodypose backends                                   # list backends
wholebodypose image data/images/image.jpg -b mediapipe   # -> outputs/image_mediapipe.{png,json}
wholebodypose video data/videos/videoTest.mp4 -b rtmpose --fps 15
wholebodypose download                                   # pre-fetch every weight

The core install (NumPy, OpenCV, ONNX Runtime) runs RTMPose and ViTPose. Extras: [mediapipe], [video], [filters] (NLMS), [datasets], [gpu] (uninstall onnxruntime first). Model weights are downloaded on first use from the Hugging Face mirror (sha256-verified), with the upstream URLs as fallback. More in the README.

Demo

Upload a video or an image, pick a backend and get the skeleton overlay plus the keypoints, without installing anything.

The demo runs on a free Hugging Face Space and may take a minute to wake up.

Open the Space

Paper and citation

Please cite. If the library, the demo or the pose datasets help your work, cite the paper below.

First page of the WholeBodyPose paper.

WholeBodyPose: A Unified End-to-End Framework for Sign Language Recognition and Pose-Based Training Data

Cristian Lazo-Quispe, Renato Castro-Cruz, Mauricio Salazar-Espinosa, Gissella Bejarano

Eighteenth International Conference on Machine Vision (ICMV 2025), Proc. SPIE 14114, 141140A (25 Feb 2026), pp. 69–79. doi:10.1117/12.3096312

PDF (0.3 MB)

BibTeX
@inproceedings{lazo2026wholebodypose,
  title        = {WholeBodyPose: a unified end-to-end framework for sign language recognition and pose-based training data},
  author       = {Lazo-Quispe, Cristian and Castro-Cruz, Renato and Bejarano, Gissella and Salazar-Espinosa, Mauricio},
  booktitle    = {Eighteenth International Conference on Machine Vision (ICMV 2025)},
  volume       = {14114},
  pages        = {69--79},
  year         = {2026},
  organization = {SPIE},
  doi          = {10.1117/12.3096312}
}

Please also cite the backend you use (RTMW, ViTPose++, MediaPipe), COCO-WholeBody, and the source dataset of every pose subset you use; the backend entries are in the documentation and the dataset entries in the dataset card.