PalmTrack

PalmTrack
On-Palm Finger Tracking Using a Wrist-Worn Camera

PalmTrack originates from my vision to expand the limits of human-
computer interaction, transforming the palm into
an absolute positioning digitizer.

[data-wf-bgvideo-fallback-img] { display: none; } @media (prefers-reduced-motion: reduce) { [data-wf-bgvideo-fallback-img] { position: absolute; z-index: -100; display: inline-block; height: 100%; width: 100%; object-fit: cover; } }

Full video on
YouTube

Timeframe
‍Mar-Sep2025

Role
‍Developer
Experimenter
Designer

Publication
PDF

Author rank
First Author

Academic recognition
IMWUT 2026
Manuscript Submitted

Support

Professor Jianjiang Feng

*This paper is submitted to the CHI Conference on Human Factors in Computing Systems (CHI ’26). It is currently under review and temporarily open for application. People who see this page are kindly requested not to spread it.

Motivation‍

‍Thinking of input, conventional devices such as mice and touchpads depend on dedicated surfaces, limiting their suitability for ubiquitous computing. While alternatives like gaze and mid-air hand tracking exist, they suffer from issues such as fatigue and lack of tactile feedback.

Mice
‍Depending on dedicated surface

Graphics tablet
‍Bulky and complex

Gazing
‍Lack of tactile feedback

Hand tracking
‍Fatigue and inaccurate

Define user story‍
PalmTrack is a wristband camera-based pointing technique that track the dominant hand’s finger position on the non-dominant hand’s palm, enabling always-available and eyes-free operation.
‍The four panels demonstrate: (1) A flip-up camera design
(2) Finger-palm contact detection
(3) Free positioning across the entire palm area including fingers
(4) Support for precision-demanding interactions like handwriting input.

Form factor and interaction
‍
‍‍PalmTrack’s hardware prototype includes a wristband with an infrared camera with IR LEDs positioned under the wrist.

The study defines three interaction modes—Left Yaw (LY), Vertical Touch (VT), and Right Yaw (RY)—enabling touch interaction across the palm and fingers with different motion constraints. The pitch angle between the finger and palm surface can be freely adjusted across all three interaction modes.

Applications

Using this device can accomplish a wide range of tasks, including writing, typing, drawing, and VR control. It also demonstrates significant potential in the field of accessibility.

Tech Stack

Interaction: The user touches the non-dominant palm with the dominant finger.
‍
Processing: A real-time multi-task Transformer encoder–decoder detects contact and estimates absolute fingertip coordinates.
‍
Output: The system maps these coordinates to application commands for direct control.

Offline Evaluation:

We evaluate model performance from two perspectives:
‍• Contact Region: Evaluation of finger touch detection and localization performance across two regions: palm and fingers.
• Interaction Mode: Analysis of model performance under three finger interaction patterns (LY, VT, RY), as introduced in Form factor and interaction part.

For finger touch positioning, we report the Mean Absolute Error (MAE) and Standard Deviation (SD) for errors in both 𝑥 and 𝑦 directions, denoted as 𝑥error and 𝑦error, along with the Euclidean distance error 𝑙error between the predicted point and the ground truth. Simply put, PalmTrack achieved a 5.9mm MAE.

For contact detection, we use Accuracy (ACC) and F1-score (F1) as evaluation metrics. The overall accuracy achieves 98.7%.

User Study:

Indoor scenarios (Study A)
In three studies we conducted, we chose smartphone touchscreens as the baseline input method and compared PalmTrack with it. Smartphone touchscreens were selected because they provide absolute positioning capabilities and their screen size is comparable to that of a human hand.

Study A.1: Fitts’ Law Study
‍‍
‍We cmpared PalmTrack to a smartphone touchscreen under eyes-free use via a GUI, aligning coordinate systems so targets appeared the same size. In each trial, participants hit a yellow target (20–40 mm) then dragged it into a fixed green area (60 mm); we recorded hit time and drag time. Each method used 5 blocks × 11 trials (first trial per block excluded), with practice provided, identical target sequences across methods, and short breaks between blocks.

Across ~600 drag trials per method, linear regression confirmed Fitts’ Law. PalmTrack’s total/hit/drag times (1572/798/774 ms) were close to touchscreen (1362/643/719 ms); drag times were nearly identical (+55 ms) with no significant difference (F₁,₁₁=1.30, p=0.279).

Study A.2: Digit Input Evaluation
‍‍
We then compared eyes-free digit entry on PalmTrack versus a phone touchscreen: participants first learned a 0–9 keypad mapped to 12 finger knuckles (recording phase), then eyes-free entered five random 10-digit strings using that mapping; PalmTrack recognized touches with a simple 3-NN classifier. PalmTrack was much more accurate overall than the touchscreen (93.71% vs. 74.86%) and remained reliable across different digit-to-digit transitions.

Study A.3: Handwriting Task
‍‍
We tested input with 10 digits (0–9) and 26 uppercase letters (A–Z). Each participant wrote 20 prompted characters in their preferred palm posture; recognition used GPT-4o. The touchscreen baseline used the same strings. In total, we collected 480 characters (12 participants × 2 methods × 20). PalmTrack reached 92.5% average handwriting accuracy, with 26/36 characters at 100%; main confusions involved ‘0’ vs. ‘D/O’. Touchscreen baseline scored 94.58%, statistics showed no significant difference.

PalmTrack
On-Palm Finger Tracking Using a Wrist-Worn Camera

PalmTrack

Pebbl

Wristp^2

Eatinter

Beyond Qomolangma

Alibaba : Xing Yun AI

Mercedes-Benz: China one app

PalmTrack On-Palm Finger Tracking Using a Wrist-Worn Camera

PalmTrack

Pebbl

Wristp^2

Eatinter

Beyond Qomolangma

Alibaba : Xing Yun AI

Mercedes-Benz: China one app

PalmTrack
On-Palm Finger Tracking Using a Wrist-Worn Camera