DexterTwin will use camera tracking to create a virtual leader arm and five-finger hand from the operator’s movements. Instead of building a mechanical leader device, the system will track/create a simulated arm and hand, then translate those movements into commands for a follower robot with a dexterous hand. Tracking tools like MediaPipe will estimate the operator’s shoulder, elbow, wrist, and hand landmarks. Software will connect those landmarks into a simulated leader arm, with individual thumb and finger joints for the five-finger hand.
DexterTwin is motivated by the goal of making dexterous robot control feel as natural as using your own arm and hand. Although a robot with a five-finger hand can perform delicate tasks, controlling each arm joint and finger separately is difficult. By using MediaPipe to track an operator’s movements and create a virtual leader, DexterTwin would let the operator demonstrate motions with their own arm and hand, then map those motions to a follower robot. An Oculus headset would provide a view of the robot’s workspace to help the operator judge position and contact. This approach could make teleoperation easier to learn, reduce the cost and complexity of a mechanical leader arm, and support safer remote manipulation in environments where direct human access is undesirable. It would also provide a platform for studying how camera-based tracking, VR feedback, and dexterous robot control work together.
Python, AI Development Courses, PyTorch, OpenCV
Physical AI, Large Language Models (LLM) application to Robotics, Experience with OSU HPC Cluster, Experience with robotics simulation of any kind (MuJoCo, Isaac SIM, etc or Unity)
