IEEE Paper
Toward Zero-Shot Sim-to-Real Transfer Learning for Pneumatic Soft Robot 3D Proprioceptive Sensing

Soft robotics is a subfield in robotics that specializes in the building and controlling of bodies with flexible and deformable materials instead of traditional rigid links. The compliant materials allow the robot to change shape either passively or actively and provide a high degree of sensitivity to external factors. This allows soft robots to perform tasks like grasping and manipulating irregular surfaces in a safer and more effective way than traditional rigid robotics. Nonetheless, due to its multidimensional deformation capabilities, proprioceptive sensing has proven to be a great challenge, and very few methods have provided reliable 3D representation using embedded sensors.
​
In this paper, NYU assistant professor Chen Feng, Ph.D. candidate Hanwen Zhao and me, in collaboration with Carnegie Mellon assistant professor Wenzhen Yuan and Ph.D. candidate Uksang Yoo present a method to accurately reconstruct a soft finger by using vision-based proprioception and deep learning models. This method proved to be efficient and reliable with just under 3% reliable error. This confirms that the approach can be utilized to correctly control and track a multidimensional body, therefore opening endless opportunities for real-world applications, like precision object handling for fragile objects, and new areas of research, like utilizing vision-based proprioception to conduct force estimation. 

​

Below you can find the paper, and also a detailed explanation of my contributions to the project. 

 

I hope you enjoy the paper!