Employing a teleoperation system for gathering demonstrations offers the potential for more efficient learning of robot manipulation. However, teleoperating a robot arm equipped with a dexterous hand or gripper, via a teleoperation system poses significant challenges due to its high dimensionality, complex motions, and differences in physiological structure.In this study, we introduce a novel system for joint learning between human operators and robots, that enables human operators to share control of a robot end-effector with a learned assistive agent, facilitating simultaneous human demonstration collection and robot manipulation teaching. In this setup, as data accumulates, the assistive agent gradually learns. Consequently, less human effort and attention are required, enhancing the efficiency of the data collection process. It also allows the human operator to adjust the control ratio to achieve a trade-off between manual and automated control.We conducted experiments in both simulated environments and physical real-world settings. Through user studies and quantitative evaluations, it is evident that the proposed system could enhance data collection efficiency and reduce the need for human adaptation while ensuring the collected data is of sufficient quality for downstream tasks.
@inproceedings{luo2024humanagentjointlearningefficient,title={Human-Agent Joint Learning for Efficient Robot Manipulation Skill Acquisition},author={Luo, Shengcheng and Peng, Quanquan and Lv, Jun and Hong, Kaiwen and Driggs-Campbell, Katherine Rose and Lu, Cewu and Li, Yong-Lu},year={2025},eprint={2407.00299},booktitle={International Conference on Robotics and Automation},archiveprefix={arXiv},primaryclass={cs.RO},url={https://arxiv.org/abs/2407.00299},}
2024
ECCV 2024
DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-Level Control
Xinyu Xu, Shengcheng Luo, Yanchao Yang, Yong-Lu Li, and Cewu Lu
@inproceedings{xu2024disco,title={DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-Level Control},author={Xu, Xinyu and Luo, Shengcheng and Yang, Yanchao and Li, Yong-Lu and Lu, Cewu},booktitle={European Conference on Computer Vision},pages={108--125},year={2024},organization={Springer},}
2021
ACM MM 2021
Syntropic Counterpoints: Metaphysics of The Machines
Predrag K Nikolic, Ruiyang Liu, and Shengcheng Luo
In Proceedings of the 29th ACM International Conference on Multimedia, 2021
@inproceedings{nikolic2021syntropic,title={Syntropic Counterpoints: Metaphysics of The Machines},author={Nikolic, Predrag K and Liu, Ruiyang and Luo, Shengcheng},booktitle={Proceedings of the 29th ACM International Conference on Multimedia},pages={1443--1445},year={2021},}
DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-Level ControlIn European Conference on Computer Vision, 2024
Syntropic Counterpoints: Metaphysics of The MachinesIn Proceedings of the 29th ACM International Conference on Multimedia, 2021