Gesture KINECT capture system for virtual surgical robots manipulation
Keywords:
Kinect device, natural interfaces, surgical robots, surgical simulator
Abstract
This article presents a gesture capture system for manipulating two virtual surgical robots. Gesture capture uses Kinect device to detect the movement of the user ́s hand to move the surgical robots, and his right knee to change the needed surgical tools. This natural interface capture system is proved on a surgical simulator for laparoscopic, that has a holder endoscopic robot manipulated by a joystick, and two surgical robots that are guided by the Kinect device. Kinematics model is used to transform Kinect captured signals to articular movements for the robots. System is tested on a robotic cholecystectomy, allowing a more natural manipulation to the user.
Downloads
Download data is not yet available.
References
[1] D.B. Jones, Laparoscopic Surgery: Principles and Procedures. New York: Marcel Dekker, 2004.
[2] N. Katkhouda, Advanced Laparoscopic Surgery: Techniques and Tips. New York: Springer, 2010.
[3] J. Marescaux, and F. Rubino, “The Zeus robotic system: experimental and clinical applications,” Surgical Clinics of North America, vol. 83, pp. 1305-1345, 2003.
[4] G.H. Ballantyne, and F. Molle, “The Da Vinci telerobotic surgical system: the virtual operative field and telepresence surgery,” Surgical Clinics of North America, vol. 83, pp. 1293-1304, 2003.
[5] C. Vaessen, “Location of robotic surgical systems worldwide and in France,” Journal of Visceral Surgery, vol. 148, pp. 9-11, 2011.
[6] A. Liu, F. Tendick, K. Cleary, and C. Kaufmann, “A survey of surgical simulation: applications, Technology, and education,” Presence, vol. 12, no. 6, pp. 599-614, 2003.
[7] H. Delingette and N. Ayache, “Hepatic Surgery Simulation,” Communications of the ACM, vol. 48, pp. 31-36, no. 2, 2005.
[8] J. García, M. Arias y E. Valencia, “Diseño de prototipo de simulador para entrenamiento en cirugía laparoscópica,” Revista Ingeniería Biomédica, vol. 5, no. 9, pp. 13-19, 2011.
[9] R. Valero, Y.H. Ko, S. Chauhan, O. Schatloff, A. Sivaraman, R.F. Coelho, F. Ortega, K.J. Palmer, R. SanchezSalas, H. Davila, X. Cathelineau, and V.R. Patel, “Cirugía robótica: Historia e impacto en la enseñanza,” Actas Urológicas Españolas, vol. 35, no. 9, pp. 540-545, 2011.
[10] C. Lallas and J. Davis, “Robotic surgery training with commercially available simulation systems in 2011: a current review and practice pattern survey from the society of urologic robotic surgeons”, Journal of Endourology, vol. 26, no. 3, pp. 283-293. 2012.
[11] S. Salinas y A. Vivas, “Modelado, simulación y control del robot para cirugía laparoscópica Lapbot,” Revista Chilena de Ingeniería, vol. 17, pp. 317-328, 2009.
[12] V. Torres, C. Méndez, A. Vivas y S. Salinas, “Diseño y simulación en 3D de un robot porta endoscopio para operaciones de laparoscopia,” en V Seminario Internacional de Ingeniería Electrónica, Bucaramanga, Colombia, 2011.
[13] D. Guzmán, “Herramienta software para la práctica y experimentación de la robótica quirúrgica,” Tesis Ingeniería en Automática Industrial, Universidad del Cauca, Popayán, Colombia, 2013.
[14] N. Petersen, “Continuous natural user interface: Reducing the gap between real and digital world,” presented at 8th IEEE International Symposium of Mixed and Augmented Reality, Orlando, FL, pp. 23-26, 2009.
[15] Y, Li, “Hand gesture recognition using Kinect,” presented at 3rd International Conference on Software Engineering and Software Science, Beijing, China, 2012.
[16] S. Lang, “Sign language Recognition whit Kinect,” Tesis de licenciatura, Freie Univesität Berlin, Berlin, Germany, 2011.
[17] K. Jahrmann, “3D Reconstruction with the Kinect-Camera,” Tesis de licenciatura, Technischen Universität Wien, Wien, Austria, 2013.
[18] E. Trilles, “Desarrollo de interfaces de usuarios naturales con Kinect,” Tesis Ingeniería Informática, Universidad Politécnica de Valencia, Valencia, España, 2012.
[19] S. Pfeiffer, “Guiado gestual de un robot humanoide mediante un sensor Kinect,” Tesis Ingeniería Técnica en Informática de Sistemas, Universidad Politécnica de Cataluña, Barcelona, España, 2011.
[20] G. Du and P. Zhang, “Markerless human–robot interface for dual robot manipulators using Kinect sensor,” Robotics and Computer-Integrated Manufacturing, vol. 30, no. 2, pp. 150-159, 2014.
[21] Y. Chang, B. Lange, M. Zhang, S. Koenig, P. Requejo, N. Somboon, A. Sawchuk and A. Rizzo, “Towards pervasive physical rehabilitation using Microsoft Kinect,” presented at Pervasive Computing Technologies for Healthcare (PervasiveHealth), San Diego, CA, 2012.
[22] Y. Chang, W. Han and Y. Tsai, “A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy,” Research in Developmental Disabilities, vol. 34, no. 11, pp. 3654-3659, 2013.
[23] R. Wen, W. Tay, B. Nguyen, C. Chng and C. Chui, “Hand gesture guided robot-assisted surgery based on a direct augmented reality interface,” Computer Methods and Programs in Biomedicine, vol. 116, no. 2, pp. 68-80, 2014.
[24] S. Dargar, A. Nunno and G. Sankaranarayanan, “Microsoft Kinect based head tracking for Life Size Collaborative Surgical Simulation Environments (LS-CollaSSLE),” Studies in Health Technology and Informatics, vol. 184, pp. 109-113, 2013.
[25] M. Jacob and J. Wachs, “Context-based hand gesture recognition for the operating room,” Pattern Recognition Letters, vol. 35, pp. 196-203, 2014.
[26] W. Souba, F. Mitchell, G. Jurkovich, L. Kaiser, W. Pearce, J. Pemberton, and N. Soper, ACS Surgery: Principles and Practice, 6th edition. Hamilton: Decker Publishing, 2007.
[27] D. Litwin, and M. Cahan, “Laparoscopic cholecystectomy,” Surgical Clinics of North America, vol. 88, pp. 1295-1313, 2008.
[2] N. Katkhouda, Advanced Laparoscopic Surgery: Techniques and Tips. New York: Springer, 2010.
[3] J. Marescaux, and F. Rubino, “The Zeus robotic system: experimental and clinical applications,” Surgical Clinics of North America, vol. 83, pp. 1305-1345, 2003.
[4] G.H. Ballantyne, and F. Molle, “The Da Vinci telerobotic surgical system: the virtual operative field and telepresence surgery,” Surgical Clinics of North America, vol. 83, pp. 1293-1304, 2003.
[5] C. Vaessen, “Location of robotic surgical systems worldwide and in France,” Journal of Visceral Surgery, vol. 148, pp. 9-11, 2011.
[6] A. Liu, F. Tendick, K. Cleary, and C. Kaufmann, “A survey of surgical simulation: applications, Technology, and education,” Presence, vol. 12, no. 6, pp. 599-614, 2003.
[7] H. Delingette and N. Ayache, “Hepatic Surgery Simulation,” Communications of the ACM, vol. 48, pp. 31-36, no. 2, 2005.
[8] J. García, M. Arias y E. Valencia, “Diseño de prototipo de simulador para entrenamiento en cirugía laparoscópica,” Revista Ingeniería Biomédica, vol. 5, no. 9, pp. 13-19, 2011.
[9] R. Valero, Y.H. Ko, S. Chauhan, O. Schatloff, A. Sivaraman, R.F. Coelho, F. Ortega, K.J. Palmer, R. SanchezSalas, H. Davila, X. Cathelineau, and V.R. Patel, “Cirugía robótica: Historia e impacto en la enseñanza,” Actas Urológicas Españolas, vol. 35, no. 9, pp. 540-545, 2011.
[10] C. Lallas and J. Davis, “Robotic surgery training with commercially available simulation systems in 2011: a current review and practice pattern survey from the society of urologic robotic surgeons”, Journal of Endourology, vol. 26, no. 3, pp. 283-293. 2012.
[11] S. Salinas y A. Vivas, “Modelado, simulación y control del robot para cirugía laparoscópica Lapbot,” Revista Chilena de Ingeniería, vol. 17, pp. 317-328, 2009.
[12] V. Torres, C. Méndez, A. Vivas y S. Salinas, “Diseño y simulación en 3D de un robot porta endoscopio para operaciones de laparoscopia,” en V Seminario Internacional de Ingeniería Electrónica, Bucaramanga, Colombia, 2011.
[13] D. Guzmán, “Herramienta software para la práctica y experimentación de la robótica quirúrgica,” Tesis Ingeniería en Automática Industrial, Universidad del Cauca, Popayán, Colombia, 2013.
[14] N. Petersen, “Continuous natural user interface: Reducing the gap between real and digital world,” presented at 8th IEEE International Symposium of Mixed and Augmented Reality, Orlando, FL, pp. 23-26, 2009.
[15] Y, Li, “Hand gesture recognition using Kinect,” presented at 3rd International Conference on Software Engineering and Software Science, Beijing, China, 2012.
[16] S. Lang, “Sign language Recognition whit Kinect,” Tesis de licenciatura, Freie Univesität Berlin, Berlin, Germany, 2011.
[17] K. Jahrmann, “3D Reconstruction with the Kinect-Camera,” Tesis de licenciatura, Technischen Universität Wien, Wien, Austria, 2013.
[18] E. Trilles, “Desarrollo de interfaces de usuarios naturales con Kinect,” Tesis Ingeniería Informática, Universidad Politécnica de Valencia, Valencia, España, 2012.
[19] S. Pfeiffer, “Guiado gestual de un robot humanoide mediante un sensor Kinect,” Tesis Ingeniería Técnica en Informática de Sistemas, Universidad Politécnica de Cataluña, Barcelona, España, 2011.
[20] G. Du and P. Zhang, “Markerless human–robot interface for dual robot manipulators using Kinect sensor,” Robotics and Computer-Integrated Manufacturing, vol. 30, no. 2, pp. 150-159, 2014.
[21] Y. Chang, B. Lange, M. Zhang, S. Koenig, P. Requejo, N. Somboon, A. Sawchuk and A. Rizzo, “Towards pervasive physical rehabilitation using Microsoft Kinect,” presented at Pervasive Computing Technologies for Healthcare (PervasiveHealth), San Diego, CA, 2012.
[22] Y. Chang, W. Han and Y. Tsai, “A Kinect-based upper limb rehabilitation system to assist people with cerebral palsy,” Research in Developmental Disabilities, vol. 34, no. 11, pp. 3654-3659, 2013.
[23] R. Wen, W. Tay, B. Nguyen, C. Chng and C. Chui, “Hand gesture guided robot-assisted surgery based on a direct augmented reality interface,” Computer Methods and Programs in Biomedicine, vol. 116, no. 2, pp. 68-80, 2014.
[24] S. Dargar, A. Nunno and G. Sankaranarayanan, “Microsoft Kinect based head tracking for Life Size Collaborative Surgical Simulation Environments (LS-CollaSSLE),” Studies in Health Technology and Informatics, vol. 184, pp. 109-113, 2013.
[25] M. Jacob and J. Wachs, “Context-based hand gesture recognition for the operating room,” Pattern Recognition Letters, vol. 35, pp. 196-203, 2014.
[26] W. Souba, F. Mitchell, G. Jurkovich, L. Kaiser, W. Pearce, J. Pemberton, and N. Soper, ACS Surgery: Principles and Practice, 6th edition. Hamilton: Decker Publishing, 2007.
[27] D. Litwin, and M. Cahan, “Laparoscopic cholecystectomy,” Surgical Clinics of North America, vol. 88, pp. 1295-1313, 2008.
Published
2015-06-16
How to Cite
Hurtado-Chaves, J., Nástar-Guacales, A., & Vivas-Albán, O. (2015). Gesture KINECT capture system for virtual surgical robots manipulation. ITECKNE, 12(1), 17-24. https://doi.org/https://doi.org/10.15332/iteckne.v12i1.817
Issue
Section
Research and Innovation Articles
