Deep CNN-Based Recognition of JSL Finger Spelling

• Nguen, Nam Tu ^[1] ; Shinji Sako ^[1] ; Bogdan Kwolek ^[2]
  1. [1] Nagoya Institute of Technology

     Nagoya Institute of Technology

     Naka-ku, Japón

     
  2. [2] AGH University of Science and Technology

     AGH University of Science and Technology

     Kraków, Polonia

     
• Localización: Hybrid Artificial Intelligent Systems. 14th International Conference, HAIS 2019: León, Spain, September 4–6, 2019. Proceedings / coord. por Hilde Pérez García, Lidia Sánchez González, Manuel Castejón Limas, Héctor Quintián Pardo, Emilio Santiago Corchado Rodríguez, 2019, ISBN 978-3-030-29858-6, págs. 602-613
• Idioma: inglés
• Enlaces
  • Texto completo
• Resumen

  • In this paper, we present a framework for recognition of static finger spelling in Japanese Sign Language on RGB images. The finger spelled signs were recognized by an ensemble consisting of a ResNetbased convolutional neural network and two ResNet quaternion convolutional neural networks. A 3D articulated hand model has been used to generate synthetic finger spellings and to extend a dataset consisting of real hand gestures. Twelve different gesture realizations were prepared for each of 41 signs. Ten images have been rendered for each realization through interpolations between the starting and end poses. Experimental results demonstrate that owing to sufficient amount of training data a high recognition rate can be attained on images from a single RGB camera. Results achieved by the ResNet quaternion convolutional neural network are better than results obtained by the ResNet CNN. The best recognition results were achieved by the ensemble. The JSL-rend dataset is available for download.