Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods

We present an approach to continuous American sign language (ASL) recognition, which uses as input 3D data of arm motions. We use computer vision methods for 3D object shape and motion parameter extraction and an ascension technologies 'Flock of Birds' interchangeably to obtain accurate 3D movement parameters of ASL sentences, selected from a 53-sign vocabulary and a widely varied sentence structure. These parameters are used as features for hidden Markov models (HMMs). To address coarticulation effects and improve our recognition results, we experimented with two different approaches. The first consists of training context-dependent HMMs and is inspired by speech recognition systems. The second consists of modeling transient movements between signs and is inspired by the characteristics of ASL phonology. Our experiments verified that the second approach yields better recognition results.