Allen et al -- ASL Finger Spelling

Allen, J.M.; Asselin, P.K.; Foulds, R., "American Sign Language finger spelling recognition system," Bioengineering Conference, 2003 IEEE 29th Annual, Proceedings of , vol., no., pp. 285-286, 22-23 March 2003

Summary

Allen et al. want to create a wearable computer system that is capable of translating ASL finger spelling used by the deaf into both written and spoken forms. Their intention is to lower communication barriers between deaf and hearing members of the community.

Their system consists of a CyberGlove worn by the finger speller. The glove uses sensors to detect bending in the fingers, palm, finger and wrist abduction, thumb crossover, etc. These glove is polled at a controlled sampling rate. The vector of sensor values is fed into a perceptron neural network that has been trained with examples of wach of the 24 different letters ('J' and 'Z' require hand motion, so were left out of this study). The classification output is given by the neural network, and is the right letter 90% of the time. Their experiments were only based on one user, however.

Discussion

First, the authors of this paper are very condescending toward the Deaf community. If any Deaf people were to ever read this article, they would be seriously pissed. Obviously I'm not Deaf. Obviously I can't speak for all Deaf people. That being said, the Deaf community is very strong (I capitalize Deaf on purpose, as that's the way Deaf culture sees itself). They work hard to make themselves independent, not needing the help or assistance of the hearing. The motivation for the paper is sound, and technology like this would indeed lower some of the communication barrier.

This doesn't seem too bad as a proof of concept. Motion needs to be incorporated to get the 'J' and 'Z' characters into play. This system also needs to be ***fast*** as the Deaf can finger spell incredibly rapidly, as quickly as you can spell a word verbally. Natural finger spelling is not about making every letter distinct, but about capturing the "shape" of the word (same way your brain works when it reads words on a page, remember that Cambridge study thing? http://www.jtnimoy.net/itp/cambscramb/). How distinct do the letters have to be for their approach to work? What sampling rate do they use? Can it be done real time (I guess no, since they say MATLAB stinks at real time).

Also, I would like to see results on misclassifications. Which letters do poorly (m and n look alike, so do a, e, and s)? They also point out accuracy is user specific. Finger spelling is a set form, so surely there are ways to generalize recognition. Just train on more than one person. Neural nets could also be used to train the 'in-between' stuff and give a little context for the letters before and after a transition.

BiBTeX

@inproceedings{allen2003aslFingerSpelling
,author={Jerome M. Allen and Pierre K. Asselin and Richard Foulds}
,title={{American Sign Language} finger spelling recognition system}
,booktitle={29th Annual IEEE Bioengineering Conference, 2003}
,year={2003}
,month={March}
,pages={285-286}
,doi={10.1109/NEBC.2003.1216106}
}