Sign Language Recognition - review of useful resources

A list of useful resources in the sign language recognition, translation or generation using different languages. Created during the hearai.pl project.

Contributing

Feel free to add issue with short description of new publication or create a pull request - add the new resource to the table or fill missing description.

Relevant Repositories

• Hear AI project: hearai/hearai

Table of Datasets

| Dataset | Language | Classes/task decription | Size | Data type | Adnotation type | Language level | Link | Licence |Downloaded| |----------|----------|----------------|-------|----------|-----------------|----------------|-----|-------|---------| | DGS | German SL | Signers were asked to do one of 20 activities like: describing something and storytelling | 50 hours | video | iLex, srt (subtitles), openpose (json), elan, cmdi | continous |DGS | Licence| partially| | PJM | Polish SL | Signers were asked to tell the story of picture content and clips, tell videos about themselves and talk about topics that interest them. | 400 hours | video | text | continous | PJM | no info | partialy| | Signor Corpus | Slovenian SL| Most of the recording sessions were performed on the premises of the local deaf clubs, only in some cases the mobile recording team visited the informant at home. | no info | video | tokenization, iLex, HamNoSys, gloss | continous | Signor | not avaible| not avaible yet due to datta protection issues| | Dicta Sign Lexicon |British SL <br>Greek SL <br>German SL <br>French SL | 1000 words and phrases and many videos where signers tell some stories | no info | video |gloss/translation (?) | both |Dicta | no info | no | | Galex | German SL | gardening and landscape vocabulary | 654 technical phrases | video | gloss | isolated |GaLex | no info | no | | LEDASILA | Austrian SL | words, different categories | no info |video | meaning (gloss) and description of the move | isolated | LedaSila |Creative commons | in contact with owners | | RWTH-PHOENIX | German SL | It is a sign language transcription of a weather forecast. |53GB | video (All recorded videos are at 25 frames per second and the size of the frames is 210 by 260 pixels)| gloss |continous |Phoenix| seems to be open to use (not specified directly on the page) |yes| | GSLL | Greek SL | 347 different signs/classes | 3,464 videos /42 GB | video |gloss/traslation | isolated|GSLL|publicly avaible, no info about restrictions |yes | | SIGNUM | German SL | words and phrases (used on a daily basis) | no info | videos | gloss/translation | both | SIGNUM| no info about restrictions | no| | MS-ASL | American SL | 1000 signs | 25000 videos | video | bbox, gloss, | isolated |MS ASL|publicly available | yes | | AUTSL | Turkish SL | 226 signs that are performed by 43 different signers | 38336 videos | video recorded using Microsoft Kinect v2 in RGB, depth and skeleton formats| spatial coordinates of the 25 junction points on the signer body, gloss | isolated|AUTSL |public but not uploaded yet (?) | no | | WLASL | American SL | 3126 glosses | 34404 videos| video | gloss, bbox,temporal boundary (start, end of frame), dialect of SL |isolated |WLASL | Computational Use of Data Agreement (C-UDA)| yes, partially without yt videos | | SPREAD THE SIGN | Polish SL <br>British SL <br>German SL <br>Russian SL <br>French SL <br>American SL <br>Spanish SL |25030 words in 42 languages | 574273 videos | video | gloss|isolated |Spread the sign | no info |yes | | CSL-Daily | Chinese SL | travel, shopping, medical care and daily-live words | no info |video |spoken language translations and gloss-level annotations | continous|CSL | you need to sign the agreement with the USTC link, research only, noncommercial use| no | | NMFs-CSL | Chinese SL | 1,067 Chinese sign words (610 confusing words , 457 normal words | no info | RGB videos | gloss | isolated |NMFs CSL | you need to sign the agreement with the USTC link, research only, noncommercial use| no | | ASLLVD | American SL | many words and phrases | no info |video |gloss labels, sign start and end time codes, start and end handshape labels for both hands, morphological and articulatory classifications of sign type | isolated |ASLLVD | Can be used for research and education purposes. Commercial use is not allowed. |no | | BUHMAP-DB | Turkish SL | non-manual gestures, 8 different classes | 48 annotated videos | video | gloss, ground truth of selected points | isolated |BUHMAP | available and free for academic research purposes |no | | PL-Kinect | Polish SL | 84 words in PSL (PJM).Each gesture is performed 20 times. | 84 videos (?) | point clouds videos | translation/gloss |isolated | Vision PRZ| publicly avaible, no info about restrictions | no | |LATLAB | American SL | 98 different stories | each video is 0,5-4 minutes long |video | glosses for each sign, an English translation of each passage, and details about the establishment and use of pronominal spatial reference points in space| continous|LATLAB|no info | no|

Table of Contribiutions

| Year | Paper | Dataset | Language | Task | Algorithms | Results| Code | |:----:|:-----:|:---------:|:---------:|:----:|:----------:|:------:|:----:| | 2021 | Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks | PHOENIX14T | German SL | Sign Language Production | Progressive Transformers and Mixture Density Networks | BLEU-4 ~ 13.64 | :x: | | 2021 | NetFACS: Using network science to understand facial communication systems | FACS datasets | :x: | Facial Signals Recognition | NetFACS | :x: | Github - code in R | | 2021 | ANONYSIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation | SMILE | German SL | Sign Language Production for Sign Language Video Anonymisation | AnonySign architecture | LPIPS ~ 0.243, FID ~ 49.48 | :x: | | 2021 | Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives | Pre-processed Phoenix14T | German SL | Sign Language Production | Mixture of Motion Primitives architecture | BLEU-4 ~ 12.67 | :x: | | 2021 | On-device Real-time Hand Gesture Recognition | :x: | American SL | Hand Gesture Recognition | Hand Tracking + NN | Recall=88% | :x: | | 2021 | Development of a software module for recognizing the fingerspelling of the Russian Sign Language based on LSTM | :x: | Russian SL | Sign Alphabet Recognition | LSTM Neural Network | Precision=91%, Recall=91% | :x: | | 2021 | Artificial Intelligence Technologies for Sign Language | - | - | Sign Language Recognition & Translation | - | - | - | | 2021 | A Deep Convolutional Neural Network Approach to Sign Alphabet Recognition | Sign Language MNIST | American Sign Language | Sign Alphabet Recognition | CNN | Accuracy=~94% | Kaggle | | 2021 | [Efficient sign language recognition system and dataset creatio