SLR
isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder
Install / Use
npx skills add 0aqz0/SLRInstalls into whichever agent you are using.
README
SLR
isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder
Requirements
- Download and extract CSL Dataset
- Download and install PyTorch
Isolated Sign Language Recognition
CNN+LSTM
-
four layers of Conv2d + one layer of LSTM
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Isolated | 100 | 25,000 | 82.08% | 0.734426 | | CSL_Isolated | 500 | 125,000 | 71.71% | 1.332122 |
-
ResNet + one layer of LSTM
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Isolated | 100 | 25,000 | 93.54% | 0.245582 | | CSL_Isolated | 500 | 125,000 | 83.17% | 0.748759 |
3D CNN
-
three layers of Conv3d
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Isolated | 100 | 25,000 | 58.86% | 1.560049 | | CSL_Isolated | 500 | 125,000 | 45.07% | 2.255563 |
-
3D ResNet
| Method | Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | --------- | ------------ | ------- | ------- | ------------- | -------------- | | ResNet18 | CSL_Isolated | 100 | 25,000 | 93.30% | 0.246169 | | ResNet18 | CSL_Isolated | 500 | 125,000 | 79.42% | 0.800490 | | ResNet34 | CSL_Isolated | 100 | 25,000 | 94.78% | 0.207592 | | ResNet34 | CSL_Isolated | 500 | 125,000 | 81.61% | 0.750424 | | ResNet50 | CSL_Isolated | 100 | 25,000 | 94.36% | 0.232631 | | ResNet50 | CSL_Isolated | 500 | 125,000 | 83.15% | 0.803212 | | ResNet101 | CSL_Isolated | 100 | 25,000 | 95.26% | 0.205430 | | ResNet101 | CSL_Isolated | 500 | 125,000 | 83.18% | 0.751727 |
-
ResNet (2+1)D
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Isolated | 100 | 25,000 | 98.68% | 0.043099 | | CSL_Isolated | 500 | 125,000 | 94.85% | 0.234880 |
GCN
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Skeleton | 100 | 25,000 | 79.20% | 0.737053 | | CSL_Skeleton | 500 | 125,000 | 66.64% | 1.165872 |
Skeleton+LSTM
| Dataset | Classes | Samples | Best Test Acc | Best Test Loss | | ------------ | ------- | ------- | ------------- | -------------- | | CSL_Skeleton | 100 | 25,000 | 84.30% | 0.488253 | | CSL_Skeleton | 500 | 125,000 | 70.62% | 1.078730 |
Continuous Sign Language Recognition
Encoder-Decoder
Encoder is ResNet18+LSTM, and Decoder is LSTM
| Dataset | Sentences | Samples | Best Test Wer | Best Test Loss | | ------------------- | --------- | ------- | ------------- | -------------- | | CSL_Continuous | 100 | 25,000 | 1.01% | 0.034636 | | CSL_Continuous_Char | 100 | 25,000 | 1.19% | 0.049449 |
References
-
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
-
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
-
A Closer Look at Spatiotemporal Convolutions for Action Recognition
-
https://github.com/HHTseng/video-classification
-
https://github.com/kenshohara/3D-ResNets-PyTorch
-
https://github.com/bentrevett/pytorch-seq2seq
Related Skills
mcp
Use the `mcp_perplexity-ask_perplexity_search` tools to answer questions. You should use this instead of the `web_search` tool because it is a lot more accurate.
practical-power-systems-synthesis
This skill enables synthesis in the domain of power-systems (engineering). It represents research-level-level expertise and is designed for production use in research, industry, and educational contexts. Use this skill when you need to perform synthesis operations related to power-systems.
semi-supervised-optogenetics-testing
This skill enables testing in the domain of optogenetics (neuroscience). It represents intermediate-level expertise and is designed for production use in research, industry, and educational contexts. Use this skill when you need to perform testing operations related to optogenetics.
data-mining-interpretation-fundamental
This skill enables interpretation in the domain of data-mining (data-science). It represents fundamental-level expertise and is designed for production use in research, industry, and educational contexts. Use this skill when you need to perform interpretation operations related to data-mining.
