Development of an Indonesian Sign Language Translation Application with Hand Gesture Detection

Agus Purwanto; Joko Aryanto

doi:10.58526/jsret.v4i4.918

Authors

Agus Purwanto Universitas Teknologi Yogyakarta
Joko Aryanto Universitas Teknologi Yogyakarta

DOI:

https://doi.org/10.58526/jsret.v4i4.918

Keywords:

Indonesian Sign Language, Hand Gesture Detection, YOLOv8, Artificial Intelligence, Flutter, FastAPI

Abstract

Sign language serves as an essential medium of communication for individuals with hearing impairments. However, limited public understanding continues to create barriers between deaf individuals and the broader community. This study aims to develop an AI-based mobile application capable of recognizing and translating Indonesian Sign Language (BISINDO) gestures in real time. The system integrates the YOLOv8 model for gesture detection within a mobile environment. The YOLOv8n model was trained using the BISINDO dataset and evaluated through precision, recall, and mean average precision (mAP) metrics. The model obtained a mAP@50 of 98% under the given experimental settings. These results indicate that the YOLOv8 architecture can be applied to mobile-based real-time gesture recognition and may serve as a foundation for future research on assistive communication technologies.

Downloads

Download data is not yet available.

References

Abhinand, K., Nair, A. B., Dhananjay, C., Hamza, H., MohammedFawaz, J., RahmaFahim, K., & AnoopV., S. (2024). Malayalam Sign Language Identification using Finetuned YOLOv8 and Computer Vision Techniques. ArXiv, abs/2405.06702. https://api.semanticscholar.org/CorpusID:269757552

Ariansyah, D. S. (2024). PENDETEKSI KATA DALAM BAHASA ISYARAT MENGGUNAKAN ALGORITMA YOLO VERSI 8. Jurnal Informatika Dan Teknik Elektro Terapan, 12(3). https://doi.org/10.23960/jitet.v12i3.4904

Arrozaq, A. G., & Latipah. (2024). Implementasi Computer Vision untuk Terjemahkan Abjad Bahasa Isyarat SIBI pada Aplikasi Android. Jurnal Teknologi Dan Sistem Informasi Bisnis, 6(2), 297–303. https://doi.org/10.47233/jteksis.v6i2.1260

Deli, & Hendy. (2021). Perancangan Strategi Kreatif Rebranding Program Sarjana Sistem Informasi dengan Menggunakan Metode ADDIE. Jurnal Ilmiah Teknologi Informasi Terapan, 8(1).

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. https://books.google.co.id/books?id=omivDQAAQBAJ

Hussain, M. (2024). YOLOv5, YOLOv8 and YOLOv10: The Go-To Detectors for Real-time Vision. ArXiv, abs/2407.02988. https://api.semanticscholar.org/CorpusID:270924280

Judijanto, L., Muhammadiah, M., Utami, R. N., Suhirman, L., Laka, L., Boari, Y., Lembang, S. T., Wattimena, F. Y., Astriawati, N., & Laksono, R. D. (2024). Metodologi Research and Development: Teori dan Penerapan Metodologi RnD. PT. Sonpedia Publishing Indonesia. https://books.google.co.id/books?id=y3INEQAAQBAJ

Kementrian Sosial Republik Indonesia. (2020, October 26). Kemensos Dorong Aksesibilitas Informasi Ramah Penyandang Disabilitas. Kementrian Sosial Republik Indonesia. https://kemensos.go.id/kemensos-dorong-aksesibilitas-informasi-ramah-penyandang-disabilitas

Komisi Nasional Anti Kekerasan Terhadap Perempuan. (2020, September 24). Siaran Pers Komnas Perempuan: Pekan Tuli Internasional dan Hari Bahasa Isyarat Internasional (23 -29 September 2020). Komisi Nasional Anti Kekerasan Terhadap Perempuan. https://komnasperempuan.go.id/siaran-pers-detail/siaran-pers-komnas-perempuan-pekan-tuli-internasional-dan-hari-bahasa-isyarat-internasional-23-29-september-2020

Pramono, I. M., Niswati, imatun, & Agustina, A. (2024). MODEL PENERJEMAH BAHASA ISYARAT INDONESIA DENGAN METODE CONVOLUTIONAL NEURAL NETWORK (CNN). Seminar Nasional Riset Dan Inovasi Teknologi.

Pangestu, A. B., Muttaqin, M. R., & Sunandar, M. A. (2024). SISTEM DETEKSI BAHASA ISYARAT INDONESIA (BISINDO) MENGGUNAKAN ALGORITMA YOU ONLY LOOK ONCE (YOLO)v8. JATI (Jurnal Mahasiswa Teknik Informatika). https://api.semanticscholar.org/CorpusID:273829519

Permana, D., & Sutopo, J. (2023). APLIKASI PENGENALAN ABJAD SISTEM ISYARAT BAHASA INDONESIA (SIBI) DENGAN ALGORITMA YOLOv5 MOBILE APPLICATION ALPHABET RECOGNITION OF INDONESIAN LANGUAGE SIGN SYSTEM (SIBI) USING YOLOv5 ALGORITHM. Jurnal Simantec, 11, 230–231. https://doi.org/10.21107/simantec.v11i2.19783

Pradikja, M., Tolle, H., & Brata, K. (2018). Pengembangan Aplikasi Pembelajaran Bahasa Isyarat Berbasis Android Tablet. Jurnal Pengembangan Teknologi Informasi & Ilmu Komputer, 2, 2548–2964.

Rayanto, Y. H., Rokhmawan, T., & Maulana, M. Z. A. S. (2020). PENELITIAN PENGEMBANGAN MODEL ADDIE DAN R2D2: TEORI & PRAKTEK. Lembaga Academic & Research Institute. https://books.google.co.id/books?id=pJHcDwAAQBAJ

Saleh, F., Owon, R. A. S., Muhammadiah, M., Palip, R. E. A., Launingtia, I. G. A. N., & Leu, Y. Y. M. (2024). INOVASI PEMBELAJARAN BAHASA TEORI DAN PRAKTIK (N. Mayasari & P. A.-V. Maqfirah, Eds.; Pertama). CV WIDINA MEDIA UTAMA. https://repository.penerbitwidina.com/publications/569648/inovasi-pembelajaran-bahasa-teori-dan-praktik#cite