Comparative Analysis of Adam and RMSprop Optimizers on Bi-LSTM Models for Indonesian–Ngapak Translation

Hery Setiyawan Jordi; Muhammad Zakariyah

doi:10.58526/jsret.v4i4.923

Authors

Hery Setiyawan Jordi Universitas Teknologi Yogyakarta
Muhammad Zakariyah Universitas Teknologi Yogyakarta

DOI:

https://doi.org/10.58526/jsret.v4i4.923

Keywords:

Neural Machine Translation, Bi-LSTM, Adam, RMSprop, Ngapak Language

Abstract

This study analyzes the performance comparison between the Adam and RMSprop optimization algorithms in training a Bidirectional Long Short-Term Memory (Bi-LSTM) model with an Attention mechanism for Indonesian–Ngapak language translation. A parallel corpus of 23,592 sentence pairs was used, divided into training, validation, and testing datasets. The experimental results show that the Adam optimizer achieved faster convergence with a validation accuracy of 95.5%, validation loss of 0.43, and BLEU-1 to BLEU-4 scores of 0.8775, 0.8317, 0.7887, and 0.7393, respectively. In contrast, RMSprop reached 93.6% validation accuracy, 0.49 validation loss, and BLEU scores of 0.8284, 0.7636, 0.7034, and 0.6384. These results indicate that Adam offers higher efficiency and adaptability in optimizing neural parameters compared to RMSprop. Overall, this research contributes to the development of Neural Machine Translation for low-resource local languages while supporting the preservation of Ngapak as part of Indonesia’s linguistic heritage.

Downloads

Download data is not yet available.

References

Abdulkadirov, R., & Lyakhov, P. (2023). Survey of Optimization Algorithms in Modern Neural Networks. 1–37.

Abidin, Z., Sucipto, A., & Budiman, A. (2018). PENERJEMAHAN KALIMAT BAHASA LAMPUNG-INDONESIA DENGAN PENDEKATAN NEURAL MACHINE TRANSLATION BERBASIS ATTENTION TRANSLATION OF SENTENCE LAMPUNG-INDONESIAN LANGUAGES WITH NEURAL MACHINE TRANSLATION ATTENTION BASED. 06(02), 191–206.

Ahda, F. A., Wibawa, A. P., Prasetya, D. D., & Sulistyo, D. A. (2024). Comparison of Adam Optimization and RMSprop in Minangkabau- Indonesian Bidirectional Translation with Neural Machine Translation. International Journal on Informatics Visualization, 8(1), 231–238. https://doi.org/10.62527/joiv.8.1.1818

Araabi, A., & Monz, C. (2020). Optimizing Transformer for Low-Resource Neural Machine Translation. COLING 2020 - 28th International Conference on Computational Linguistics, Proceedings of the Conference, 2019, 3429–3435. https://doi.org/10.18653/v1/2020.coling-main.304

Buchanan, L. (2023). A new condition on the Jones polynomial of a fibered positive link. Journal of Knot Theory and Its Ramifications, 32(14), 1–12. https://doi.org/10.1142/S0218216523500797

CAI, G., & ZHU, J. (2025). Interpreting Attention Mechanisms of NMT with Linguistic Features. IEICE Transactions on Information and Systems. https://doi.org/10.1587/transinf.2024edp7292

Elharrouss, O., Mahmood, Y., Bechqito, Y., Serhani, M. A., Badidi, E., Riffi, J., & Tairi, H. (2025). Task-based Loss Functions in Computer Vision: A Comprehensive Review. http://arxiv.org/abs/2504.04242

Elshamy, R., Abu-Elnasr, O., Elhoseny, M., & Elmougy, S. (2023). Improving the efficiency of RMSProp optimizer by utilizing Nestrove in deep learning. Scientific Reports, 13(1), 1–16. https://doi.org/10.1038/s41598-023-35663-x

Fauziyah, Y., Ilyas, R., & Kasyidi, F. (2022). MESIN PENTERJEMAH BAHASA INDONESIA-BAHASA SUNDA MENGGUNAKAN RECURRENT NEURAL NETWORKS. 16, 313–322.

Gunawan, W., Sujaini, H., & Tursina, T. (2021). Analisis Perbandingan Nilai Akurasi Mekanisme Attention Bahdanau dan Luong pada Neural Machine Translation Bahasa Indonesia ke Bahasa Melayu Ketapang dengan Arsitektur Recurrent Neural Network. Jurnal Edukasi Dan Penelitian Informatika (JEPIN), 7(3), 488. https://doi.org/10.26418/jp.v7i3.50287

Hosseinzadeh, R., & Sadeghzadeh, M. (2025). Attention Mechanisms in Transformers: A General Survey. Technology Journal of Artificial Intelligence and Data Mining, 13(3), 359–368. https://doi.org/10.22044/jadm.2025.15584.2679

Indonesia-banyumasan, B., Wijanarko, A., Nisa, A., Haura, A., Puspitaningrum, I., & Surya, D. I. (2023). Model Recurrent Neural Network-Gated Recurrent Unit untuk Membangun Mesin Penerjemah. 218–226. https://doi.org/10.30864/eksplora.v13i2.977

Israr, H., Khan, S. A., Tahir, M. A., Shahzad, M. K., Ahmad, M., & Zain, J. M. (2023). Neural Machine Translation Models with Attention-Based Dropout Layer. Computers, Materials and Continua, 75(2), 2981–3009. https://doi.org/10.32604/cmc.2023.035814

Keskar, N. S., & Socher, R. (2017). Improving Generalization Performance by Switching from Adam to SGD. 1. http://arxiv.org/abs/1712.07628

Lee, S., Lee, J., Moon, H., Park, C., Seo, J., Eo, S., Koo, S., & Lim, H. (2023). A Survey on Evaluation Metrics for Machine Translation. Mathematics, 11(4), 1–22. https://doi.org/10.3390/math11041006

Li, C., & Flanigan, J. (2022). Improving Neural Machine Translation with the Abstract Meaning Representation by Combining Graph and Sequence Transformers. Dlg4nlp, 12–21.

Li, R., & Liu, R. (2025). The Central Role of Adaptive Optimization Algorithms in Deep Learning: A Cross-Domain Survey from CNNs to Transformers. Applied and Computational Engineering, 158(1), 1–10. https://doi.org/10.54254/2755-2721/2025.tj23299

Loshchilov, I., & Hutter, F. (2019). Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019.

Malladi, S., Lyu, K., Panigrahi, A., & Arora, S. (2022). On the SDEs and Scaling Rules for Adaptive Gradient Algorithms. Advances in Neural Information Processing Systems, 35.

Marco, M. W. Di, & Fraser, A. (2022). Findings of the WMT 2022 Shared Tasks in Unsupervised MT and Very Low Resource Supervised MT. Conference on Machine Translation - Proceedings, 801–805.

Nugroho, C., & Kusuma, I. P. (2023). Identitas Budaya Banyumasan dalam Dialek Ngapak. Jurnal Ilmu Komunikasi, 21(2), 333. https://doi.org/10.31315/jik.v21i2.4556

Puspitaningrum, D. (2021). A Study of English-Indonesian Neural Machine Translation with Attention (Seq2Seq, ConvSeq2Seq, RNN, and MHA): A Comparative Study of NMT on English-Indonesian. ACM International Conference Proceeding Series, 271–280. https://doi.org/10.1145/3479645.3479703

Putri, F. I., Wibawa, A. P., & Collante, L. H. (2024). Refining the Performance of Indonesian-Javanese Bilingual Neural Machine Translation Using Adam Optimizer. ILKOM Jurnal Ilmiah, 16(3), 271–282. https://doi.org/10.33096/ilkom.v16i3.2467.271-282

Review, A. N. (2025). Lingua : Journal of Linguistics and Language. 2, 77–88.

Rotskoff, G., & Vanden-Eijnden, E. (2022). Trainability and Accuracy of Artificial Neural Networks: An Interacting Particle System Approach. Communications on Pure and Applied Mathematics, 75(9), 1889–1935. https://doi.org/10.1002/cpa.22074

Sihite, M. R., & Sibarani, B. (2024). Technology and Language Revitalization in Indonesia: A Literature Review of Digital Tools for Preserving Endangered Languages. International Journal of Educational Research Excellence (IJERE), 3(2), 610–620. https://doi.org/10.55299/ijere.v3i2.988

Tonja, A. L., Kolesnikova, O., Gelbukh, A., & Sidorov, G. (2023). Low-Resource Neural Machine Translation Improvement Using Source-Side Monolingual Data. Applied Sciences (Switzerland), 13(2). https://doi.org/10.3390/app13021201

Wang, R., Tan, X., Luo, R., Qin, T., & Liu, T. Y. (2021). A Survey on Low-Resource Neural Machine Translation. IJCAI International Joint Conference on Artificial Intelligence, 4636–4643. https://doi.org/10.24963/ijcai.2021/629

Wu, Y., & Xing, Y. (2024). Efficient Machine Translation with a BiLSTM-Attention Approach. http://arxiv.org/abs/2410.22335

Xie, T., Ding, W., Zhang, J., Wan, X., & Wang, J. (2023). Bi-LS-AttM: A Bidirectional LSTM and Attention Mechanism Model for Improving Image Captioning. Applied Sciences (Switzerland), 13(13). https://doi.org/10.3390/app13137916

Xie, Z., Wang, X., Zhang, H., Sato, I., & Sugiyama, M. (2022). Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentum. Proceedings of Machine Learning Research, 162, 24430–24459.

Zou, D., Cao, Y., Li, Y., & Gu, Q. (2023). Understanding the Generalization of Adam in Learning Neural Networks With Proper Regularization. 11th International Conference on Learning Representations, ICLR 2023.