An Improved Hybrid Encryption Scheme Based on the Sequence of
Reduced Residue Systems
Muhammad A. H.
1
, Ibrahim A. A.
2
, Garba A. I.
2
, Sarki M. N.
3
, Mua’azu S. B.
3
, Abubakar S. F.
3
, Shehu
S.
4
, James T. O.
3
and Abubakar T. U.
5
1
Department of Science, Mathematics Unit, State Collage of Basic & Remedial Studies, Sokoto.
2
Department of Mathematics, UsmanuDanfodio University Sokoto.
3
Department of Mathematics, Abdullahi Fodio University of Science and Technology, Aliero.
4
Department of Mathematics, Sokoto State University.
5
Department of Mathematics, ShehuShagari College of Education, Sokoto.
DOI: https://dx.doi.org/10.51244/IJRSI.2025.1210000348
Received: 10 November 2025; Accepted: 16 November 2025; Published: 22 November 2025
ABSTRACT
The rapid growth of digital communication has created an urgent demand for advanced cryptographic
techniques that ensure both security and efficiency. As reliance on digital platforms increases, so does the risk
of cyber threats and data breaches. Robust cryptography is, therefore, essential to protect sensitive information
and maintain trust in digital transactions. This research proposes an improved composed hybrid cryptosystem
that integrates Transposition, Caesar, and Hill ciphers, followed sequentially by RSA encryption. The study
examines how a hybrid of four ciphers can be attacked when treated as a composite function. To further
enhance security, a sequence of Reduced Residue System (RRS) values was introduced to replace ASCII
characters after the third cipher (Hill cipher), adding an additional layer of residue-based encryption before the
final RSA stage. The findings demonstrate that the improved hybrid cryptosystem significantly enhances data
security and key generation efficiency, adding a new level of complexity that makes it more challenging for
attackers to guess or compute decryption keys.
Keywords: Hybrid, Encryption, Decryption, Attacks and RSA.
INTRODUCTION
Cryptography is the study of secure communication techniques that allow the sender and intended recipient of
a message to view and understand its contents, (Ibrahim, et al., 2021). In today's digital world, the need for
robust data security has never been more critical as now. Information exchange proliferates across various
platforms and networks, safeguarding sensitive data from unauthorized access, modification, or disclosure
becomes paramount, (Shehu et al., 2023). Individual encryption methods, while effective, often face challenges
related to computational efficiency, key management, and vulnerable to attacks, (Yao & Su, 2021). Hybrid
encryption systems have emerged as a widely adopted solution to these challenges, (Manna, et al., 2017).
Recently, numerous research endeavours have focused on enhancing security using the hybrid cryptosystem.
For instance, Rufa’i et al., (2020) integrated RSA, Shifting, and Hill ciphers to enhance security and
robustness, while Hassan, Garko, et al., (2023) combined Hill and Transposition ciphers to improve data
protection. Despite these advancements, vulnerabilities remain in existing hybrid cryptosystems. This paper
aims to enhance data security by improving a hybrid encryption system that sequentially integrates
Transposition, Caesar, Hill, and RSA ciphers, using a sequence of Reduced Residue Systems (RRS) to replace
ASCII characters.