INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4050
www.rsisinternational.org
An Improved Hybrid Encryption Scheme Based on the Sequence of
Reduced Residue Systems
Muhammad A. H.
1
, Ibrahim A. A.
2
, Garba A. I.
2
, Sarki M. N.
3
, Mua’azu S. B.
3
, Abubakar S. F.
3
, Shehu
S.
4
, James T. O.
3
and Abubakar T. U.
5
1
Department of Science, Mathematics Unit, State Collage of Basic & Remedial Studies, Sokoto.
2
Department of Mathematics, UsmanuDanfodio University Sokoto.
3
Department of Mathematics, Abdullahi Fodio University of Science and Technology, Aliero.
4
Department of Mathematics, Sokoto State University.
5
Department of Mathematics, ShehuShagari College of Education, Sokoto.
DOI: https://dx.doi.org/10.51244/IJRSI.2025.1210000348
Received: 10 November 2025; Accepted: 16 November 2025; Published: 22 November 2025
ABSTRACT
The rapid growth of digital communication has created an urgent demand for advanced cryptographic
techniques that ensure both security and efficiency. As reliance on digital platforms increases, so does the risk
of cyber threats and data breaches. Robust cryptography is, therefore, essential to protect sensitive information
and maintain trust in digital transactions. This research proposes an improved composed hybrid cryptosystem
that integrates Transposition, Caesar, and Hill ciphers, followed sequentially by RSA encryption. The study
examines how a hybrid of four ciphers can be attacked when treated as a composite function. To further
enhance security, a sequence of Reduced Residue System (RRS) values was introduced to replace ASCII
characters after the third cipher (Hill cipher), adding an additional layer of residue-based encryption before the
final RSA stage. The findings demonstrate that the improved hybrid cryptosystem significantly enhances data
security and key generation efficiency, adding a new level of complexity that makes it more challenging for
attackers to guess or compute decryption keys.
Keywords: Hybrid, Encryption, Decryption, Attacks and RSA.
INTRODUCTION
Cryptography is the study of secure communication techniques that allow the sender and intended recipient of
a message to view and understand its contents, (Ibrahim, et al., 2021). In today's digital world, the need for
robust data security has never been more critical as now. Information exchange proliferates across various
platforms and networks, safeguarding sensitive data from unauthorized access, modification, or disclosure
becomes paramount, (Shehu et al., 2023). Individual encryption methods, while effective, often face challenges
related to computational efficiency, key management, and vulnerable to attacks, (Yao & Su, 2021). Hybrid
encryption systems have emerged as a widely adopted solution to these challenges, (Manna, et al., 2017).
Recently, numerous research endeavours have focused on enhancing security using the hybrid cryptosystem.
For instance, Rufa’i et al., (2020) integrated RSA, Shifting, and Hill ciphers to enhance security and
robustness, while Hassan, Garko, et al., (2023) combined Hill and Transposition ciphers to improve data
protection. Despite these advancements, vulnerabilities remain in existing hybrid cryptosystems. This paper
aims to enhance data security by improving a hybrid encryption system that sequentially integrates
Transposition, Caesar, Hill, and RSA ciphers, using a sequence of Reduced Residue Systems (RRS) to replace
ASCII characters.
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4051
www.rsisinternational.org
Integrating an RRS sequence into a hybrid cryptosystem enhances both data security and efficiency by
combining the strengths of hybrid encryption and residue-based techniques. This modification addresses
vulnerabilities found in the existing system and strengthens its resistance to potential attacks. The additional
layer of complexity introduced by RRS makes it considerably more difficult for attackers to decipher
ciphertexts.
LITERATURE REVIEW
The concept of hybrid cryptography has attracted considerable attention in recent years due to its ability to
provide enhanced security features. Some of the most relevant works are:
Rufa’i et al. (2020) combined three ciphersRSA, Shifting, and Hilland represented them as bijective
functions. The composition of these functions resulted in an improved hybrid cipher system with enhanced
encryption performance.
Khan, Pradhan, & Chandavarkar (2021) proposed a hybrid algorithm based on the GuillouQuisquater scheme
and RSA. Their model aims to ensure data integrity through RSA-based key generation, while the Guillou
Quisquater component handles integrity and confidentiality.
Suhasini and Bushra (2021) introduced a three-level encryption technique designed to overcome the limitations
of single-key encryption by combining the Advanced Encryption Standard (AES), Data Encryption Standard
(DES), and RSA. This multi-tier approach provides greater resistance against brute-force and key-compromise
attacks.
Saja, Zaynab, and Jaafer (2023) proposed a two-layer hybrid cryptosystem. In the first layer, plaintext is
encrypted using a modified Playfair cipher to produce ciphertext, which is subsequently re-encrypted using
RSA to generate the final ciphertext. This layered design enhances overall data confidentiality.
Prakash, Saeed, Rajan, Mohammad, and Ahmed (2023), in their paper presented a scheme combining RSA
with a Simple Symmetric Key (SSK) algorithm.
Susmitha, Kumar, and Bulla (2023) demonstrated that a hybrid cryptographic approach combining AES and
ElGamal enhances file security by integrating symmetric and asymmetric techniques. Their method ensures
confidentiality and integrity for stored data and provides valuable insights into key management strategies.
Hassan et al. (2023) enhanced data security using a combination of Hill and Transposition ciphers. In their
method, plaintext is first encrypted with the Hill cipher and then re-encrypted with the Transposition cipher,
resulting in a more complex ciphertexts.
Ariffin, Wijonarko, Suwarno, and Kristianto (2024) combined the Unimodular Hill Cipher with RSA to
produce a more secure text encryption system.
Despite these significant contributions, hybrid cryptosystems remain susceptible to certain forms of attack.
Consequently, there is a need for further improvement. This study aims to advance the composed hybrid
cryptosystem by incorporating a sequence of Reduced Residue Systems (RRS) to replace ASCII characters,
thereby increasing security complexity.
METHODOLOGY
The methodology employed in this study incorporates several fundamental concepts of number theory,
including the Greatest Common Divisor (GCD), Euler’s Totient Function 󰇛󰇜, Congruence, and the concept
of the Reduced Residue System (RRS), as well as the American Standard Code for Information Interchange
(ASCII) table. Furthermore, it integrates four cryptographic techniques: Transposition cipher, Caesar cipher,
Hill cipher, and RSA algorithm, alongside the method of composing multiple ciphers.
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4052
www.rsisinternational.org
ASCII Table
The complete 7-bit ASCII table and corresponding decimal equivalents are presented in table 1.
Table 1: ASCII Table (Google Search, 2024)
Dec
Chr
Dec
Chr
Dec
Chr
Dec
Chr
Dec
Chr
0
NUL
26
SUB
52
4
78
N
104
h
1
SOH
27
ESC
53
5
79
O
105
i
2
STX
28
FS
54
6
80
P
106
j
3
ETX
29
GS
55
7
81
Q
107
k
4
EOT
30
RS
56
8
82
R
108
l
5
ENQ
31
US
57
9
83
S
109
m
6
ACK
32
58
:
84
T
110
n
7
BEL
33
!
59
;
85
U
111
o
8
BS
34
"
60
<
86
V
112
p
9
HT
35
#
61
=
87
W
113
q
10
LF
36
$
62
>
88
X
114
r
11
VT
37
%
63
?
89
Y
115
s
12
FF
38
&
64
@
90
Z
116
t
13
CR
39
'
65
A
91
[
117
u
14
SO
40
(
66
B
92
\
118
v
15
SI
41
)
67
C
93
]
119
w
16
DLE
42
*
68
D
94
^
120
x
17
DC1
43
+
69
E
95
_
121
y
18
DC2
44
,
70
F
96
`
122
z
19
DC3
45
-
71
G
97
a
123
{
20
DC4
46
.
72
H
98
b
124
|
21
NAK
47
/
73
I
99
c
125
}
22
SYN
48
0
74
J
100
d
126
~
23
ETB
49
1
75
K
101
e
127
DEL
24
CAN
50
2
76
L
102
f
25
EM
51
3
77
M
103
g
Fundamental Concepts of Number Theory
This section presents key number theory concepts utilized in the study.
Greatest Common Divisor (GCD)
If m and n are integers, a positive integer d is the GCD of m and n if d divides both m and n, and d is the
greatest among all common divisors of m and n. For instance, the GCD of 24 and 60 is 12, denoted as (24, 60)
= 12.
Relatively Prime Integers
Two or more integers are said to be relatively prime if their GCD is 1, (Hardy & Wright, 2008). For instance,
the factors of 8 are {1, 2, 4, 8}, and the factors of 15 are {1, 3, 5, 15}. Thus, GCD (8, 15) = 1, meaning 8 and
15 are relatively prime.
Euler's Totient Function 󰇛󰇜
Euler's Totient Function, denoted by 󰇛󰇜, counts the number of positive integers less than or equal to N that
are relatively prime to N, (Rosen, 2011). Mathematically,
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4053
www.rsisinternational.org
󰇛󰇜, such that GCD󰇛󰇜, .
For instance: 󰇛󰇜, as numbers less than or equal to  are: , for which integers
≤ 10 that are coprime with 10 are: 1, 3, 7, 9.
Similarly, 󰇛󰇜, because the integers ≤ 12 that are coprime with 12 are {1, 5, 7, 11}.
It follows that
󰇛
󰇜

󰇛
󰇜

󰇛
󰇜

󰇛
󰇜

󰇛
󰇜
 thus, in general
󰇛
󰇜
, where  is prime.
Properties of 󰇛󰇜
The Euler Totient function has many useful properties:
(i) is multiplicative when 
󰇛
󰇜
 thus,
󰇛

󰇜
󰇛
󰇜
󰇛
󰇜
For instance,
󰇛

󰇜
󰇛
󰇜

󰇛
󰇜
󰇛
󰇜

󰇛

󰇜
(ii) For any prime p,
󰇛
󰇜
󰇡
󰇢

󰇛
󰇜
, hence,
if
then,
󰇛
󰇜
󰇡
󰇢󰇡
󰇢󰇡
󰇢, (Rosen, 2011)
Congruence
If and are integers and is a positive integer, then is said to be congruent to modulo if divides
󰇛 󰇜. This is written as 󰇛󰇜, (Rosen, 2011).
For instance, 󰇛󰇜, since  divides  .
Least Residue
If and is the remainder when dividing n by m then is called the least residue of modulo , (Rosen,
2011).
Residue Classes
A residue class modulo is a set of integers that are congruent to each other modulo . Each residue class
contains exactly one integer from 󰇝 󰇞. For modulo 4, there are four residue classes 󰇟󰇠
󰇟󰇠 󰇟󰇠 󰇟󰇠 where
Complete Residue System
A complete residue system modulo is any set of  integers







such that


󰇛

󰇜
, (Rosen, 2011).
Reduced Residue System (RRS)
A Reduced Residue System (RRS) modulo is a set of integers that represent all possible residue classes
modulo that are relatively prime to (Rosen, 2011).
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4054
www.rsisinternational.org
A set S is an RRS modulo m if:
i)
󰇛
󰇜
󰇛
󰇜
(ii)
󰇛

󰇜
(iii) GCD
󰇛
󰇜
The set 󰇝󰇞 forms an RRS modulo 9; since
(i)
󰇛
󰇜
and
󰇛
󰇜
(ii) 󰇝󰇞󰇝󰇞
󰇛

󰇜
(iii)
󰇛
󰇜
,
3.3 Composition of Four Ciphers
The four ciphers: Transposition, Caesar, Hill, and RSA were modelled as mathematical functions and
composed sequentially using the concept of function composition.
Encryption Process
Step I: Transposition Cipher () - Rearrange the plaintext according to a specific key to obtain ciphertexts
as:
󰇛󰇜
that is 
Step II: Caesar Cipher () - Shift each letter in
by a fixed number 󰇛
󰇜 of positions down the ASCII table
to obtain

as:
󰇛󰇛󰇜󰇜 that is

.
Step III: Convert

into its ASCII numerical equivalents, denoted

󰆒
.
Step IV: Hill Cipher (): Encrypt

󰆒
using an invertible key matrix
to obtain

as:
󰇛󰇛󰇛󰇜󰇜󰇜󰇛󰇛󰇜󰇜
that is

󰆒

,
Step V: RSA Encryption () - Encrypt

using modular exponentiation with public key to obtain

as:
󰇛󰇛󰇛󰇛󰇜󰇜󰇜󰇜󰇛󰇛󰇛󰇛󰇜󰇜󰇜
󰇜󰇛󰇜 that is


Thus, the composed cipher is:
󰇛
󰇜
󰇛
󰇛
󰇛
󰇜
󰇜
󰇜
Where:

ciphertexts from Caesar encryption

󰆒
ASCII numerical equivalent of



ciphertexts from Hill encryption

ciphertexts from RSA encryption
 the RSA modulus
Decryption Process
Decryption reverses the encryption steps as follows:
Step I: RSA Decryption - decrypt

using private key to obtain

as:

󰇛

󰇜

󰇛󰇜


INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4055
www.rsisinternational.org
Step II: Hill Decryption - Multiply

inverse key matrix

to recover

󰆒
as:

󰇛

󰇜




󰆒
Step III: Convert

󰆒
to ASCII characters to obtain

Step IV: Caesar Decryption - shift each character of

upward by
positions:

󰇛

󰇜
󰇛

󰇜󰇛

󰇜

Step V: Transposition Decryption - rearrange
back to its original order using the transposition key
:

󰇛
󰇜
Hence, the complete decryption process is represented as:



󰇡


󰇛

󰇜
󰇢
Where:

RSA decryption function

Hill decryption function

Caesar decryption function

Transposition decryption function

inverse of Hill key matrix 

ciphertexts of the composed system
RESULT AND ANALYSIS
This section presents a comprehensive analysis of the findings, highlighting the performance and security
enhancements achieved through the proposed approach.
Cryptanalysis of the Combined Four Ciphers
If an attacker knows the cipher order (Transposition Caesar Hill RSA), the hybrid can possibly be
attacked by peeling the layers in reverse: first break RSA, then the Hill cipher, then Caesar, and finally the
transposition, as illustrated below:
Step I: Apply RSA cryptanalytic methods to recover the decrypted block stream
Step II: Use Hill-specific attacks (known-plaintext, linear algebraic key recovery, or brute force) on the result
to recover the message before Hill.
Step III: Break the Caesar shift via frequency analysis, known plaintext, or simple brute force to get the pre-
Caesar text.
Step IV: Determine and reverse the transposition pattern to reconstruct the original plaintext.
Mathematically, the process can be illustrated as follows:
Step I: RSA Attack:

󰇛

󰇜


(1)
Step II: Hill Attack:

󰇛

󰇜
(From equation 1)


(2)
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4056
www.rsisinternational.org
Step III: Caesar Attack:

󰇛

󰇜
(From equation 2)

(3)
Step IV: Transposition Attack:

󰇛
󰇜
(From equation 3)

Where:
Ciphertext obtained using the Transposition method

Ciphertext obtained after applying the Caesar cipher

Ciphertexts obtained after applying the Hill cipher

Ciphertexts obtained after applying the RSA algorithm
,
,
,
Attacks on RSA, Hill, Caesar, and Transposition ciphers respectively
Generating RRS Sequence
To generate a Sequence of Reduce Residue System 󰇛
󰇜 a modulus is selected such that the number of
elements in the set corresponds to
󰇛
󰇜
, where  represents the total number of ASCII characters,
and
󰇛
󰇜
denotes Euler’s Totient Function.
Let 
 
Then: 
󰇛

󰇜
󰇛

󰇜

󰇛
󰇜
󰇛
󰇜
󰇛

󰇜
 

Thus, the RRS sequence modulo 255 can be represented as:






 

Assigning this generated RRS sequence
to the corresponding ASCII character values allows the
replacement of standard ASCII encoding with residue-based encoding. This substitution increases encryption
complexity, as each ASCII value is represented by its equivalent in the residue sequence, thus enhancing
security before the final RSA encryption phase.
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4057
www.rsisinternational.org
Table 2: Integrated ASCII Table with RRS
ASCII
Value
RRS
Value
ASCII
Value
Chr
RRS
Value
ASCII
Value
Chr
RRS
Value
0
1
43
+
88
86
V
172
1
2
44
,
89
87
W
173
2
4
45
-
91
88
X
176
3
7
46
.
92
89
Y
178
4
8
47
/
94
90
Z
179
5
11
48
0
97
91
[
181
6
13
49
1
98
92
\
182
7
14
50
2
101
93
]
184
8
16
51
3
103
94
^
188
9
19
52
4
104
95
_
191
10
22
53
5
106
96
`
193
11
23
54
6
107
97
a
194
12
26
55
7
109
98
b
196
13
28
56
8
112
99
c
197
14
29
57
9
113
100
d
199
15
31
58
:
116
101
e
202
16
32
59
;
118
102
f
203
17
37
60
<
121
103
g
206
18
38
61
=
122
104
h
208
19
41
62
>
124
105
i
209
20
43
63
?
127
106
j
211
21
44
64
@
128
107
k
212
22
46
65
A
131
108
l
214
23
47
66
B
133
109
m
217
24
49
67
C
134
110
n
218
25
52
68
D
137
111
o
223
26
53
69
E
139
112
p
224
27
56
70
F
142
113
q
226
28
58
71
G
143
114
r
227
29
59
72
H
146
115
s
229
30
61
73
I
148
116
t
232
31
62
74
J
149
117
u
233
32
64
75
K
151
118
v
236
33
67
76
L
152
119
w
239
34
71
77
M
154
120
x
241
35
73
78
N
157
121
y
242
36
74
79
O
158
122
z
244
37
76
80
P
161
123
{
247
38
77
81
Q
163
124
|
248
39
79
82
R
164
125
}
251
40
82
83
S
166
126
~
253
41
83
84
T
167
127
DEL
254
42
86
85
U
169
Improved Composed Hybrid
The improved composed hybrid encryption method involves a two-stage processes. Firstly, the generated
sequence of Reduced Residue System (RRS) will be applied to replace the characters of the ciphertexts
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4058
www.rsisinternational.org
obtained after third encryption (Hill cipher encryption,
) with their corresponding assigned numerical values
as in Table 2. In the second stage, the resulting numerical string is encrypted using RSA encryption techniques.
Procedure for Encryption /Decryption Process of the Modified Hybrid
Encryption Process:
1. Select a transposition key
2. Rearrange the plaintext characters base on the key
to have ciphertexts
as:

󰇛
󰇜
that is
󰇛
󰇜
(4)
Where is the plaintext,
,  and is the number of characters in the plaintext.
3. Select Caesar key
4. Compute
, to obtain a new ciphertexts as:

󰇛
󰇜
that is
󰇛



󰇜
(5)
where
 .
5. Split equation (4.5) as column vectors:
󰇡


󰇢󰇡


󰇢󰇡



󰇢 (6)
6. Replace the column vectors of (4.6) with their corresponding ASCII values to have:
󰇡
󰇢󰇡
󰇢󰇡

󰇢 (7)
7. Choose an invertible key matrix




8. Compute


󰇛

󰇜
, to have:

󰇛
󰇜
that is
󰇛
󰇜
(8)
9. Convert the values of equation (4.8) to their corresponding characters in ASCII to obtain

󰇛
󰇜
(9)
10. Select
such that
󰇛
󰇜
, that is value greater than or equals to total number of characters in
ASCII.
11. Generate sequence of Reduce Residue System then, assign each character of the ASCII 󰇛󰇜 with a
sequential value of RRS, denoted as:


󰇛
󰇜
that is
󰇛
󰇜
(10)
12. Replace each letter of equation (4.9) with the corresponding value of
, to have:
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4059
www.rsisinternational.org



󰇛
󰇜

that is


󰇛
󰇜
(11)
13. Select RSA public key 󰇛󰇜 such that GCD
󰇛
󰇜
, where is the encryption key, is the RSA
modulus and
󰇛
󰇜
is the Euler number.
14. Encrypt equation (4.11) using RSA method, with a public key 󰇛󰇜, to obtain a final ciphertexts
as:
󰇛
󰇜
󰇛󰇜 (12)
Thus, the improved composed cipher becomes:
󰇛

󰇜

󰇛
󰇛
󰇛
󰇜
󰇜
󰇜
Decryption Process
The decryption process involves reversing the above encryption process:
1. Compute RSA private key using RSA key equation  
󰇛
󰇜

2. Decrypt the ciphertexts in equation (4.12) to recover

󰇛
󰇜
as in equation (11):

󰇛
󰇜󰇛
󰇜


3. Map each value of

with the corresponding value in RRS󰇛
󰇜then, replace it with corresponding values in
ASCII to recover
as:
󰇛
󰇜
(as in equation (9))
where
and ASCII characters.
4. Convert the characters in
to numerals corresponding to ASCII characters to recover
which is the Hill
ciphertexts, as:
󰇛
󰇜
(as in (8))
5. Decrypt the resulting ciphertexts
󰇛
󰇜
using inverse key matrix to recover
:


󰇛
󰇜

󰇛

󰇜
to have:

󰇡
󰇢󰇡
󰇢󰇡

󰇢 (as in equation (7))
6. Convert the values of
to corresponding characters in ASCII to recover
:
󰇛



󰇜
(as in equation (6))
7. Shift each character of
back to a fixed number of positions upward the ASCII based on the Caesar key
to recover
as:

󰇛
󰇜󰇛
󰇜󰇛󰇜.

(as in equation (4))
8. Rearrange the letters of
back to the original order using the transposition key
to recover the plaintexts
as:




󰇛
󰇜
 that is:

󰇛
󰇜
󰇛
󰇜
Thus, the combined decryption process becomes:
󰇛




󰇜

󰇛

󰇛

󰇛

󰇛
󰇜󰇜󰇜󰇜
Where:

RSA decryption function

Hill decryption function
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4060
www.rsisinternational.org

Caesar decryption function

Transposition decryption function

Inverse of Hill cipher key matrix
ASCII values of the Hill ciphertexts

RRS value of the ciphertexts
is the ciphertexts of the modified hybrid.
 RSA secret key RSA modulus
Caesar key
Application of the Improved Hybrid Cryptosystem
Suppose, to encrypt the message “The message is confidential. Please, keep it secret.
Firstly, using Transposition to encrypt the statement with the transposition key
, we have:
Table 3: Encryption Table for Transposition Cipher


The ciphertexts consists of the characters read from the top left box going down the column to have ciphertexts
as: Tsifil irhssiaekteea dlae t gce.ses.meon epee ntP, c
Secondly, encrypt the ciphertexts
using Caesar method with key,
 to obtain
as:
󰇛
󰇜
 , illustrated below:
  (From ASCII Table)
  
To have a ciphertexts
as: Wvlilo#lukvvldhnwhhd#godh#w#jfh1vhv1phrq#hshh#qwS/#f
Thirdly, encrypt ciphertexts
using Hill cipher technique with encryption matrix key
󰇡
󰇢 as:
󰇡
󰇢󰇡


󰇢
󰇛
 
󰇜
󰇡
 
 
󰇢
󰇛
 
󰇜
󰇡


󰇢
󰇛
 
󰇜
󰇡


󰇢󰇡

󰇢
󰇡
󰇢󰇡


󰇢
󰇛
 
󰇜
󰇡


󰇢
󰇛
 
󰇜
󰇡


󰇢
󰇛
 
󰇜
󰇡


󰇢
󰇝
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4061
www.rsisinternational.org
To obtain a new ciphertexts
as:
03106711004712206500705703111605802800502710003403512108000412504211505307806502504209402
4106107030114048077000072028113127045015093078065035124125082123039
USCn/zABEL9USt:FSENQESCd”#yPEOT}*s5NAEM*^CANjkRSr0MNULHFSqDEL-SI]NA#|}R{‘
Fourthly, apply Sequence of Reduce Residue System (RRS) with , to obtain

as:
, to obtain

as:
06213421809424413101411306223211605801105619907107324216100825108622910615713105208618804
9211212061227043197001146058226254091031184157131073248251218247079
Next, generate RSA Public key
󰇛

󰇜
and Private key
󰇛
󰇜
as:
Let  and 
󰇛
󰇜
󰇛
󰇜󰇛
󰇜
 
with  then compute the decryption exponent , such that  
󰇛
󰇜
, with ,
󰇛
󰇜

to have .
Thus, Public key
󰇛
󰇜
is 󰇛 󰇜, Private key 󰇛
󰇛
󰇜
󰇜 is 󰇛 󰇜
Table 4: Converted ciphertexts
to RRS
ASCII
Value
031
067
110
047
122
065
007
057
031
116
058
028
005
Chr

A
BEL

t
FS

RRS
Value
062
134
218
094
244
131
014
113
062
232
116
058
011
ASCII
Value
027
100
034
035
121
080
004
125
042
115
053
078
065
Chr


󰇞
RRS
Value
056
199
071
073
242
161
008
251
086
229
106
157
131
ASCII
Value
025
042
094
024
102
101
030
114
020
099
000
072
028
Chr


j
K



RRS
Value
052
086
188
049
211
212
061
227
043
197
001
146
058
ASCII
Value
113
127
045
015
093
078
065
035
124
101
110
123
039
Chr


󰇠
󰇞
R
󰇝
RRS
Value
226
254
091
031
184
157
131
073
248
251
218
247
079
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4062
www.rsisinternational.org
From Table 4, the new ciphertexts becomes:

06213421809424413101411306223211605801105619907107324216100825108622910615713105208618804
9211212061227043197001146058226254091031184157131073248251218247079
Finally, encrypt ciphertexts

using the public key to obtain final ciphertexts
as:
󰇛

󰇜
󰇛
 
󰇜
that is

  

  

  
To have final ciphertexts:
1563143290360650412953091561053461311221291262661462002532193782733213872680412562733
07025284270379320274124001311131061254252380023268041146328378329342037
Decryption process
Reverse the process of the above encryption process to recover the original message.
SUMMARY AND CONCLUSION
This study introduces an improved hybrid cryptosystem that inserts a Reduced Residue System (RRS)
substitution between Hill and RSA. The implemented pipeline is Transposition Caesar Hill RRS
RSA (baseline omits RRS). Using modulus M = 255, Hill outputs (mapped to ASCII indices) are replaced by
residues from a secret RRS mapping, severing direct numeric correlation with ASCII before RSA encryption.
Even if RSA is compromised, an adversary recovers only RRS-coded residues, forcing a combinatorial
mapping reconstruction rather than direct plaintext recovery. The scheme was tested by encrypting and
decrypting a short message; empirical results show the RRS layer increases attacker workload and enhances
key-generation/management, security, performance, and overall efficiencyproducing a more robust hybrid
encryption solution.
REFERENCES
1. Ariffin, S., Wijonarko, D., Suwarno, & Kristianto, E. S. (2024). Application of unimodular Hill cipher
and RSA methods to text encryption algorithms using Python. Journal of Computer Science, 20(5),
548563. https://doi.org/10.3844/jcssp.2024.548.563
2. ASCII Table. (2024). Google Search. https://www.google.com/search?q=ascii+table&client=ms-
opera-mini-android&channel=new&tpsf=opiminienpf
3. Hardy, G. H., & Wright, E. M. (2008). An introduction to the theory of numbers (6th ed.). Oxford
University Press.
4. Hassan, A., Garko, A., Sani, S., Abdullahi, U., & Sahalu, S. (2023). Combined techniques of Hill
cipher and transposition cipher. Journal of Mathematics Letters, 1(1), 5764.
https://www.scipublications.com/journal/index.php/jml/article/view/822
5. Ibrahim, A. A., Muhammad, A. H., Shehu, S., Abubakar, T. U., Zaid, I., & Bello, U. (2021).
Cryptanalysis on RSA using decryption exponent. IOSR Journal of Mathematics (IOSR-JM), 17(5),
18.
6. Khan, H. K., Pradhan, R., & Chandavarkar, B. R. (2021). Hybrid cryptography for cloud computing.
In 2nd International Conference for Emerging Technology (INCET) (pp. 15).
INTERNATIONAL JOURNAL OF RESEARCH AND SCIENTIFIC INNOVATION (IJRSI)
ISSN No. 2321-2705 | DOI: 10.51244/IJRSI |Volume XII Issue X October 2025
Page 4063
www.rsisinternational.org
https://doi.org/10.1109/INCET51464.2021.9456210
7. Manna, S., Prajapati, M., Sett, A., Banerjee, J., and S. Dutta, (2017). Design and implementation of a
two-layered hybrid cryptosystem. Third International Conference on Research in Computational
Intelligence and Communication Networks (ICRCICN), 327331, DOI:
10.1109/ICRCICN.2017.8234529
8. Prakash, K., Saeed, Q. Y. A., Rajan, J., Mohammad, H., & Ahmed, A. S. M. (2023). A hybrid
encryption system for communication and financial transactions using RSA and a novel symmetric
key algorithm. Bulletin of Electrical Engineering and Informatics, 12(2), 11481158.
https://doi.org/10.11591/eei.v12i2.4967
9. Rosen, K. H. (2011). Elementary number theory and its applications (6th ed.). Addison-Wesley
Publishing Company, Pearson.
10. Rufa’i, A., Balarabe, A. T., Muazu, I., & Sirajo, M. (2020). Formulation of an improved hybrid cipher
system. International Journal of Innovative Science and Research Technology, 5(12).
11. Saja, M. S., Zaynab, A. A., & Jaafer, A. H. (2023). Proposed hybrid cryptosystems based on
modifications of Playfair cipher and RSA cryptosystem. Baghdad Science Journal.
https://doi.org/10.21123/bsj.2023.8361
12. Shehu, S., Abdullahi, H., Ibrahim, A. A., & Ahmad, R. (2023). Breaking modulus of the form
with improved polynomial attacks. Journal of Advances in Mathematics and Computer Science,
38(8), 3346. https://doi.org/10.9734/JAMCS/2023/v38i81788
13. Suhasini, C. A., & Bushra, S. N. (2021). Securing of cloud data with duplex data encryption
algorithm. In 5th International Conference on Computing Methodologies and Communication 252
256. https://doi.org/10.1109/ICCMC51019.2021.9418247
14. Susmitha, C., Kumar, S. K., & Bulla, S. (2023). Hybrid cryptography for secure file storage. In 7th
International Conference on Computing Methodologies and Communication (ICCMC).
https://doi.org/10.1109/ICCMC56507.2023.10084073.
15. Yao, F. and Su, J., (2021). Hybrid Encryption Scheme for Hospital Financial Data Based on Noekeon
Algorithm. Security and Communication Network. DOI: 10.1155/2021/7578752.