Classical Ciphers and Cryptanalysis

Introduction and Context

Cryptography is the practice of protecting information by transforming readable data, called plaintext, into an unintelligible form, called ciphertext. The reverse transformation is decryption. Classical ciphers are historically important because they introduce the central design choices that still influence modern cryptography: substitution, permutation, key management, attack resistance, and the distinction between a secret algorithm and a secret key.

Classical systems were generally designed for manual or mechanical operation. They are not suitable for protecting contemporary confidential information because their key spaces are small, their patterns are often visible, and their mathematical structures can usually be attacked efficiently. Nevertheless, studying them gives cybersecurity students a practical foundation for understanding cryptanalysis, information leakage, brute-force search, and the evolution toward modern algorithms.

A cryptographic system normally consists of a plaintext message P, an encryption algorithm E, a key K, and a ciphertext C. The encryption and decryption relationships are:

Here, E_K means encryption using key K, and D_K means decryption using K or, in some systems, a related decryption key. These expressions describe the intended operation; security depends on whether an attacker can recover P or K without authorization.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain plaintext, ciphertext, keys, encryption, decryption, cryptanalysis, and cryptology.
2. Distinguish substitution ciphers from transposition ciphers.
3. Apply Caesar, affine, monoalphabetic, Vigenère, and columnar transposition methods.
4. Explain frequency analysis, brute-force attacks, known-plaintext attacks, and related methods.
5. Use Kerckhoffs's principle to evaluate cryptographic design.
6. Analyse why classical ciphers fail against modern adversaries.
7. Relate historical cipher weaknesses to the design goals of modern cryptography.

1. Essential Terminology and the Cryptographic Model

A message before encryption is plaintext. The encrypted output is ciphertext. A key is a value that controls the transformation. Encryption converts plaintext to ciphertext, while decryption recovers plaintext. An adversary who studies ciphertext and attempts to recover the message or key is conducting cryptanalysis. Cryptology combines cryptography and cryptanalysis.

A cipher is an algorithm for transforming information, usually using a key. A code replaces words or phrases with other symbols or expressions, whereas a cipher operates systematically on letters, symbols, or bits. In everyday language the terms are sometimes confused, but the distinction is useful academically.

Classical ciphers commonly operate on an alphabet. For illustration, assign A = 0, B = 1, ..., Z = 25. Arithmetic is performed modulo 26, meaning that values wrap around after 25. For example, 24 + 5 = 29, and 29 mod 26 = 3, corresponding to D.

The notation C = E_K(P) does not itself guarantee confidentiality. A secure system must resist realistic attacks, including guessing, statistical analysis, implementation mistakes, and stolen keys. The algorithm may be public; protection should depend primarily on the key.

2. Substitution Ciphers

A substitution cipher replaces each plaintext symbol with another symbol. The order of symbols is normally preserved, but their identities change.

2.1 Caesar Cipher

The Caesar cipher shifts each letter by a fixed amount. With letters represented by numbers, encryption is:

Decryption is:

P and C are numerical representations of a plaintext and ciphertext letter, and k is the shift, with 0 ≤ k ≤ 25. The formula applies to a 26-letter alphabet and assumes consistent letter encoding.

With k = 3, A becomes D, B becomes E, and X becomes A. Thus HELLO becomes KHOOR. Decryption subtracts 3 and restores HELLO.

The Caesar cipher has only 26 possible shifts. A cryptanalyst can try every key, so it is vulnerable to brute force. The shift k = 0 produces no encryption, and some shifts produce equivalent patterns under relabelling.

2.2 Monoalphabetic Substitution

A monoalphabetic substitution uses a scrambled alphabet. For example, the plaintext alphabet ABCDEFGHIJKLMNOPQRSTUVWXYZ might map to QWERTYUIOPASDFGHJKLZXCVBNM. Each plaintext letter always maps to the same ciphertext letter.

The number of possible full-alphabet substitutions is 26!, approximately 4.03 × 10²⁶. This appears much larger than the Caesar key space, but the cipher remains vulnerable because English letter frequencies and structural patterns are preserved. A common ciphertext symbol may represent E, T, or A; repeated symbols may represent repeated plaintext letters; and common digrams such as TH or HE may remain identifiable.

The key-space calculation is not a complete security analysis. A large nominal key space does not help if the ciphertext leaks useful statistical information.

2.3 Affine Cipher

The affine cipher combines multiplication and addition:

Here, P and C are letter values, a and b are key parameters, and a must be relatively prime to 26. This condition is required so that a has a multiplicative inverse modulo 26 and decryption is possible. The decryption formula is:

where a⁻¹ satisfies a a⁻¹ ≡ 1 mod 26. The valid values of a are 1, 3, 5, 7, 9, 11, 15, 17, 19, 21, 23, and 25. Although more complex than Caesar, affine encryption is still monoalphabetic and therefore susceptible to frequency analysis.

2.4 Homophonic Substitution

A homophonic cipher allows several ciphertext symbols to represent a frequent plaintext letter. For example, E might be represented by several symbols. This can flatten frequency distributions and make simple frequency analysis more difficult. However, homophonic systems require larger symbol sets, careful design, and sufficient ciphertext. They do not automatically provide modern security.

3. Polyalphabetic Substitution

A polyalphabetic cipher uses multiple substitution alphabets. The substitution changes according to a key or position, reducing the direct relationship between one plaintext letter and one ciphertext letter.

3.1 Vigenère Cipher

The Vigenère cipher uses a repeating keyword. If letters are represented as values from 0 to 25, encryption is:

Decryption is:

P_i is the value of the i-th plaintext letter, C_i is the corresponding ciphertext value, and K_i is the repeating key value. The formula assumes that spaces and punctuation are either removed or handled by an agreed convention.

For plaintext ATTACKATDAWN and key LEMON repeated as LEMONLEMONLE, the ciphertext is LXFOPVEFRNHR. The same plaintext letter can encrypt to different ciphertext letters because its result depends on the current key letter.

The Vigenère cipher is stronger than a Caesar cipher when the key is long and nonrepeating. With a short repeating key, however, the repetition creates exploitable structure. Kasiski examination searches for repeated ciphertext sequences and measures distances between them. Common factors of those distances may suggest the key length. Index-of-coincidence analysis provides another way to estimate periodicity.

3.2 One-Time Pad

A one-time pad is a special polyalphabetic system in which the key is truly random, at least as long as the message, used only once, and kept secret. Under these conditions it provides information-theoretic confidentiality: ciphertext alone does not reveal which plaintext is correct.

The conditions are demanding. The key must be generated randomly, distributed securely, stored safely, used once, and destroyed after use. Reusing a pad is dangerous because two ciphertexts can be combined to cancel the key, exposing relationships between the plaintexts. Practical communication systems therefore use computationally secure algorithms rather than literal one-time pads in most applications.

4. Transposition Ciphers

A transposition cipher does not replace symbols. Instead, it rearranges their positions. Letter frequencies are preserved, but the sequence and word structure are disrupted.

4.1 Rail Fence Cipher

In a rail fence cipher, plaintext is written diagonally across a selected number of rows and then read row by row. With two rails, alternating letters are placed in the first and second rows. For example, the arrangement of MEETMEATNOON with two rails is formed by writing alternating characters on two lines and concatenating the lines.

The cipher has a small number of possible rail counts and preserves letter frequencies exactly. It is therefore vulnerable to testing likely rail numbers and examining whether the resulting text forms meaningful language.

4.2 Columnar Transposition

In a columnar transposition, plaintext is written beneath a keyword, usually row by row. Columns are then read in the alphabetical order of the keyword letters. Padding may be added to complete the final row.

Suppose the key is CAT. Number its columns alphabetically: A = 1, C = 2, T = 3. The plaintext is placed under C, A, T, and columns are read in the order A, C, T. Repeated key letters require a tie-breaking convention, such as ordering equal letters from left to right.

Columnar transposition preserves the frequency of every character. Its security depends on the permutation and key length, but ordinary versions are vulnerable to anagramming, probable-word searches, and statistical analysis of language.

4.3 Product Ciphers

A product cipher combines substitution and transposition. The combination can conceal both symbol identities and positions more effectively than either operation alone. Modern block ciphers are sophisticated product-cipher designs, applying multiple rounds of substitution, permutation, mixing, and key-dependent transformation.

5. Historical Cipher Systems

5.1 The Scytale

The scytale, associated with ancient Sparta, used a strip wrapped around a rod. Writing appeared correctly ordered only when the strip was wrapped around a rod of the correct diameter. It is an early transposition idea: the symbols remain unchanged, but their spatial arrangement changes.

5.2 The Caesar Cipher and Military Communication

The Caesar cipher illustrates the historical value of secrecy and the modern weakness of small key spaces. It was appropriate in a context where many recipients were unlikely to understand cryptanalysis, but it provides no meaningful protection against systematic testing.

5.3 The Enigma Machine

The Enigma machine used rotors, a plugboard, and changing electrical paths to create a changing substitution. Rotor movement made the substitution depend on position. In practice, weaknesses included operational procedures, repeated message formats, predictable plaintext, captured material, and design limitations. Allied cryptanalysis combined mathematics, engineering, intelligence, and disciplined exploitation of operator mistakes.

The Enigma story demonstrates that cryptographic security is a system property. A complicated mechanism can fail because of key management, predictable messages, poor procedures, or implementation weaknesses.

6. Cryptanalysis

Cryptanalysis studies methods for recovering plaintext, keys, or useful information without authorized access. The attacker model matters. In a ciphertext-only attack, the adversary has only ciphertext. In a known-plaintext attack, the adversary knows some plaintext and its corresponding ciphertext. In a chosen-plaintext attack, the adversary can obtain ciphertexts for selected plaintexts. In a chosen-ciphertext attack, the adversary can obtain decryptions of selected ciphertexts. Modern cryptographic designs are expected to withstand powerful models appropriate to their use.

6.1 Frequency Analysis

Frequency analysis exploits the unequal distribution of letters in natural language. In typical English text, E, T, A, O, I, N, and other letters occur at different rates. A monoalphabetic substitution preserves these frequency relationships under relabelling.

Single-letter frequency is only a starting point. Effective analysis also considers:

• repeated-letter patterns, such as the pattern ABBA in “noon”;
• common digrams such as TH, HE, IN, and ER;
• common trigrams such as THE, AND, and ING;
• likely word lengths and spaces;
• grammatical context;
• consistency of a proposed substitution.

Frequency analysis works best on sufficiently long, ordinary-language ciphertext. It becomes less reliable with short messages, unusual vocabulary, technical terms, intentionally distorted language, or multiple plaintext languages.

6.2 Brute-Force Search

A brute-force attack tries candidate keys until a result appears valid. If a cipher has N equally likely keys and an attacker tests keys in a random order, the expected number of trials is approximately N/2. The worst case is N trials. This assumes that each key is tested once and that the correct key is equally likely to occur at any position.

For a Caesar cipher, N = 26, so the expected search is about 13 trials. For a password or key space of size 2^n, exhaustive search has up to 2^n trials and an average of approximately 2^(n−1), assuming uniform keys and one trial per candidate. Real attacks may be faster because users choose weak keys, keys may be reused, or partial verification may eliminate candidates early.

6.3 Statistical and Pattern-Based Attacks

Statistical attacks measure features such as letter frequencies, repeated sequences, coincidence rates, and likely word patterns. A substitution solver may use scoring functions based on n-gram frequencies. For example, candidate plaintexts containing common trigrams receive higher scores than strings containing improbable sequences.

These methods are not magic. They depend on assumptions about language, message length, and formatting. A competent analyst combines automated scoring with human interpretation.

6.4 Attack Classification

The following table summarises common attack models.

7. Kerckhoffs's Principle

Kerckhoffs's principle states that a cryptosystem should remain secure even if everything except the key is public. The algorithm should not depend on secrecy of its design. This principle is often summarised as: the enemy knows the system.

The principle does not mean that keys should be public. It means that security should survive publication of the algorithm and inspection by independent experts. Public algorithms can receive peer review, testing, standardisation, and vulnerability analysis. By contrast, a secret algorithm may contain undiscovered weaknesses and creates operational problems if many users must receive confidential algorithm descriptions.

Security through obscurity is not a reliable substitute for sound cryptographic design. Concealing an implementation detail may reduce casual discovery, but it should not be treated as the fundamental security boundary.

8. Why Classical Ciphers Fail

Classical ciphers fail for several related reasons:

1. Small or structured key spaces: Caesar and rail fence systems can be searched quickly.
2. Information leakage: Monoalphabetic substitution preserves frequency and repetition patterns.
3. Short or repeating keys: Vigenère reveals periodic structure when its keyword repeats.
4. Predictable language: Common words, greetings, names, and message formats help analysts.
5. Poor key management: A strong transformation is ineffective if the key is disclosed or reused.
6. Lack of authentication: Traditional ciphers generally do not detect modification or impersonation.
7. No protection against modern computation: Contemporary attackers can automate testing at high speed.

Confidentiality is also not the only security requirement. Integrity ensures that data has not been altered; authentication verifies identity or message origin; availability ensures authorised access when needed; and non-repudiation supports evidence about an action or origin. Classical encryption generally addresses only a limited part of confidentiality.

9. Applied Case: Recovering a Monoalphabetic Substitution

Background

The following is an illustrative training case, not a real incident. A small organisation discovers an old archive containing a message encrypted with a monoalphabetic substitution cipher. The key is not recorded.

Problem

The ciphertext is:

“GSRH RH Z HVXIVG NVHHZTV”

The analyst must determine whether the system can be recovered without the key.

Analysis

The message contains repeated word patterns. The first word has four letters, the second has two letters, and the third has a two-letter word followed by a six-letter word. The repeated ciphertext symbol Z appears in the two-letter word and inside the final word. In ordinary English, a two-letter word may be OF, TO, IN, IS, or other common forms.

The letter pattern of GSRH is four distinct letters, while RH repeats as a two-letter word later in the message. Testing likely substitutions and observing consistency suggests:

G → T, S → H, R → I, H → S, Z → A, V → E, X → C, I → R, N → M, H → S, Z → A, T → G, V → E.

The resulting plaintext is:

“THIS IS A SECRET MESSAGE”

Approach and Solution

The analyst used word boundaries, repeated patterns, likely short words, and consistency checks. The recovered mapping was then applied to every occurrence, rather than guessing each letter independently. This is essential: a substitution key is global for the message.

Outcome

The message was recovered without exhaustive testing of all 26! permutations. The archive is therefore treated as confidentially compromised. The organisation should not rely on the recovered scheme for future protection and should assess whether other messages used the same key.

Learning Points

• A large theoretical permutation count does not guarantee practical security.
• Repeated words and language statistics can reduce the search dramatically.
• Once part of a substitution mapping is known, it can expose many other letters.
• Historical ciphertext should be handled as potentially sensitive data.
• Modern systems must provide confidentiality, integrity, authentication, and secure key management.

10. Practical Laboratory Method

A useful classroom exercise is to implement small ciphers and attack them ethically using instructor-provided messages.

A disciplined workflow is:

1. Define the alphabet and treatment of spaces, punctuation, and case.
2. Implement encryption and independently implement decryption.
3. Test round-trip correctness: decrypt(encrypt(P)) must equal P under the chosen conventions.
4. Generate ciphertext from several message lengths and languages.
5. Attempt brute-force search where the key space is small.
6. Calculate letter and n-gram frequencies.
7. Compare attack performance on short and long messages.
8. Document assumptions, results, limitations, and ethical boundaries.

For an encryption implementation, test cases should include the empty string, repeated letters, alphabet boundaries such as Z followed by A, punctuation, and invalid input. In real systems, cryptographic code should normally use established, reviewed libraries rather than student-created algorithms.

11. Contemporary Relevance and Professional Practice

Classical ciphers are not deployed for serious modern protection, but their concepts appear in current security work. Frequency leakage resembles metadata and traffic-analysis problems. Key reuse remains a practical failure mode. Predictable plaintext appears in protocol headers and structured files. Brute-force resistance remains a central requirement for password storage and encryption keys.

Modern cryptography uses large keys, carefully analysed constructions, secure random generation, authenticated encryption, and formal security goals. Examples include block ciphers such as AES and authenticated modes such as GCM or modern misuse-resistant alternatives. Public-key systems address key-establishment and digital-signature problems that classical shared-key ciphers do not solve. These systems remain dependent on correct implementation, secure random numbers, patching, access control, and sound operational procedures.

Artificial intelligence can assist analysts by ranking candidate plaintexts, identifying language patterns, and automating historical-cipher exercises. It does not remove the need for cryptographic reasoning. Automated guesses can be confidently wrong, and using AI to analyse real confidential ciphertext may create privacy and legal risks. Cybersecurity professionals must obtain authorisation, protect evidence, document methods, and follow applicable law and organisational policy.

Best-Practice Principles

• Assume the algorithm may become public; protect keys instead.
• Use established, peer-reviewed cryptographic standards.
• Generate keys with a cryptographically secure random generator.
• Never reuse one-time pads or nonce values where the algorithm forbids reuse.
• Use authenticated encryption when confidentiality and integrity are both required.
• Separate encryption keys from passwords and manage their lifecycle.
• Minimise plaintext exposure and protect keys at rest and in memory.
• Test implementations, but do not treat testing as proof of security.
• Retire obsolete algorithms and document migration plans.
• Treat historical and recovered ciphertext as potentially sensitive information.

Chapter Summary

Classical cryptography introduces two fundamental transformation families: substitution changes symbol identities, while transposition changes symbol positions. Caesar, affine, monoalphabetic, Vigenère, rail fence, and columnar transposition ciphers demonstrate increasing complexity but also recurring weaknesses. Monoalphabetic systems preserve frequency and repetition patterns; repeating-key systems expose periodic structure; and small key spaces permit brute-force attacks.

Cryptanalysis uses attack models, statistical evidence, probable words, repeated patterns, and computational search. The security of a cipher depends not only on its apparent complexity but also on key size, randomness, key reuse, implementation, and operational procedures. Kerckhoffs's principle requires security to depend on the key rather than secrecy of the algorithm.

Historical systems such as the scytale, Caesar cipher, and Enigma show the evolution from manual transformations to mechanical and electronic cryptography. Their failures motivate modern requirements for strong algorithms, secure key management, authentication, integrity, peer review, and resistance to realistic adversaries.

Key Terms

Plaintext: Readable information before encryption.

Ciphertext: Transformed information produced by encryption.

Cipher: A systematic method for encrypting and decrypting information.

Key: A value controlling a cryptographic transformation.

Encryption: Conversion of plaintext into ciphertext.

Decryption: Recovery of plaintext from ciphertext.

Cryptanalysis: Study of methods for attacking cryptographic systems.

Substitution cipher: A cipher that replaces symbols with other symbols.

Transposition cipher: A cipher that rearranges symbol positions.

Monoalphabetic cipher: A substitution system using one fixed alphabet mapping.

Polyalphabetic cipher: A substitution system using multiple changing mappings.

Frequency analysis: Use of statistical symbol patterns to infer plaintext or a key.

Brute-force attack: Systematic testing of possible keys.

Known-plaintext attack: An attack using known plaintext and matching ciphertext.

One-time pad: A random, single-use key at least as long as the message.

Kerckhoffs's principle: The principle that security should depend on the key, not algorithm secrecy.

Key space: The set of all possible keys for a cryptosystem.

Product cipher: A cipher combining substitution and transposition operations.

Authenticated encryption: Encryption providing confidentiality and integrity together.

Review Questions

5 Short Answer Questions

1. What is the difference between plaintext and ciphertext?
2. Why is a Caesar cipher vulnerable to brute-force attack?
3. What information does frequency analysis exploit?
4. State Kerckhoffs's principle in your own words.
5. What condition must the multiplier satisfy in an affine cipher, and why?

5 Long Answer Questions

1. Explain substitution and transposition ciphers, using at least two examples of each.
2. Describe how a cryptanalyst could attack a monoalphabetic substitution cipher.
3. Explain the Vigenère cipher and why repeating keywords create vulnerability.
4. Discuss the lessons that the Enigma system provides for cryptographic design and operations.
5. Compare classical cryptographic systems with modern authenticated encryption.

3 Analytical / Application Questions

1. An organisation uses a short repeating keyword to encrypt long English messages. Design an analysis plan using repeated sequences, key-length estimation, and frequency evidence. Explain the limitations of your plan.
2. A developer argues that a proprietary encryption algorithm is secure because outsiders cannot inspect it. Evaluate this claim using Kerckhoffs's principle and professional security practice.
3. A recovered archive contains messages encrypted with a monoalphabetic substitution. Propose a responsible technical and governance process for analysing, preserving, and reporting the material.

Multiple Choice Questions with Answers

Question 1

Which operation is characteristic of a substitution cipher?

A. Rearranging the positions of symbols

B. Replacing symbols according to a mapping

C. Deleting repeated symbols

D. Compressing plaintext before encryption

Correct Answer: B. Replacing symbols according to a mapping

Question 2

For a Caesar cipher with encryption C = (P + k) mod 26, what is the ciphertext value when P = 24 and k = 5?

A. A (0)

B. C (2)

C. D (3)

D. Z (25)

Correct Answer: C. D (3)

Question 3

Why must a in an affine cipher be relatively prime to 26?

A. To make the ciphertext longer

B. To ensure a has a modular multiplicative inverse

C. To eliminate all spaces

D. To make frequency analysis impossible

Correct Answer: B. To ensure a has a modular multiplicative inverse

Question 4

Which feature most directly enables frequency analysis against a monoalphabetic substitution cipher?

A. It preserves relationships between letter frequencies

B. It uses a truly random key for every letter

C. It changes the message length randomly

D. It destroys all repeated-letter patterns

Correct Answer: A. It preserves relationships between letter frequencies

Question 5

What is the main weakness of a repeating-key Vigenère cipher?

A. It cannot encrypt alphabetic text

B. Its key repetition can reveal periodic structure

C. It has no decryption operation

D. It always uses only one substitution alphabet

Correct Answer: B. Its key repetition can reveal periodic structure

Question 6

Which statement is required for a theoretically secure one-time pad?

A. The key is shorter than the message and reused often

B. The key is publicly available but randomly generated

C. The key is truly random, message-length, secret, and used once

D. The key consists only of common English words

Correct Answer: C. The key is truly random, message-length, secret, and used once

Question 7

What does a transposition cipher primarily change?

A. The identity of every symbol

B. The positions or order of symbols

C. The key length only

D. The alphabet size only

Correct Answer: B. The positions or order of symbols

Question 8

An attacker who systematically tests every possible key is performing what type of attack?

A. Brute-force attack

B. Traffic shaping

C. Data compression

D. Digital signing

Correct Answer: A. Brute-force attack

Question 9

What does Kerckhoffs's principle require?

A. The encryption algorithm must always be secret

B. Security should depend on secrecy of the key rather than the algorithm

C. Keys should be shared publicly

D. Only historical algorithms should be used

Correct Answer: B. Security should depend on secrecy of the key rather than the algorithm

Question 10

Which statement best explains why a large nominal key space may not guarantee security?

A. All large numbers are impossible to calculate

B. Statistical leakage or implementation weaknesses may reduce the practical search

C. Large keys cannot be stored digitally

D. A larger key always makes plaintext longer

Correct Answer: B. Statistical leakage or implementation weaknesses may reduce the practical search