Classical Ciphers and Cryptanalysis

Introduction

Cryptography is the practice of protecting information by transforming readable data, called plaintext, into an unintelligible form, called ciphertext. The transformation is controlled by a key and an algorithm. Classical ciphers are historically important systems that illustrate the central ideas of encryption, decryption, keyspace, secrecy, and cryptanalysis. Although they are unsuitable for protecting modern systems, they provide an accessible foundation for understanding why contemporary cryptography requires large keys, carefully analysed algorithms, and resistance to attacks based on statistical patterns.

A classical cipher generally operates on letters or small groups of letters. Two major design families are substitution and transposition. A substitution cipher replaces symbols with other symbols, while a transposition cipher rearranges the positions of symbols without changing the symbols themselves. Caesar and Vigenère ciphers are important substitution examples. Cryptanalysis is the study of recovering plaintext or keys from ciphertext without authorised access to the key.

This chapter treats classical ciphers as learning models rather than deployable security mechanisms. Their simplicity makes their weaknesses visible: small keyspaces permit brute-force search, natural-language redundancy enables frequency analysis, and repeated structures often reveal information about the key or plaintext.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish substitution from transposition;
• explain encryption, decryption, key, keyspace, and cryptanalysis;
• encrypt and decrypt text using Caesar and Vigenère methods;
• calculate and interpret classical-cipher keyspaces;
• apply frequency analysis and brute-force reasoning;
• explain why classical ciphers fail to provide modern confidentiality;
• evaluate an applied scenario involving the recovery of a classical-cipher message; and
• identify ethical and professional limits on cryptanalytic activity.

1. The Classical Cipher Model

A cipher is a reversible transformation. Let P denote the plaintext, C the ciphertext, E the encryption function, D the decryption function, and K the key. The general model is:

Correctness requires that, for valid messages and keys, decryption reverses encryption:

These relationships assume that the algorithm, ciphertext alphabet, key format, and message-processing rules are agreed by the communicating parties. In a sound cryptographic design, the algorithm may be public; security should depend primarily on the secrecy of the key rather than on keeping the method secret. This principle is commonly associated with Kerckhoffs's principle.

A classical system often simplifies text by converting letters to a fixed alphabet. For explanations in this chapter, the English alphabet is represented by integers from 0 to 25, where A = 0, B = 1, and so on. Spaces and punctuation may be removed or retained according to the system's rules. Any such preprocessing rule must be known to both sender and receiver.

Important terminology

• Plaintext: The original readable message.
• Ciphertext: The transformed message produced by encryption.
• Cipher: An algorithm for transforming plaintext and ciphertext.
• Key: A value that selects a particular transformation.
• Encryption: Transformation from plaintext to ciphertext.
• Decryption: Transformation from ciphertext to plaintext.
• Cryptanalysis: Analysis intended to recover plaintext, a key, or information about a cryptosystem.
• Keyspace: The set of all possible keys.
• Brute force: Systematically trying possible keys until a plausible result is found.
• Cryptanalysis attack model: The attacker's assumptions about available plaintext, ciphertext, keys, or encryption access.

2. Substitution and Transposition

Substitution ciphers

A substitution cipher replaces each plaintext symbol with another symbol. In a monoalphabetic substitution, the same plaintext letter is always mapped to the same ciphertext letter throughout a message. For example, a substitution might map A to Q, B to W, and C to E. The mapping is a permutation of the alphabet.

A general monoalphabetic substitution over an alphabet of 26 distinct symbols has:

possible one-to-one mappings. This is approximately 4.03 × 10²⁶, a very large nominal keyspace. However, keyspace size alone does not guarantee security. Natural language has strong statistical structure, and an attacker can use letter frequencies, repeated words, common patterns, and grammatical constraints to reduce the search dramatically.

Transposition ciphers

A transposition cipher retains the original symbols but changes their order. A simple columnar transposition writes plaintext into a rectangular grid and reads columns according to a key-defined order. For example, the letters of a message may be written in rows under a keyword and then read down the columns.

Transposition preserves the frequency of every symbol exactly. Therefore, single-letter frequency analysis can still reveal the language, although the relationships between adjacent letters are disrupted. In contrast, substitution preserves word lengths and repeated-letter patterns but changes the identities of letters.

Some systems combine substitution and transposition. Modern block ciphers use repeated, carefully designed forms of both ideas, together with larger blocks, nonlinear operations, and extensive key schedules. The classical examples therefore introduce design principles but do not provide modern protection.

3. The Caesar Cipher

The Caesar cipher is a shift substitution cipher. Each letter is shifted by a fixed number of positions through the alphabet. If a letter is represented by x and the key shift by k, encryption is:

Decryption is:

Here, x and C are integers from 0 to 25, k is normally an integer from 0 to 25, and “mod 26” means that values wrap around after 25. The equations apply to letters encoded with A = 0 through Z = 25. Spaces and punctuation require a separate convention.

Worked example: Caesar encryption

Encrypt the word MEET using k = 3.

• M = 12; (12 + 3) mod 26 = 15 = P
• E = 4; (4 + 3) mod 26 = 7 = H
• E = 4; (4 + 3) mod 26 = 7 = H
• T = 19; (19 + 3) mod 26 = 22 = W

The ciphertext is PHHW. Decrypting PHHW with k = 3 subtracts 3 from each value and returns MEET.

The Caesar keyspace is only 26 shifts, although shifts 0 and 26 are equivalent and the shift of 0 leaves the message unchanged. Thus, there are only 25 useful non-identity shifts. An attacker can try all possibilities quickly, inspect the results, and select the meaningful plaintext. This is a brute-force attack.

Caesar cryptanalysis

A Caesar cipher can be attacked in two main ways:

1. Exhaustive search: Try every possible shift.
2. Frequency analysis: Identify the most frequent ciphertext letters and compare them with expected language frequencies.

Exhaustive search is especially effective because the keyspace is tiny. Frequency analysis is not even necessary for a short message if all shifts can be inspected. The example demonstrates an important security lesson: a cipher may be easy to use but still provide almost no confidentiality.

4. The Vigenère Cipher

The Vigenère cipher improves on Caesar by using a sequence of shifts derived from a keyword. Instead of applying one fixed k to every letter, the system applies a different shift at each position. If xᵢ is the plaintext value at position i and kᵢ is the corresponding keyword value, then:

Decryption is:

The subscript i identifies the position in the message. The keyword is repeated as necessary, and non-letter handling must be specified. The formulas assume A = 0, a 26-letter alphabet, and that the same repeated-key alignment is used for encryption and decryption.

Worked example: Vigenère encryption

Encrypt ATTACKATDAWN using the keyword LEMON. Repeat the keyword beneath the plaintext:

For the first position, A = 0 and L = 11, so C₁ = (0 + 11) mod 26 = 11, which is L. The resulting ciphertext is LXFOPVEFRNHR.

The Vigenère cipher is a polyalphabetic substitution cipher because the same plaintext letter may encrypt to different ciphertext letters depending on its position in the keyword cycle. This weakens simple single-letter frequency analysis. However, a repeated keyword creates periodic structure. If the keyword length can be estimated, the ciphertext can be separated into groups, each behaving approximately like a Caesar cipher.

5. Keyspace, Work Factor, and Brute Force

The work factor is the practical effort required to test possible keys or otherwise defeat a system. A keyspace of N possible keys has a maximum exhaustive-search workload of N trials, and an average of approximately N ÷ 2 trials if keys are tested in random order and the correct key is equally likely to be anywhere in the list. This average assumes that every key is equally likely and that each trial has approximately the same cost.

For a repeated Vigenère keyword of length m, if every position can independently contain one of 26 letters, the nominal keyspace is:

For m = 5:

This is much larger than the Caesar keyspace, but it is not automatically secure. The calculation assumes all five-letter keys are equally likely and that repeated keywords are allowed. Human-chosen keywords are not uniformly random; dictionary words and familiar phrases make the effective search space much smaller. In addition, repeated-key structure can be attacked without testing every possible key.

The distinction between nominal and effective keyspace is important. Nominal keyspace counts all formal keys. Effective keyspace reflects the keys that are realistically selected and the information available to an attacker. Good modern cryptographic keys are generated from high-quality randomness and are used with algorithms designed to resist statistical and computational attacks.

6. Frequency Analysis

Frequency analysis exploits the unequal distribution of symbols in natural language. In typical English text, letters such as E, T, A, O, I, N, and S occur more often than letters such as Q, X, and Z, although exact frequencies vary by text type. In a monoalphabetic substitution, the most common plaintext symbol is consistently replaced by one ciphertext symbol. Consequently, the frequency profile is preserved in a rearranged form.

Frequency analysis is more reliable for long messages than short ones. It is also affected by writing style, topic, vocabulary, spelling conventions, and whether spaces have been removed. A competent analyst combines frequency counts with:

• repeated-letter patterns, such as the pattern in “BALLOON”;
• common short words, such as “A,” “I,” “OF,” and “TO”;
• common digrams and trigrams, such as TH, HE, and ING;
• likely word boundaries;
• grammatical context; and
• trial substitutions checked against the complete message.

A useful statistical measure for comparing symbol distributions is the index of coincidence. For a text containing n symbols, where fᵢ is the count of symbol i, it is:

The sum is over all alphabet symbols. The index estimates the probability that two symbols selected without replacement are identical. It tends to be higher for natural-language text than for a uniformly random sequence. The measure is informative only when enough text is available and when the alphabet and preprocessing assumptions are understood. It should not be treated as a guaranteed identifier of a language or cipher.

For Vigenère ciphertext, an analyst may search for repeated ciphertext sequences. If repetitions occur at distances that share common factors, those factors may suggest the keyword length. This is the basis of the Kasiski examination. Once a likely period is obtained, letters in each position of the period can be analysed as a Caesar-like stream. The method is most useful when the message is sufficiently long and the keyword is actually repeated.

7. Cryptanalysis as a Reasoning Process

Cryptanalysis is not simply guessing. It is a structured process that combines assumptions, observations, hypotheses, tests, and verification.

Step 1: Identify the likely cipher family

Ask whether symbols have changed, positions have changed, or both. Preserved frequencies suggest transposition; consistent symbol replacement suggests monoalphabetic substitution. Uniform-looking output may indicate a stronger modern cipher, compression, or encoding rather than a classical cipher.

Step 2: Establish preprocessing rules

Determine whether spaces, punctuation, case, and digits were removed. Incorrect assumptions can make a correct key appear incorrect.

Step 3: Estimate the keyspace or period

For Caesar, enumerate the shifts. For a substitution, examine patterns and frequency. For Vigenère, examine repeated sequences, possible periods, and coincidence statistics.

Step 4: Test candidate keys

Apply a candidate key and inspect the result. Automated scoring can rank outputs according to dictionary matches, n-gram frequencies, or language models, but a high score is not proof of correctness.

Step 5: Validate independently

A plausible fragment may be accidental. Confirm that the same key decrypts the whole message, that the result is grammatically coherent, and that re-encryption reproduces the original ciphertext.

This workflow illustrates a general cybersecurity principle: an attack succeeds by combining algorithmic weaknesses with information about implementation, usage, and human behaviour.

8. Worked Comparison of Classical Attacks

Suppose an analyst receives three ciphertexts.

• Ciphertext A uses a fixed alphabetic shift. The analyst can test 25 meaningful Caesar keys.
• Ciphertext B uses a monoalphabetic substitution. The nominal keyspace is 26!, but frequency and pattern analysis may reduce the practical effort greatly.
• Ciphertext C uses a repeated five-letter Vigenère key. The nominal keyspace is 26⁵, but repeated periods and non-random human keywords create exploitable structure.

The comparison shows that increasing the formal keyspace does not automatically solve every weakness. Ciphertext B has a much larger nominal keyspace than Ciphertext C, yet both can be vulnerable to language statistics. Security depends on the interaction of key generation, algorithm structure, message length, implementation, and attack model.

9. Applied Case: Recovering a Historical Training Message

Background

This illustrative case concerns a cybersecurity training exercise at a small museum. A curator discovers an old note containing the ciphertext LXFOPVEFRNHR and a reference to the keyword LEMON. The exercise is designed to demonstrate historical cryptanalysis, not to access a live system.

Problem

Students must determine whether the note was produced using a Vigenère cipher and recover the plaintext. They must also explain why the method would not be acceptable for protecting current confidential information.

Analysis

The ciphertext contains letters only and does not display an obvious single fixed shift. The supplied keyword has five letters, suggesting a repeated-key Vigenère system. The students encode LEMON as 11, 4, 12, 14, 13 and subtract these values from the ciphertext values modulo 26.

For the first character, L = 11 and L = 11, so (11 − 11) mod 26 = 0, or A. For the second, X = 23 and E = 4, so (23 − 4) mod 26 = 19, or T. Continuing produces ATTACKATDAWN.

Approach and solution

The students verify the result by re-encrypting ATTACKATDAWN with LEMON. The original ciphertext is reproduced, confirming the key and plaintext under the stated conventions.

Outcome

The exercise succeeds because the key is supplied and the algorithm is historically documented. In a more realistic cryptanalysis task, students would need to infer the key length, exploit repeated patterns, and test candidate keys. The recovered message has no operational value, and the exercise is conducted under explicit educational authorisation.

Learning points

1. Encryption and decryption must use complementary operations.
2. A repeated keyword creates exploitable periodic structure.
3. Re-encryption is a useful correctness check.
4. Historical success does not imply contemporary security.
5. Cryptanalytic skills must be applied only with permission and within defined scope.

10. Limitations and Professional Implications

Classical ciphers fail against modern security requirements for several reasons. Their keyspaces are small or structurally weak; they expose language patterns; they generally lack integrity protection; and they do not authenticate the sender. Even if an attacker cannot immediately read a message, the attacker may alter it without detection.

Confidentiality, integrity, authentication, and availability are separate security goals. Classical encryption primarily illustrates confidentiality and does so poorly. Modern applications normally require authenticated encryption, secure key management, protected random-number generation, safe implementation, and lifecycle planning. A technically strong algorithm can still fail if keys are reused, disclosed, stored insecurely, or selected predictably.

Cryptanalysis also has ethical and legal boundaries. Students and professionals should analyse only data they own or are explicitly authorised to examine. Testing a cipher on a museum exercise, laboratory dataset, or instructor-provided ciphertext is appropriate. Intercepting private communications, bypassing access controls, or recovering operational keys without permission may violate law, policy, and professional codes. Document scope, preserve evidence responsibly, avoid unnecessary disclosure, and report findings through the authorised channel.

11. Contemporary Relevance

Classical methods are not used to secure internet traffic, financial transactions, or enterprise databases. Their continuing value is educational and conceptual. They explain confusion between encoding and encryption, demonstrate the importance of randomness, and provide intuitive examples of statistical leakage and key reuse.

Modern cryptographic systems use algorithms such as AES or ChaCha20-Poly1305, public-key mechanisms for key establishment, cryptographic hash functions, digital signatures, and authenticated protocols. These systems are not simply classical ciphers with larger keys. They are engineered to resist known attack models and are evaluated through extensive public analysis. AI-assisted language models and automated scoring can help rank candidate decryptions in an authorised laboratory, but they do not eliminate the need for correct assumptions, validation, lawful access, and human judgement. AI output can also produce plausible but false plaintext and must therefore be independently verified.

Best Practices for Learning and Laboratory Use

• State the alphabet, numeric mapping, preprocessing rules, and key convention.
• Separate encoding, encryption, hashing, and authentication in technical explanations.
• Use classical ciphers only for education, demonstrations, or historical analysis.
• Prefer reproducible calculations and verify results by re-encryption.
• Record the attack model and assumptions before interpreting results.
• Never use predictable human words as production cryptographic keys.
• Do not treat a large nominal keyspace as proof of security.
• Obtain written authorisation before testing any system or message.
• Use modern, standardised authenticated cryptography for real protection.

Chapter Summary

Classical cryptography introduces the central model of plaintext, ciphertext, key, encryption, and decryption. Substitution changes symbol identities, whereas transposition changes their positions. The Caesar cipher uses a fixed modular shift and is defeated by a tiny keyspace or simple frequency analysis. The Vigenère cipher uses a repeated sequence of shifts and therefore reduces basic frequency leakage, but its periodic structure and human-selected keywords remain vulnerable to cryptanalysis.

Keyspace describes possible keys, but effective security also depends on randomness, algorithm structure, message statistics, implementation, and the attacker's information. Brute force is effective against small keyspaces. Frequency analysis, repeated-pattern analysis, and the index of coincidence reveal weaknesses in classical systems. These methods are valuable foundations for understanding modern cryptographic design, but classical ciphers do not provide contemporary confidentiality, integrity, or authentication. Their use should be limited to authorised educational and historical contexts.

Key Terms

Plaintext: The original readable message before encryption.

Ciphertext: The transformed message produced by encryption.

Cipher: An algorithm that transforms plaintext and ciphertext.

Key: A value controlling a cryptographic transformation.

Encryption: The process of converting plaintext into ciphertext.

Decryption: The process of recovering plaintext from ciphertext.

Substitution: Replacing symbols with other symbols.

Transposition: Rearranging symbols while retaining their identities.

Caesar cipher: A fixed-shift substitution cipher.

Vigenère cipher: A polyalphabetic substitution cipher based on repeated shifts from a keyword.

Cryptanalysis: The study of recovering plaintext, keys, or system information without authorised key knowledge.

Keyspace: The complete set of possible keys.

Brute force: Systematic testing of possible keys.

Frequency analysis: Using symbol-frequency patterns to analyse ciphertext.

Period: The length of a repeated key cycle.

Index of coincidence: A statistic measuring the likelihood that two selected symbols are identical.

Kasiski examination: A method using repeated ciphertext sequences to estimate a Vigenère key period.

Kerckhoffs's principle: The principle that security should rely on key secrecy rather than algorithm secrecy.

Work factor: The practical computational effort required for an attack.

Authenticated encryption: Encryption that also provides integrity and authenticity protection.

Review Questions

Short Answer Questions

1. What is the difference between substitution and transposition?
2. Why is a Caesar cipher vulnerable to exhaustive search?
3. What does the modulus operation accomplish in Caesar and Vigenère encryption?
4. Why does a repeated Vigenère keyword create a cryptanalytic weakness?
5. What is meant by effective keyspace?

Long Answer Questions

1. Explain the classical encryption model and the roles of plaintext, ciphertext, algorithm, and key.
2. Describe how frequency analysis can attack a monoalphabetic substitution cipher.
3. Explain the Vigenère cipher, including its encryption and decryption equations and assumptions.
4. Compare the security weaknesses of Caesar, monoalphabetic substitution, and Vigenère ciphers.
5. Discuss why modern cryptography requires more than confidentiality alone.

Analytical / Application Questions

1. A message has been encrypted with a Caesar cipher, but the shift is unknown. Design a lawful laboratory procedure for recovering the plaintext and validating the result.
2. A long ciphertext contains repeated sequences at regular intervals. Explain how this observation could support an estimate of a Vigenère key period, and identify limitations of the inference.
3. A manager proposes using a five-letter English word as a production Vigenère key because its nominal keyspace is 26⁵. Evaluate the proposal using keyspace, human key selection, periodicity, and modern security requirements.

Multiple Choice Questions with Answers

Question 1

Which operation is performed by a transposition cipher?

A. Replacing every letter with a fixed shifted letter

B. Rearranging symbols without changing their identities

C. Hashing the message into a fixed-length digest

D. Generating a digital signature

Correct Answer: B. Rearranging symbols without changing their identities

Question 2

Using A = 0, what is the Caesar encryption of M with key shift 3?

A. N

B. O

C. P

D. Q

Correct Answer: C. P

Question 3

Which equation represents Caesar encryption under the chapter's alphabet convention?

A. C = (x − k) mod 26

B. C = x ÷ k

C. C = (x + k) mod 26

D. C = x × k

Correct Answer: C. C = (x + k) mod 26

Question 4

What is the nominal keyspace of a repeated five-letter Vigenère key when each position may independently contain any of 26 letters?

A. 130

B. 26²

C. 26⁴

D. 26⁵

Correct Answer: D. 26⁵

Question 5

Why is simple single-letter frequency analysis less direct against Vigenère ciphertext than against monoalphabetic substitution?

A. Vigenère removes all letter frequencies from the language

B. The same plaintext letter can map to different ciphertext letters

C. Vigenère always encrypts whole words as single symbols

D. Vigenère uses no key

Correct Answer: B. The same plaintext letter can map to different ciphertext letters

Question 6

What does brute-force cryptanalysis mean?

A. Trying possible keys systematically

B. Deleting all repeated letters

C. Reordering the alphabet randomly

D. Compressing ciphertext before decryption

Correct Answer: A. Trying possible keys systematically

Question 7

What information does the index of coincidence estimate?

A. The probability that two selected symbols are identical

B. The exact length of every encryption key

C. The time required to decrypt a message

D. The number of words in plaintext

Correct Answer: A. The probability that two selected symbols are identical

Question 8

Which feature commonly makes a repeated-key Vigenère cipher vulnerable?

A. It uses no alphabet

B. Its repeated period can reveal structural information

C. It cannot encrypt repeated plaintext letters

D. Its keyspace is always one

Correct Answer: B. Its repeated period can reveal structural information

Question 9

For a Caesar cipher with 26 possible shifts, how many trials are required in the worst case for exhaustive search?

A. 13

B. 25

C. 26

D. 676

Correct Answer: C. 26

Question 10

Which practice is appropriate when conducting classical cryptanalysis?

A. Testing private messages without permission

B. Using recovered keys on unrelated systems

C. Analysing an authorised educational ciphertext

D. Publishing sensitive plaintext immediately

Correct Answer: C. Analysing an authorised educational ciphertext