Entropy Calculator

Type a message and watch Shannon entropy grow, one letter at a time. Compare it with random letters and with English.

EntropyFrequency analysisdemo

Objectives

Count the draws in a message and explain why a space is not one.
Compute a letter's contribution to the entropy sum and say why a common letter contributes little.
Explain why three draws over 26 outcomes carry three different amounts of entropy.
Convert a difference in bits into the factor it changes an attacker's guessing work by.
Relate the gap between a uniform letter and an English letter to what frequency analysis spends.

20%

One message, counted, weighed, and compared against English.

Entropy is measured per draw, so the first question is what counts as a draw. Here a draw is one letter. A space is not a letter, so it is not a draw, and it is removed before anything is counted.

The message

Message

Anything you type. Only A to Z is counted. Every step below reads this one field.

what is kept

W E H A V E A M O L E A B O R T T H E D E A L

23 letters counted

12 of 26 letters appear

6 characters dropped

Where the count comes from

Letters per word, and the total.
word	letters
We	2
have	4
a	1
mole	4
abort	5
the	3
deal	4
total	23

That total is the number the slide quotes. Count the letters in each word and add them up. The spaces between the words have no row, because they are not draws.

The letters that never appear

C F G I J K N P Q S U X Y Z

The alphabet still has 26 letters, so a draw from it still has 26 possible answers. But 14 of them have probability zero here. A letter that never appears contributes nothing to the entropy, and that is already most of the reason a short message carries less per letter than the language does.

Each letter contributes its own term to one sum. A letter you see often has a large probability, and a large probability contributes little, because seeing it was not much of a surprise.

H(X) = − ∑ p_i log₂ p_i

Every term of the sum

Every letter that appears in the message, with its count, its probability, and the bits it contributes. Sorted by contribution.
letter	count	p	−p log₂ p
E	5	0.2174	0.479
A	4	0.1739	0.439
H	2	0.0870	0.306
L	2	0.0870	0.306
O	2	0.0870	0.306
T	2	0.0870	0.306
B	1	0.0435	0.197
D	1	0.0435	0.197
M	1	0.0435	0.197
R	1	0.0435	0.197
V	1	0.0435	0.197
W	1	0.0435	0.197

One row, worked out

The row for E, in full.
quantity	value
letter	E
count	5 of 23
probability p	5 / 23 = 0.2174
log₂ p	-2.2016
contribution −p log₂ p	0.2174 × 2.2016 = 0.4786 bits

Every row of the table above is this calculation. The probability is a count divided by the total. The bits are that probability multiplied by how surprising it is.

The whole sum

H = 0.479 + 0.439 + 0.306 + 0.306 + 0.306 + 0.306 + 0.197 + 0.197 + 0.197 + 0.197 + 0.197 + 0.197 = 3.3231 bits

The entropy of one letter of this message, and the two counts behind it.
quantity	value
letters counted	23
distinct letters	12
H, per letter	3.3231 bits

The most common letter here is E, contributing 0.479 bits. The rarest is W, contributing 0.197. A common letter is not the same as an informative one. Type the same letter twenty times and H falls to zero: the alphabet never changed size, so its size was never what the entropy measured.

Uniform, English, and your message, letter by letter

left bar: uniform, every letter 1/26

middle bar: English frequency

right bar: your message

All three draw one letter out of the same 26, and they are not the same draw. Each letter has three bars in a fixed left-to-right order, so the shape reads without relying on the colours. The flatter the row of bars, the higher the entropy.

Three letters, across the three distributions

The probability of three letters under each of the three distributions in the chart above.
letter	uniform	English	your message
E	0.0385	0.1270	0.2174
T	0.0385	0.0906	0.0870
Z	0.0385	0.0007	0.0000

Under a uniform draw, E and Z are equally likely, so guessing E first buys nothing. Under English, E is roughly 180 times more likely than Z, so guessing E first is a real advantage. That advantage is what the entropy difference measures.

What the difference is worth

4.70 − 4.18 = 0.52 bits, and 2^0.52 = 1.44× the guessing work

The cost of knowing English letter frequencies, per letter.
quantity	value
uniform letter	4.70 bits
English letter	4.18 bits
difference	0.52 bits
in guessing work	a factor of 1.44

Bits are exponents, so 0.52 bits per letter is a factor of 1.44 in the work an attacker does. That gap is exactly what frequency analysis spends, and every attack in the previous session was a way of collecting it.

step 1/5

CPE 645 / CS 685

Entropy Calculator

Score Card

Tests Cannot See Prediction