Information and coding
Compute entropy and expected code length, decode a prefix code, and explain why uncertainty is not meaning.
Start with: Probability. log₂ counts powers of two: log₂ 8 = 3.
01 · Commit to a prediction
Responses stay in this page only. Reloading or closing may discard them. Nothing is transmitted, saved or synchronized.
02 · Change an assumption
Before changing a control, say what should move and why. Start with the experiments below. Reset restores the starting model; it preserves your written responses.
Change only A probability from 50% to 90%. The remaining probability is split B:C:D = 2:1:1. Predict entropy and code efficiency.
The source becomes more predictable. Entropy is about 0.619 bits/symbol and this fixed code costs 1.15. It is still the best symbol-by-symbol code. The 0.53-bit gap is the cost of whole-bit codewords, and coding blocks of symbols can close it. Below 33% A, four 2-bit codewords would beat this code.
Set A probability to 100%. Does this fixed code reach zero bits per symbol?
No, it still writes one zero for every A. Entropy is zero. If the receiver knows the constant symbol and message length, no symbol content needs transmission.
0 to 100 · step 1
Calculated model output. The table gives the same values. Displayed values are rounded; calculations keep full precision.
03 · Connect the mechanism
A prefix code gives no symbol a codeword that begins another codeword. That lets a receiver find boundaries while reading left to right. Frequent symbols can use shorter words. Entropy describes the probability-weighted surprise of the source. It constrains achievable average lossless coding, not the meaning or value of a message.
p(x) is a symbol probability and all probabilities sum to one. ℓ(x) is the integer codeword length. I and H are in bits; H and L here are bits per emitted symbol. Use 0 log₂ 0 = 0 by a limit in entropy sums. Surprise of an impossible event is not a finite observed value. pA = p, pB = (1−p)/2 and pC = pD = (1−p)/4.
At the starting distribution, surprisals are 1, 2, 3 and 3 bits. Weight them: 0.5 × 1 + 0.25 × 2 + 0.125 × 3 + 0.125 × 3 = 1.75. The displayed code has exactly those lengths. For A,D,B the stream is 0|111|10 = 011110; reading its prefix tree recovers the boundaries.
The source emits independent identically distributed symbols. The codebook is fixed as you move the control; it is not rebuilt optimally. For a binary symbol-by-symbol prefix code, H ≤ L. Suitable block coding can approach the entropy rate for long independent sequences, with framing and codebook costs treated separately. Correlations require conditional probabilities. Shannon entropy in bits and thermodynamic entropy in J/K have related mathematics but different definitions and units.
The annotated sources distinguish established results from this lesson’s original examples.
04 · Follow the structure
Common sensor symbols can have short descriptions.
Boundary: Correlations, framing and error protection change the practical rate.
An unlikely answer eliminates more uncertainty than an expected one.
Boundary: Shannon information does not judge whether the answer is useful, true or meaningful.
05 · Retrieve without hints
Write an answer before opening its feedback. Later, return directly here without rereading above. Recognition, explanation and transfer are separate outcomes. No page action or answer reveal measures mastery.
Recognition and explanation
The A word is a prefix of the B word. A received zero does not yet tell you whether A finished or B is coming.
Self-check: Identify the ambiguous boundary, not merely unequal lengths.
Calculation
H = −2 × 0.5 log₂(0.5) = 1 bit/symbol. Assign 0 and 1.
Self-check: Weight both possibilities and state the unit.
Novel transfer
No. Only the initial phase is uncertain. One bit determines the whole fixed-length stream. The per-symbol entropy rate tends to zero as length grows. Independence was the missing assumption.
Self-check: Distinguish marginal entropy from sequence entropy and count the two possible streams.
On a later day, try again and record actual evidence in the curriculum. A later unaided explanation and a fresh transfer problem give stronger evidence than immediate familiarity. No reminder is scheduled.
Sources & scope
Original teaching examples. Reference links need a connection; the lesson itself does not. Built 2026-10-11. Learner understanding is not assessed.