Shannon Entropy Calculator

Shannon entropy, introduced by Claude E. Shannon in his 1948 paper 'A Mathematical Theory of Communication', is a measure of the uncertainty or randomness in a set of data. It quantifies the average minimum number of bits needed to encode a sequence of symbols, given their probability distribution. Higher entropy means more unpredictability; lower entropy means more structure or redundancy.

The formula is H(X) = −∑ P(xᵢ) · log₂ P(xᵢ), summed over all possible symbols or events. Each term P(xᵢ) · log₂ P(xᵢ) represents the contribution of symbol xᵢ to the total entropy. The negative sign is used because log₂ of a probability (a value between 0 and 1) is always negative, making the entropy a positive value measured in bits.

Shannon entropy is fundamental to data compression, cryptography, and communications. It sets a theoretical lower bound on lossless data compression — you cannot compress a source below its entropy without losing information. It is also used to measure the strength of passwords, evaluate randomness in random number generators, and assess feature importance in machine learning algorithms like decision trees.

Maximum entropy occurs when all symbols in the alphabet are equally probable. For an alphabet of n symbols, the maximum entropy is log₂(n) bits per symbol. Any deviation from a uniform distribution reduces entropy. The relative entropy (efficiency) shown by this calculator tells you what fraction of the theoretical maximum your distribution achieves.

Shannon entropy (bits per symbol) describes the average information content of a single symbol. Metric entropy (total bits) is simply Shannon entropy multiplied by the total number of symbols in the string — it represents the total information content of the entire sequence.

Shannon entropy is defined over a valid probability distribution, where all probabilities are non-negative and sum to exactly 1. If your probabilities don't sum to 1, the calculation is mathematically undefined. This calculator automatically normalises your entered probabilities so they sum to 1 before computing entropy.

Password entropy estimates how difficult a password is to guess by brute force. A password drawn from a larger, more varied character set has higher entropy. For a password of length L using an alphabet of N characters (assuming uniform distribution), the entropy is L · log₂(N) bits. Higher entropy passwords are exponentially harder to crack.

An entropy of 0 bits per symbol means there is absolutely no uncertainty — the same symbol always appears (probability = 1 for one symbol, 0 for all others). For example, the string 'AAAAAAA' has zero entropy because you can perfectly predict every character. Real-world strings almost always have entropy greater than 0.