Information Theory and Its Real World Applications
Information Theory and Its Real World Applications
Written by: Pranshu Nautiyal
Introduction
One of the most qualitative measures of recent centuries is information and how it is stored. Information theory lets us see how information can be a qualitative data factor, and how to reduce the amount of entropy you have. While information theory might not seem that important in the real world, it has many real-world applications, such as in data storage and security. However, one of the most fascinating ways to interact with information theory is with board games, as it helps you exploit the rules of them in many ways.
Information Theory and Entropy
While information is commonly referred to as an abstract concept relating to facts and truth, it can also be expressed as the arrangement of what is conveyed. For example, combining both the sounds humans can make and the gestures we do forms the very basis of language that humans have used for thousands of years. Many animals often use their vocalizations to convey information to other animals in their species, like mating calls or calls for danger. Telegraphs, which are another form of how information is conveyed, use a simple system called Morse code involving two symbols to send messages over long distances. Information can be measured in simplest terms by bytes, which are singular units that hold either the values true or false. In computers, bits are also stored in binary, which is the most basic programming language. It uses a series of 0’s and 1’s to show information through the unique combinations of the two. Every unit of communication can eventually be simplified into a certain number of bytes at its core. Information theory also includes the value of entropy. It is used to measure the average uncertainty or information content in a probability distribution. The formula used to calculate entropy is H(P) = −∑p(x) log₂ p(x), where H(P) is the entropy, and p(x) is the probability of a specific event happening. Entropy is important to measure because you always want it to be as high as possible, which signifies the most knowledge known in a situation. Due to all of these, entropy and information theory is very important to our normal lives, and can be seen in many aspects like games or security.
Graph by https://machinelearningmastery.com/
This graph shows how the greater the probability of a given situation is, the less information is actually available
How to exploit entropy
The first person to exploit entropy in games like poker was a scientist named John Kelley Jr., when he studied how a gambler can maximize the expected earnings out of every bet based on information learnt in that round. In this case, a byte was the amount of entropy in a bettable event with two possible outcomes, like winning or losing. Kelley’s point was that to make our money grow exponentially from bets, the value of the side information we need can be calculated as I ( X ; Y ) = Ey [ DKL ( P( X | Y ) | | P ( X | I ) ) } where Y is the side information, X is the outcome of the betable event, and I is the state of the bookmaker’s knowledge. Since you do not know all the information you could have, the formula won’t guarantee a win every single time. This principle and formula can be applied to a wide variety of games, from the original poker to even Wordle. If you knew the patterns of the letters and how they are arranged in the word, then it would be easier to figure out what the word is. In most gambling, this principle can be exploited to figure out the winner, especially if you have prior knowledge of the topic.
Graph by https://machinelearningmastery.com/
This graph shows how the lessened probability distribution contributes to more entropy there is. The lower the range of probability, the higher the entropy, and therefore, the higher the chance you win.
Real World Applications
While information theory certainly is interesting to exploit for games and gambling, the more practical use of information theory comes from its roles in the real world, like security. When a code is encrypted, it is the hacker’s job to decrypt the message and steal whatever data was in the message. A perfect ciphertext would not reveal anything about the cipher used, thereby reducing the amount of entropy to foil the hacker. This is why cybersecurity specialists often have to use information theory.
Conclusion
Information theory is one of the most fascinating theories, and it can be applied to many different situations due to its broad scope. Information is one of the most important things around, and having a fundamental grasp on it and how entropy works is important for topics as mundane as games to topics as important as security.
References and Sources