Exponentials and logarithms

Logarithms answer "what power?"

Start with 1 and keep doubling: 2, 4, 8. How many doublings did it take to reach 8? Three. That count is a logarithm. It answers the question "how many times do I multiply to get here?"

In symbols, the log has a small number underneath called the base: the number you keep multiplying by. $\log_2(8)$ reads "log base 2 of 8". It is the power you raise 2 to in order to get 8.

In plain words: $\log_2(8) = 3$ because 2 × 2 × 2 = 8. The log is the number of times you multiply the base. $\ln$ (say "L N", the natural log) is the log whose base is $e$.

Each base has its own NumPy function. np.log is the natural log. np.log2 and np.log10 use bases 2 and 10.

import numpy as np

print(np.log2(8), np.log10(1_000_000), np.log(np.e))

This prints 3.0 6.0 1.0. Three 2s make 8. Six 10s make a million. And one e makes e.

Notice how slowly logs grow. A million turns into 6. That is why logs are used for lopsided data, like spend per customer or follower counts, where a few huge values sit far above the rest.

import math
print(round(math.log(1000, 10)))

How long until it doubles?

Logs let you run growth backwards. Your list grows 5% a month. How many months until it doubles? You need the $t$ where $1.05^t = 2$. Taking the log of both sides brings $t$ down where you can reach it:

In plain words: divide the log of 2 (the "doubling") by the log of 1.05 (one month of growth). The answer is how many months of growth fit into one doubling.

With small numbers: $\ln 2$ is about 0.693 and $\ln 1.05$ is about 0.0488. Divide them and you get about 14.2. The worked example below checks it in code.

import math

rate = 0.05
t = math.log(2) / math.log(1 + rate)
print(round(t, 1))
print(round(1000 * 1.05 ** 14), round(1000 * 1.05 ** 15))

The rule that matters most: products become sums

Logs follow three rules. In them, $a$ is any positive number. $b$ is another positive number. $k$ is any power.

In plain words: the log of a product is the sum of the logs. The log of a division is the difference of the logs. And the log of a power is the power times the log.

With small numbers, $\ln 2$ is about 0.693 and $\ln 3$ is about 1.099. Add them and you get 1.792, which is $\ln 6$. The code checks all three rules with a = 2, b = 3 and a power of 4.

import math

a, b = 2, 3
print(round(math.log(a * b), 4), round(math.log(a) + math.log(b), 4))
print(round(math.log(a / b), 4), round(math.log(a) - math.log(b), 4))
print(round(math.log(a ** 4), 4), round(4 * math.log(a), 4))

Each line prints two equal numbers: 1.7918, then -0.4055, then 2.7726.

Why language models need the first rule

A language model gives each word in a sentence a probability. The chance of the whole sentence is all those probabilities multiplied together. They are small numbers, so the product shrinks fast.

Think of a calculator with a limited screen. Show it a number with too many zeros after the decimal point and it gives up and shows 0. A computer does the same. When a number gets too small to store, it underflows: it becomes exactly zero, and the information is gone.

Adding logs avoids this. Each small probability turns into an ordinary negative number, and you add those up instead.

import numpy as np

probs = np.full(1000, 0.01)          # 1,000 words, each given probability 0.01
print("product:", np.prod(probs))    # underflows to 0.0
print("sum of logs:", np.sum(np.log(probs)))
print("as a power of 10:", np.sum(np.log10(probs)))

The product prints 0.0. The sum of logs is about −4605.17, a normal number to store. The last line says the true product is $10^{-2000}$: a 1 that sits 2,000 places after the decimal point. A computer's usual floor is roughly $10^{-308}$, so the product drops far below it. The log version keeps all the information. That is why models report log-probabilities, and why the standard training loss is a negative log (you'll meet it in the cross-entropy lesson).

Going back: exponentials undo logs

A log and an exponential cancel each other out, like walking 10 steps forward and 10 steps back.

$e^{\ln x} = x$

In plain words: take the log of a number, raise $e$ to that result, and you are back where you started.

For example, $\ln 7$ is about 1.9459. Raise $e$ to 1.9459 and you get 7 again.

import math

x = 7
print(round(math.log(x), 4))
print(round(math.exp(math.log(x)), 6))

This prints 1.9459 and then 7.0. So you can do all the arithmetic in "log land", where nothing underflows, and step back with exp only at the end. Perplexity, a score you'll meet in Phase 5, works this way: it is the exponential of an average negative log-probability.

For marketers: when a metric spans a huge range (spend per customer, page views per URL), plot it on a log scale or analyze np.log(value). Percentage changes like "+20%" become steps of the same size, and one giant customer stops squashing the rest of the chart flat.