LLMs¹ model text by outputting a probability distribution for the next token, over the \(M\) tokens in a vocabulary. Funnily enough, it is a well-proven fact that this output can only express a limited set of values. In fact, nearly all \(M\)-element probability distributions cannot be output by any LLM no matter what we input.

That can be a problem, because it means that, although we want LLMs to perfectly capture all patterns in natural language, they probably can’t.²

I will be elaborating on the reason for this bottleneck in another blog post. But for this project, we’re not mainly interested in why there is a bottleneck. We’re more interested in the characteristics of distributions that this bottleneck limits.

Experiment

Wang et al. (2025) points out that the entropy of a distribution seems to be a governing factor in how closely an LLM is able to approximate it. I reproduced the experiment on Llama 3.2 1B and confirmed the results in my figure below:

This is surprisingly clean. Low-entropy and high-entropy distributions both prove easy to produce, whereas the LLM is not able to closely approximate medium entropy distributions.

Wang et al. notes that the same phenomenon occurs in randomly-initialized models, and so I may rule out the role of pre-training and limit its cause to the architecture. But for the most part, the why question remains unanswered; I take a stab at it in the next blog post.

For ease of reading, I’ve separated this derivation into sections using horizontal lines, where each section builds upon the conclusion of the previous section. It goes something like this.

Let \(\{F_n\}_{n=0}^\infty\) be the Fibonacci Sequence. This is defined by base cases

and for all \(n \geq 2\), the Sequence is defined recurrently by

We first establish that \(F_n\) is an increasing sequence (you can prove this rigorously using induction), and therefore, that \(F_n \geq 0.\)

Then, for all \(n \geq 2\), we have from the definition that \(F_n=F_{n-1}+F_{n-2} \leq F_{n-1} + F_{n-1} = 2F_{n-1}\), so \(F_n \leq 2F_{n-1}\). A little intuition tells us that since the next item in the Sequence is less than or equal to double the previous, the growth speed of this function at most follows \(2^n\). Indeed, \(F_n \leq 2^n\), and you can again prove this using induction if you like.

We bound \(0 \leq F_n \leq 2^n\) from the last two paragraphs. Then, for \(n \geq 0\),

Therefore, by the comparison test, since \(\sum\limits_{n=0}^\infty\frac1{2^n}\) converges, the series \(\sum\limits_{n=0}^\infty\frac{F_n}{4^n}\) also converges.

Okay, stay with me.

Consider now a power series with Fibonacci coefficients:

Note that we can plug in \(\frac14\) to get \(\sum\limits_{n=0}^\infty\frac{F_n}{4^n}\), which we just proved converges, and the series converges to \(0\) trivially when \(x=0\). Using a few theorems from Calculus on radii of convergence, we know that this series converges on at least a radius of \(\frac14\) around \(0\). So for some function \(f\),

when \(-\frac14 < x \leq \frac14\).

By the uniqueness of power series representations, any function that adheres to its power series on an interval has a unique power series representation on that interval. We can thus claim that within \((-\frac14, \frac14]\), \(\sum\limits_{n=0}^\infty F_nx^n\) is not only a power series representation of \(f(x)\), it is the unique one.

That’s a cool result, but let’s try and obtain more information about this mysterious \(f(x)\). Of course, the below calculations are bounded on our working interval of \((-\frac14, \frac14]\). We first separate out the first two elements of the series:

Substituting \(F_0, F_1,\) and \(F_n\) out for their definitions from earlier and simplifying, we have

We then split up the remaining series by distributive property and reindex them:

Factor out \(x\) and \(x^2\):

We can then substitute in \(\sum\limits_{n=0}^\infty F_{n} x^n = f(x)\) to obtain:

completely removing any infinite sequences from the equation.

Let’s solve for \(f(x)\). Rearranging the equation yields

We then decompose the fraction to get

(You’re just gonna have to take my prof’s word on this one, because I don’t want to spend time decomposing this)

At this point, we notice that these forms may look very familiar from the formula \(\frac{a}{1-r} = \sum\limits_{n=0}^\infty ar^n\), so we apply it on the fractions above to get back to infinite series:

We now have two power series representations for \(f(x)\) on its domain:

Since \(f(x)\) has a unique power series representation, these two series must have the same coefficients; so, we can equate

and bam, we have a closed form for the Fibonacci Sequence! \(~\square\)

It turns out that, after some further reading, this technique of using generating functions to find closed forms for sequences is not uncommon. But it did surprise me, and I hope you found it interesting too.

Thanks for reading!

In the meantime, we can approximate \(e^x\) by

from which we see that MathJax seems to be working.

I can also paste some code.

def entropy(x: torch.Tensor, dim: int, alpha: float = 1) -> torch.Tensor:
    if alpha == 1: 
        return -torch.sum(x * torch.log(x), dim=dim)
    elif alpha == 0: 
        return torch.log((x != 0).sum(dim=dim).float())
    return torch.log(torch.sum(x ** alpha, dim=dim)) / (1 - alpha)

Neat.