The code for this project can be found in the repo.

LLMs1 model text by outputting a probability distribution for the next token, over the \(M\) tokens in a vocabulary. Funnily enough, it is a well-proven fact that this output can only express a limited set of values. In fact, nearly all \(M\)-element probability distributions cannot be output by any LLM no matter what we input.

That can be a problem, because it means that, although we want LLMs to perfectly capture all patterns in natural language, they probably can’t.2

I will be elaborating on the reason for this bottleneck in another blog post. But for this project, we’re not mainly interested in why there is a bottleneck. We’re more interested in the characteristics of distributions that this bottleneck limits.

Experiment

Wang et al. (2025) points out that the entropy of a distribution seems to be a governing factor in how closely an LLM is able to approximate it. I reproduced the experiment on Llama 3.2 1B and confirmed the results in my figure below:

Figure

This is surprisingly clean. Low-entropy and high-entropy distributions both prove easy to produce, whereas the LLM is not able to closely approximate medium entropy distributions.

More details on methods

My methods reproduce soft prompt tuning for vanilla distributions as laid out by Wang et al. (2025).

  1. I synthesize a set of target probability distributions over the vocabulary, with entropies ranging from the theoretical minimum to the maximum. I use gradient descent to optimize for the entropy of each distribution.
  2. I use soft prompt tuning to find the input that would elicit an LLM output matching each target distributions as closely as possible.
  3. I compute the KL-divergence between the LLM’s output and the target, and plot it relative to the entropy of the target.

Wang et al. notes that the same phenomenon occurs in randomly-initialized models, and so I may rule out the role of pre-training and limit its cause to the architecture. But for the most part, the why question remains unanswered; I take a stab at it in the next blog post.

\[\blacksquare\]

  1. In this post, I use LLMs to refer more specifically to transformer-based autoregressive language models.

  2. Though, as Basri and Jacobs (2026) demonstrates, it’s not as huge of a problem in practice.