Natively Controllable Language Models
We believe the future belongs to capable and controllable language models. The current autoregressive, next-token paradigm stands as a fundamental bottleneck to the model’s controllability. We illustrate the bottleneck of the autoregressive model design with a simple task: “Respond in n words”. In contrast, non-autoregressive models can succeed without any additional training.
waiting for a prompt…
same model, same seed, constraint off
autoregressive baseline
Comparison with autoregressive decoding
We baseline against multiple autoregressive steering methods: (1) prompting—Respond in exactly n words (2) best-of-k generation—generate k samples and select the best (3) constrained decoding—force the model to end its sequence after outputting n words (4) constrained decoding + prompting.
For a fair comparison we use two matched models: the autoregressive 7B Mistral-7B-Instruct, and a 7B diffusion language model TESS-2, which was adapted from Mistral-7B-Instruct.
Because the autoregressive model predicts the next token, the user can only steer a single token at a time. This is a big limitation, because in practice we care about the entire sequence satisfying certain properties. In contrast, the diffusion language model predicts the entire sequence, allowing the user to steer the entire sequence. Specifically, we can steer our sequence to better satisfy a plug-in objective of our choosing. We define ours as the squared distance towards our target number of words: v = -(E[words] - target)² , where the expected number of words can be read from the model’s prediction of the sequence:
The objective function may also be a trained model (ex: sentiment/safety classifier).
Looking at word count alone, constrained decoding method performs well, since it prevents the model from exceeding the target number of words. However, when incorporating completeness, whether the model’s response actually completed in n words, we see that the autoregressive methods fail: 12.9% with constrained decoding plus prompting versus 67.6% with guided steering of a diffusion language model.
Comparing steering methods
Table 1. We compare steering methods using n=256 generations per method, from 32 open-domain prompts with 4 target lengths and 2 seeds. Both is our success criteria: exactly n words, a finished sentence, and ends with real text. Perplexity is scored by GPT-2 Large.
Why this matters
Model will continue to become more capable. However, there is still a lot of gains to be had for improving the model’s controllability. Non-autoregressive models will allow users to easily steer model’s outputs to their desired objectives. As models become more capable, their ability to be steered by the user becomes increasingly important to ensure an aligned future.
At Lateral Intelligence, we are working towards advancing the next generation of language modelling with more natively controllable language models. We’re a small team based in San Francisco.
If you’re building in this direction and want to join, reach out at join@lateralintelligence.org.
If you’re interested in supporting the work, get in touch at invest@lateralintelligence.org.