N-Gram Speculation

What Is N-Gram Speculation?

N-Gram speculation is a technique used to optimize inference performance, primarily for accelerating the generation process of large language models (LLMs). It predicts subsequent tokens using N-Gram matching, reducing the computational load of the model and thus improving generation speed.

Scenarios

Applicable Scenarios

Long text generation, such as stories and code completion.
High repetitiveness tasks, such as batch question answering and translation.

Inapplicable Scenario

Short text generation, for which the benefits of speculation are not significant.
High randomness tasks, such as creative writing, where the N-Gram matching rate is low.

How It Works

Cached historical n-grams: Record sequences of already generated tokens (for example, ["the", "quick", "brown"]).
Matching-based prediction: When a new input partially matches an n-gram prefix (for example, ["the", "quick"]), the subsequent token (for example, "brown") is directly predicted.
Speculative execution: If the predicted token is accepted, part of the computation is skipped; otherwise, it falls back to standard decoding.

Enabling N-Gram Speculation Parameters

Configuration Item	Parameter	Type	Description
--speculative-config	num_speculative_tokens	int	The number of tokens to predict each time.
	method	str	Speculative method: ngram.
	prompt_lookup_min	int	Minimum match length.
	prompt_lookup_max	int	Maximum match length.