Inference: Definition and Examples

Inference is the stage where an artificial intelligence model moves from theory to practice. After being trained on vast amounts of data, the model uses the patterns and associations it has learned to process new inputs and produce results. This is precisely what happens every time you send a prompt to ChatGPT, Claude, or any other LLM: the model performs inference.

Concretely, inference involves passing an input (text, image, audio) through the layers of the neural network to obtain an output. In the case of large language models, this output is generated token by token: the model predicts the most likely next word or subword, then uses that prediction to generate the next, and so on until completing the response.

Inference fundamentally differs from training. Training is the learning phase, computationally expensive and time-consuming, where the model adjusts its parameters. Inference, on the other hand, is the usage phase: the model is frozen and simply applies what it has learned. That is why we often speak of 'inference cost' to refer to the resources required for each request.

In prompt engineering, understanding inference is essential because it helps you grasp why the wording of a prompt directly influences the quality of the response. The model does not 'think': it computes conditional probabilities at each generation step. A well-designed prompt steers these calculations toward more relevant and accurate results.