Vision Language Model: Definition and Examples

A Vision Language Model (VLM) is an AI architecture that combines visual and linguistic understanding capabilities within a single system. Unlike traditional models specialized in a single modality (text or image), VLMs can analyze an image, extract meaning from it, and produce relevant textual responses based on what they "see."

VLMs generally rely on the association of a visual encoder (often based on a Vision Transformer) and a large language model (LLM). The visual encoder transforms the image into a numerical representation that the language model can interpret. Notable examples include GPT-4o, Claude (with vision), Gemini, or LLaVA. These models are trained on vast corpora of image-text pairs to learn the correspondences between the two modalities.

In prompt engineering, VLMs open up considerable possibilities: one can submit an image accompanied by a textual instruction to obtain a description, analysis, data extraction, or even code generation from a mockup. The quality of results heavily depends on the precision of the textual prompt that accompanies the image.

Practical applications are numerous: accessibility (image description for the visually impaired), document analysis, visual content moderation, medical imaging assistance, robotics, or automation of tasks requiring a joint understanding of visual and textual information.