Neural Architecture Search: Definition and Examples

Neural Architecture Search, or NAS, refers to the set of methods that automate the design of artificial neural network architectures. Traditionally, creating a high-performing neural network relies on human expertise: a researcher or engineer manually selects the number of layers, their types, connections, and hyperparameters. NAS replaces this artisanal process with systematic algorithmic search.

The fundamental principle of NAS rests on three components: a search space that defines the set of candidate architectures, a search strategy that explores this space (evolutionary algorithms, reinforcement learning, gradient-based methods), and an evaluation method that measures the performance of each candidate architecture. The algorithm iteratively explores different configurations, evaluates their performance on a target task, and then refines its search toward the most promising regions of the space.

Early NAS approaches, such as the one proposed by Google Brain in 2017, required enormous computational resources—sometimes thousands of GPU-days. Since then, more efficient techniques have emerged, such as DARTS (Differentiable Architecture Search), which makes the search space continuous and differentiable, or weight-sharing methods that avoid training each candidate architecture from scratch.

In prompt engineering, understanding NAS helps to grasp how language models like GPT or Claude were designed. These architectures are not random but result from sophisticated optimization processes. This understanding allows you to better formulate your prompts by accounting for the structural strengths and limitations of the models you use.