Neural Architecture Search

Neural Architecture Search (NAS) automates the design of neural network architectures (e.g., number of layers, type of activation, and connections). its goal is to find the optimal architecture that performs well on the data.

Models designed with NAS are on par with or outperform those created by hand.

Components of NAS:

• Search Space: The search space defines the possible components for building different architectures.

  • The macro search space comprises individual layers (e.g., convolutional, pooling) and connection types, and the best model is found by stacking layers sequentially to form a chain structured space.
  • Micro search space: NAS builds a neural network from cells where each cell is a smaller neural network.

  The micro approach has been shown to have significant performance advantages over the macro approach.

• Search Strategy: NAS searches through the search space based on specific strategies to find the architectures to test to find the most performant one.
  Five common strategies include:
  - Grid search
  - Random Search
  - Bayesian Optimization
  - Evolutionary Algorithms
  - Reinforcement Learning.

• Performance Estimation: NAS depends on measuring the performance of the different architectures that it tries. The most straightforward approach to estimating performance is to evaluate the validation accuracy of each architecture. However, calculating validation accuracy can be computationally heavy given the large search spaces and complex networks.

  Strategies to reduce computation costs:

  • Lower fidelity estimates: Trains a subset of data using lower resolution data or using fewer filters per layer. It greatly reduces computational cost, but underestimates performance.
  • Learning curve extrapolation: given that we can predict learning curve, we can terminate poorly performing architecture.
  • Network morphism: Modifies architecture without changing underlying functions.