Gradient Methods

When to Use

Use this skill when working on gradient-methods problems in optimization.

Decision Tree

1. Basic Gradient Descent

   • Update: x_{k+1} = x_k - alpha * grad f(x_k)
   • Step size alpha: fixed, diminishing, or line search
   • Convergence: O(1/k) for convex, linear for strongly convex

2. Step Size Selection

   Method	Approach
   Fixed	alpha constant (requires tuning)
   Backtracking	Armijo condition: f(x - alphagrad) <= f(x) - calpha*