Gradient Descent

Gradient descent is like hiking down a mountain in thick fog. You can't see the bottom (the lowest error), so you feel around to find the steepest downward slope at your feet and take a step in that direction. Repeat until you're as low as you can get, hoping you've found the true bottom and not just a local dip.