吴恩达机器学习 课堂笔记 Chapter8 正则化（Regularization）

吴恩达机器学习 课堂笔记 Chapter8 正则化（Regularization）

• The problem of overfitting
• Methods for addressing overfitting
• Cost function
• Intuition
• Cost function
• Regularized linear regression
• Gradient descent
• Normal equation
• Regularized logistic regression
• Cost function
• Gradient descent

The problem of overfitting

If we have too many features, the learned hypothesis may fit the training set very well, but fail to generalize to new examples.

• Underfitting => high bias
• Overfitting => high variance

Methods for addressing overfitting

• Reduce number of features
  Manually choose which features to keep => Drop some information meanwhile.
  Model selection algorithm
• Regularization
  Keep all the features, but reduce magnitude/values of parameter $\theta_j$θj​
  Works well when we have lots of features, each of which contributes a bit to predicting y.

Cost function

Intuition

Small values for parameters $\theta_0, \theta_1, ..., \theta_n$θ0​,θ1​,...,θn​

• “Simpler” hypothesis
• Less prone to overfitting
• We cannot know in advance which ones to pick. So we ask for every parameter to be small.

Cost function

$J(\theta) = \frac{1}{2m}(\sum_{i=0}^{m}(h_\theta(x^{(i)}-y^{i})^2) + \lambda \sum_{i=1}^n\theta_j^2)$J(θ)=2m1​(i=0∑m​(hθ​(x(i)−yi)2)+λi=1∑n​θj2​)
NOTE:
We do not penalize $\theta_0$θ0​.
$\lambda$λ:regularization parameter. Need to choose it well.

Regularized linear regression

Gradient descent

Normal equation

Regularized logistic regression

Cost function

$J(\theta)=-\frac{1}{m}\sum(ylogh_\theta(x)(1-y)log(1-h_\theta(x)))+\frac{1}{2m}\sum_{i=1}^n\theta_j^2$J(θ)=−m1​∑(yloghθ​(x)(1−y)log(1−hθ​(x)))+2m1​i=1∑n​θj2​

Gradient descent

Same as above.