神经网络的前向传播和反向传播推导

神经网络的前向传播和反向传播推导

$x_{1}$x1​和 $x_{2}$x2​表示输入
$w_{ij}$wij​表示权重
$b_{ij}$bij​表示偏置
$\sigma_{i}$σi​表示激活函数，这里使用sigmoid激活函数
$out$out表示输出
$y$y表示真实值
$\eta$η表示学习率

前向传播
$h_{1}=w_{11}x_{1}+w_{13}x_{2}+b_{11}$h1​=w11​x1​+w13​x2​+b11​， $\alpha_{1}=\sigma(h1)=\frac{1}{1+e^{-h1}}$α1​=σ(h1)=1+e−h11​

$h_{2}=w_{12}x_{1}+w_{14}x_{2}+b_{12}$h2​=w12​x1​+w14​x2​+b12​， $\alpha_{2}=\sigma(h2)=\frac{1}{1+e^{-h2}}$α2​=σ(h2)=1+e−h21​

$z=w_{21}\alpha_{1}+w_{22}\alpha_{2}+b_{21}$z=w21​α1​+w22​α2​+b21​， $out=\sigma(z)=\frac{1}{1+e^{-z}}$out=σ(z)=1+e−z1​

损失函数

$E=\frac{1}{2}(out-y)^2$E=21​(out−y)2

反向传播
求导
$\bigtriangleup w_{21}=\frac{\partial E}{\partial w_{21}}=\frac{\partial E}{\partial out}\frac{{\partial out}}{\partial z}\frac{\partial z}{\partial w_{21}}=(out-y)\sigma(z)(1-\sigma(z))\alpha_{1}$△w21​=∂w21​∂E​=∂out∂E​∂z∂out​∂w21​∂z​=(out−y)σ(z)(1−σ(z))α1​

$\bigtriangleup w_{22}=\frac{\partial E}{\partial w_{22}}=\frac{\partial E}{\partial out}\frac{{\partial out}}{\partial z}\frac{\partial z}{\partial w_{22}}=(out-y)\sigma(z)(1-\sigma(z))\alpha_{2}$△w22​=∂w22​∂E​=∂out∂E​∂z∂out​∂w22​∂z​=(out−y)σ(z)(1−σ(z))α2​

$\bigtriangleup b_{21}=\frac{\partial E}{\partial b_{21}}=\frac{\partial E}{\partial out}\frac{{\partial out}}{\partial z}\frac{\partial z}{\partial b_{21}}=(out-y)\sigma(z)(1-\sigma(z))$△b21​=∂b21​∂E​=∂out∂E​∂z∂out​∂b21​∂z​=(out−y)σ(z)(1−σ(z))

更新 $w_{21}、w_{22}、b_{21}$w21​、w22​、b21​

$w_{21}=w_{21}-\eta \bigtriangleup w_{21}$w21​=w21​−η△w21​

$w_{22}=w_{22}-\eta \bigtriangleup w_{22}$w22​=w22​−η△w22​

$b_{21}=b_{21}-\eta \bigtriangleup b_{21}$b21​=b21​−η△b21​

求导

$\bigtriangleup w_{12}=\frac{\partial \alpha_{2}}{\partial h_{2}}\frac{{\partial h_{2}}}{\partial w_{12}}=\sigma(h_{2})(1-\sigma(h_{2}))x_{1}$△w12​=∂h2​∂α2​​∂w12​∂h2​​=σ(h2​)(1−σ(h2​))x1​

$\bigtriangleup w_{14}=\frac{\partial \alpha_{2}}{\partial h_{2}}\frac{{\partial h_{2}}}{\partial w_{14}}=\sigma(h_{2})(1-\sigma(h_{2}))x_{2}$△w14​=∂h2​∂α2​​∂w14​∂h2​​=σ(h2​)(1−σ(h2​))x2​

$\bigtriangleup b_{12}=\frac{\partial \alpha_{2}}{\partial h_{2}}\frac{{\partial h_{2}}}{\partial b_{12}}=\sigma(h_{2})(1-\sigma(h_{2}))$△b12​=∂h2​∂α2​​∂b12​∂h2​​=σ(h2​)(1−σ(h2​))

$\bigtriangleup w_{11}=\frac{\partial \alpha_{1}}{\partial h_{1}}\frac{{\partial h_{1}}}{\partial w_{11}}=\sigma(h_{1})(1-\sigma(h_{1}))x_{1}$△w11​=∂h1​∂α1​​∂w11​∂h1​​=σ(h1​)(1−σ(h1​))x1​

$\bigtriangleup w_{13}=\frac{\partial \alpha_{1}}{\partial h_{1}}\frac{{\partial h_{1}}}{\partial w_{13}}=\sigma(h_{1})(1-\sigma(h_{1}))x_{2}$△w13​=∂h1​∂α1​​∂w13​∂h1​​=σ(h1​)(1−σ(h1​))x2​

$\bigtriangleup b_{11}=\frac{\partial \alpha_{1}}{\partial h_{1}}\frac{{\partial h_{1}}}{\partial b_{11}}=\sigma(h_{1})(1-\sigma(h_{1}))$△b11​=∂h1​∂α1​​∂b11​∂h1​​=σ(h1​)(1−σ(h1​))

更新 $w_{12}、w_{14}、b_{12}$w12​、w14​、b12​

$w_{12}=w_{12}-\eta \bigtriangleup w_{12}$w12​=w12​−η△w12​

$w_{14}=w_{14}-\eta \bigtriangleup w_{14}$w14​=w14​−η△w14​

$b_{12}=b_{12}-\eta \bigtriangleup b_{12}$b12​=b12​−η△b12​

更新 $w_{11}、w_{13}、b_{11}$w11​、w13​、b11​

$w_{11}=w_{11}-\eta \bigtriangleup w_{11}$w11​=w11​−η△w11​

$w_{13}=w_{13}-\eta \bigtriangleup w_{13}$w13​=w13​−η△w13​

$b_{11}=b_{11}-\eta \bigtriangleup b_{11}$b11​=b11​−η△b11​