理解MAE损失梯度震荡

MAE损失梯度震荡完整示例重点关注求导公式几乎只依赖x误差e只有在大于0和小于0时会变成±1 梯度大小恒定不随误差缩小。一、统一符号与基础公式1. 变量定义符号含义xxx模型输入特征WWW网络权重y^\hat{y}y^​模型预测值yyy真实标签eee误差ey^−ye\hat{y}-yey^​−yLLLMAE损失$Lη\etaη学习率2. 核心公式线性预测简化无偏置b0b0b0y^W⋅x\hat{y}W \cdot xy^​W⋅x链式梯度求导dLdWdLde⋅dedy^⋅dy^dW\frac{dL}{dW}\frac{dL}{de} \cdot \frac{de}{d\hat{y}} \cdot \frac{d\hat{y}}{dW}dWdL​dedL​⋅dy^​de​⋅dWdy^​​dedy^1\displaystyle \frac{de}{d\hat{y}}1dy^​de​1dy^dWx\displaystyle \frac{d\hat{y}}{dW}xdWdy^​​xe0e0e0dLde1\displaystyle \frac{dL}{de}1dedL​1e0e0e0dLde−1\displaystyle \frac{dL}{de}-1dedL​−1权重更新规则WnewWold−η⋅dLdWW_{new}W_{old} - \eta \cdot \frac{dL}{dW}Wnew​Wold​−η⋅dWdL​二、案例1输入x1x1x1学习率η0.1\eta0.1η0.1不会跨过零点固定已知参数真实标签y5y5y5初始权重Wold5.2W_{old}5.2Wold​5.2步骤1计算初始预测值与初始误差y^oldWold⋅x5.2×15.2eoldy^old−y5.2−50.2 \begin{align} \hat{y}_{old} W_{old} \cdot x 5.2 \times 1 5.2 \\ e_{old} \hat{y}_{old} - y 5.2 - 5 0.2 \end{align}y^​old​eold​​Wold​⋅x5.2×15.2y^​old​−y5.2−50.2​​eold0.20e_{old}0.20eold​0.20步骤2计算梯度dLdW1×1×11 \frac{dL}{dW} 1 \times 1 \times 1 1dWdL​1×1×11步骤3更新权重WnewWold−η⋅dLdW5.2−0.1×15.1 W_{new} W_{old} - \eta \cdot \frac{dL}{dW} 5.2 - 0.1 \times 1 5.1Wnew​Wold​−η⋅dWdL​5.2−0.1×15.1步骤4计算更新后的预测值与新误差y^newWnew⋅x5.1×15.1enew5.1−50.1 \begin{align} \hat{y}_{new} W_{new} \cdot x 5.1 \times 1 5.1 \\ e_{new} 5.1 - 5 0.1 \end{align}y^​new​enew​​Wnew​⋅x5.1×15.15.1−50.1​​误差由0.20.20.2缩小至0.10.10.1向0靠近未跨零点。再迭代一轮e0.10e0.10e0.10梯度仍为1W5.1−0.15.0,e5.0−50 W 5.1 - 0.1 5.0,\quad e5.0-50W5.1−0.15.0,e5.0−50刚好到达最优误差0。三、案例2输入x3x3x3学习率η0.1\eta0.1η0.1一步跨过零点产生震荡固定已知参数真实标签y5y5y5初始权重Wold1.7333W_{old}1.7333Wold​1.7333步骤1初始预测、初始误差y^oldWold⋅x1.7333×35.2eold5.2−50.20 \begin{align} \hat{y}_{old} W_{old} \cdot x 1.7333 \times 3 5.2 \\ e_{old} 5.2 - 5 0.20 \end{align}y^​old​eold​​Wold​⋅x1.7333×35.25.2−50.20​​步骤2计算梯度dLdW1×1×33 \frac{dL}{dW}1 \times 1 \times 3 3dWdL​1×1×33步骤3更新权重WnewWold−η⋅dLdW1.7333−0.1×31.4333 W_{new}W_{old} - \eta \cdot \frac{dL}{dW}1.7333 - 0.1 \times 3 1.4333Wnew​Wold​−η⋅dWdL​1.7333−0.1×31.4333步骤4计算新预测、新误差y^new1.4333×34.3enew4.3−5−0.1 \begin{align} \hat{y}_{new} 1.4333 \times 3 4.3 \\ e_{new} 4.3 - 5 -0.1 \end{align}y^​new​enew​​1.4333×34.34.3−5−0.1​​现象误差从正数0.20.20.2直接变为负数−0.1-0.1−0.1跨过最优零点。下一轮迭代梯度符号反转反向拉回当前e−0.10e-0.10e−0.10dLde−1\dfrac{dL}{de}-1dedL​−1dLdW−1×1×3−3 \frac{dL}{dW}-1 \times 1 \times 3-3dWdL​−1×1×3−3Wnext1.4333−0.1×(−3)1.7333 W_{next}1.4333 - 0.1 \times (-3)1.7333Wnext​1.4333−0.1×(−3)1.7333权重回到初始值误差变回0.20.20.2形成来回横跳、损失持续震荡。核心总结MAE梯度大小由输入特征xxx固定不会随误差变小自动缩小输入特征数值较大时单次更新修正幅度超过当前误差会直接冲过误差0的最优零点跨过零点后梯度符号翻转下一轮反向更新误差在0两侧反复跳动无法稳定精准拟合MSE无此问题梯度随误差同步缩小越靠近最优值更新步长越平缓收敛稳定。