一般来说，Ensemble模型适合于过拟合的模型，包括bagging和boosting.
其中Bagging是单独训练每个分类器，然后用平均或者投票的方法组合，boosting的方法则是分类器之前存在强依赖，前一个分类器预测的解构会影响后一个分类器。随机森林就是DT的bagging。
boosting的目的是用迭代的方法提高弱分类器的性能(improving weak classifier)
boosting的架构如下：
f
1
(
x
)
f_1(x)
f 1 ( x )
f
2
(
x
)
f_2(x)
f 2 ( x )
f
1
(
x
)
f_1(x)
f 1 ( x )
f
2
(
x
)
f_2(x)
f 2 ( x )
f
1
(
x
)
f_1(x)
f 1 ( x )
f
2
(
x
)
f_2(x)
f 2 ( x )
f
1
(
x
)
f_1(x)
f 1 ( x )
u
(
i
)
u^{(i)}
u ( i )
L
(
f
)
=
∑
i
=
1
m
u
(
i
)
l
(
f
(
x
(
i
)
)
,
y
^
(
i
)
)
L(f) = \sum_{i=1}^{m} u^{(i)} l(f(x^{(i)}),\hat{y}^{(i)})
L ( f ) = i = 1 ∑ m u ( i ) l ( f ( x ( i ) ) , y ^ ( i ) )
其中
l
(
f
(
x
(
i
)
)
,
y
^
(
i
)
)
l(f(x^{(i)}),\hat{y}^{(i)})
l ( f ( x ( i ) ) , y ^ ( i ) )
f
1
,
f
2
,
.
.
.
,
f
k
f_1,f_2,...,f_k
f 1 , f 2 , . . . , f k
{
x
(
i
)
,
y
^
(
i
)
,
u
1
(
i
)
}
\{x^{(i)},\hat{y}^{(i)},u^{(i)}_1\}
{ x ( i ) , y ^ ( i ) , u 1 ( i ) }
f
1
f_1
f 1
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
{
x
(
i
)
,
y
^
(
i
)
,
u
2
(
i
)
}
\{x^{(i)},\hat{y}^{(i)},u^{(i)}_2\}
{ x ( i ) , y ^ ( i ) , u 2 ( i ) }
f
1
f_1
f 1
f
2
f_2
f 2
f
2
f_2
f 2 错误率 ：
ϵ
1
=
∑
i
=
1
m
u
1
(
i
)
δ
(
f
1
(
x
(
i
)
)
≠
y
^
(
i
)
)
Z
1
\epsilon_1 = \frac{ \sum_{i=1}^{m}u^{(i)}_1\delta(f_1(x^{(i)})\not=\hat{y}^{(i)})}{Z_1}
ϵ 1 = Z 1 ∑ i = 1 m u 1 ( i ) δ ( f 1 ( x ( i ) ) = y ^ ( i ) )
其中
δ
(
f
1
(
x
(
i
)
)
≠
y
^
(
i
)
)
\delta(f_1(x^{(i)})\not=\hat{y}^{(i)})
δ ( f 1 ( x ( i ) ) = y ^ ( i ) )
Z
1
Z_1
Z 1
Z
1
=
∑
i
=
1
m
u
1
(
i
)
Z_1 = \sum_{i=1}^{m}u^{(i)}_1
Z 1 = i = 1 ∑ m u 1 ( i )
Z
1
=
m
Z_1=m
Z 1 = m
m
m
m
m
m
m
f
1
f_1
f 1
t
t
t
ϵ
1
\epsilon_1
ϵ 1
t
t
t
t
m
\frac{t}{m}
m t
u
1
(
i
)
Z
1
\frac{u^{(i)}_1}{Z_1}
Z 1 u 1 ( i )
ϵ
1
<
0.5
\epsilon_1<0.5
ϵ 1 < 0 . 5
ϵ
1
<
1
/
K
\epsilon_1<1/K
ϵ 1 < 1 / K
u
1
(
i
)
u^{(i)}_1
u 1 ( i )
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
∑
i
=
1
m
u
2
(
i
)
δ
(
f
1
(
x
(
i
)
)
≠
y
^
(
i
)
)
Z
2
=
0.5
\frac{ \sum_{i=1}^{m}u^{(i)}_2\delta(f_1(x^{(i)})\not=\hat{y}^{(i)})}{Z_2}=0.5
Z 2 ∑ i = 1 m u 2 ( i ) δ ( f 1 ( x ( i ) ) = y ^ ( i ) ) = 0 . 5
f
1
f_1
f 1
f
1
f_1
f 1
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
{
x
(
i
)
,
y
^
(
i
)
,
u
2
(
i
)
}
\{x^{(i)},\hat{y}^{(i)},u^{(i)}_2\}
{ x ( i ) , y ^ ( i ) , u 2 ( i ) }
那如何让
f
1
f_1
f 1
f
1
f_1
f 1
f
1
f_1
f 1
f
1
f_1
f 1
1
/
3
1/\sqrt3
1 / 3
3
\sqrt3
3
f
1
f_1
f 1
如果
f
1
f_1
f 1
f
1
(
x
i
)
≠
y
^
(
i
)
f_1(x^{i})\not =\hat{y}^{(i)}
f 1 ( x i ) = y ^ ( i )
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
d
1
d_1
d 1
u
2
(
i
)
=
u
1
(
i
)
/
d
1
u^{(i)}_2=u^{(i)}_1/d_1
u 2 ( i ) = u 1 ( i ) / d 1
如果
f
1
f_1
f 1
f
1
(
x
i
)
=
y
^
(
i
)
f_1(x^{i})=\hat{y}^{(i)}
f 1 ( x i ) = y ^ ( i )
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
d
1
d_1
d 1
u
2
(
i
)
=
u
1
(
i
)
d
1
u^{(i)}_2=u^{(i)}_1 d_1
u 2 ( i ) = u 1 ( i ) d 1
d
1
d_1
d 1
∑
i
=
1
m
u
2
(
i
)
δ
(
f
1
(
x
(
i
)
)
≠
y
^
(
i
)
)
Z
2
=
0.5
\frac{ \sum_{i=1}^{m}u^{(i)}_2\delta(f_1(x^{(i)})\not=\hat{y}^{(i)})}{Z_2}=0.5
Z 2 ∑ i = 1 m u 2 ( i ) δ ( f 1 ( x ( i ) ) = y ^ ( i ) ) = 0 . 5
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
u
1
(
i
)
d
1
u^{(i)}_1d_1
u 1 ( i ) d 1
u
1
(
i
)
/
d
1
u^{(i)}_1/d_1
u 1 ( i ) / d 1
∑
f
1
(
x
(
i
)
)
=
y
^
(
i
)
u
1
(
i
)
/
d
1
=
∑
f
1
(
x
(
i
)
)
≠
y
^
(
i
)
u
1
(
i
)
d
1
\sum_{f_1(x^{(i)})=\hat{y}^{(i)}} u^{(i)}_1/d_1=\sum_{f_1(x^{(i)})\not=\hat{y}^{(i)}} u^{(i)}_1d_1
f 1 ( x ( i ) ) = y ^ ( i ) ∑ u 1 ( i ) / d 1 = f 1 ( x ( i ) ) = y ^ ( i ) ∑ u 1 ( i ) d 1
也就是那些分类错误的数据的权重的和要等于分类正确的数据的权重的和，在经过推导得到
d
1
d_1
d 1
d
1
=
1
−
ϵ
1
ϵ
1
>
1
d_1 = \sqrt{\frac{1-\epsilon_1}{\epsilon_1}}>1
d 1 = ϵ 1 1 − ϵ 1
> 1
d
1
d_1
d 1
a
1
=
log
d
1
=
log
(
1
−
ϵ
1
ϵ
1
)
=
1
2
log
1
−
ϵ
1
ϵ
1
a_1 = \log d_1 = \log (\sqrt{\frac{1-\epsilon_1}{\epsilon_1}})= \frac{1}{2}\log \frac{1-\epsilon_1}{\epsilon_1}
a 1 = log d 1 = log ( ϵ 1 1 − ϵ 1
) = 2 1 log ϵ 1 1 − ϵ 1
如果
f
1
f_1
f 1
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
u
2
(
i
)
=
u
1
(
i
)
e
−
a
1
u^{(i)}_2=u^{(i)}_1 e^{-a_1}
u 2 ( i ) = u 1 ( i ) e − a 1
如果
f
1
f_1
f 1
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
u
2
(
i
)
=
u
1
(
i
)
e
a
1
u^{(i)}_2=u^{(i)}_1 e^{a_1}
u 2 ( i ) = u 1 ( i ) e a 1
u
2
(
i
)
u^{(i)}_2
u 2 ( i )
u
2
(
i
)
=
u
1
(
i
)
×
exp
(
−
y
^
(
i
)
f
1
(
x
(
i
)
)
a
1
)
u^{(i)}_2 = u^{(i)}_1 \times \exp(-\hat{y}^{(i)}f_1(x^{(i)})a_1)
u 2 ( i ) = u 1 ( i ) × exp ( − y ^ ( i ) f 1 ( x ( i ) ) a 1 )
当预测值和真实值相同时，
−
y
^
(
i
)
f
1
(
x
(
i
)
)
=
−
1
-\hat{y}^{(i)}f_1(x^{(i)})=-1
− y ^ ( i ) f 1 ( x ( i ) ) = − 1
综上，二分类的Adaboost算法就可以概况为：
{
(
x
(
1
)
,
y
^
(
1
)
,
u
1
(
1
)
)
,
（
x
(
2
)
,
y
^
(
2
)
,
u
1
(
2
)
)
,
.
.
.
,
（
x
(
m
)
,
y
^
(
m
)
,
u
1
(
m
)
)
}
\{(x^{(1)},\hat{y}^{(1)},u^{(1)}_1),（x^{(2)},\hat{y}^{(2)},u^{(2)}_1),...,（x^{(m)},\hat{y}^{(m)},u^{(m)}_1)\}
{ ( x ( 1 ) , y ^ ( 1 ) , u 1 ( 1 ) ) , （ x ( 2 ) , y ^ ( 2 ) , u 1 ( 2 ) ) , . . . , （ x ( m ) , y ^ ( m ) , u 1 ( m ) ) }
y
(
i
)
=
±
1
y^{(i)}=\pm 1
y ( i ) = ± 1
u
1
(
i
)
=
1
u^{(i)}_1=1
u 1 ( i ) = 1
对于弱分类器
t
=
1
,
2
,
.
.
.
,
T
t=1,2,...,T
t = 1 , 2 , . . . , T
{
u
1
(
1
)
,
u
1
(
2
)
,
.
.
.
,
u
1
(
m
)
}
\{u^{(1)}_1,u^{(2)}_1,...,u^{(m)}_1\}
{ u 1 ( 1 ) , u 1 ( 2 ) , . . . , u 1 ( m ) }
ϵ
t
\epsilon_t
ϵ t
a
t
a_t
a t
u
2
(
i
)
=
u
1
(
i
)
×
exp
(
−
y
^
(
i
)
f
1
(
x
(
i
)
)
a
1
)
u^{(i)}_2 = u^{(i)}_1 \times \exp(-\hat{y}^{(i)}f_1(x^{(i)})a_1)
u 2 ( i ) = u 1 ( i ) × exp ( − y ^ ( i ) f 1 ( x ( i ) ) a 1 )
训练得到一系列的弱分类器
f
1
,
f
2
,
.
.
.
,
f
T
f_1,f_2,...,f_T
f 1 , f 2 , . . . , f T
H
(
x
)
=
s
i
g
n
(
∑
t
=
1
T
a
t
f
t
(
x
)
)
H(x) = sign(\sum_{t=1}^T a_t f_t(x))
H ( x ) = s i g n ( t = 1 ∑ T a t f t ( x ) )
a
t
a_t
a t
a
t
a_t
a t
组合的策略从直观上解释就是错误率低的分类器权重更高，反之权重更低。
至此adaboost讲完
