本文做为第2部分，主要根据原始论文介绍几篇基础的工做，主要包括GNN，GCN及变体，DCNN，Tree-LSTM,包括模型的详解和模型训练，以及模型评价。html
这篇论文是第一个提出Graph Neural Network模型的论文，它将神经网络使用在图结构数据上，并细述了神经网络模型告终构组成、计算方法、优化算法、流程实现等等。论文后面还对模型的复杂度进行了评估，以及在现实任务上进行了实验和比较(比较算法为NL、L、FNN)。该报告暂时主要关注模型设计部分和实验结果部分，忽略复杂性评估部分 。node
对于图领域问题，假设函数
τ
\tau
τ
G
G
G
n
n
n
τ
(
G
,
n
)
∈
R
m
\tau(G,n)\in{R^m}
τ ( G , n ) ∈ R m python
图领域的应用主要能够分为两种类型：专一于图的应用(graph-focused)和专一于节点的应用(node-focused) 。对于graph-focused的应用，函数
τ
\tau
τ
τ
(
G
)
\tau(G)
τ ( G )
τ
\tau
τ
n
n
n
τ
(
G
,
n
)
\tau(G,n)
τ ( G , n ) web
图 (a) 是一个化学分子结构，可以使用图
G
G
G
τ
(
G
)
\tau(G)
τ ( G )
图 (b) 是一张城堡的图片，图片中的每一种结构都由节点表示，函数
τ
(
G
,
n
)
\tau(G,n)
τ ( G , n )
GNN模型基于信息传播机制，每个节点经过相互交换信息来更新本身的节点状态，直到达到某一个稳定值 ，GNN的输出就是在每一个节点处，根据当前节点状态分别计算输出。算法
有以下定义：json
一个图
G
G
G
(
N
,
E
)
(\boldsymbol{N}, \boldsymbol{E})
( N , E )
N
\boldsymbol{N}
N
E
\boldsymbol{E}
E 网络
n
e
[
n
]
ne[n]
n e [ n ]
n
n
n app
c
o
[
n
]
co[n]
c o [ n ]
n
n
n 框架
l
n
∈
R
l
N
\boldsymbol{l}_{n} \in \mathbb{R}^{l_{N}}
l n ∈ R l N
n
n
n ide
l
(
n
1
,
n
2
)
∈
R
l
E
\boldsymbol{l}_{\left(n_{1}, n_{2}\right)} \in \mathbb{R}^{l_{E}}
l ( n 1 , n 2 ) ∈ R l E
(
n
1
,
n
2
)
(n_1,n_2)
( n 1 , n 2 )
l
\boldsymbol{l}
l
注 ：原论文里面
l
\boldsymbol{l}
l features of objects related to nodes and features of the relationships between the objects，也就是相关特征，因此这里一概使用特征向量翻译。
论文将图分为positional graph和nonpositional graph，对于positional graph，对于每个节点
n
n
n
u
u
u
ν
n
(
u
)
\nu_{n}(u)
ν n ( u )
ν
n
:
n
e
[
n
]
→
{
1
,
…
∣
N
∣
}
\nu_{n} : \mathrm{ne}[n] \rightarrow\{1, \ldots|\mathbf{N}|\}
ν n : n e [ n ] → { 1 , … ∣ N ∣ }
假设存在一个图-节点对的集合
D
=
G
×
N
\mathcal{D}=\mathcal{G} \times \mathcal{N}
D = G × N
G
\mathcal{G}
G
N
\mathcal{N}
N 图领域问题能够表示成一个有以下数据集的监督学习框架
L
=
{
(
G
i
,
n
i
,
j
,
t
i
,
j
)
∣
G
i
=
(
N
i
,
E
i
)
∈
G
;
n
i
,
j
∈
N
i
;
t
i
,
j
∈
R
m
,
1
≤
i
≤
p
,
1
≤
j
≤
q
i
}
\mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i }
n
i
,
j
∈
N
i
n_{i, j} \in \boldsymbol{N}_{i}
n i , j ∈ N i
N
i
∈
N
\boldsymbol{N}_{i} \in \mathcal{N}
N i ∈ N
j
j
j
t
i
,
j
\boldsymbol{t}_{i, j}
t i , j
n
i
j
n_{ij}
n i j
节点
n
n
n
x
n
∈
R
s
\boldsymbol{x}_{n} \in \mathbb{R}^{s}
x n ∈ R s
o
n
\boldsymbol{o}_{\boldsymbol{n}}
o n
f
w
f_{\boldsymbol{w}}
f w local transition function ，
g
w
g_{\boldsymbol{w}}
g w local output function ，那么
x
n
\boldsymbol{x}_{n}
x n
o
n
\boldsymbol{o}_{\boldsymbol{n}}
o n
x
n
=
f
w
(
l
n
,
l
c
o
[
n
]
,
x
n
e
[
n
]
,
l
n
e
[
n
]
)
o
n
=
g
w
(
x
n
,
l
n
)
\begin{array}{l}{\boldsymbol{x}_{n}=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}\left[n\right]}\right)} \\ {\boldsymbol{o}_{n}=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}, \boldsymbol{l}_{n}\right)}\end{array}
x n = f w ( l n , l c o [ n ] , x n e [ n ] , l n e [ n ] ) o n = g w ( x n , l n )
l
n
,
l
co
[
n
]
,
x
n
e
[
n
]
,
l
n
e
[
n
]
\boldsymbol{l}_{n}, \boldsymbol{l}_{\operatorname{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}, \boldsymbol{l}_{\mathrm{ne}[n]}
l n , l c o [ n ] , x n e [ n ] , l n e [ n ]
n
n
n
n
n
n
n
n
n
n
n
n
假设
x
,
o
,
l
,
l
N
\boldsymbol{x}, \boldsymbol{o}, \boldsymbol{l}, \boldsymbol{l}_{N}
x , o , l , l N
x
=
F
w
(
x
,
l
)
o
=
G
w
(
x
,
l
N
)
\begin{array}{l}{\boldsymbol{x}=F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})} \\ {\boldsymbol{o}=\boldsymbol{G}_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)}\end{array}
x = F w ( x , l ) o = G w ( x , l N )
F
w
F_{\boldsymbol{w}}
F w global transition function ，
G
w
G_{\boldsymbol{w}}
G w global output function ，分别是
f
w
f_{\boldsymbol{w}}
f w
g
w
g_{\boldsymbol{w}}
g w
根据Banach的不动点理论，假设
F
w
F_{\boldsymbol{w}}
F w
x
(
t
+
1
)
=
F
w
(
x
(
t
)
,
l
)
\boldsymbol{x}(t+1)=F_{\boldsymbol{w}}(\boldsymbol{x}(t), \boldsymbol{l})
x ( t + 1 ) = F w ( x ( t ) , l )
x
(
t
)
\boldsymbol{x}(t)
x ( t )
x
\boldsymbol{x}
x
t
t
t
x
n
(
t
+
1
)
=
f
w
(
l
n
,
l
c
o
[
n
]
,
x
n
e
[
n
]
(
t
)
,
l
n
e
[
n
]
)
o
n
(
t
)
=
g
w
(
x
n
(
t
)
,
l
n
)
,
n
∈
N
\begin{aligned} \boldsymbol{x}_{n}(t+1) &=f_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{\mathrm{co}[n]}, \boldsymbol{x}_{\mathrm{ne}[n]}(t), \boldsymbol{l}_{\mathrm{ne}[n]}\right) \\ \boldsymbol{o}_{n}(t) &=g_{\boldsymbol{w}}\left(\boldsymbol{x}_{n}(t), \boldsymbol{l}_{n}\right), \quad n \in \boldsymbol{N} \end{aligned}
x n ( t + 1 ) o n ( t ) = f w ( l n , l c o [ n ] , x n e [ n ] ( t ) , l n e [ n ] ) = g w ( x n ( t ) , l n ) , n ∈ N
T
T
T
GNN的学习就是估计参数
w
\boldsymbol{w}
w
φ
w
\varphi_{\boldsymbol{w}}
φ w
L
=
{
(
G
i
,
n
i
,
j
,
t
i
,
j
)
∣
G
i
=
(
N
i
,
E
i
)
∈
G
;
n
i
,
j
∈
N
i
;
t
i
,
j
∈
R
m
,
1
≤
i
≤
p
,
1
≤
j
≤
q
i
}
\mathcal{L}=\left\{\left(\boldsymbol{G}_{i}, n_{i, j}, \boldsymbol{t}_{i, j}\right)| \boldsymbol{G}_{i}=\left(\boldsymbol{N}_{i}, \boldsymbol{E}_{i}\right) \in \mathcal{G}\right.;n_{i, j} \in \boldsymbol{N}_{i} ; \boldsymbol{t}_{i, j} \in \mathbb{R}^{m}, 1 \leq i \leq p, 1 \leq j \leq q_{i} \}
L = { ( G i , n i , j , t i , j ) ∣ G i = ( N i , E i ) ∈ G ; n i , j ∈ N i ; t i , j ∈ R m , 1 ≤ i ≤ p , 1 ≤ j ≤ q i }
q
i
q_i
q i
G
i
G_{i}
G i
e
w
=
∑
i
=
1
p
∑
j
=
1
q
i
(
t
i
,
j
−
φ
w
(
G
i
,
n
i
,
j
)
)
2
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2
按照迭代方程迭代
T
T
T
x
n
(
t
)
x_{n}(t)
x n ( t )
x
(
T
)
≈
x
\boldsymbol{x}(T) \approx \boldsymbol{x}
x ( T ) ≈ x
计算参数权重的梯度
∂
e
w
(
T
)
/
∂
w
\partial e_{\boldsymbol{w}}(T) / \partial \boldsymbol{w}
∂ e w ( T ) / ∂ w
使用该梯度来更新权重
w
\boldsymbol{w}
w
这里假设函数
F
w
F_{\boldsymbol{w}}
F w backpropagation-through-time algorithm 。
为了代表前面的方法是可行的，论文接着证实了两个结论
理论1(可微性)：令
F
w
F_{\boldsymbol{w}}
F w
G
w
G_{\boldsymbol{w}}
G w
F
w
(
x
,
l
)
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
F w ( x , l )
G
w
(
x
,
l
N
)
G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)
G w ( x , l N )
x
\boldsymbol{x}
x
w
\boldsymbol{w}
w
φ
w
\varphi_{\boldsymbol{w}}
φ w
w
\boldsymbol{w}
w
理论2(反向传播)：令
F
w
F_{\boldsymbol{w}}
F w
G
w
G_{\boldsymbol{w}}
G w
F
w
(
x
,
l
)
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
F w ( x , l )
G
w
(
x
,
l
N
)
G_{\boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{\boldsymbol{N}}\right)
G w ( x , l N )
x
\boldsymbol{x}
x
w
\boldsymbol{w}
w
z
(
t
)
\boldsymbol{z}(t)
z ( t )
z
(
t
)
=
z
(
t
+
1
)
⋅
∂
F
w
∂
x
(
x
,
l
)
+
∂
e
w
∂
o
⋅
∂
G
w
∂
x
(
x
,
l
N
)
z(t)=z(t+1) \cdot \frac{\partial F_{w}}{\partial x}(x, l)+\frac{\partial e_{w}}{\partial o} \cdot \frac{\partial G_{w}}{\partial x}\left(x, l_{N}\right)
z ( t ) = z ( t + 1 ) ⋅ ∂ x ∂ F w ( x , l ) + ∂ o ∂ e w ⋅ ∂ x ∂ G w ( x , l N )
z
(
T
)
,
z
(
T
−
1
)
,
…
\boldsymbol{z}(T), \boldsymbol{z}(T-1), \ldots
z ( T ) , z ( T − 1 ) , …
z
=
lim
t
→
−
∞
z
(
t
)
z=\lim _{t \rightarrow-\infty} z(t)
z = lim t → − ∞ z ( t )
z
(
T
)
\boldsymbol{z}(T)
z ( T )
∂
e
w
∂
w
=
∂
e
w
∂
o
⋅
∂
G
w
∂
w
(
x
,
l
N
)
+
z
⋅
∂
F
w
∂
w
(
x
,
l
)
\frac{\partial e_{w}}{\partial \boldsymbol{w}}=\frac{\partial e_{\boldsymbol{w}}}{\partial \boldsymbol{o}} \cdot \frac{\partial G_{\boldsymbol{w}}}{\partial \boldsymbol{w}}\left(\boldsymbol{x}, \boldsymbol{l}_{N}\right)+\boldsymbol{z} \cdot \frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})
∂ w ∂ e w = ∂ o ∂ e w ⋅ ∂ w ∂ G w ( x , l N ) + z ⋅ ∂ w ∂ F w ( x , l )
x
\boldsymbol{x}
x
算法流程以下FORWARD 用于迭代计算出收敛点，BACKWARD 用于计算梯度。
在GNN中，函数
g
w
g_{\boldsymbol{w}}
g w
f
w
f_{\boldsymbol{w}}
f w
f
w
f_{\boldsymbol{w}}
f w
Linear(nonpositional) GNN ：
对于节点
n
n
n
f
w
f_{\boldsymbol{w}}
f w
x
n
=
∑
u
∈
ne
∣
n
]
h
w
(
l
n
,
l
(
n
,
u
)
,
x
u
,
l
u
)
,
n
∈
N
\boldsymbol{x}_{n}=\sum_{u \in \text { ne } | n ]} h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right), \quad n \in \boldsymbol{N}
x n = u ∈ ne ∣ n ] ∑ h w ( l n , l ( n , u ) , x u , l u ) , n ∈ N
n
n
n
h
w
h_{\boldsymbol{w}}
h w
n
n
n
由此，对上式中的函数
h
w
h_{\boldsymbol{w}}
h w
h
w
(
l
n
,
l
(
n
,
a
)
,
x
u
,
l
u
)
=
A
n
,
u
x
u
+
b
n
h_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, \mathfrak{a})}, \boldsymbol{x}_{u}, \boldsymbol{l}_{u}\right) = \boldsymbol{A}_{n, u} \boldsymbol{x}_{u}+\boldsymbol{b}_{n}
h w ( l n , l ( n , a ) , x u , l u ) = A n , u x u + b n
b
n
∈
R
s
\boldsymbol{b}_{n} \in \mathbb{R}^{s}
b n ∈ R s
A
n
,
u
∈
R
s
×
s
\boldsymbol{A}_{n, u} \in \mathbb{R}^{s \times s}
A n , u ∈ R s × s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u transition network ，产生向量
b
n
\boldsymbol{b}_{n}
b n forcing network
transition network表示为
ϕ
w
\phi_{\boldsymbol{w}}
ϕ w
ϕ
w
:
R
2
l
N
+
l
E
→
R
s
2
\phi_{\boldsymbol{w}} : \mathbb{R}^{2 l_{N}+l_{E}} \rightarrow \mathbb{R}^{s^{2}}
ϕ w : R 2 l N + l E → R s 2
ρ
w
\rho_{\boldsymbol{w}}
ρ w
ρ
w
:
R
l
N
→
R
s
\rho_{\boldsymbol{w}} : \mathbb{R}^{l_{N}} \rightarrow \mathbb{R}^{s}
ρ w : R l N → R s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
A
n
,
u
=
μ
s
∣
ne
[
u
]
∣
⋅
Ξ
b
w
=
ρ
w
(
l
n
)
\begin{aligned} \boldsymbol{A}_{\boldsymbol{n}, \boldsymbol{u}} &=\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \boldsymbol{\Xi} \\ \boldsymbol{b}_{\boldsymbol{w}} &=\rho_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}\right) \end{aligned}
A n , u b w = s ∣ n e [ u ] ∣ μ ⋅ Ξ = ρ w ( l n )
μ
∈
(
0
,
1
)
\mu \in(0,1)
μ ∈ ( 0 , 1 )
Ξ
=
resize
(
ϕ
w
(
l
n
,
l
(
n
,
u
)
,
l
u
)
)
\Xi=\operatorname{resize}\left(\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(n, u)}, \boldsymbol{l}_{u}\right)\right)
Ξ = r e s i z e ( ϕ w ( l n , l ( n , u ) , l u ) )
resize
(
⋅
)
\text{resize}(\cdot)
resize ( ⋅ )
s
2
s^2
s 2
s
×
s
s\times{s}
s × s
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
在这里，假定
∥
ϕ
w
(
l
n
,
l
(
n
,
u
)
,
l
u
)
∥
1
≤
s
\left\|\phi_{\boldsymbol{w}}\left(\boldsymbol{l}_{n}, \boldsymbol{l}_{(\boldsymbol{n}, \boldsymbol{u})}, \boldsymbol{l}_{u}\right)\right\|_{1} \leq \boldsymbol{s}
∥ ∥ ϕ w ( l n , l ( n , u ) , l u ) ∥ ∥ 1 ≤ s
t
a
n
h
(
)
tanh()
t a n h ( )
F
w
(
x
,
l
)
=
A
x
+
b
F_{\boldsymbol{w}}(\boldsymbol{x}, \boldsymbol{l})=\boldsymbol{A} \boldsymbol{x}+\boldsymbol{b}
F w ( x , l ) = A x + b
A
\boldsymbol{A}
A
b
\boldsymbol{b}
b
A
n
,
u
\boldsymbol{A}_{n, u}
A n , u
b
n
\boldsymbol{b}_{n}
b n
∥
∂
F
w
∂
x
∥
1
=
∥
A
∥
1
≤
max
u
∈
N
(
∑
n
∈
ne
[
u
]
∥
A
n
,
u
∥
1
)
≤
max
u
∈
N
(
μ
s
∣
ne
[
u
]
∣
⋅
∑
n
∈
n
e
[
u
]
∥
Ξ
∥
1
)
≤
μ
\begin{aligned}\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|_{1} &=\|\boldsymbol{A}\|_{1} \leq \max _{u \in \boldsymbol{N}}\left(\sum_{n \in \operatorname{ne}[u]}\left\|\boldsymbol{A}_{n, u}\right\|_{1}\right) \\ & \leq \max _{u \in N}\left(\frac{\mu}{s|\operatorname{ne}[u]|} \cdot \sum_{n \in \mathrm{ne}[u]}\|\mathbf{\Xi}\|_{1}\right) \leq \mu \end{aligned}
∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ 1 = ∥ A ∥ 1 ≤ u ∈ N max ⎝ ⎛ n ∈ n e [ u ] ∑ ∥ A n , u ∥ 1 ⎠ ⎞ ≤ u ∈ N max ⎝ ⎛ s ∣ n e [ u ] ∣ μ ⋅ n ∈ n e [ u ] ∑ ∥ Ξ ∥ 1 ⎠ ⎞ ≤ μ
F
w
F_{\boldsymbol{w}}
F w
w
\boldsymbol{w}
w
矩阵
M
M
M
∥
M
∥
1
=
max
j
∑
i
∣
m
i
,
j
∣
\|M\|_{1}=\max _{j} \sum_{i}\left|m_{i, j}\right|
∥ M ∥ 1 = j max i ∑ ∣ m i , j ∣
Nonelinear(nonpositional) GNN ：在这个结构中，
h
w
h_{\boldsymbol{w}}
h w
w
\boldsymbol{w}
w
F
w
F_{\boldsymbol{w}}
F w
e
w
=
∑
i
=
1
p
∑
j
=
1
q
i
(
t
i
,
j
−
φ
w
(
G
i
,
n
i
,
j
)
)
2
+
β
L
(
∥
∂
F
w
∂
x
∥
)
e_{\boldsymbol{w}}=\sum_{i=1}^{p} \sum_{j=1}^{q_{i}}\left(\boldsymbol{t}_{i, j}-\varphi_{\boldsymbol{w}}\left(\boldsymbol{G}_{i}, n_{i, j}\right)\right)^{2}+\beta L\left(\left\|\frac{\partial F_{\boldsymbol{w}}}{\partial \boldsymbol{x}}\right\|\right)
e w = i = 1 ∑ p j = 1 ∑ q i ( t i , j − φ w ( G i , n i , j ) ) 2 + β L ( ∥ ∥ ∥ ∥ ∂ x ∂ F w ∥ ∥ ∥ ∥ )
L
(
y
)
L(y)
L ( y )
y
>
μ
y>\mu
y > μ
(
y
−
μ
)
2
(y-\mu)^2
( y − μ ) 2
y
≤
μ
y\le{\mu}
y ≤ μ
μ
∈
(
0
,
1
)
\mu\in(0,1)
μ ∈ ( 0 , 1 )
F
w
F_{\boldsymbol{w}}
F w
论文将GNN模型在三个任务上进行了实验：子图匹配(subgraph matching)任务，诱变(mutagenesis)任务和网页排序(web page ranking)任务。在这些任务上使用linear和nonlinear的模型测试，其中nonlinear模型中的激活函数使用sigmoid函数。
子图匹配任务为在一个大图
G
G
G
S
S
S
τ
\tau
τ
n
i
,
j
n_{i,j}
n i , j
G
G
G
τ
(
G
i
,
n
i
,
j
)
=
1
\tau(G_i,n_{i,j})=1
τ ( G i , n i , j ) = 1
τ
(
G
i
,
n
i
,
j
)
=
−
1
\tau(G_i,n_{i,j})=-1
τ ( G i , n i , j ) = − 1
诱变问题任务是对化学分子进行分类，识别出诱变化合物，采用二分类方法。实验结果是nonlinear效果较好，但不是最好。
网页排序任务是学会网页排序。实验代表虽然训练集只包含50个网页，可是仍然没有产生过拟合的现象。
在模拟的节点分类任务上实现该论文的GNN模型。
任务要求 为输入一个graph，该graph的全部节点都有标签，而后对部分节点进行训练，在验证阶段使用另一部分节点进行验证。输入的数据图以下图：
{
n
1
,
n
2
,
.
.
.
,
n
18
}
\{n1,n2,...,n18\}
{ n 1 , n 2 , . . . , n 1 8 }
{
0
,
1
,
2
}
\{0,1,2\}
{ 0 , 1 , 2 }
{
n
1
,
n
2
,
n
3
,
n
7
,
n
8
,
n
9
,
n
13
,
n
14
,
n
15
}
\{n1,n2,n3,n7,n8,n9,n13,n14,n15\}
{ n 1 , n 2 , n 3 , n 7 , n 8 , n 9 , n 1 3 , n 1 4 , n 1 5 }
输入的数据为(node,label)列表和(node1, node2)边列表，表示以下
N = [ ( "n{}" . format ( i) , 0 ) for i in range ( 1 , 7 ) ] + \
[ ( "n{}" . format ( i) , 1 ) for i in range ( 7 , 13 ) ] + \
[ ( "n{}" . format ( i) , 2 ) for i in range ( 13 , 19 ) ]
E = [ ( "n1" , "n2" ) , ( "n1" , "n3" ) , ( "n1" , "n5" ) ,
( "n2" , "n4" ) ,
( "n3" , "n6" ) , ( "n3" , "n9" ) ,
( "n4" , "n5" ) , ( "n4" , "n6" ) , ( "n4" , "n8" ) ,
( "n5" , "n14" ) ,
( "n7" , "n8" ) , ( "n7" , "n9" ) , ( "n7" , "n11" ) ,
( "n8" , "n10" ) , ( "n8" , "n11" ) , ( "n8" , "n12" ) ,
( "n9" , "n10" ) , ( "n9" , "n14" ) ,
( "n10" , "n12" ) ,
( "n11" , "n18" ) ,
( "n13" , "n15" ) , ( "n13" , "n16" ) , ( "n13" , "n18" ) ,
( "n14" , "n16" ) , ( "n14" , "n18" ) ,
( "n15" , "n16" ) , ( "n15" , "n18" ) ,
( "n17" , "n18" ) ]
N
N
N
E
E
E
模型部分使用论文的linear函数来设计
f
w
f_w
f w
g
w
g_w
g w
Ξ
\Xi
Ξ
ρ
\rho
ρ
class Xi ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Rou ( nn. Module) :
def __init__ ( self, ln, s) :
. . .
def forward ( self, X) :
. . .
class Hw ( nn. Module) :
def __init__ ( self, ln, s, mu= 0.9 ) :
. . .
def forward ( self, X, H, dg_list) :
. . .
class AggrSum ( nn. Module) :
def __init__ ( self, node_num) :
. . .
def forward ( self, H, X_node) :
. . .
class OriLinearGNN ( nn. Module) :
def __init__ ( self, node_num, feat_dim, stat_dim, T) :
. . .
def forward ( self, X_Node, X_Neis, dg_list) :
. . .
for t in range ( self. T) :
H = torch. index_select( self. node_states, 0 , X_Node)
H = self. Hw( X, H, dg_list)
self. node_states = self. Aggr( H, X_Node)
. . . 能够看出，在模型训练阶段，每次forward，都会直接循环计算T次
f
w
f_w
f w
模型训练部分按照常规的分类模型进行训练，采用Adam优化器，学习率保持为0.01，权重衰减为0.01，使用交叉熵做为损失函数，模型训练部分代码以下
def CalAccuracy ( output, label) :
. . .
def train ( node_list, edge_list, label_list, T, ndict_path= "./node_dict.json" ) :
. . .
. . .
. . .
. . .
train_node_list = [ 0 , 1 , 2 , 6 , 7 , 8 , 12 , 13 , 14 ]
train_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
test_node_list = [ 3 , 4 , 5 , 9 , 10 , 11 , 15 , 16 , 17 ]
test_node_label = [ 0 , 0 , 0 , 1 , 1 , 1 , 2 , 2 , 2 ]
model = OriLinearGNN( node_num= len ( node_list) ,
feat_dim= 2 ,
stat_dim= 2 ,
T= T)
optimizer = torch. optim. Adam( model. parameters( ) , lr= 0.01 , weight_decay= 0.01 )
criterion = nn. CrossEntropyLoss( size_average= True )
min_loss = float ( 'inf' )
node_inds_tensor = Variable( torch. Tensor( node_inds) . long ( ) )
node_neis_tensor = Variable( torch. Tensor( node_neis) . long ( ) )
train_label = Variable( torch. Tensor( train_node_label) . long ( ) )
for ep in range ( 500 ) :
res = model( node_inds_tensor, node_neis_tensor, dg_list)
train_res = torch. index_select( res, 0 , torch. Tensor( train_node_list) . long ( ) )
test_res = torch. index_select( res, 0 , torch. Tensor( test_node_list) . long ( ) )
loss = criterion( input = train_res,
target= train_label)
loss_val = loss. item( )
train_acc = CalAccuracy( train_res. cpu( ) . detach( ) . numpy( ) , np. array( train_node_label) )
test_acc = CalAccuracy( test_res. cpu( ) . detach( ) . numpy( ) , np. array( test_node_label) )
optimizer. zero_grad( )
loss. backward( retain_graph= True )
optimizer. step( )
if loss_val < min_loss:
min_loss = loss_val
print ( "==> [Epoch {}] : loss {:.4f}, min_loss {:.4f}, train_acc {:.3f}, test_acc {:.3f}" . format ( ep, loss_val, min_loss, train_acc, test_acc) )
模型的训练和评估结果以下图:
2
3
\frac{2}{3}
3 2
按照图傅里叶变换的性质，能够获得以下图卷积的定义
(
f
∗
h
)
G
=
Φ
diag
[
h
^
(
λ
1
)
,
…
,
h
^
(
λ
n
)
]
Φ
T
f
(\boldsymbol{f} * \boldsymbol{h})_{\mathcal{G}}=\boldsymbol{\Phi} \operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right] \mathbf{\Phi}^{T} \boldsymbol{f}
( f ∗ h ) G = Φ d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ] Φ T f
对于图$ \boldsymbol{f}
的
傅
里
叶
变
换
为
的傅里叶变换为
的 傅 里 叶 变 换 为
对于卷积核的图傅里叶变换：
h
^
=
(
h
^
1
,
…
,
h
^
n
)
\hat{h}=\left(\hat{h}_{1}, \ldots, \hat{h}_{n}\right)
h ^ = ( h ^ 1 , … , h ^ n )
h
^
k
=
⟨
h
,
ϕ
k
⟩
,
k
=
1
,
2
…
,
n
\hat{h}_{k}=\left\langle h, \phi_{k}\right\rangle, k=1,2 \ldots, n
h ^ k = ⟨ h , ϕ k ⟩ , k = 1 , 2 … , n
h
^
=
Φ
T
h
\hat{\boldsymbol{h}}=\mathbf{\Phi}^{T} \boldsymbol{h}
h ^ = Φ T h
对二者的傅里叶变换向量
f
^
∈
R
N
×
1
\hat{f} \in \mathbb{R}^{N \times 1}
f ^ ∈ R N × 1
h
^
∈
R
N
×
1
\hat{h} \in \mathbb{R}^{N \times 1}
h ^ ∈ R N × 1 等价于将
h
\boldsymbol{h}
h ，即
diag
[
h
^
(
λ
k
)
]
∈
R
N
×
N
\operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right] \in \mathbb{R}^{N \times N}
d i a g [ h ^ ( λ k ) ] ∈ R N × N
diag
[
h
^
(
λ
k
)
]
\operatorname{diag}\left[\hat{h}\left(\lambda_{k}\right)\right]
d i a g [ h ^ ( λ k ) ]
f
\boldsymbol{f}
f
求上述结果的傅里叶逆变换，即左乘
Φ
\mathbf{\Phi}
Φ
深度学习中的卷积就是要设计trainable的卷积核 ，从上式能够看出，就是要设计
diag
[
h
^
(
λ
1
)
,
…
,
h
^
(
λ
n
)
]
\operatorname{diag}\left[\hat{h}\left(\lambda_{1}\right), \ldots, \hat{h}\left(\lambda_{n}\right)\right]
d i a g [ h ^ ( λ 1 ) , … , h ^ ( λ n ) ]
diag
[
θ
1
,
…
,
θ
n
]
\operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right]
d i a g [ θ 1 , … , θ n ] 不须要再将卷积核进行傅里叶变换 ，由此，至关于直接将变换后的参量进行学习。
第一代GCN为
y
output
=
σ
(
Φ
g
θ
Φ
T
x
)
=
σ
(
Φ
diag
[
θ
1
,
…
,
θ
n
]
Φ
T
x
)
\boldsymbol{y}_{\text {output}}=\sigma\left(\mathbf{\Phi} \boldsymbol{g}_{\theta} \mathbf{\Phi}^{T} \boldsymbol{x}\right)=\sigma\left(\boldsymbol{\Phi} \operatorname{diag}\left[\theta_{1}, \ldots, \theta_{n}\right] \mathbf{\Phi}^{T} \boldsymbol{x}\right)
y output = σ ( Φ g θ Φ T x ) = σ ( Φ d i a g [ θ 1 , … , θ n ] Φ T x )
x
\boldsymbol{x}
x
x
=
(
x
1
,
x
2
,
…
,
x
n
)
x=\left(x_{1}, x_{2}, \ldots, x_{n}\right)
x = ( x 1 , x 2 , … , x n )
y
output
\boldsymbol{y}_{\text {output}}
y output
一些缺点：
须要对拉普拉斯矩阵进行谱分解来求
Φ
\mathbf{\Phi}
Φ
O
(
n
2
)
O(n^2)
O ( n 2 )
卷积核参数为
n
n
n
n
n
n
卷积核的spatial localization很差。
图傅里叶变换是关于特征值(至关于普通傅里叶变换的频率)的函数，也就是
F
(
λ
1
)
,
…
,
F
(
λ
n
)
F\left(\lambda_{1}\right), \ldots, F\left(\lambda_{n}\right)
F ( λ 1 ) , … , F ( λ n )
F
(
Λ
)
F(\mathbf{\Lambda})
F ( Λ )
g
θ
\boldsymbol{g}_{\theta}
g θ
g
θ
(
Λ
)
\boldsymbol{g}_{\theta}(\Lambda)
g θ ( Λ )
g
θ
(
Λ
)
\boldsymbol{g}_{\theta}(\Lambda)
g θ ( Λ ) k阶多项式
g
θ
′
(
Λ
)
≈
∑
k
=
0
K
θ
k
′
Λ
k
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k}
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ Λ k
g
θ
′
∗
x
≈
Φ
∑
k
=
0
K
θ
k
′
Λ
k
Φ
T
x
=
∑
k
=
0
K
θ
k
′
(
Φ
Λ
k
Φ
T
)
x
=
∑
k
=
0
K
θ
k
′
(
Φ
Λ
Φ
T
)
k
x
=
∑
k
=
0
K
θ
k
′
L
k
x
\begin{aligned} g_{\theta^{\prime}} * x & \approx \Phi \sum_{k=0}^{K} \theta_{k}^{\prime} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T} \boldsymbol{x} \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda}^{k} \mathbf{\Phi}^{T}\right) x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} \mathbf{\Lambda} \mathbf{\Phi}^{T}\right)^{k} x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime} \boldsymbol{L}^{k} x \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ Λ k Φ T x = k = 0 ∑ K θ k ′ ( Φ Λ k Φ T ) x = k = 0 ∑ K θ k ′ ( Φ Λ Φ T ) k x = k = 0 ∑ K θ k ′ L k x
L
k
\boldsymbol{L}^{k}
L k
O
(
K
n
2
)
O(Kn^2)
O ( K n 2 )
对于每一次Laplacian矩阵
L
\boldsymbol{L}
L
x
\mathbf{x}
x
n
n
n
n
e
[
n
]
ne[n]
n e [ n ]
n
n
n
k
k
k
n
n
n 所以，实际上只利用了节点的K-Localized信息 。
另外，可使用切比雪夫展开式来近似
L
k
\boldsymbol{L}^{k}
L k 任何k次多项式均可以使用切比雪夫展开式来近似 ，由此，引入切比雪夫多项式的
K
K
K
L
k
\boldsymbol{L}^{k}
L k
g
θ
(
Λ
)
g_{\theta}(\mathbf{\Lambda})
g θ ( Λ )
g
θ
′
(
Λ
)
≈
∑
k
=
0
K
θ
k
′
T
k
(
Λ
~
)
g_{\theta^{\prime}}(\mathbf{\Lambda}) \approx \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\mathbf{\Lambda}})
g θ ′ ( Λ ) ≈ k = 0 ∑ K θ k ′ T k ( Λ ~ )
Λ
~
=
2
λ
max
Λ
−
I
n
\tilde{\mathbf{\Lambda}}=\frac{2}{\lambda_{\max }} \mathbf{\Lambda}-\boldsymbol{I}_{n}
Λ ~ = λ max 2 Λ − I n
θ
′
∈
R
K
\boldsymbol{\theta}^{\prime} \in \mathbb{R}^{K}
θ ′ ∈ R K
θ
k
′
\theta_{k}^{\prime}
θ k ′
k
k
k
T
k
(
x
)
T_{k}(x)
T k ( x )
T
k
(
x
)
=
2
x
T
k
−
1
(
x
)
−
T
k
−
2
(
x
)
T_{k}(x)=2 x T_{k-1}(x)-T_{k-2}(x)
T k ( x ) = 2 x T k − 1 ( x ) − T k − 2 ( x )
T
0
(
x
)
=
1
,
T
1
(
x
)
=
x
T_{0}(x)=1, T_{1}(x)=x
T 0 ( x ) = 1 , T 1 ( x ) = x
此时，带入到卷积公式
g
θ
′
∗
x
≈
Φ
∑
k
=
0
K
θ
k
′
T
k
(
Λ
~
)
Φ
T
x
≈
∑
k
=
0
K
θ
k
′
(
Φ
T
k
(
Λ
~
)
Φ
T
)
x
=
∑
k
=
0
K
θ
k
′
T
k
(
L
~
)
x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \mathbf{\Phi} \sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{\Lambda}}) \mathbf{\Phi}^{T} \boldsymbol{x} \\ &\approx \sum_{k=0}^{K} \theta_{k}^{\prime}\left(\mathbf{\Phi} T_{k}(\tilde{\mathbf{\Lambda}}) \mathbf{\Phi}^{T}\right) x \\ &=\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ Φ k = 0 ∑ K θ k ′ T k ( Λ ~ ) Φ T x ≈ k = 0 ∑ K θ k ′ ( Φ T k ( Λ ~ ) Φ T ) x = k = 0 ∑ K θ k ′ T k ( L ~ ) x
L
~
=
2
λ
max
L
−
I
n
\tilde{\boldsymbol{L}}=\frac{2}{\lambda_{\max }} \boldsymbol{L}-\boldsymbol{I}_{n}
L ~ = λ max 2 L − I n
所以，能够获得输出为
y
output
=
σ
(
∑
k
=
0
K
θ
k
′
T
k
(
L
~
)
x
)
\boldsymbol{y}_{\text {output}}=\sigma\left(\sum_{k=0}^{K} \theta_{k}^{\prime} T_{k}(\tilde{\boldsymbol{L}}) \boldsymbol{x}\right)
y output = σ ( k = 0 ∑ K θ k ′ T k ( L ~ ) x )
这一代GCN直接取切比雪夫多项式中
K
=
1
K=1
K = 1
将
K
=
1
K=1
K = 1
λ
max
=
2
\lambda_{\max }=2
λ max = 2
g
θ
′
∗
x
≈
θ
0
′
x
+
θ
1
′
(
L
−
I
n
)
x
=
θ
0
′
x
+
θ
1
′
(
L
−
I
n
)
x
=
θ
0
′
x
−
θ
1
′
(
D
−
1
/
2
W
D
−
1
/
2
)
x
\begin{aligned} \boldsymbol{g}_{\boldsymbol{\theta}^{\prime}} * \boldsymbol{x} & \approx \boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\boldsymbol{\theta}_{0}^{\prime} \boldsymbol{x}+\theta_{1}^{\prime}\left(\boldsymbol{L}-\boldsymbol{I}_{n}\right) \boldsymbol{x} \\ &=\theta_{0}^{\prime} \boldsymbol{x}-\theta_{1}^{\prime}\left(\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}\right) \boldsymbol{x} \end{aligned}
g θ ′ ∗ x ≈ θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x + θ 1 ′ ( L − I n ) x = θ 0 ′ x − θ 1 ′ ( D − 1 / 2 W D − 1 / 2 ) x
L
=
D
−
1
/
2
(
D
−
W
)
D
−
1
/
2
=
I
n
−
D
−
1
/
2
W
D
−
1
/
2
\boldsymbol{L}=\boldsymbol{D}^{-1 / 2}(\boldsymbol{D}-\boldsymbol{W}) \boldsymbol{D}^{-1 / 2}=\boldsymbol{I}_{n}-\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}
L = D − 1 / 2 ( D − W ) D − 1 / 2 = I n − D − 1 / 2 W D − 1 / 2 为了进一步简化，令
θ
0
′
=
−
θ
1
′
\theta_{0}^{\prime}=-\theta_{1}^{\prime}
θ 0 ′ = − θ 1 ′
θ
\theta
θ
g
θ
′
∗
x
=
θ
(
I
n
+
D
−
1
/
2
W
D
−
1
/
2
)
x
g_{\theta^{\prime}} * x=\theta\left(I_{n}+D^{-1 / 2} W D^{-1 / 2}\right) x
g θ ′ ∗ x = θ ( I n + D − 1 / 2 W D − 1 / 2 ) x
I
n
+
D
−
1
/
2
W
D
−
1
/
2
\boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2}
I n + D − 1 / 2 W D − 1 / 2 谱半径
[
0
,
2
]
[0,2]
[ 0 , 2 ]
I
n
+
D
−
1
/
2
W
D
−
1
/
2
→
D
~
−
1
/
2
W
~
D
~
−
1
/
2
\boldsymbol{I}_{n}+\boldsymbol{D}^{-1 / 2} \boldsymbol{W} \boldsymbol{D}^{-1 / 2} \rightarrow \tilde{\boldsymbol{D}}^{-1 / 2} \tilde{\boldsymbol{W}} \tilde{\boldsymbol{D}}^{-1 / 2}
I n + D − 1 / 2 W D − 1 / 2 → D ~ − 1 / 2 W ~ D ~ − 1 / 2
W
~
=
W
+
I
n
\tilde{\boldsymbol{W}}=\boldsymbol{W}+\boldsymbol{I}_{n}
W ~ = W + I n
D
~
i
j
=
Σ
j
W
~
i
j
\tilde{D}_{i j}=\Sigma_{j} \tilde{W}_{i j}
D ~ i j = Σ j W ~ i j
由此，带入卷积公式
g
θ
′
∗
x
⏟
R
n
×
1
=
θ
(
D
~
−
1
/
2
W
~
D
~
−
1
/
2
⏟
R
n
×
n
)
x
⏟
R
n
×
1
\underbrace{g_{\theta^{\prime}} * x}_{\mathbb{R}^{n \times 1}}=\theta\left(\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{n \times n}}\right) \underbrace{x}_{\mathbb{R}^{n \times 1}}
R n × 1
g θ ′ ∗ x = θ ( R n × n
D ~ − 1 / 2 W ~ D ~ − 1 / 2 ) R n × 1
x 推广到多通道 ，至关于每个节点的信息是向量
x
∈
R
N
×
1
→
X
∈
R
N
×
C
x \in \mathbb{R}^{N \times 1} \rightarrow X \in \mathbb{R}^{N \times C}
x ∈ R N × 1 → X ∈ R N × C
N
N
N 数量 ，
C
C
C 维度数 。
X
\mathbf{X}
X 特征矩阵 。
相应的卷积核参数变化
θ
∈
R
→
Θ
∈
R
C
×
F
\theta \in \mathbb{R} \rightarrow \Theta \in \mathbb{R}^{C \times F}
θ ∈ R → Θ ∈ R C × F
F
F
F
那么卷积结果写成矩阵形式为
Z
⏟
R
N
×
F
=
D
~
−
1
/
2
W
~
D
~
−
1
/
2
⏟
R
N
×
N
X
⏟
R
N
×
C
Θ
⏟
R
C
×
F
\underbrace{Z}_{\mathbb{R}^{N \times F}}=\underbrace{\tilde{D}^{-1 / 2} \tilde{W} \tilde{D}^{-1 / 2}}_{\mathbb{R}^{N \times N}} \underbrace{X}_{\mathbb{R}^{N \times C}} \underbrace{\mathbf{\Theta}}_{\mathbb{R}^{C \times F}}
R N × F
Z = R N × N
D ~ − 1 / 2 W ~ D ~ − 1 / 2 R N × C
X R C × F
Θ 能够叠加多层，对上述输出激活一下，就能够做为下一层节点的特征矩阵 。
这一代GCN特色：
取
K
=
1
K=1
K = 1
3
×
3
3\times{3}
3 × 3
因为卷积核宽度减少，能够经过增长卷积层数来扩大感觉野，从而加强网络的表达能力。
增长了参数约束，好比
λ
max
≈
2
\lambda_{\max } \approx 2
λ max ≈ 2
论文采用两层的GCN，用来在graph上进行半监督的节点分类任务，邻接矩阵为
A
A
A
A
^
=
D
~
−
1
2
A
~
D
~
−
1
2
\hat{A}=\tilde{D}^{-\frac{1}{2}} \tilde{A} \tilde{D}^{-\frac{1}{2}}
A ^ = D ~ − 2 1 A ~ D ~ − 2 1
Z
=
f
(
X
,
A
)
=
softmax
(
A
^
ReLU
(
A
^
X
W
(
0
)
)
W
(
1
)
)
Z=f(X, A)=\operatorname{softmax}\left(\hat{A} \operatorname{ReLU}\left(\hat{A} X W^{(0)}\right) W^{(1)}\right)
Z = f ( X , A ) = s o f t m a x ( A ^ R e L U ( A ^ X W ( 0 ) ) W ( 1 ) )
W
(
0
)
∈
R
C
×
H
W^{(0)} \in \mathbb{R}^{C \times H}
W ( 0 ) ∈ R C × H
H
H
H
W
(
1
)
∈
R
H
×
F
W^{(1)} \in \mathbb{R}^{H \times F}
W ( 1 ) ∈ R H × F
softmax
(
x
i
)
=
1
Z
exp
(
x
i
)
\operatorname{softmax}\left(x_{i}\right)=\frac{1}{\mathcal{Z}} \exp \left(x_{i}\right)
s o f t m a x ( x i ) = Z 1 exp ( x i )
Z
=
∑
i
exp
(
x
i
)
\mathcal{Z}=\sum_{i} \exp \left(x_{i}\right)
Z = ∑ i exp ( x i ) 交叉熵损失函数 为
L
=
−
∑
l
∈
Y
L
∑
f
=
1
F
Y
l
f
ln
Z
l
f
\mathcal{L}=-\sum_{l \in \mathcal{Y}_{L}} \sum_{f=1}^{F} Y_{l f} \ln Z_{l f}
L = − l ∈ Y L ∑ f = 1 ∑ F Y l f ln Z l f
Y
L
\mathcal{Y}_{L}
Y L
C
C
C
F
F
F
Y
1
Y_1
Y 1
Y
2
Y_2
Y 2
论文在以下几个任务中进行实验
在citation network中进行半监督的document classification。
在从knowledge graph中提取的bipartite graph中进行半监督的entity classification
实验数听说明以下
结果以下：
能够看出，在比较的几种算法中，论文GCN的在准确率和时间上都最好。
该模型对每个节点(或边、或图)采用H个hop的矩阵进行表示，每个hop都表示该邻近范围的邻近信息，由此，对于局部信息的获取效果比较好，获得的节点的representation的表示能力很强。
假设有以下定义
一个graph数据集
G
=
{
G
t
∣
t
∈
1
…
T
}
\mathcal{G}=\left\{G_{t} | t \in 1 \ldots T\right\}
G = { G t ∣ t ∈ 1 … T }
graph定义为
G
t
=
(
V
t
,
E
t
)
G_{t}=\left(V_{t}, E_{t}\right)
G t = ( V t , E t )
V
t
V_t
V t
E
t
E_t
E t
全部节点的特征矩阵定义为
X
t
X_t
X t
N
t
×
F
N_t\times{F}
N t × F
N
t
N_t
N t
G
t
G_t
G t
F
F
F
边信息
E
t
E_t
E t
N
t
×
N
t
N_t\times{}N_t
N t × N t
A
t
A_t
A t 节点度(degree)归一化 的转移几率矩阵
P
t
P_t
P t
i
i
i
j
j
j
对于graph来讲没有任何限制，graph能够是带权重的或不带权重的，有向的或无向的。
模型的目标为预测
Y
Y
Y 在每一种状况中，模型输入部分带有标签的数据集合，而后预测剩下的数据的标签 。
DCNN模型输入图
G
\mathcal{G}
G
Y
Y
Y
P
(
Y
∣
X
)
\mathbb{P}(Y|X)
P ( Y ∣ X )
H
×
F
H\times{}F
H × F
H
H
H
t
t
t
N
t
×
H
×
F
N_t\times{H}\times{F}
N t × H × F
Z
t
Z_t
Z t
Z
t
Z_t
Z t
H
×
F
H\times{F}
H × F
Z
t
Z_t
Z t
M
t
×
H
×
F
M_t\times{H}\times{F}
M t × H × F
节点分类任务 ，假设
P
t
∗
P_t^*
P t ∗
P
t
P_t
P t
N
t
×
H
×
N
t
N_t\times{H}\times{N_t}
N t × H × N t
t
t
t
i
i
i
j
j
j
k
k
k
Z
t
i
j
k
Z_{tijk}
Z t i j k
Z
t
i
j
k
=
f
(
W
j
k
c
⋅
∑
l
=
1
N
t
P
t
i
j
l
∗
X
t
l
k
)
Z_{t i j k}=f\left(W_{j k}^{c} \cdot \sum_{l=1}^{N_{t}} P_{t i j l}^{*} X_{t l k}\right)
Z t i j k = f ( W j k c ⋅ l = 1 ∑ N t P t i j l ∗ X t l k )
Z
t
=
f
(
W
c
⊙
P
t
∗
X
t
)
Z_{t}=f\left(W^{c} \odot P_{t}^{*} X_{t}\right)
Z t = f ( W c ⊙ P t ∗ X t )
⊙
\odot
⊙
H
H
H
O
(
H
×
F
)
O(H\times{F})
O ( H × F ) 使得diffusion-convolutional representation不受输入大小的限制 。
在计算出
Z
Z
Z
Y
Y
Y
Y
^
\hat{Y}
Y ^
P
(
Y
∣
X
)
\mathbb{P}(Y|X)
P ( Y ∣ X )
Y
^
=
arg
max
(
f
(
W
d
⊙
Z
)
)
\hat{Y}=\arg \max \left(f\left(W^{d} \odot Z\right)\right)
Y ^ = arg max ( f ( W d ⊙ Z ) )
P
(
Y
∣
X
)
=
softmax
(
f
(
W
d
⊙
Z
)
)
\mathbb{P}(Y | X)=\operatorname{softmax}\left(f\left(W^{d} \odot Z\right)\right)
P ( Y ∣ X ) = s o f t m a x ( f ( W d ⊙ Z ) )
对于图分类任务 ，直接采用全部节点表示的均值做为graph的representation
Z
t
=
f
(
W
c
⊙
1
N
t
T
P
t
∗
X
t
/
N
t
)
Z_{t}=f\left(W^{c} \odot 1_{N_{t}}^{T} P_{t}^{*} X_{t} / N_{t}\right)
Z t = f ( W c ⊙ 1 N t T P t ∗ X t / N t )
1
N
t
1_{N_t}
1 N t
N
t
×
1
N_t\times{1}
N t × 1
对于边分类任务 ，经过将每一条边转化为一个节点来进行训练和预测 ，这个节点与原来的边对应的首尾节点相连，转化后的图的邻接矩阵
A
t
′
A_t'
A t ′
A
t
A_t
A t incidence matrix 获得
A
t
′
=
(
A
t
B
t
T
B
t
0
)
A_{t}^{\prime}=\left(\begin{array}{cc}{A_{t}} & {B_{t}^{T}} \\ {B_{t}} & {0}\end{array}\right)
A t ′ = ( A t B t B t T 0 )
A
t
′
A_t'
A t ′
P
t
′
P_t'
P t ′
P
t
P_t
P t
对于模型训练 ，使用梯度降低法，并采用early-stop方式获得最终模型。
节点分类任务的实验数据集使用Cora和Pubmed数据集，包含scientific papers(至关于node)、citations(至关于edge)和subjects(至关于label)。实验评估标准使用分类准确率以及F1值。
节点分类的实验结果 以下
能够看出使用各类评估标准，DCNN效果都是最好的。
图分类任务 的实验结果以下
优势 ：
缺点 ：
内存占用大 ：DCNN创建在密集的张量计算上，须要存储大量的张量，须要
O
(
N
t
2
H
)
O(N_t^2H)
O ( N t 2 H ) 长距离信息传播不足 ：模型对于局部的信息获取较好，可是远距离的信息传播不足。
将序列型的LSTM模型扩展到树型的LSTM模型 ，简称Tree-LSTM，并根据孩子节点是否有序，论文提出了两个模型变体，Child-Sum Tree-LSTM模型和N-ary Tree-LSTM模型 。和序列型的LSTM模型的主要不一样点在于，序列型的LSTM从前一时刻获取隐藏状态
h
t
h_t
h t
Tree-LSTM模型对于每个孩子节点都会产生一个“遗忘门”
f
j
k
f_{jk}
f j k 从全部的孩子节点选择性地获取信息和结合信息 。
该模型的更新方程以下
h
~
j
=
∑
k
∈
C
(
j
)
h
k
i
j
=
σ
(
W
(
i
)
x
j
+
U
(
i
)
h
~
j
+
b
(
i
)
)
f
j
k
=
σ
(
W
(
f
)
x
j
+
U
(
f
)
h
k
+
b
(
f
)
)
o
j
=
σ
(
W
(
o
)
x
j
+
U
(
o
)
h
~
j
+
b
(
o
)
)
u
j
=
tanh
(
W
(
u
)
x
j
+
U
(
u
)
h
~
j
+
b
(
u
)
)
c
j
=
i
j
⊙
u
j
+
∑
k
∈
C
(
j
)
f
j
k
⊙
c
k
h
j
=
o
j
⊙
tanh
(
c
j
)
\begin{aligned} \tilde{h}_{j} &=\sum_{k \in C(j)} h_{k} \\ i_{j} &=\sigma\left(W^{(i)} x_{j}+U^{(i)} \tilde{h}_{j}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+U^{(f)} h_{k}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+U^{(o)} \tilde{h}_{j}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+U^{(u)} \tilde{h}_{j}+b^{(u)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{k \in C(j)} f_{j k} \odot c_{k} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
h ~ j i j f j k o j u j c j h j = k ∈ C ( j ) ∑ h k = σ ( W ( i ) x j + U ( i ) h ~ j + b ( i ) ) = σ ( W ( f ) x j + U ( f ) h k + b ( f ) ) = σ ( W ( o ) x j + U ( o ) h ~ j + b ( o ) ) = tanh ( W ( u ) x j + U ( u ) h ~ j + b ( u ) ) = i j ⊙ u j + k ∈ C ( j ) ∑ f j k ⊙ c k = o j ⊙ tanh ( c j )
C
(
j
)
C(j)
C ( j )
j
j
j
h
k
h_k
h k
k
k
k
i
j
i_j
i j
j
j
j
f
j
k
f_{jk}
f j k
j
j
j
k
k
k
o
j
o_j
o j
j
j
j
这里的关键点在于第三个公式的
f
j
k
f_{jk}
f j k
j
j
j
k
k
k
c
j
c_j
c j
因为该模型是对全部的孩子节点求和，因此这个模型对于节点顺序不敏感的 ，适合于孩子节点无序的状况。
假如一个树的最大分支数为
N
N
N
N
N
N
j
j
j
k
k
k
h
j
k
h_{jk}
h j k
c
j
k
c_{jk}
c j k
i
j
=
σ
(
W
(
i
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
i
)
h
j
ℓ
+
b
(
i
)
)
f
j
k
=
σ
(
W
(
f
)
x
j
+
∑
ℓ
=
1
N
U
k
ℓ
(
f
)
h
j
ℓ
+
b
(
f
)
)
o
j
=
σ
(
W
(
o
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
o
)
h
j
ℓ
+
b
(
o
)
)
u
j
=
tanh
(
W
(
u
)
x
j
+
∑
ℓ
=
1
N
U
ℓ
(
a
)
h
j
ℓ
+
b
(
a
)
)
c
j
=
i
j
⊙
u
j
+
∑
ℓ
=
1
N
f
j
ℓ
⊙
c
j
ℓ
h
j
=
o
j
⊙
tanh
(
c
j
)
\begin{aligned} i_{j} &=\sigma\left(W^{(i)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(i)} h_{j \ell}+b^{(i)}\right) \\ f_{j k} &=\sigma\left(W^{(f)} x_{j}+\sum_{\ell=1}^{N} U_{k \ell}^{(f)} h_{j \ell}+b^{(f)}\right) \\ o_{j} &=\sigma\left(W^{(o)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(o)} h_{j \ell}+b^{(o)}\right) \\ u_{j} &=\tanh \left(W^{(u)} x_{j}+\sum_{\ell=1}^{N} U_{\ell}^{(a)} h_{j \ell}+b^{(a)}\right) \\ c_{j} &=i_{j} \odot u_{j}+\sum_{\ell=1}^{N} f_{j \ell} \odot c_{j \ell} \\ h_{j} &=o_{j} \odot \tanh \left(c_{j}\right) \end{aligned}
i j f j k o j u j c j h j = σ ( W ( i ) x j + ℓ = 1 ∑ N U ℓ ( i ) h j ℓ + b ( i ) ) = σ ( W ( f ) x j + ℓ = 1 ∑ N U k ℓ ( f ) h j ℓ + b ( f ) ) = σ ( W ( o ) x j + ℓ = 1 ∑ N U ℓ ( o ) h j ℓ + b ( o ) ) = tanh ( W ( u ) x j + ℓ = 1 ∑ N U ℓ ( a ) h j ℓ + b ( a ) ) = i j ⊙ u j + ℓ = 1 ∑ N f j ℓ ⊙ c j ℓ = o j ⊙ tanh ( c j ) 该模型为每一个孩子节点都单独地设置了参数
U
l
U_{l}
U l 。
分类任务定义为在类别集
Y
\mathcal{Y}
Y
y
^
\hat{y}
y ^
j
j
j
y
^
j
\hat{y}_j
y ^ j
h
j
h_j
h j
p
^
θ
(
y
∣
{
x
}
j
)
=
softmax
(
W
(
s
)
h
j
+
b
(
s
)
)
y
^
j
=
arg
max
y
p
^
θ
(
y
∣
{
x
}
j
)
\begin{aligned} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) &=\operatorname{softmax}\left(W^{(s)} h_{j}+b^{(s)}\right) \\ \hat{y}_{j} &=\arg \max _{y} \hat{p}_{\theta}\left(y |\{x\}_{j}\right) \end{aligned}
p ^ θ ( y ∣ { x } j ) y ^ j = s o f t m a x ( W ( s ) h j + b ( s ) ) = arg y max p ^ θ ( y ∣ { x } j )
J
(
θ
)
=
−
1
m
∑
k
=
1
m
log
p
^
θ
(
y
(
k
)
∣
{
x
}
(
k
)
)
+
λ
2
∥
θ
∥
2
2
J(\theta)=-\frac{1}{m} \sum_{k=1}^{m} \log \hat{p}_{\theta}\left(y^{(k)} |\{x\}^{(k)}\right)+\frac{\lambda}{2}\|\theta\|_{2}^{2}
J ( θ ) = − m 1 k = 1 ∑ m log p ^ θ ( y ( k ) ∣ { x } ( k ) ) + 2 λ ∥ θ ∥ 2 2
m
m
m
λ
\lambda
λ
L
2
L2
L 2
该任务给定一个句子对(sentence pair)，模型须要预测出一个范围在
[
1
,
K
]
[1,K]
[ 1 , K ]
论文首先对每个句子产生一个representation，两个句子的表示分别用
h
L
h_L
h L
h
R
h_R
h R
(
h
L
,
h
R
)
(h_L,h_R)
( h L , h R )
h
×
=
h
L
⊙
h
R
h
+
=
∣
h
L
−
h
R
∣
h
s
=
σ
(
W
(
×
)
h
×
+
W
(
+
)
h
+
+
b
(
h
)
)
p
^
θ
=
softmax
(
W
(
p
)
h
s
+
b
(
p
)
)
y
^
=
r
T
p
^
θ
\begin{aligned} h_{ \times} &=h_{L} \odot h_{R} \\ h_{+} &=\left|h_{L}-h_{R}\right| \\ h_{s} &=\sigma\left(W^{( \times)} h_{ \times}+W^{(+)} h_{+}+b^{(h)}\right) \\ \hat{p}_{\theta} &=\operatorname{softmax}\left(W^{(p)} h_{s}+b^{(p)}\right) \\ \hat{y} &=r^{T} \hat{p}_{\theta} \end{aligned}
h × h + h s p ^ θ y ^ = h L ⊙ h R = ∣ h L − h R ∣ = σ ( W ( × ) h × + W
web