无所不能的Embedding 1 - Word2vec模型详解&代码实现
word2vec是google 2013年提出的，从大规模语料中训练词向量的模型，在许多场景中都有应用，信息提取类似度计算等等。也是从word2vec开始，embedding在各个领域的应用开始流行，因此拿word2vec来做为开篇再合适不过了。本文但愿能够较全面的给出Word2vec从模型结构概述，推导，训练，和基于tf.estimator实现的具体细节。完整代码戳这里https://github.com/DSXiangLi/Embeddinghtml
模型概述
word2vec模型结构比较简单，是为了可以在大规模数据上训练，下降了模型复杂度，移除了非线性隐藏层。根据不一样的输入输出形式又分红CBOW和SG两种方法。node
让咱们先把问题简化成1v1的bigram问题，单词i做为context,单词j是target。V是单词总数，N是词向量长度，D是训练词对，输入\(x_i \in R ^{1*V}\)是one-hot向量。python
模型训练两个权重矩阵,\(W \in R ^{V*N}\)是输入矩阵，每一行对应输入单词的词向量,\(W^{'} \in R ^{V*N}\)是输出矩阵，每一行对应输出单词的词向量。词i和词j的共现信息用词向量的内积来表达，经过softmax获得每一个单词的几率以下git
对每一个训练样本，模型的目标是最大化条件几率\(p(w_j|w_I)\), 所以咱们的对数损失函数以下github
CBOW : Continuous bag of words
CBOW是把bigram的输入context，扩展成了目标单词周围2*window_size内的单词，用中心词先后的语境来预测中心词。app
对比bigram, CBOW只多作了一步操做，对输入的2 * Window_size个单词，在映射获得词向量后，须要作average_pooling获得1*N的输入向量, 因此差别只在h的计算。假定$C = 2 * \text{window_size}$ $$ \begin{align} h & = \frac{1}{C}W^T(x_1 + x_2 +... + x_C) \\ & = \frac{1}{C}(v_{w1} + v_{w2} + ... + v_{wc}) ^T \\ E &= -log \, p(w_O|w_{I,1}...w_{I,C}) \\ & = -u_j^* + log\sum_{j^{'}=1}^Vexp(u_{j^{'}}) \end{align} $$
SG : Skip Gram
SG是把bigram的输出target，扩展成了输入单词周围2*window_size内的单词，用中心词来预测周围单词的出现几率。dom
对比bigram，SG的差别只在于输出几率多项分布再也不是一个而是C个函数
模型推导：word embedding是如何获得的？
下面咱们从back propogation推导下以上模型结构是如何学到词向量的，为简化咱们仍是先从bigram来看，\(\eta\)是learning rate。学习
首先是hidden->output \(W^{'}\)的词向量的更新优化
\(e_j\)是单词j的预测几率偏差，因此\(W^{'}\)的更新能够理解为若是单词j被高估就从\(v_{w^{'}j}\)中减去\(\eta \cdot e_j \cdot h\)，下降h和\(v_{w^{'}j}\)的向量内积(similarity)，反之被低估则在\(v_{w^{'}j}\)上叠加\(\eta \cdot e_j \cdot h\)增长内积类似度，偏差越大更新的幅度越大。
而后是input->hidden W的词向量的更新
每一个输入单词对应的词向量\(v_{wI}\)，都用全部单词的输出词向量按预测偏差加权平均获得的向量进行更新。和上述的逻辑相同高估作subtraction，低估的作addition而后按偏差大小进行加权来更新输入词向量。
因此模型学习过程会是输入词向量更新输出词向量，输出词向量再更新输入词向量，而后back-and-forth到达稳态。
把bigram拓展到CBOW，惟一的变化在于更新input-hidden的词向量时，不是每次更新一个单词对应的向量，而是用相同的幅度同时更新C个单词的词向量.
把bigram拓展到SG，惟一的变化在于更新hidden-output的词向量时，再也不是用单词j的预测偏差，而是用C个单词的预测偏差之和
模型训练
虽然模型结构已经作了优化，移除了非线性的隐藏层，可是模型训练起来并不高效,瓶颈在于Word2vec本质是多分类任务，类别有整个vocabulary这么多，因此\(p(w_j|w_I) = \frac{exp(u_j)}{\sum_{j^{'}=1}^Vexp(u_{j^{'}})}\)每次须要计算整个vocabulary的几率\(O(VN)\)。即使batch只有1个训练样本，也须要更新全部单词hidden->output的embedding矩阵。针对这个问题有两种解决方案
Hierarchical Softmax
若是把softmax看做一个1-layer tree,每一个单词都是一个叶节点, 由于须要归一化因此计算每一个单词的几率的复杂度是\(O(V)\)。Hierarchical Softmax只是把1-layer变成了multi-layer，在不增长embedding大小的状况下（V个叶节点，树有V-1个inner node), 把计算每一个单词几率的复杂度下降到\(O(logV)\)，直接用从root到叶节点的路径来计算每一个单词的几率。树的构造做者选用了huffman tree,优势在于高频词从root到leaf的路径会比低频词更短，这样能够进一步加速训练，具体细节能够来看这篇博客human coding
例以下图（图片来源）
$$ \begin{align} P(Horse) &= P(0,left)\cdot P(1,right)\cdot P(2,left) \end{align} $$
那具体上面的p(0,left)要如何计算呢？
每个node都有本身的embedding \(v_n{(w,j)}\), 既单词w路径上第j个node的embedding,输入输出的单词内积，变为输入单词和node的内积， 每一个单词的几率计算以下
不得不说这个式子写的真是生怕别人能看懂>_<
\([n(w,j+1) = ch(n(w,j))]\) 是个啥？ch是left child，\([\cdot]\)只是用来判断path是往左仍是往右
因此
对应上面的模型推导，hidden->ouput的部分发生变化, 损失函数变为如下
每次output单词对应的路径上的embedding会被更新，预测任务变为该路径上每一个inner_node应该往左仍是往右。
简单的huffman Hierarchy softmax的实现以下
class TreeNode(object):
total_node = 0
def __init__(self, frequency, char = None , word_index = None, is_leaf = False):
self.frequency = frequency
self.char = char # word character
self.word_index = word_index # word look up index
self.left = None
self.right = None
self.is_leaf = is_leaf
self.counter(is_leaf)
def counter(self, is_leaf):
# node_index will be used for embeeding_lookup
self.node_index = TreeNode.total_node
if not is_leaf: TreeNode.total_node += 1
def __lt__(self, other):
return self.frequency < other.frequency
def __repr__(self):
if self.is_leaf:
return 'Leaf Node char = [{}] index = {} freq = {}'.format(self.char, self.word_index, self.frequency)
else:
return 'Inner Node [{}] freq = {}'.format(self.node_index, self.frequency)
class HuffmanTree(object):
def __init__(self, freq_dic):
self.nodes = []
self.root = None
self.max_depth = None
self.freq_dic = freq_dic
self.all_paths = {}
self.all_codes = {}
self.node_index = 0
@staticmethod
def merge_node(left, right):
parent = TreeNode(left.frequency + right.frequency)
parent.left = left
parent.right = right
return parent
def build_tree(self):
"""
Build huffman tree with word being leaves
"""
TreeNode.total_node = 0 # avoid train_and_evaluate has different node_index
heap_nodes = []
for word_index, (char, freq) in enumerate(self.freq_dic.items()):
tmp = TreeNode( freq, char, word_index, is_leaf=True )
heapq.heappush(heap_nodes, tmp )
while len(heap_nodes)>1:
node1 = heapq.heappop(heap_nodes)
node2 = heapq.heappop(heap_nodes)
heapq.heappush(heap_nodes, HuffmanTree.merge_node(node1, node2))
self.root = heapq.heappop(heap_nodes)
@property
def num_node(self):
return self.root.node_index + 1
def traverse(self):
"""
Compute all node to leaf path and direction: list of node_id, list of 0/1
"""
def dfs_helper(root, path, code):
if root.is_leaf :
self.all_paths[root.word_index] = path
self.all_codes[root.word_index] = code
return
if root.left :
dfs_helper(root.left, path + [root.node_index], code + [0])
if root.right :
dfs_helper(root.right, path + [root.node_index], code + [1])
dfs_helper(self.root, [], [] )
self.max_depth = max([len(i) for i in self.all_codes.values()])
class HierarchySoftmax(HuffmanTree):
def __init__(self, freq_dic):
super(HierarchySoftmax, self).__init__(freq_dic)
def convert2tensor(self):
# padded to max_depth and convert to tensor
with tf.name_scope('hstree_code'):
self.code_table = tf.convert_to_tensor([ code + [INVALID_INDEX] * (self.max_depth - len(code)) for word, code
in sorted( self.all_codes.items(), key=lambda x: x[0] )],
dtype = tf.float32)
with tf.name_scope('hstree_path'):
self.path_table = tf.convert_to_tensor([path + [INVALID_INDEX] * (self.max_depth - len(path)) for word, path
in sorted( self.all_paths.items(), key=lambda x: x[0] )],
dtype = tf.int32)
def get_loss(self, input_embedding_vector, labels, output_embedding, output_bias, params):
"""
:param input_embedding_vector: [batch * emb_size]
:param labels: word index [batch * 1]
:param output_embedding: entire embedding matrix []
:return:
loss
"""
loss = []
labels = tf.unstack(labels, num = params['batch_size']) # list of [1]
inputs = tf.unstack(input_embedding_vector, num = params['batch_size']) # list of [emb_size]
for label, input in zip(labels, inputs):
path = self.path_table[tf.squeeze(label)]# (max_depth,)
code = self.code_table[tf.squeeze(label)] # (max_depth,)
path = tf.boolean_mask(path, tf.not_equal(path, INVALID_INDEX)) # (real_path_length,)
code = tf.boolean_mask(code, tf.not_equal(code, INVALID_INDEX) ) # (real_path_length,)
output_embedding_vector = tf.nn.embedding_lookup(output_embedding, path) # real_path_length * emb_size
bias = tf.nn.embedding_lookup(output_bias, path) # (real_path_length,)
logits = tf.matmul(tf.expand_dims(input, axis=0), tf.transpose(output_embedding_vector) ) + bias # (1,emb_size) *(emb_size, real_path_length)
loss.append(tf.nn.sigmoid_cross_entropy_with_logits(labels = code, logits = tf.squeeze(logits) ))
loss = tf.reduce_mean(tf.concat(loss, axis = 0), axis=0, name = 'hierarchy_softmax_loss') # batch -> scaler
return loss
Negative Sampling
Negative Sampling理解起来更加直观，由于模型的目标是训练出高质量的word embedding，也就是input word embedding，那是否每一个batch都更新所有的output word embedding并不重要，咱们能够每次只sample K个embedding来作更新。原始的正样本保留，咱们再采样 K组负样原本进行训练，模型只须要学习正样本vs负样本，也就绕过了用V个单词来作归一化的问题，把多分类问题成功简化为二分类问题。做者表示小样本K=5~20，大样本k=2~5。
对应上述的模型推导，hidden->output的部分发生变化, 损失函数变为
每一个iteration只有K个embedding被更新
而input->hidden的部分,只有k个embedding的加权向量会用于输入embedding的更新
tensorflow有几种candidate sample的实现，两种比较经常使用的是nn.sampled_softmax_loss和nn.nce_loss, 它们调用了相同的采样函数。差别在于sampled_softmax_loss用的是softmax（排他单分类)，而nce_loss是求logistic (不排他多分类）。这两种实现都和negative sampling有些许差别，细节能够看下Notes on Noise Contrastive Estimation and Negative Sampling。而这两者之间比较是有观点说nce更适合skip-gram, sample更适合CBOW，具体差别我也还得再多用用试试看。
Subsampling
论文还有一个重点是subsampling，针对出现频率高的词，对于它们过多的训练样本不能进一步提升表现，所以能够对这些样本进行downsample。t是词频阈值， \(f(w_i)\)是单词在corpus里的出现频率，全部出现频率高于t的单词，都会按照如下几率被降采样
模型实现
手残党现实体验是word2vec比较复杂的部分不是模型。。。而是input_pipe和loss function，因此在实现的时候也但愿尽量把dataset, model_fn, 和train的部分分割开来。如下只给出model_fn的核心部分
def avg_pooling_embedding(embedding, features, params):
"""
:param features: (batch, 2*window_size)
:param embedding: (vocab_size, emb_size)
:return:
input_embedding : average pooling of context embedding
"""
input_embedding= []
samples = tf.unstack(features, params['batch_size'])
for sample in samples:
sample = tf.boolean_mask(sample, tf.not_equal(sample, INVALID_INDEX), axis=0) # (real_size,)
tmp = tf.nn.embedding_lookup(embedding, sample) # (real_size, emb_size)
input_embedding.append(tf.reduce_mean(tmp, axis=0)) # (emb_size, )
input_embedding = tf.stack(input_embedding, name = 'input_embedding_vector') # batch * emb_size
return input_embedding
def model_fn(features, labels, mode, params):
if params['train_algo'] == 'HS':
# If Hierarchy Softmax is used, initialize a huffman tree first
hstree = HierarchySoftmax( params['freq_dict'] )
hstree.build_tree()
hstree.traverse()
hstree.convert2tensor()
if params['model'] == 'CBOW':
features = tf.reshape(features, shape = [-1, 2 * params['window_size']])
labels = tf.reshape(labels, shape = [-1,1])
else:
features = tf.reshape(features, shape = [-1,])
labels = tf.reshape(labels, shape = [-1,1])
with tf.variable_scope( 'initialization' ):
w0 = tf.get_variable( shape=[params['vocab_size'], params['emb_size']],
initializer=tf.truncated_normal_initializer(), name='input_word_embedding' )
if params['train_algo'] == 'HS':
w1 = tf.get_variable( shape=[hstree.num_node, params['emb_size']],
initializer=tf.truncated_normal_initializer(), name='hierarchy_node_embedding' )
b1 = tf.get_variable( shape = [hstree.num_node],
initializer=tf.random_uniform_initializer(), name = 'bias')
else:
w1 = tf.get_variable( shape=[params['vocab_size'], params['emb_size']],
initializer=tf.truncated_normal_initializer(), name='output_word_embedding' )
b1 = tf.get_variable( shape=[params['vocab_size']],
initializer=tf.random_uniform_initializer(), name='bias')
add_layer_summary( w0.name, w0)
add_layer_summary( w1.name, w1 )
add_layer_summary( b1.name, b1 )
with tf.variable_scope('input_hidden'):
# batch_size * emb_size
if params['model'] == 'CBOW':
input_embedding_vector = avg_pooling_embedding(w0, features, params)
else:
input_embedding_vector = tf.nn.embedding_lookup(w0, features, name = 'input_embedding_vector')
add_layer_summary(input_embedding_vector.name, input_embedding_vector)
with tf.variable_scope('hidden_output'):
if params['train_algo'] == 'HS':
loss = hstree.get_loss( input_embedding_vector, labels, w1, b1, params)
else:
loss = negative_sampling(mode = mode,
output_embedding = w1,
bias = b1,
labels = labels,
input_embedding_vector =input_embedding_vector,
params = params)
optimizer = tf.train.AdagradOptimizer( learning_rate = params['learning_rate'] )
update_ops = tf.get_collection( tf.GraphKeys.UPDATE_OPS )
with tf.control_dependencies( update_ops ):
train_op = optimizer.minimize( loss, global_step= tf.train.get_global_step() )
return tf.estimator.EstimatorSpec( mode, loss=loss, train_op=train_op )
留言，评论，吐槽代码的都欢迎哈～
Ref
- [Word2Vec A]Tomas Mikolov et al, 2013, Efficient Edtimation of Word Representations in Vector Space
- [Word2Vec B]Tomas Mikolow et al, 2013, Distributed Representations of Words and Phrases and their Compositionality
- Yoav GoldBerg, Omer Levy, 2014, Wor2Vec Explained: Deribing Mikolow et al's Negative-Sampling Word Embedding Method
- Xin Rong, 2016, word2vec ParameterLearning Explained
- [Candidate Sampling]https://www.tensorflow.org/extras/candidate_sampling.pdf
- [Negative Sampling]Chris Dyer, 2014, Notes on Noise Contrastive Estimation and Negative Sampling
- https://github.com/chao-ji/tf-word2vec
- https://github.com/akb89/word2vec
- https://ruder.io/word-embeddings-softmax/index.html#negativesampling
- http://www.javashuo.com/article/p-ovsaqrgx-mw.html