word2vec学习笔记
word2vec学习笔记
前言
最近一个月事情多，心力交瘁，临近过年这几天进入到啥也不想干的状态，要想摆脱这种状态最好的方法就是赶忙看书写东西，给本身一些正反馈，走出负面循环。过完年要作一些NLP相关的事情了，全部要大体了解下相关内容，第一个准备深刻了解的就是word2vec，这是一种词嵌入模型主要做用就是为语言单词寻找一种尽量合理的向量化表示，一方面能保持单词的一些语义特征(如类似性)；另外一方面能是向量维度大小比较合理。Word2vec是身兼这两种特色的词嵌入表示。固然没有免费的午饭，咱们要经过训练获得这种表达。NLP和CV对待特征的思路很不同，这也是我刚入NLP的感受。前端
word2vec理论
这部分要仔细写起来很纠结，网上也有一堆相似的教程，我就不作详细介绍了，这里只讲个大概。一下内容大多来自standford CS224d lecture1。NLP须要先将文档进行分词而后对分词进行编码，编码最简单的就是One-hot vector一个单词占一个坑，可是这样一方面一个单词的维度太高，另外一方面没法表达向量之间的关系。word2vec有前端和后端之分，前端有CBOW和SKIP-GRAM这两种模型,后端有负采样和哈弗曼树这两种模型，前端和后端能够自由组合。不过经常使用的高效实现都是采用Skip-gram + 负采样.python
Skip-gram
Skip-gram的原理是对输入的单词预测其上下文，好比有一句话是{“The”, “cat”, ”jumped”，”over”, “the”, “puddle”}，skip-gram模型对输入中心词语"jumped"进行预测输出"jumped"的上下文“The”, “cat”, ”over”, “the”, “puddle”，听起来感受很神奇。下面这张图片表示了Skip-gram模型运行的过程。Skip-gram本质上就是一个逻辑回归。
Skip-gram的运行方式主要有如下几步骤:git
- 对单词生成one-hot输入向量\(x_k\)
- 获得上下文的嵌入词向量\(v_c = Vx\)
- 经过\(u = Uu_c\)产生2m个得分向量\(u_{c-m},...,u_{c-1},u_{c+1},...,u_{c+m}\)
- 将分向量转换成几率分布\(y=softmax(u)\)
- 最后将产生的几率与真实的几率分布作匹配
Skip-gram的目标/损失函数以下:
\[ \begin{eqnarray} minimize L &=& -logP(w_{c-m},...,w_{c-1},w_{c+1},...,w_{c+m}|w_c) \\ &=& -log\prod_{j=0,j\not=m}^{2m}P(w_{c-m+j}|w_c)\\ &=& -log\prod_{j=0,j\not=m}^{2m}P(u_{c-m+j}|u_c)\\ &=& -log\prod_{j=0,j\not=m}^{2m}\frac{exp(u^T_{c-m+j}v_c)}{\sum^{|V|}_{k=1}exp(u_k^Vv_c)}\\ &=& -\sum_{j=0,j\not=m}^{2m}u^T_{c-m+j}v_c + 2mlog\sum_{k=1}^{|V|}exp(u_k^Tv_c) \end{eqnarray} \]
负采样
上面的目标/损失函数须要对整个词汇表\(|V|\)进行计算，代价很是的高，所以引入了负采样。负采样的思想是：咱们不用去循环整个单词表，而只是采样一些负面的样本就够了，其几率分布与单词表中的频率相匹配。考虑一个词的"词－上下文"对\((w,c)\)，令\(P(D=1|w,c)\)为\((w,c)\)来自语料库的几率，则\(P(D=1|w,c)\)为不是来自语料库的几率，咱们有:后端
\[ P(D=1|w,c,\theta)=\frac{1}{1+e^{-v^T_cv_w}} \]
咱们须要创建一个新的目标函数。若是\((w,c)\)真是来自与语料库，目标函数可以最大化\(P(D=1|w,c)\)。咱们能够采用最大似然估计来获得模型参数。session
\[ \begin{eqnarray} \theta &=&\mathop{argmax}_{\theta}\prod_{(w,c)\in D}P(D=1|w,c,\theta)\prod_{(w,c)\in \tilde{D}}P(D=0|w,c,\theta)\\ &=&\mathop{argmax}_{\theta}\prod_{(w,c)\in D}P(D=1|w,c,\theta)\prod_{(w,c)\in \tilde{D}}(1-P(D=1|w,c,\theta))\\ &=&\mathop{argmax}_{\theta}\sum_{(w,c)\in D}log\frac{1}{1+exp(-u^T_wv_c)}+\sum_{(w,c)\in \tilde{D}}log(1-\frac{1}{1+exp(-u^T_wv_c)}) \\ &=&\mathop{argmax}_{\theta}\sum_{(w,c)\in D}log\frac{1}{1+exp(-u^T_wv_c)}+\sum_{(w,c)\in \tilde{D}}log\frac{1}{1+exp(u^T_wv_c)} \\ &=&\mathop{argmax}_{\theta}\sum_{(w,c)\in D}log\sigma(-u^T_wv_c)+\sum_{(w,c)\in \tilde{D}}log\sigma(u^T_wv_c)\\ \end{eqnarray} \]app
这是的\(\theta\)能够看作是上面的\(U,V\)，\(\tilde{D}\)表示负面的语料库。咱们可一进一步把目标函数写成：cors
\[ \begin{eqnarray} log\sigma(-u_{c-m+j}^Tv_c) + \sum^{K}_{k=1}log\sigma(\tilde{u}^T_kv_c) \end{eqnarray} \]
这里\(\tilde{u}_k\)是由负采样获得。dom
基于tensorflow的word2vec实现
上面大概介绍了一下word2vec的原理，讲的很简略，要想仔细了解仍是去看看网上的《word2vec的数学原理》一文，下面介绍tensorflow里面自带的例子word2vec的实现。ide
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
%matplotlib inline
from __future__ import print_function
import collections
import math
import numpy as np
import os
import random
import tensorflow as tf
import zipfile
import seaborn as sbn
from matplotlib import pylab
%config InlineBackend.figure_format = 'svg'
from six.moves import range
from six.moves.urllib.request import urlretrieve
from sklearn.manifold import TSNE
url = 'http://mattmahoney.net/dc/'
def maybe_download(filename, expected_bytes):
"""Download a file if not present, and make sure it's the right size."""
if not os.path.exists(filename):
filename, _ = urlretrieve(url + filename, filename)
statinfo = os.stat(filename)
if statinfo.st_size == expected_bytes:
print('Found and verified %s' % filename)
else:
print(statinfo.st_size)
raise Exception(
'Failed to verify ' + filename + '. Can you get to it with a browser?')
return filename
filename = maybe_download('text8.zip', 31344016)
def read_data(filename):
"""Extract the first file enclosed in a zip file as a list of words"""
with zipfile.ZipFile(filename) as f:
data = tf.compat.as_str(f.read(f.namelist()[0])).split()
return data
words = read_data(filename)
print('Data size %d' % len(words))
上面的代码主要功能是下载数据集而且读取数据，载入内存的是一个很长的文本序列。svg
vocabulary_size = 50000
def build_dataset(words):
count = [['UNK', -1]]
count.extend(collections.Counter(words).most_common(vocabulary_size - 1))
dictionary = dict()
for word, _ in count:
dictionary[word] = len(dictionary)
data = list()
unk_count = 0
for word in words:
if word in dictionary:
index = dictionary[word]
else:
index = 0 # dictionary['UNK']
unk_count = unk_count + 1
data.append(index)
count[0][1] = unk_count
reverse_dictionary = dict(zip(dictionary.values(), dictionary.keys()))
return data, count, dictionary, reverse_dictionary
data, count, dictionary, reverse_dictionary = build_dataset(words)
print('Most common words (+UNK)', count[:5])
print('Sample data', data[:10])
del words # Hint to reduce memory.
上面的代码短主要功能是为数据集进行编码，其中使用了most_common，因此单词会按照在文档中出现的次数进行编码，具体来讲就是出现次数多的单词的编码会相对小一些，这个在后面负采样中会用到。
data_index = 0
def generate_batch(batch_size, num_skips, skip_window):
global data_index
assert batch_size % num_skips == 0
assert num_skips <= 2 * skip_window
batch = np.ndarray(shape=(batch_size), dtype=np.int32)
labels = np.ndarray(shape=(batch_size, 1), dtype=np.int32)
span = 2 * skip_window + 1 # [ skip_window target skip_window ]
buffer = collections.deque(maxlen=span) # deque窗口 大小为 ２*skip_window + 1
for _ in range(span):
buffer.append(data[data_index])
data_index = (data_index + 1) % len(data)
for i in range(batch_size // num_skips): #两层循环，一个batch有batch/num_skips个数据,每一个数据的label大小为num_skips
target = skip_window # target label at the center of the buffer
targets_to_avoid = [ skip_window ]
for j in range(num_skips):
while target in targets_to_avoid:
target = random.randint(0, span - 1)
targets_to_avoid.append(target)
batch[i * num_skips + j] = buffer[skip_window]
labels[i * num_skips + j, 0] = buffer[target]
buffer.append(data[data_index])
data_index = (data_index + 1) % len(data)
return batch, labels
print('data:', [reverse_dictionary[di] for di in data[:8]])
for num_skips, skip_window in [(2, 1), (4, 2)]:
data_index = 0
batch, labels = generate_batch(batch_size=8, num_skips=num_skips, skip_window=skip_window)
print('\nwith num_skips = %d and skip_window = %d:' % (num_skips, skip_window))
print(batch)
print(' batch:', [reverse_dictionary[bi] for bi in batch])
print(' labels:', [reverse_dictionary[li] for li in labels.reshape(8)])
对于data: ['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first']上面的操做会造成一个这样的输出 batch中存储的是id， 假设咱们去skip_size = 4, skip_window = 2那么，单词 as 所对应的context的word个数就是4个，因此batch中有4个as， 所对应的就是context中的word
12 as -> 195 term
12 as -> 5239 anarchism
12 as -> 6 a
12 as -> 3084 originated
6 a -> 12 as
6 a -> 3084 originated
6 a -> 2 of
6 a -> 195 term
batch_size = 128
embedding_size = 128 # Dimension of the embedding vector.
skip_window = 1 # How many words to consider left and right.
num_skips = 2 # How many times to reuse an input to generate a label.
# We pick a random validation set to sample nearest neighbors. here we limit the
# validation samples to the words that have a low numeric ID, which by
# construction are also the most frequent.
valid_size = 16 # Random set of words to evaluate similarity on.
valid_window = 100 # Only pick dev samples in the head of the distribution.
valid_examples = np.array(random.sample(range(valid_window), valid_size))
num_sampled = 64 # Number of negative examples to sample.
graph = tf.Graph()
with graph.as_default(), tf.device('/cpu:0'):
# Input data.
train_dataset = tf.placeholder(tf.int32, shape=[batch_size])
train_labels = tf.placeholder(tf.int32, shape=[batch_size, 1])
valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
# Variables.
embeddings = tf.Variable(
tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
softmax_weights = tf.Variable(
tf.truncated_normal([vocabulary_size, embedding_size],
stddev=1.0 / math.sqrt(embedding_size)))
softmax_biases = tf.Variable(tf.zeros([vocabulary_size]))
# Model.
# Look up embeddings for inputs.
embed = tf.nn.embedding_lookup(embeddings, train_dataset) #其实就是按照train_dataset顺序返回embeddings中的第train_dataset行。
# Compute the softmax loss, using a sample of the negative labels each time.
loss = tf.reduce_mean(
tf.nn.nce_loss(softmax_weights, softmax_biases, embed,
train_labels, num_sampled, vocabulary_size))#是对类别太多的状况下loss计算的一种加速方法，具体能够参考文档
# Optimizer.
# Note: The optimizer will optimize the softmax_weights AND the embeddings.
# This is because the embeddings are defined as a variable quantity and the
# optimizer's `minimize` method will by default modify all variable quantities
# that contribute to the tensor it is passed.
# See docs on `tf.train.Optimizer.minimize()` for more details.
optimizer = tf.train.AdagradOptimizer(1.0).minimize(loss)
# Compute the similarity between minibatch examples and all embeddings.
# We use the cosine distance:
norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True))
normalized_embeddings = embeddings / norm
valid_embeddings = tf.nn.embedding_lookup(
normalized_embeddings, valid_dataset)
similarity = tf.matmul(valid_embeddings, tf.transpose(normalized_embeddings))
上面的代码就是tensorflow实现的word2vec的skip-gram模型，本质上就是一个逻辑回归啊，和上面的理论仍是有区别的，不过这里用的到了nce_loss，这个函数里面包括了negtive sample，后面会详细介绍。
num_steps = 100001
with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print('Initialized')
average_loss = 0
for step in range(num_steps):
batch_data, batch_labels = generate_batch(
batch_size, num_skips, skip_window)
feed_dict = {train_dataset : batch_data, train_labels : batch_labels}
_, l = session.run([optimizer, loss], feed_dict=feed_dict)
average_loss += l
if step % 2000 == 0:
if step > 0:
average_loss = average_loss / 2000
# The average loss is an estimate of the loss over the last 2000 batches.
print('Average loss at step %d: %f' % (step, average_loss))
average_loss = 0
# note that this is expensive (~20% slowdown if computed every 500 steps)
if step % 10000 == 0:
sim = similarity.eval()
for i in range(valid_size):
valid_word = reverse_dictionary[valid_examples[i]]
top_k = 8 # number of nearest neighbors
nearest = (-sim[i, :]).argsort()[1:top_k+1]
log = 'Nearest to %s:' % valid_word
for k in range(top_k):
close_word = reverse_dictionary[nearest[k]]
log = '%s %s,' % (log, close_word)
print(log)
final_embeddings = normalized_embeddings.eval()
num_points = 400
tsne = TSNE(perplexity=30, n_components=2, init='pca', n_iter=5000)
two_d_embeddings = tsne.fit_transform(final_embeddings[1:num_points+1, :])
def plot(embeddings, labels):
assert embeddings.shape[0] >= len(labels), 'More labels than embeddings'
pylab.figure(figsize=(15,15)) # in inches
for i, label in enumerate(labels):
x, y = embeddings[i,:]
pylab.scatter(x, y)
pylab.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',
ha='right', va='bottom')
pylab.savefig('softmax_loss.svg', format='svg')
pylab.show()
words = [reverse_dictionary[i] for i in range(1, num_points+1)]
plot(two_d_embeddings, words)
最后获得的结果以下
nce_loss
nce_loss的源码以下
def nce_loss(weights, #[num_classes, dim] dim就是emdedding_size
biases, #[num_classes] num_classes就是word的个数（不包括重复的）
inputs， #[batch_size, dim]
labels, #[batch_size, num_true] 这里，咱们的num_true设置为1，就是一个输入对应一个输出
num_sampled,#要取的负样本的个数（per batch）
num_classes,#类别的个数（在这里就是word的个数（不包含重复的））
num_true=1,
sampled_values=None,
remove_accidental_hits=False,
partition_strategy="mod",
name="nce_loss"):
logits, labels = _compute_sampled_logits(
weights,
biases,
inputs,
labels,
num_sampled,
num_classes,
num_true=num_true,
sampled_values=sampled_values,
subtract_log_q=True,
remove_accidental_hits=remove_accidental_hits,
partition_strategy=partition_strategy,
name=name)
sampled_losses = sigmoid_cross_entropy_with_logits(
logits, labels, name="sampled_losses")
#此函数返回的tensor与输入logits同维度。 _sum_rows以后，就获得了每一个样本的corss entropy。
# sampled_losses is batch_size x {true_loss, sampled_losses...}
# We sum out true and sampled losses.
return _sum_rows(sampled_losses)
#在word2vec中对此函数的返回调用了reduce_mean() 就得到了平均 cross entropy
# _compute_sampled_logits源码以下
def _compute_sampled_logits(weights,
biases,
inputs,
labels,
num_sampled,
num_classes,
num_true=1,
sampled_values=None,
subtract_log_q=True,
remove_accidental_hits=False,
partition_strategy="mod",
name=None):
if not isinstance(weights, list):
weights = [weights]
with ops.op_scope(weights + [biases, inputs, labels], name,
"compute_sampled_logits"):
if labels.dtype != dtypes.int64:
labels = math_ops.cast(labels, dtypes.int64)
labels_flat = array_ops.reshape(labels, [-1])
# Sample the negative labels.
# sampled shape: [num_sampled] tensor
# true_expected_count shape = [batch_size, 1] tensor
# sampled_expected_count shape = [num_sampled] tensor
if sampled_values is None:
sampled_values = candidate_sampling_ops.log_uniform_candidate_sampler(
true_classes=labels,
num_true=num_true,
num_sampled=num_sampled,
unique=True,
range_max=num_classes)
NOTE：这个函数是经过log-uniform进行取样的\(P(class)=\frac{(log(class+2)−log(class+1))}{log(rang\_max+1)}\)，取样范围是[0, range_max] ,用这种方法取样就要求咱们的word是按照频率从高到低排列的。以前对word的处理的确是这样,class越小取的几率越大。
sampled_softmax_loss
tensorflow的word2vec有的版本的损失函数用到了sampled_softmax_loss他和nce_loss很类似，参数是如出一辙的。
def sampled_softmax_loss(weights,
biases,
labels,
inputs,
num_sampled,
num_classes,
num_true=1,
sampled_values=None,
remove_accidental_hits=True,
partition_strategy="mod",
name="sampled_softmax_loss"):
logits, labels = _compute_sampled_logits(
weights=weights,
biases=biases,
labels=labels,
inputs=inputs,
num_sampled=num_sampled,
num_classes=num_classes,
num_true=num_true,
sampled_values=sampled_values,
subtract_log_q=True,
remove_accidental_hits=remove_accidental_hits,
partition_strategy=partition_strategy,
name=name)
sampled_losses = nn_ops.softmax_cross_entropy_with_logits(labels=labels,
logits=logits)
# sampled_losses is a [batch_size] tensor.
return sampled_losses
主要区别就是sigmoid_cross_entropy_with_logits和softmax_cross_entropy_with_logits，前者不要求类别之间是互斥的，后者要求是互斥的。nce_loss获得的结果会更加平滑一些。下面贴出了用sampled_softmax_loss获得的结果
参考
暂略
- 1. word2vec学习笔记
- 2. Word2Vec学习笔记
- 3. word2Vec学习笔记
- 4. Word2Vec --学习总结笔记
- 5. Word2vec：理论学习笔记
- 6. Tensorflow学习笔记——word2vec
- 7. 【NLP学习笔记】word2vec
- 8. word2vec学习笔记 转
- 9. CS224N学习笔记（一）Word2Vec
- 10. Word2vec学习笔记 ING
- 更多相关文章...
- • 您已经学习了 XML Schema，下一步学习什么呢？ - XML Schema 教程
- • 我们已经学习了 SQL，下一步学习什么呢？ - SQL 教程
- • Tomcat学习笔记（史上最全tomcat学习笔记）
- • 适用于PHP初学者的学习线路和建议
-
每一个你不满意的现在，都有一个你没有努力的曾经。
- 1. word2vec学习笔记
- 2. Word2Vec学习笔记
- 3. word2Vec学习笔记
- 4. Word2Vec --学习总结笔记
- 5. Word2vec：理论学习笔记
- 6. Tensorflow学习笔记——word2vec
- 7. 【NLP学习笔记】word2vec
- 8. word2vec学习笔记 转
- 9. CS224N学习笔记（一）Word2Vec
- 10. Word2vec学习笔记 ING