CS224n assignment1 Q3 word2vec

(a) 求解预测词向量 Vc的所对应的梯度。

Answer: 在Q2的(b)中，咱们已经获得了softmax关于输入向量的梯度=y_hat-y 因此本题的结果为 其中U=[μ1,μ2,…,μW]是全体输出向量造成的矩阵 即

---

(b) 求解输出词向量μw的梯度（包括μo在内）

Answer: 相似于上题，结果为 即

---

(c) 梯度求解

Answer: 一样是计算梯度，使用咱们以前的结果便可。

---

(d)计算由skip-gram和CBOW算法分别算出的所有词向量的梯度

Answer: 咱们用F(O,Vc)(其中O表示词汇)来做为损失函数的占位符。题目当中已经有提示，对于skip-gram，以c为中心的周围内容的损失计算为: 其中，Vc使咱们的词向量。

题目中提到，CBOW的损失函数定义为:

---

(e) 完成word2vec模型

首先完成归一化函数:

def normalizeRows(x): #行归一化函数
    """ Row normalization function

    Implement a function that normalizes each row of a matrix to have
    unit length.
    """
    n = x.shape[0]
    x /= np.sqrt(np.sum(x**2,axis=1)).reshape((n,1)) + 1e-30 #防止除0加个小数
    return x

完成word2vec的softmax损失函数:

def softmaxCostAndGradient(predicted, target, outputVectors, dataset):
    """ Softmax cost function for word2vec models

    Implement the cost and gradients for one predicted word vector
    and one target word vector as a building block for word2vec
    models, assuming the softmax prediction function and cross
    entropy loss.

    Arguments:
    predicted -- numpy ndarray, predicted word vector (\hat{v} in
                 the written component)
    target -- integer, the index of the target word
    outputVectors -- "output" vectors (as rows) for all tokens
    dataset -- needed for negative sampling, unused here.

    Return:
    cost -- cross entropy cost for the softmax word prediction
    gradPred -- the gradient with respect to the predicted word
           vector
    grad -- the gradient with respect to all the other word
           vectors

    We will not provide starter code for this function, but feel
    free to reference the code you previously wrote for this
    assignment!
    """
    #针对一个predict word和当前的的target word，完成一个传播过程

    #计算预测结果
    v_hat = predicted
    z = np.dot(outputVectors,v_hat)
    preds = softmax(z)

    cost = -np.log(preds[target])

    #计算梯度
    z = preds.copy()
    z[target] -= 1.0

    grad = np.outer(z,v_hat)
    #np.outer函数:
    #①对于多维向量，所有展开变为一维向量
    #②第一个参数表示倍数，使得第二个向量每次变为几倍
    #③第一个参数肯定结果的行，第二个参数肯定结果的列
    gradPred = np.dot(outputVectors.T,z)
    return cost, gradPred, grad

Word2vec模型负例采样后的损失函数和梯度:

def negSamplingCostAndGradient(predicted, target, outputVectors, dataset,
                               K=10):
    """ Negative sampling cost function for word2vec models

    Implement the cost and gradients for one predicted word vector
    and one target word vector as a building block for word2vec
    models, using the negative sampling technique. K is the sample
    size.

    Note: See test_word2vec below for dataset's initialization.

    Arguments/Return Specifications: same as softmaxCostAndGradient
    """
    # Sampling of indices is done for you. Do not modify this if you
    # wish to match the autograder and receive points!
    #为每一个窗口取k个负样本
    indices = [target]
    indices.extend(getNegativeSamples(target, dataset, K))

    grad = np.zeros(outputVectors.shape)
    gradPred = np.zeros(predicted.shape)
    cost = 0

    z = sigmoid(np.dot(outputVectors[target],predicted))

    cost -= np.log(z)
    grad[target] += predicted*(z-1.0)
    gradPred = outputVectors[target] * (z-1.0)

    #最小化这些词随中心词出如今中心词附近的几率
    for k in range(K):
        sample = indices[k+1]
        z = sigmoid(np.dot(outputVectors[sample],predicted))
        cost -= np.log(1.0-z)
        grad[sample] += predicted*z
        gradPred += outputVectors[sample] * z
    return cost, gradPred, grad

完成skipgram:

def skipgram(currentWord, C, contextWords, tokens, inputVectors, outputVectors,
             dataset, word2vecCostAndGradient=softmaxCostAndGradient):
    """ Skip-gram model in word2vec

    Implement the skip-gram model in this function.

    Arguments:
    currentWord -- a string of the current center word
    C -- integer, context size
    contextWords -- list of no more than 2*C strings, the context words
    tokens -- a dictionary that maps words to their indices in
              the word vector list
    inputVectors -- "input" word vectors (as rows) for all tokens
    outputVectors -- "output" word vectors (as rows) for all tokens
    word2vecCostAndGradient -- the cost and gradient function for
                               a prediction vector given the target
                               word vectors, could be one of the two
                               cost functions you implemented above.

    Return:
    cost -- the cost function value for the skip-gram model
    grad -- the gradient with respect to the word vectors
    """
    cost = 0.0
    gradIn = np.zeros(inputVectors.shape)
    gradOut = np.zeros(outputVectors.shape)

    cword_idx = tokens[currentWord]
    v_hat = inputVectors[cword_idx]
    #skipgram即根据当前词预测必定范围内的上下文词汇，选择让几率分部值最大的向量
    for i in contextWords:#对于窗口中的每一个单词
        idx = tokens[i] #target的下标(要预测的单词的下标)
        c_cost,c_grad_in,c_grad_out = word2vecCostAndGradient(v_hat,idx,outputVectors,dataset)
        #更新cost、grad 即便用k个单词来训练这个向量
        cost += c_cost
        gradOut += c_grad_out
        gradIn[cword_idx] += c_grad_in
    return cost, gradIn, gradOut

---

(f)完成SGD

注意在python3中已经不用cPickle，直接用pickle就能够了。

def sgd(f, x0, step, iterations, postprocessing=None, useSaved=False,
        PRINT_EVERY=10):
    """ Stochastic Gradient Descent

    Implement the stochastic gradient descent method in this function.

    Arguments:
    f -- the function to optimize, it should take a single
         argument and yield two outputs, a cost and the gradient
         with respect to the arguments
    x0 -- the initial point to start SGD from
    step -- the step size for SGD
    iterations -- total iterations to run SGD for
    postprocessing -- postprocessing function for the parameters
                      if necessary. In the case of word2vec we will need to
                      normalize the word vectors to have unit length.
    PRINT_EVERY -- specifies how many iterations to output loss

    Return:
    x -- the parameter value after SGD finishes
    """

    # Anneal learning rate every several iterations
    ANNEAL_EVERY = 20000

    if useSaved:
        start_iter, oldx, state = load_saved_params()
        if start_iter > 0:
            x0 = oldx
            step *= 0.5 ** (start_iter / ANNEAL_EVERY)

        if state:
            random.setstate(state)
    else:
        start_iter = 0

    x = x0

    if not postprocessing:
        postprocessing = lambda x: x

    expcost = None

    for iter in range(start_iter + 1, iterations + 1):
        # Don't forget to apply the postprocessing after every iteration!
        # You might want to print the progress every few iterations.

        cost = None
        ### YOUR CODE HERE
        cost,grad = f(x)
        x -= step*grad
        postprocessing(x)
        ### END YOUR CODE

        if iter % PRINT_EVERY == 0:
            if not expcost:
                expcost = cost
            else:
                expcost = .95 * expcost + .05 * cost
            print ("iter %d: %f" % (iter, expcost))

        if iter % SAVE_PARAMS_EVERY == 0 and useSaved:
            save_params(iter, x)

        if iter % ANNEAL_EVERY == 0:
            step *= 0.5

    return x

---

(f)训练

这里为了使用python3，须要改较多地方，根据报错的地方逐渐修改便可 训练一共花了8.09个小时

---

(h)完成cbow

def cbow(currentWord, C, contextWords, tokens, inputVectors, outputVectors,
         dataset, word2vecCostAndGradient=softmaxCostAndGradient):
    """CBOW model in word2vec

    Implement the continuous bag-of-words model in this function.

    Arguments/Return specifications: same as the skip-gram model

    Extra credit: Implementing CBOW is optional, but the gradient
    derivations are not. If you decide not to implement CBOW, remove
    the NotImplementedError.
    """
    #经过周围单词向量的和来预测中心单词

    cost = 0
    gradIn = np.zeros(inputVectors.shape)
    gradOut = np.zeros(outputVectors.shape)
    D = inputVectors.shape[1]
    predicted = np.zeros((D,))

    indices = [tokens[cwd] for cwd in contextWords]
    #输入为周围单词的向量和
    for idx in indices:
        predicted += inputVectors[idx, :]

    #tokens[currentWord]中心词的下标
    cost, gp, gradOut = word2vecCostAndGradient(predicted, tokens[currentWord], outputVectors, dataset)
    #即用这个单词来训练周围的向量
    gradIn = np.zeros(inputVectors.shape)
    for idx in indices:
        gradIn[idx, :] += gp
    return cost, gradIn, gradOut

参考: https://blog.csdn.net/longxinchen_ml/article/details/51765418