[Tensorflow] Practice - The Tensorflow Way

该系列主要是《Tensorflow 实战Google深度学习框架 》阅读笔记；有了Cookbook的热身后，以这本书做为基础造成我的知识体系。

 

Ref: [Tensorflow] Cookbook - The Tensorflow Way

---

 

第一章，简介（略）

第二章，安装（仅记录个别要点）

Protocol buffer

Bazel, similar with Makefile for complile.

Install steps: 

　　(1) Docker

　　(2) Tensorflow

Source code --> pip install package --> pip install.

 

第三章，入门

 

计算图

1. 定义计算

2. 执行计算

 

In [1]: import tensorflow as tf

In [2]: a = tf.constant([1.0, 2.0], name = "a")

In [3]: b = tf.constant([2.0, 3.0], name = "b")

In [4]: result = a+b

# 必须sess才能执行，这里只是定义
In [5]: result
Out[5]: <tf.Tensor 'add:0' shape=(2,) dtype=float32>

 

系统默认了一个计算图：

In [6]: print(a.graph is tf.get_default_graph())
True

In [7]: print(b.graph is tf.get_default_graph())
True

 

两个图，两个name = 'v'的variable；但这里不冲突。

import tensorflow as tf g1 = tf.Graph()　　#自定了一个图 with g1.as_default():　　#设置为当前要操做的 v = tf.get_variable("v", [1])
 g2 = tf.Graph() with g2.as_default(): v = tf.get_variable("v", [1])
 
# 定义结构图
# 执行结构图
 with tf.Session(graph = g1) as sess:　　# 执行图g1 tf.global_variables_initializer().run()
 with tf.variable_scope("", reuse=True): print(sess.run(tf.get_variable("v"))) with tf.Session(graph = g2) as sess:　　# 执行图g2 tf.global_variables_initializer().run() with tf.variable_scope("", reuse=True): print(sess.run(tf.get_variable("v")))

经过图，指定运行图的设备
g = tf.Graph()
with g.device('/gpu:0'):
　　result = a + b

 

集合

　　-- 将资源加入集合

 

张量

　　-- 仅保存了如何获得这些数字的计算过程

import tensorflow as tf a = tf.constant([1.0, 2.0], name="a") b = tf.constant([2.0, 3.0], name="b") result = a + b print(result) sess = tf.InteractiveSession() print(result.eval()) sess.close()

 

获得的是：对结果的一个引用。【一个张量的结构】

【add:0 表示result这个张量是计算节点“add"输出的第一个结果】

【2, 表示是一维数组，长度为2】

Tensor("add:0", shape=(2,), dtype=float32) [ 3.  5.]

基本概念：

零阶张量：scalar

一阶张量：vector

二阶张量：matrix

三阶张量：super matrix :-p

 

会话

 将全部计算放在“with"的内部：

with tf.Session() as sess: print(sess.run(result))

 

NB: Graph有默认的，自动生成；但session没有！The sess you create will be added autometically into this default Graph.

设置默认会话：【sess过程当中有一次with就能够了】

sess = tf.Session() with sess.as_default(): print(result.eval()) Output:
[ 3.  7.]

 

指定为默认会话的意义是什么？获取张量的取值更加方便。

sess = tf.Session() with sess.as_default():　　# 注册的过程 print(result.eval())

 经过InteractiveSession自动将会话注册为默认会话。

sess = tf.InteractiveSession ()　　# create session即同时注册 print(result.eval()) sess.close()　　# 但岂不是多了一行代码？方便在了哪里，不解

 

会话配置的修改

config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=True)
 sess1 = tf.InteractiveSession(config=config) sess2 = tf.Session(config=config)

 

矩阵计算

 a = tf.matmul(x, w1)　　# 已经默认考虑了转置问题，故比较方便

 

变量

【cookbook有详细实例】

w1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))

NB：seed的意义在于：保证每次运行获得的结果是同样的。

得到shape：

w1.get_shape()
Out[51]: TensorShape([Dimension(2), Dimension(3)])

w1.get_shape()[0]
Out[52]: Dimension(2)

w1.get_shape()[1]
Out[53]: Dimension(3)

 

• 经过“拷贝”初始化

w2 = tf.Variable(w1.initialized_value())　　# 直接拷贝别人家的初始值 w3 = tf.Variable(w1.initialized_value() * 2.0)

• 经过"随机数"初始化

• 经过"常数"初始化

 

变量初始化的执行

经过 tf.global_variables_initializer() 真正执行对变量初始化的设定。

w1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1)) w2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1)) x = tf.placeholder(tf.float32, shape=(1, 2), name="input")　　// 没有初始值，但最好给出自身“容器”的大小，未来给feed瞧 a = tf.matmul(x, w1) y = tf.matmul(a, w2) sess = tf.Session() init_op = tf.global_variables_initializer() sess.run(init_op) print(sess.run(y, feed_dict={x: [[0.7,0.9]]}))

 

例如：w₁在Graph中的解析

 

Assign

变量维度的改变，但基本不用，也不会给本身找麻烦。

tf.assign( w1, w2, validate_shape=False )

 

 

第四章，深层神经网络

激活函数让神经网络再也不线性化。

实现代码，可见极其简洁：

a = tf.nn.relu(tf.matmul(x, w1) + biases1) y = tf.nn.relu(tf.matmul(a, w2) + biases2)

  

• Cross-entropy

避免log值太小的方式：clip_by_value

cross_entropy = -tf.reduce_mean(t * tf.log(tf.clip_by_value(y, 1e-10, 1.0))) 

 

Before cross-entropy, we always use softmax: X * W -->  softmax --> cross-entropy

softmax_cross_entropy_with_logits( _sentinel=None, labels =None, logits =None, dim =-1, name =None ) sparse_softmax_cross_entropy_with_logits( _sentinel=None, labels =None, logits =None, name =None )

 

若是只是关心前向传播的预测值，那么其实只关心logits部分，而后须要取出最大几率的那个label。

 

• MSE - L2 loss

NB: Classification by xentropy; For regression, we use MSE as following:

mse = tf.reduce_mean(tf.square(y_ - y))

 

Loss最终的归宿：

train_step = tf.train.AdamOptimizer(0.001).minimize(loss)

  

tensorflow api for LOSS: 

absolute_difference(...): Adds an Absolute Difference loss to the training procedure.

add_loss(...): Adds a externally defined loss to the collection of losses.

compute_weighted_loss(...): Computes the weighted loss.

cosine_distance(...): Adds a cosine-distance loss to the training procedure.

get_losses(...): Gets the list of losses from the loss_collection.

get_regularization_loss(...): Gets the total regularization loss.

get_regularization_losses(...): Gets the list of regularization losses.

get_total_loss(...): Returns a tensor whose value represents the total loss.

hinge_loss(...): Adds a hinge loss to the training procedure.

huber_loss(...): Adds a Huber Loss term to the training procedure.

log_loss(...): Adds a Log Loss term to the training procedure.

mean_pairwise_squared_error(...): Adds a pairwise-errors-squared loss to the training procedure.

mean_squared_error(...): Adds a Sum-of-Squares loss to the training procedure.

sigmoid_cross_entropy(...): Creates a cross-entropy loss using tf.nn.sigmoid_cross_entropy_with_logits.

softmax_cross_entropy(...): Creates a cross-entropy loss using tf.nn.softmax_cross_entropy_with_logits.

sparse_softmax_cross_entropy(...): Cross-entropy loss using tf.nn.sparse_softmax_cross_entropy_with_logits.

 

 

• 高级点的问题

CNN--两个Loss层计算的数值问题 (overflow...)

　　From: https://zhuanlan.zhihu.com/p/22260935

在计算Loss部分是可能出现的一些小问题以及如今的解决方法。

其实也是仔细阅读下Caffe代码中有关Softmax loss和sigmoid cross entropy loss两个部分的真实计算方法。

 

• exp这个函数实在是有毒

指数函数是一个很容易让数值爆炸的函数，那么输入大概到多少会溢出呢？蛋疼的我仍是作了一个实验：

np.exp(709) 8.2184074615549724e+307

 

出现以下问题：

def naive_softmax(x): y = np.exp(x) return y / np.sum(y)

#b取值很大，部分值大于了709
b = np.random.rand(10) * 1000
print b print naive_softmax(b)

[ 497.46732916  227.75385779  537.82669096  787.54950048  663.13861524
  224.69389572  958.39441314  139.09633232  381.35034548  604.08586655]
[  0.   0.   0.  nan   0.   0.  nan   0.   0.   0.]

 

那么如何解决呢？咱们只要给每一个数字除以一个大数，保证它不溢出，问题不就解决了？

老司机给出的方案是找出输入数据中最大的数，而后除以e的最大数次幂，至关于下面的代码：

def high_level_softmax(x): max_val = np.max(x) x -= max_val return naive_softmax(x)

 

However，scale太大，个别值过小了！

b = np.random.rand(10) * 1000
print b print high_level_softmax(b) [ 903.27437996  260.68316085   22.31677464  544.80611744  506.26848644
  698.38019158  833.72024087  200.55675076  924.07740602  909.39841128] [ 9.23337324e-010   7.79004225e-289   0.00000000e+000   
   1.92562645e-165   3.53094986e-182   9.57072864e-099   
   5.73299537e-040   6.01134555e-315   9.99999577e-001   
   4.21690097e-007]

 

使用一点平滑的小技巧仍是颇有必要的，因而代码又变成：

def practical_softmax(x): max_val = np.max(x) x -= max_val y = np.exp(x) y[y < 1e-20] = 1e-20
    return y / np.sum(y)

 

Result: 至关于加了个下限

[  9.23337325e-10   9.99999577e-21   9.99999577e-21   9.99999577e-21
   9.99999577e-21   9.99999577e-21   9.99999577e-21   9.99999577e-21
   9.99999577e-01   4.21690096e-07]

【但，貌似一个简单的封装好的 preds  = tf.nn.softmax(z)，便可解决这个问题】

 

• sigmoid也是中毒专业户

由于其中包含了exp，*_*b

def naive_sigmoid_loss(x, t): y = 1 / (1 + np.exp(-x)) return -np.sum(t * np.log(y) + (1 - t) * np.log(1 - y)) / y.shape[0] 

a = np.random.rand(10)* 1000 b = a > 500
print a print b print naive_sigmoid_loss(a,b)

[  63.20798359  958.94378279  250.75385942  895.49371345  965.62635077
   81.1217712   423.36466749  532.20604694  333.45425951  185.72621262]
[False  True False  True  True False False  True False False]
nan

 

改进方法：

对应代码：

def high_level_sigmoid_loss(x, t): first = (t - (x > 0)) * x second = np.log(1 + np.exp(x - 2 * x * (x > 0))) return -np.sum(first - second) / x.shape[0]

a = np.random.rand(10)* 1000 - 500 b = a > 0 print a print b print high_level_sigmoid_loss(a,b) [-173.48716596  462.06216262 -417.78666769    6.10480948  340.13986055
   23.64615392  256.33358957 -332.46689674  416.88593348 -246.51402684] [False True False True True True True False True False] 0.000222961919658

 

 

NN的进一步优化问题

• 学习率的设置

没有label，求得的值 y = x² 就直接是lost function。

对于learning_rate = 1的理解：

导数是2x，故w变化是10，这就是震荡的缘由。

import tensorflow as tf
 TRAINING_STEPS = 10 LEARNING_RATE = 1 　　#尝试改变学习率，查看收敛效果

# x here denotes w x = tf.Variable(tf.constant(5, dtype=tf.float32), name="x") y = tf.square(x)　　# y = x2 train_op = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(y) with tf.Session() as sess: sess.run(tf.global_variables_initializer())
 for i in range(TRAINING_STEPS): sess.run(train_op) x_value = sess.run(x) print "After %s iteration(s): x%s is %f."% (i+1, i+1, x_value) 

 

指数递减学习率

TRAINING_STEPS = 100 global_step = tf.Variable(0) LEARNING_RATE = tf.train.exponential_decay(0.1, global_step, 1, 0.96, staircase=True) # 初始学习率
# 没1次训练学习率衰减为原来的0.96
 x = tf.Variable(tf.constant(5, dtype=tf.float32), name="x") y = tf.square(x) train_op = tf.train.GradientDescentOptimizer(LEARNING_RATE).minimize(y, global_step=global_step) with tf.Session() as sess: sess.run(tf.global_variables_initializer()) for i in range(TRAINING_STEPS): sess.run(train_op) if i % 10 == 0: LEARNING_RATE_value = sess.run(LEARNING_RATE) x_value = sess.run(x) print("After %s iteration(s): x%s is %f, learning rate is %f."% (i+1, i+1, x_value, LEARNING_RATE_value))  

 

 

• 过拟合问题

 

画出这两个图，感受很好玩的样子，怎么画呢？

 

import tensorflow as tf import matplotlib.pyplot as plt import numpy as np data = [] label = [] np.random.seed(0)

 

# 以原点为圆心，半径为1的圆把散点划分红红蓝两部分，并加入随机噪音。

剩下就是给data, label对儿不断添加一对对儿数据的过程。

for i in range(150): x1 = np.random.uniform(-1,1) x2 = np.random.uniform(0,2)
 if x1**2 + x2**2 <= 1: data.append([np.random.normal(x1, 0.1),np.random.normal(x2,0.1)]) label.append(0) else: data.append([np.random.normal(x1, 0.1), np.random.normal(x2, 0.1)]) label.append(1) data = np.hstack(data ).reshape(-1,2)　　# 这里的2对应了二维空间的x,y两个坐标值 label = np.hstack(label).reshape(-1,1)
 plt.scatter(data[:,0], data[:,1], c=label, cmap="RdBu", vmin=-.2, vmax=1.2, edgecolor="white") plt.show()

 

np.hstack 用法

>>> a = np.array((1,2,3)) >>> b = np.array((2,3,4)) 
 >>> np.hstack((a,b)) array([1, 2, 3, 2, 3, 4]) 
 >>> a = np.array([[1],[2],[3]]) >>> b = np.array([[2],[3],[4]]) 
 >>> np.hstack((a,b)) array([[1, 2], [2, 3], [3, 4]])  

 

np.reshape 用法

a=array([[1,2,3],[4,5,6]]) reshape(a, 6) 

 Out[202]: 
  array([1, 2, 3, 4, 5, 6])

 NB：这里的 ‘-1’

reshape(a, (3, -1)) #为指定的值将被推断出为2  

Out[204]:
array([[1, 2],
[3, 4],
[5, 6]])

 

循环生成网络结构，好巧妙的技巧！

x  = tf.placeholder(tf.float32, shape=(None, 2)) y_ = tf.placeholder(tf.float32, shape=(None, 1)) sample_size = len(data) # 每层节点的个数：比较有意思的构建网络方法
layer_dimension = [2,10,5,3,1] n_layers = len(layer_dimension) cur_layer = x # 循环生成网络结构
for i in range(1, n_layers):　　# NB：这是是从2nd layer开始，也就是第一个out_layer in_dimension = layer_dimension[i-1]
 out_dimension = layer_dimension[i]
 weight = get_weight([in_dimension, out_dimension], 0.003)　　# 正则参数 ----> bias = tf.Variable(tf.constant(0.1, shape=[out_dimension]))
 cur_layer = tf.nn.elu(tf.matmul(cur_layer, weight) + bias)
  y= cur_layer # 损失函数的定义。
# 这里只须要计算"刻画模型在训练数据集上的表现"的损失函数
mse_loss = tf.reduce_sum(tf.pow(y_ - y, 2)) / sample_size tf.add_to_collection('losses', mse_loss)　　# 尚未正则的loss

# 获得了最终的损失函数 - 同时也结合了get_weight中的add_to_collection loss = tf.add_n(tf.get_collection('losses'))

 

tf.get_collection('losses') 的内容以下：

[<tf.Tensor 'l2_regularizer:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_1:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_2:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_3:0' shape=() dtype=float32>, <tf.Tensor 'truediv:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_4:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_5:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_6:0' shape=() dtype=float32>, <tf.Tensor 'l2_regularizer_7:0' shape=() dtype=float32>, <tf.Tensor 'truediv_1:0' shape=() dtype=float32>]

 

将“L2正则后的权重变量var”加入到集合中：tf.add_to_collecdtion。

def get_weight(shape, lambda1): var = tf.Variable(tf.random_normal(shape), dtype=tf.float32) tf.add_to_collection('losses', tf.contrib.layers.l2_regularizer(lambda1)(var)) return var

 

训练不带正则项的损失函数mse_loss

# 定义训练的目标函数mse_loss，训练次数及训练模型
train_op = tf.train.AdamOptimizer(0.001).minimize(mse_loss) TRAINING_STEPS = 40000 with tf.Session() as sess: tf.global_variables_initializer().run() for i in range(TRAINING_STEPS): sess.run(train_op, feed_dict={x: data, y_: label}) if i % 2000 == 0: print("After %d steps, mse_loss: %f" % (i,sess.run(mse_loss, feed_dict={x: data, y_: label}))) 
 # 画出训练后的分割曲线 - 颇有意思！ 
 # 1. 画网格
    xx, yy = np.mgrid[-1.2:1.2:.01, -0.2:2.2:.01] grid = np.c_[xx.ravel(), yy.ravel()]
 # 2. probs = sess.run(y, feed_dict={x:grid})　　# y在这里表明了最后一层 probs = probs.reshape(xx.shape) plt.scatter(data[:,0], data[:,1], c=label, cmap="RdBu", vmin=-.2, vmax=1.2, edgecolor="white") plt.contour(xx, yy, probs, levels=[.5], cmap="Greys", vmin=0, vmax=.1) plt.show()

Ref: http://blog.csdn.net/u013534498/article/details/51399035

这篇博文我喜欢，数据表现也须要开专题学习。

 

np.mgrid用法

np.mgrid[-1.2:1.2:.01, -0.2:2.2:.01]
参数格式：行，列，间隙

Out[217]: 
array([[[-1.2 , -1.2 , -1.2 , ..., -1.2 , -1.2 , -1.2 ],
        [-1.19, -1.19, -1.19, ..., -1.19, -1.19, -1.19],
        [-1.18, -1.18, -1.18, ..., -1.18, -1.18, -1.18],
        ..., 
        [ 1.17,  1.17,  1.17, ...,  1.17,  1.17,  1.17],
        [ 1.18,  1.18,  1.18, ...,  1.18,  1.18,  1.18],
        [ 1.19,  1.19,  1.19, ...,  1.19,  1.19,  1.19]],

       [[-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ],
        [-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ],
        [-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ],
        ..., 
        [-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ],
        [-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ],
        [-0.2 , -0.19, -0.18, ...,  2.18,  2.19,  2.2 ]]])

 

 

• 滑动平均模型

衰减率：模型更新的速度

 

变量 --> 影子变量 (share init)

影子变量 = 衰减率*影子变量+(1-衰减率)*变量

衰减率越大，变量更新越快！

decay整体上不但愿更新太快，但前期但愿更新快些的衰减率设置办法：

查看不一样迭代中变量取值的变化

import tensorflow as tf v1 = tf.Variable(0, dtype=tf.float32) step = tf.Variable(0, trainable=False)
 ema = tf.train.ExponentialMovingAverage(0.99, step) # step：控制衰减率的变量 maintain_averages_op = ema.apply([v1]) # 更新列表中的变量 with tf.Session() as sess: # 初始化
    init_op = tf.global_variables_initializer() sess.run(init_op) print(sess.run([v1, ema.average(v1)]))

     [0.0, 0.0]

 # 更新变量v1的取值
    sess.run(tf.assign(v1, 5)) sess.run(maintain_averages_op) print(sess.run([v1, ema.average(v1)]))

     [5.0, 4.5]

 # 更新step和v1的取值
    sess.run(tf.assign(step, 10000)) 
 sess.run(tf.assign(v1, 10)) sess.run(maintain_averages_op) print(sess.run([v1, ema.average(v1)]))

     [10.0, 4.5549998]

 # 更新一次v1的滑动平均值
 sess.run(maintain_averages_op) print(sess.run([v1, ema.average(v1)]))     

     [10.0, 4.6094499]

仍是不太了解其目的：难道就是为了ema.average(v1) 这个返回结果？

 

---

疑难杂症

 

版本查看：

python  -c 'import tensorflow as tf; print(tf.__version__)'  # for Python 2
python3 -c 'import tensorflow as tf; print(tf.__version__)'  # for Python 3

安装升级：

unsw@unsw-UX303UB$ pip3 install --upgrade tensorflow Requirement already up-to-date: tensorflow in /usr/local/anaconda3/lib/python3.5/site-packages Requirement already up-to-date: six>=1.10.0 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow) Requirement already up-to-date: tensorflow-tensorboard<0.2.0,>=0.1.0 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow) Requirement already up-to-date: wheel>=0.26 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow) Requirement already up-to-date: protobuf>=3.3.0 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow) Requirement already up-to-date: numpy>=1.11.0 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow) Requirement already up-to-date: werkzeug>=0.11.10 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow-tensorboard<0.2.0,>=0.1.0->tensorflow) Requirement already up-to-date: markdown>=2.6.8 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow-tensorboard<0.2.0,>=0.1.0->tensorflow) Requirement already up-to-date: bleach==1.5.0 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow-tensorboard<0.2.0,>=0.1.0->tensorflow) Requirement already up-to-date: html5lib==0.9999999 in /usr/local/anaconda3/lib/python3.5/site-packages (from tensorflow-tensorboard<0.2.0,>=0.1.0->tensorflow) Requirement already up-to-date: setuptools in /usr/local/anaconda3/lib/python3.5/site-packages (from protobuf>=3.3.0->tensorflow)  unsw@unsw-UX303UB$ python3 -c 'import tensorflow as tf; print(tf.__version__)'
1.3.0

 忽略警告：https://github.com/tensorflow/tensorflow/issues/7778

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'