舒适提示,TensorFlow更新的太快了,有些代码实现方式可能变了,可是思想仍是没有变滴,主要仍是理解 原文地址git
介绍github
前面的教程代表,简单的线性模型具备大约91%的分类准确度,用于识别MNIST数据集中的手写数字。windows
在本教程中,咱们将在TensorFlow中实现一个简单的卷积神经网络,若是您进行一些建议的练习,其分类精度约为99%或更高。api
卷积网络经过在输入图像上移动小滤镜来工做。 这意味着过滤器被从新用于识别整个输入图像中的模式。 这使得卷积网络比具备相同数量的变量的彻底链接网络更强大。 这反过来使Convolutional Networks更快地进行训练。bash
您应该熟悉基本的线性代数,Python和Jupyter Notebook编辑器。 TensorFlow的初学者也可能但愿在继续学习以前学习第一篇教程。网络
Flowchart 下图大体显示了下面实现的卷积神经网络中数据的流动方式。session
而后在第二卷积层中处理这16个较小的图像。 咱们须要为这16个通道中的每一个通道提供滤波器权重,而且咱们须要对该层的每一个输出通道使用滤波器权重。 有36个输出通道,所以在第二个卷积层中总共有16 x 36 = 576个滤波器。 生成的图像再次下采样到7x7像素。dom
第二卷积层的输出是每一个7×7像素的36个图像。 而后将它们展平为长度为7×7×36 = 1764的单个矢量,其用做具备128个神经元(或元件)的彻底链接层的输入。 这将进入另外一个具备10个神经元的彻底链接的层,每一个类对应一个类,用于肯定图像的类别,即图像中描绘的数字。编辑器
卷积滤波器最初是随机选择的,所以分类是随机进行的。 输入图像的预测类和真实类之间的偏差被测量为所谓的交叉熵。 而后,优化器使用区分链规则自动将此错误传播回卷积网络,并更新过滤器权重,以改善分类错误。 这反复进行数千次,直到分类偏差足够低。ide
这些特定的滤镜权重和中间图像是一次优化运行的结果,若是从新运行此Notebook,可能会有所不一样。
请注意,TensorFlow中的计算其实是在一批图像而不是单个图像上完成的,这使得计算更有效。 这意味着当在TensorFlow中实现时,流程图实际上还有一个数据维度。
Convolutional Layer
下图显示了在第一个卷积层中处理图像的基本思想。 输入图像描绘了数字7,此处显示了四个图像副本,所以咱们能够更清楚地看到滤镜如何移动到图像的不一样位置。 对于滤镜的每一个位置,在滤镜和滤镜下的图像像素之间计算点积,这致使输出图像中的单个像素。 所以,在整个输入图像上移动滤镜会致使生成新图像。
红色滤镜权重意味着滤镜对输入图像中的黑色像素具备正反应,而蓝色像素意味着滤镜对黑色像素具备负反应。
在这种状况下,过滤器能够识别出7位数的水平线,从输出图像中对该线的较强反应能够看出。
在输入上移动过滤器的步长称为步幅。 水平移动过滤器(x轴)和另外一个垂直移动步幅(y轴)有一个跨度。
在下面的源代码中,步幅在两个方向上都设置为1,这意味着滤镜从输入图像的左上角开始,并在每一个步骤中向右移动1个像素。 当滤镜到达图像右侧的末尾时,滤镜将向后移动到图像的左侧和1个像素。 这将一直持续到滤镜到达输入图像的右下角并生成整个输出图像。
当滤镜到达输入图像的右侧和底部的末端时,能够用零填充(白色像素)。 这会使输出图像与输入图像具备彻底相同的尺寸。
此外,卷积的输出能够经过所谓的整流线性单元(ReLU),其仅确保输出为正,由于负值被设置为零。 输出也能够经过所谓的max-pooling进行下采样,max-pooling考虑2x2像素的小窗口而且仅保留这些像素中的最大像素。 这使输入图像的分辨率减半,例如 从28x28到14x14像素。
注意,第二卷积层更复杂,由于它须要16个输入通道。 咱们想为每一个输入通道分别使用一个过滤器,所以咱们须要16个过滤器而不是一个过滤器。 此外,咱们须要来自第二卷积层的36个输出通道,所以总共须要16 x 36 = 576个滤波器用于第二个卷积层。 了解其工做原理可能有点难度。
Imports
%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math
复制代码
Configuration of Neural Network
为方便起见,此处定义了卷积神经网络的配置,所以您能够轻松查找和更改这些数字并从新运行Notebook。
# Convolutional Layer 1.
filter_size1 = 5 # Convolution filters are 5 x 5 pixels.
num_filters1 = 16 # There are 16 of these filters.
# Convolutional Layer 2.
filter_size2 = 5 # Convolution filters are 5 x 5 pixels.
num_filters2 = 36 # There are 36 of these filters.
# Fully-connected layer.
fc_size = 128 # Number of neurons in fully-connected layer.
复制代码
Load Data
MNIST数据集大约为12 MB,若是它不在给定路径中,将自动下载。
from mnist import MNIST
data = MNIST(data_dir="data/MNIST/")
复制代码
复制代码MNIST数据集现已加载,由70,000个图像和图像的类号组成。 数据集被分红3个互斥的子集。 咱们将仅使用本教程中的培训和测试集。
print("Size of:")
print("- Training-set:\t\t{}".format(data.num_train))
print("- Validation-set:\t{}".format(data.num_val))
print("- Test-set:\t\t{}".format(data.num_test))
Size of:
- Training-set: 55000
- Validation-set: 5000
- Test-set: 10000
复制代码
复制代码为方便起见,复制一些数据维度。
# The number of pixels in each dimension of an image.
img_size = data.img_size
# The images are stored in one-dimensional arrays of this length.
img_size_flat = data.img_size_flat
# Tuple with height and width of images used to reshape arrays.
img_shape = data.img_shape
# Number of classes, one class for each of 10 digits.
num_classes = data.num_classes
# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = data.num_channels
复制代码
Helper-function for plotting images
用于在3x3网格中绘制9个图像,并在每一个图像下面写入真实和预测类的函数。
def plot_images(images, cls_true, cls_pred=None):
assert len(images) == len(cls_true) == 9
# Create figure with 3x3 sub-plots.
fig, axes = plt.subplots(3, 3)
fig.subplots_adjust(hspace=0.3, wspace=0.3)
for i, ax in enumerate(axes.flat):
# Plot image.
ax.imshow(images[i].reshape(img_shape), cmap='binary')
# Show true and predicted classes.
if cls_pred is None:
xlabel = "True: {0}".format(cls_true[i])
else:
xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])
# Show the classes as the label on the x-axis.
ax.set_xlabel(xlabel)
# Remove ticks from the plot.
ax.set_xticks([])
ax.set_yticks([])
# Ensure the plot is shown correctly with multiple plots
# in a single Notebook cell.
plt.show()
复制代码
Plot a few images to see if data is correct
# Get the first images from the test-set.
images = data.x_test[0:9]
# Get the true classes for those images.
cls_true = data.y_test_cls[0:9]
# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)
复制代码
TensorFlow Graph
#Helper-functions for creating new variables(辅助函数用于建立新变量)
用于在给定形状中建立新TensorFlow变量并使用随机值初始化它们的函数。 请注意,此时实际上并未完成初始化,它仅在TensorFlow图中定义。
def new_weights(shape):
return tf.Variable(tf.truncated_normal(shape, stddev=0.05))
def new_biases(length):
return tf.Variable(tf.constant(0.05, shape=[length]))
复制代码
#Helper-function for creating a new Convolutional Layer(辅助功能,用于建立新的卷积层)
此函数在TensorFlow的计算图中建立一个新的卷积层。 这里没有实际计算,咱们只是将数学公式添加到TensorFlow图。
假设输入是具备如下尺寸的4维张量:
图像编号。
每一个图像的Y轴。
每一个图像的X轴。
每一个图像的通道。
复制代码
注意,输入通道能够是颜色通道,或者若是输入是从先前的卷积层产生的,则它能够是滤波器通道。
输出是另外一个4-dim张量,具备如下尺寸:
图像编号,与输入相同。
每一个图像的Y轴。 若是使用2x2池,则输入图像的高度和宽度除以2。
每一个图像的X轴。 同上。
卷积滤波器产生的通道。
复制代码
def new_conv_layer(input, # The previous layer.
num_input_channels, # Num. channels in prev. layer.
filter_size, # Width and height of each filter.
num_filters, # Number of filters.
use_pooling=True): # Use 2x2 max-pooling.
# Shape of the filter-weights for the convolution.
# This format is determined by the TensorFlow API.
shape = [filter_size, filter_size, num_input_channels, num_filters]
# Create new weights aka. filters with the given shape.
weights = new_weights(shape=shape)
# Create new biases, one for each filter.
biases = new_biases(length=num_filters)
# Create the TensorFlow operation for convolution.
# Note the strides are set to 1 in all dimensions.
# The first and last stride must always be 1,
# because the first is for the image-number and
# the last is for the input-channel.
# But e.g. strides=[1, 2, 2, 1] would mean that the filter
# is moved 2 pixels across the x- and y-axis of the image.
# The padding is set to 'SAME' which means the input image
# is padded with zeroes so the size of the output is the same.
layer = tf.nn.conv2d(input=input,
filter=weights,
strides=[1, 1, 1, 1],
padding='SAME')
# Add the biases to the results of the convolution.
# A bias-value is added to each filter-channel.
layer += biases
# Use pooling to down-sample the image resolution?
if use_pooling:
# This is 2x2 max-pooling, which means that we
# consider 2x2 windows and select the largest value
# in each window. Then we move 2 pixels to the next window.
layer = tf.nn.max_pool(value=layer,
ksize=[1, 2, 2, 1],
strides=[1, 2, 2, 1],
padding='SAME')
# Rectified Linear Unit (ReLU).
# It calculates max(x, 0) for each input pixel x.
# This adds some non-linearity to the formula and allows us
# to learn more complicated functions.
layer = tf.nn.relu(layer)
# Note that ReLU is normally executed before the pooling,
# but since relu(max_pool(x)) == max_pool(relu(x)) we can
# save 75% of the relu-operations by max-pooling first.
# We return both the resulting layer and the filter-weights
# because we will plot the weights later.
return layer, weights
复制代码
#Helper-function for flattening a layer(辅助功能,用于展平图层)
卷积层产生具备4维的输出张量。 咱们将在卷积层以后添加彻底链接的层,所以咱们须要将4-dim张量减少到2-dim,这能够用做彻底链接层的输入。
def flatten_layer(layer):
# Get the shape of the input layer.
layer_shape = layer.get_shape()
# The shape of the input layer is assumed to be:
# layer_shape == [num_images, img_height, img_width, num_channels]
# The number of features is: img_height * img_width * num_channels
# We can use a function from TensorFlow to calculate this.
num_features = layer_shape[1:4].num_elements()
# Reshape the layer to [num_images, num_features].
# Note that we just set the size of the second dimension
# to num_features and the size of the first dimension to -1
# which means the size in that dimension is calculated
# so the total size of the tensor is unchanged from the reshaping.
layer_flat = tf.reshape(layer, [-1, num_features])
# The shape of the flattened layer is now:
# [num_images, img_height * img_width * num_channels]
# Return both the flattened layer and the number of features.
return layer_flat, num_features
复制代码
#Helper-function for creating a new Fully-Connected Layer(辅助功能,用于建立新的全链接层)
此函数在TensorFlow的计算图中建立一个新的彻底链接层。 这里没有实际计算,咱们只是将数学公式添加到TensorFlow图。
假设输入是形状[di_images,num_inputs]的二维张量。 输出是2-dim张量的形状[num_images,num_outputs]。
def new_fc_layer(input, # The previous layer.
num_inputs, # Num. inputs from prev. layer.
num_outputs, # Num. outputs.
use_relu=True): # Use Rectified Linear Unit (ReLU)?
# Create new weights and biases.
weights = new_weights(shape=[num_inputs, num_outputs])
biases = new_biases(length=num_outputs)
# Calculate the layer as the matrix multiplication of
# the input and weights, and then add the bias-values.
layer = tf.matmul(input, weights) + biases
# Use ReLU?
if use_relu:
layer = tf.nn.relu(layer)
return layer
复制代码
#Placeholder variables(占位符变量) 占位符变量用做TensorFlow计算图的输入,每次执行图时咱们均可以更改。 咱们称之为占位符变量,并在下面进一步说明。
首先,咱们为输入图像定义占位符变量。 这容许咱们更改输入到TensorFlow图形的图像。 这是一个所谓的张量,这意味着它是一个多维向量或矩阵。 数据类型设置为float32,形状设置为[None,img_size_flat],其中None表示张量能够保持任意数量的图像,每一个图像是长度为img_size_flat的向量。
x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')
复制代码
卷积层指望x被编码为4-dim张量,所以咱们必须对其进行整形,使其形状为[num_images,img_height,img_width,num_channels]。 请注意,img_height == img_width == img_size和num_images可使用-1做为第一个维度的大小自动推断。 因此重塑操做是:
x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])
复制代码
接下来,咱们有占位符变量,用于与占位符变量x中输入的图像相关联的真实标签。 此占位符变量的形状为[None,num_classes],这意味着它能够包含任意数量的标签,每一个标签是长度为num_classes的向量,在这种状况下为10。
y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')
复制代码
咱们也能够为label设置占位符变量,但咱们将使用argmax计算它。 请注意,这是一个TensorFlow运算符,所以此时不计算任何内容。
y_true_cls = tf.argmax(y_true, axis=1)
复制代码
Convolutional Layer 1
建立第一个卷积层。 它将x_image做为输入并建立num_filters1个不一样的过滤器,每一个过滤器的宽度和高度等于filter_size1。 最后,咱们但愿经过使用2x2 max-pooling对图像进行下采样,使其大小减半。
layer_conv1, weights_conv1 = \
new_conv_layer(input=x_image,
num_input_channels=num_channels,
filter_size=filter_size1,
num_filters=num_filters1,
use_pooling=True)
复制代码
检查将由卷积层输出的张量的形状。 它是(?,14,14,16),这意味着有任意数量的图像(这是?),每一个图像宽14像素,高14像素,有16个不一样的通道,每一个通道一个通道 过滤器
layer_conv1
<tf.Tensor 'Relu:0' shape=(?, 14, 14, 16) dtype=float32>
复制代码
Convolutional Layer 2
建立第二个卷积层,它将第一个卷积层的输出做为输入。 输入通道的数量对应于第一卷积层中的滤波器的数量。
layer_conv2, weights_conv2 = \
new_conv_layer(input=layer_conv1,
num_input_channels=num_filters1,
filter_size=filter_size2,
num_filters=num_filters2,
use_pooling=True)
复制代码
检查将今后卷积层输出的张量的形状。 形状是(?,7,7,36)? 再次表示存在任意数量的图像,每一个图像的宽度和高度为7个像素,而且有36个通道,每一个滤波器一个。
layer_conv2
<tf.Tensor 'Relu_1:0' shape=(?, 7, 7, 36) dtype=float32>
复制代码
Flatten Layer
卷积层输出4-dim张量。 咱们如今但愿在彻底链接的网络中使用这些做为输入,这须要将张量从新整形或展平为2-dim张量。
layer_flat, num_features = flatten_layer(layer_conv2)
复制代码
检查张量如今是否具备形状(?,1764),这意味着有任意数量的图像已被展平为每一个长度为1764的向量。 请注意,1764 = 7 x 7 x 36。
In [23]:layer_flat
Out[23]:<tf.Tensor 'Reshape_1:0' shape=(?, 1764) dtype=float32>
In [24]:num_features
Out[24]:1764
复制代码
Fully-Connected Layer 1
将彻底链接的层添加到网络中。 输入是前一卷积的平坦层。 彻底链接层中的神经元或节点的数量是fc_size。 使用ReLU,咱们能够学习非线性关系。
layer_fc1 = new_fc_layer(input=layer_flat,
num_inputs=num_features,
num_outputs=fc_size,
use_relu=True)
复制代码
检查彻底链接层的输出是否为形状(?,128)的张量? 意味着有任意数量的图像,fc_size == 128。
In [26]:layer_fc1
Out[26]:<tf.Tensor 'Relu_2:0' shape=(?, 128) dtype=float32>
复制代码
Fully-Connected Layer 2
添加另外一个彻底链接的层,输出长度为10的向量,用于肯定输入图像属于哪10个类。 请注意,此层中不使用ReLU。
In [27]:layer_fc2 = new_fc_layer(input=layer_fc1,
num_inputs=fc_size,
num_outputs=num_classes,
use_relu=False)
In [28]:layer_fc2
Out[28]:<tf.Tensor 'add_3:0' shape=(?, 10) dtype=float32>
复制代码
Predicted Class
第二个彻底链接的层估计输入图像属于10个类中的每个的可能性。 然而,这些估计有点粗糙且难以解释,由于数字可能很是小或很大,所以咱们但愿将它们标准化,以便每一个元素限制在0和1之间,10个元素总和为1。 这是使用所谓的softmax函数计算的,结果存储在y_pred中。
y_pred = tf.nn.softmax(layer_fc2)
复制代码
class-number是最大元素的索引。
y_pred_cls = tf.argmax(y_pred, axis=1)
复制代码
Cost-function to be optimized
为了使模型更好地分类输入图像,咱们必须以某种方式改变全部网络层的变量。 为此,咱们首先须要经过将模型y_pred的预测输出与指望输出y_true进行比较来了解模型当前的执行状况。
交叉熵是用于分类的性能度量。 交叉熵是一个始终为正的连续函数,若是模型的预测输出与指望输出彻底匹配,则交叉熵等于零。 所以,优化的目标是最小化交叉熵,使其经过改变网络层的变量尽量接近零。
TensorFlow具备用于计算交叉熵的内置函数。 请注意,该函数在内部计算softmax,所以咱们必须直接使用layer_fc2的输出,而不是已经应用了softmax的y_pred。
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
labels=y_true)
复制代码
咱们如今已经计算了每一个图像分类的交叉熵,所以咱们能够测量模型对每一个图像的单独执行状况。 可是为了使用交叉熵来指导模型变量的优化,咱们须要一个标量值,所以咱们只须要对全部图像分类采用交叉熵的平均值。
cost = tf.reduce_mean(cross_entropy)
复制代码
Optimization Method 如今咱们有一个必须最小化的成本度量,而后咱们能够建立一个优化器。 在这种状况下,它是AdamOptimizer,它是Gradient Descent的高级形式。
请注意,此时不执行优化。 事实上,根本没有计算任何东西,咱们只需将optimizer-object添加到TensorFlow图中以便之后执行。
optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)
复制代码
Performance Measures
correct_prediction = tf.equal(y_pred_cls, y_true_cls)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
复制代码
TensorFlow Run
接下来都是训练、测试、调试啦~ 方式不少,这里我就只贴代码了
session = tf.Session()
session.run(tf.global_variables_initializer())
train_batch_size = 64
# Counter for total number of iterations performed so far.
total_iterations = 0
def optimize(num_iterations):
# Ensure we update the global variable rather than a local copy.
global total_iterations
# Start-time used for printing time-usage below.
start_time = time.time()
for i in range(total_iterations,
total_iterations + num_iterations):
# Get a batch of training examples.
# x_batch now holds a batch of images and
# y_true_batch are the true labels for those images.
x_batch, y_true_batch, _ = data.random_batch(batch_size=train_batch_size)
# Put the batch into a dict with the proper names
# for placeholder variables in the TensorFlow graph.
feed_dict_train = {x: x_batch,
y_true: y_true_batch}
# Run the optimizer using this batch of training data.
# TensorFlow assigns the variables in feed_dict_train
# to the placeholder variables and then runs the optimizer.
session.run(optimizer, feed_dict=feed_dict_train)
# Print status every 100 iterations.
if i % 100 == 0:
# Calculate the accuracy on the training-set.
acc = session.run(accuracy, feed_dict=feed_dict_train)
# Message for printing.
msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"
# Print it.
print(msg.format(i + 1, acc))
# Update the total number of iterations performed.
total_iterations += num_iterations
# Ending time.
end_time = time.time()
# Difference between start and end-times.
time_dif = end_time - start_time
# Print the time-usage.
print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))
复制代码
def plot_example_errors(cls_pred, correct):
# This function is called from print_test_accuracy() below.
# cls_pred is an array of the predicted class-number for
# all images in the test-set.
# correct is a boolean array whether the predicted class
# is equal to the true class for each image in the test-set.
# Negate the boolean array.
incorrect = (correct == False)
# Get the images from the test-set that have been
# incorrectly classified.
images = data.x_test[incorrect]
# Get the predicted classes for those images.
cls_pred = cls_pred[incorrect]
# Get the true classes for those images.
cls_true = data.y_test_cls[incorrect]
# Plot the first 9 images.
plot_images(images=images[0:9],
cls_true=cls_true[0:9],
cls_pred=cls_pred[0:9])
复制代码
def plot_confusion_matrix(cls_pred):
# This is called from print_test_accuracy() below.
# cls_pred is an array of the predicted class-number for
# all images in the test-set.
# Get the true classifications for the test-set.
cls_true = data.y_test_cls
# Get the confusion matrix using sklearn.
cm = confusion_matrix(y_true=cls_true,
y_pred=cls_pred)
# Print the confusion matrix as text.
print(cm)
# Plot the confusion matrix as an image.
plt.matshow(cm)
# Make various adjustments to the plot.
plt.colorbar()
tick_marks = np.arange(num_classes)
plt.xticks(tick_marks, range(num_classes))
plt.yticks(tick_marks, range(num_classes))
plt.xlabel('Predicted')
plt.ylabel('True')
# Ensure the plot is shown correctly with multiple plots
# in a single Notebook cell.
plt.show()
复制代码
# Split the test-set into smaller batches of this size.
test_batch_size = 256
def print_test_accuracy(show_example_errors=False,
show_confusion_matrix=False):
# Number of images in the test-set.
num_test = data.num_test
# Allocate an array for the predicted classes which
# will be calculated in batches and filled into this array.
cls_pred = np.zeros(shape=num_test, dtype=np.int)
# Now calculate the predicted classes for the batches.
# We will just iterate through all the batches.
# There might be a more clever and Pythonic way of doing this.
# The starting index for the next batch is denoted i.
i = 0
while i < num_test:
# The ending index for the next batch is denoted j.
j = min(i + test_batch_size, num_test)
# Get the images from the test-set between index i and j.
images = data.x_test[i:j, :]
# Get the associated labels.
labels = data.y_test[i:j, :]
# Create a feed-dict with these images and labels.
feed_dict = {x: images,
y_true: labels}
# Calculate the predicted class using TensorFlow.
cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)
# Set the start-index for the next batch to the
# end-index of the current batch.
i = j
# Convenience variable for the true class-numbers of the test-set.
cls_true = data.y_test_cls
# Create a boolean array whether each image is correctly classified.
correct = (cls_true == cls_pred)
# Calculate the number of correctly classified images.
# When summing a boolean array, False means 0 and True means 1.
correct_sum = correct.sum()
# Classification accuracy is the number of correctly classified
# images divided by the total number of images in the test-set.
acc = float(correct_sum) / num_test
# Print the accuracy.
msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
print(msg.format(acc, correct_sum, num_test))
# Plot some examples of mis-classifications, if desired.
if show_example_errors:
print("Example errors:")
plot_example_errors(cls_pred=cls_pred, correct=correct)
# Plot the confusion matrix, if desired.
if show_confusion_matrix:
print("Confusion Matrix:")
plot_confusion_matrix(cls_pred=cls_pred)
复制代码
print_test_accuracy()
optimize(num_iterations=900)
print_test_accuracy(show_example_errors=True)
复制代码