{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "阅读资料\n",
    "\n",
    "- [Gaussic讲述卷积神经网络mnist](https://gaussic.github.io/2017/08/14/tensorflow-cnn/)\n",
    "- [tensorflow中文社区](http://www.tensorfly.cn/tfdoc/tutorials/mnist_pros.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ../../data/mnist/train-images-idx3-ubyte.gz\n",
      "Extracting ../../data/mnist/train-labels-idx1-ubyte.gz\n",
      "Extracting ../../data/mnist/t10k-images-idx3-ubyte.gz\n",
      "Extracting ../../data/mnist/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# 初始化工作\n",
    "import tensorflow as tf\n",
    "\n",
    "# 准备数据，mnist\n",
    "old_v = tf.logging.get_verbosity()\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"../../data/mnist/\", one_hot=True)\n",
    "\n",
    "tf.logging.set_verbosity(old_v)\n",
    "\n",
    "x = tf.placeholder(\"float\", shape=[None, 784])\n",
    "y_ = tf.placeholder(\"float\", shape=[None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建一个多层卷积网络\n",
    "\n",
    "本小节在 mnist softmax 的基础上修改，构建一个多层卷积网络。最终大概达到 99.2% 的准确率。\n",
    "\n",
    "## 权重初始化\n",
    "\n",
    "多层卷积网络，需要创建大量的权重与偏置项。\n",
    "\n",
    "- 权重需要在初始化时加入少量的噪声，以打破对称性以及避免0梯度。\n",
    "- 本例使用 ReLu 神经元，即使用 线性整流函数（Rectified Linear Unit, ReLU）作为神经元的激活函数，所以建议使用一个较小的正数来初始化偏置项，避免神经元输出恒为0的问题(dead neurons)\n",
    "- 预先定义两个函数用于初始化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tf.truncated_normal(shape, mean, stddev)\n",
    "# truncate 意译截断\n",
    "# shape表示生成张量的维度，mean是均值，stddev是标准差。\n",
    "# 这个函数产生正太分布，均值和标准差自己设定，并截断的产生正太分布。\n",
    "\n",
    "def weight_variable(shape):\n",
    "  initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "  return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "  initial = tf.constant(0.1, shape=shape) # 创建 shape 的 常量，并使用 value 填充\n",
    "  return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 第一层卷积\n",
    "\n",
    "### 先认识 tf.nn.conv2d() 函数的使用\n",
    "\n",
    "![](https://img-blog.csdn.net/20171229215216572?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvTG9zZUluVmFpbg==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n",
    "\n",
    "```python\n",
    "conv2d(\n",
    "    input,\n",
    "    filter,\n",
    "    strides,\n",
    "    padding,\n",
    "    use_cudnn_on_gpu=None,\n",
    "    data_format=None,\n",
    "    name=None\n",
    ")\n",
    "```\n",
    "\n",
    "- input 输入，4d参数 `[batch_size, in_height, in_width, n_channels]`，表示图片的批数，大小和通道。\n",
    "- filter 过滤器，4d参数`[filter_height, filter_width, in_channels, out_channels]`，表示kernel的大小，输入通道数和输出通道数，其中输出通道数表示从上一层提取多少特征。\n",
    "- strides 步长，1d参数，长度为4，其中`stride[0]`和`stride[3]`必须为1，一般格式为`[1, stride[1], stride[2], 1]`，在大部分情况下，因为在height和width上的步进设为一样，因此通常为`[1, stride, stride, 1]`。 \n",
    "    + 计算公式为： \n",
    "    ```\n",
    "    output[b,i,j,k]=∑di,dj,qinput[b,strides[1]∗i+di,strides[2]∗j+dj,q]∗filter[di,dj,q,k]\n",
    "    ```\n",
    "    \n",
    "- padding 是一个字符串输入，分为 SAME 和 VALID 分别表示是否需要填充，因为卷积完之后因为周围的像素没有卷积到，因此一般是会出现卷积完的输出尺寸小于输入的现象的\n",
    "\n",
    "\n",
    "### 第一层卷积的计算过程\n",
    "\n",
    "\n",
    "第一层，由一个卷积接着一个 max pooling 完成，卷积在每个`5*5`的 patch 中算出32个特征。卷积的权重张量形状是`[5,5,1,32]`，前两个维度是 patch 的大小，接着是输入的通道数目，最后是输入的通道数目。而对于每一个输出通道都有一个对应的偏置量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_conv1 = weight_variable([5, 5, 1, 32]) # 权重即是卷积中的 filter\n",
    "b_conv1 = bias_variable([32])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了适应这一层卷积，输入 x 改为一个四维向量，第2,3维度对应图片的宽高，第4维度对应图片的颜色通道——灰度图为1 （如果是rgb彩色图，则为3）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.placeholder(\"float\", [None, 784])\n",
    "x_image = tf.reshape(x, [-1,28,28,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在这个实例里，我们会一直使用vanilla版本。我们的卷积使用1步长（stride size），0边距（padding size）的模板，保证输出和输入是同一个大小。我们的池化用简单传统的2x2大小的模板做max pooling。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def conv2d(x, W):\n",
    "  return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "def max_pool_2x2(x):\n",
    "  return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],\n",
    "                        strides=[1, 2, 2, 1], padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们把x_image和权值向量进行卷积，加上偏置项，然后应用ReLU激活函数，最后进行max pooling。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1) # 卷积的输出 + 偏移量 后，再经过 relu 置零张量中的负数\n",
    "h_pool1 = max_pool_2x2(h_conv1) # 池化，提取特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 第二层卷积\n",
    "\n",
    "为了构建一个更深的网络，我们会把几个类似的层堆叠起来。第二层中，每个5x5的patch会得到64个特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 这里将第一层的 32 个特征，作为第二层的 32 个通道的输入，相应的，filter 的 depth = 32，有 64 个 filter，得出 64 个 depth 的输出\n",
    "W_conv2 = weight_variable([5, 5, 32, 64])\n",
    "b_conv2 = bias_variable([64])\n",
    "\n",
    "h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)\n",
    "h_pool2 = max_pool_2x2(h_conv2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 密集连接层\n",
    "\n",
    "现在，图片尺寸经过两层卷积中的池化后，从28x28减小到7x7，我们加入一个有1024个神经元的全连接层，用于处理整个图片。我们把池化层输出的张量reshape成一些向量，乘上权重矩阵，加上偏置，然后对其使用ReLU。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_fc1 = weight_variable([7 * 7 * 64, 1024])\n",
    "b_fc1 = bias_variable([1024])\n",
    "\n",
    "# 第二层卷积输出的 7*7*64 的张量，平铺成一维数组，分别与 1024 组 7*7*64 个数字相乘后相加，得出 1024 个输出，再加上偏移量、置零负数\n",
    "h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])\n",
    "h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropout\n",
    "\n",
    "为了减少过拟合，我们在输出层之前加入dropout。我们用一个placeholder来代表一个神经元的输出在dropout中保持不变的概率。这样我们可以在训练过程中启用dropout，在测试过程中关闭dropout。 TensorFlow的tf.nn.dropout操作除了可以屏蔽神经元的输出外，还会自动处理神经元输出值的scale。所以用dropout的时候可以不用考虑scale。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "keep_prob = tf.placeholder(\"float\") # 保留神经元的概率\n",
    "h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob) # 不同的训练过程中随机扔掉一部分神经元"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输出层\n",
    "\n",
    "最后，我们添加一个softmax层，就像前面的单层softmax regression一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_fc2 = weight_variable([1024, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "\n",
    "y_conv=tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 执行与评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/lightfish/tf/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py:189: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.\n",
      "Instructions for updating:\n",
      "Use `tf.global_variables_initializer` instead.\n",
      "step 0, training accuracy 0.06\n",
      "step 100, training accuracy 0.76\n",
      "step 200, training accuracy 0.9\n",
      "step 300, training accuracy 0.94\n",
      "step 400, training accuracy 0.96\n",
      "step 500, training accuracy 0.98\n",
      "step 600, training accuracy 0.92\n",
      "step 700, training accuracy 0.96\n",
      "step 800, training accuracy 0.94\n",
      "step 900, training accuracy 0.96\n",
      "step 1000, training accuracy 1\n",
      "step 1100, training accuracy 1\n",
      "step 1200, training accuracy 0.98\n",
      "step 1300, training accuracy 0.94\n",
      "step 1400, training accuracy 0.98\n",
      "step 1500, training accuracy 0.98\n",
      "step 1600, training accuracy 1\n",
      "step 1700, training accuracy 0.98\n",
      "step 1800, training accuracy 1\n",
      "step 1900, training accuracy 0.98\n",
      "step 2000, training accuracy 0.98\n",
      "step 2100, training accuracy 0.88\n",
      "step 2200, training accuracy 0.98\n",
      "step 2300, training accuracy 0.98\n",
      "step 2400, training accuracy 1\n",
      "step 2500, training accuracy 0.98\n",
      "step 2600, training accuracy 0.96\n",
      "step 2700, training accuracy 1\n",
      "step 2800, training accuracy 0.98\n",
      "step 2900, training accuracy 1\n",
      "step 3000, training accuracy 0.98\n",
      "step 3100, training accuracy 0.98\n",
      "step 3200, training accuracy 0.98\n",
      "step 3300, training accuracy 1\n",
      "step 3400, training accuracy 1\n",
      "step 3500, training accuracy 1\n",
      "step 3600, training accuracy 0.96\n",
      "step 3700, training accuracy 0.98\n",
      "step 3800, training accuracy 0.98\n",
      "step 3900, training accuracy 1\n",
      "step 4000, training accuracy 1\n",
      "step 4100, training accuracy 1\n",
      "step 4200, training accuracy 1\n",
      "step 4300, training accuracy 0.98\n",
      "step 4400, training accuracy 0.98\n",
      "step 4500, training accuracy 1\n",
      "step 4600, training accuracy 0.98\n",
      "step 4700, training accuracy 0.98\n",
      "step 4800, training accuracy 1\n",
      "step 4900, training accuracy 1\n",
      "step 5000, training accuracy 1\n",
      "step 5100, training accuracy 0.98\n",
      "step 5200, training accuracy 0.96\n",
      "step 5300, training accuracy 1\n",
      "step 5400, training accuracy 1\n",
      "step 5500, training accuracy 0.98\n",
      "step 5600, training accuracy 1\n",
      "step 5700, training accuracy 0.94\n",
      "step 5800, training accuracy 1\n",
      "step 5900, training accuracy 1\n",
      "step 6000, training accuracy 1\n",
      "step 6100, training accuracy 0.98\n",
      "step 6200, training accuracy 0.98\n",
      "step 6300, training accuracy 0.98\n",
      "step 6400, training accuracy 1\n",
      "step 6500, training accuracy 0.98\n",
      "step 6600, training accuracy 0.96\n",
      "step 6700, training accuracy 1\n",
      "step 6800, training accuracy 1\n",
      "step 6900, training accuracy 1\n",
      "step 7000, training accuracy 1\n",
      "step 7100, training accuracy 1\n",
      "step 7200, training accuracy 1\n",
      "step 7300, training accuracy 1\n",
      "step 7400, training accuracy 1\n",
      "step 7500, training accuracy 1\n",
      "step 7600, training accuracy 1\n",
      "step 7700, training accuracy 1\n",
      "step 7800, training accuracy 1\n",
      "step 7900, training accuracy 1\n",
      "step 8000, training accuracy 1\n",
      "step 8100, training accuracy 1\n",
      "step 8200, training accuracy 1\n",
      "step 8300, training accuracy 1\n",
      "step 8400, training accuracy 1\n",
      "step 8500, training accuracy 1\n",
      "step 8600, training accuracy 0.98\n",
      "step 8700, training accuracy 1\n",
      "step 8800, training accuracy 1\n",
      "step 8900, training accuracy 1\n",
      "step 9000, training accuracy 1\n",
      "step 9100, training accuracy 1\n",
      "step 9200, training accuracy 1\n",
      "step 9300, training accuracy 1\n",
      "step 9400, training accuracy 1\n",
      "step 9500, training accuracy 0.98\n",
      "step 9600, training accuracy 1\n",
      "step 9700, training accuracy 0.96\n",
      "step 9800, training accuracy 1\n",
      "step 9900, training accuracy 0.98\n",
      "step 10000, training accuracy 1\n",
      "step 10100, training accuracy 1\n",
      "step 10200, training accuracy 1\n",
      "step 10300, training accuracy 1\n",
      "step 10400, training accuracy 1\n",
      "step 10500, training accuracy 1\n",
      "step 10600, training accuracy 1\n",
      "step 10700, training accuracy 1\n",
      "step 10800, training accuracy 0.98\n",
      "step 10900, training accuracy 1\n",
      "step 11000, training accuracy 1\n",
      "step 11100, training accuracy 1\n",
      "step 11200, training accuracy 1\n",
      "step 11300, training accuracy 0.98\n",
      "step 11400, training accuracy 1\n",
      "step 11500, training accuracy 0.98\n",
      "step 11600, training accuracy 1\n",
      "step 11700, training accuracy 1\n",
      "step 11800, training accuracy 1\n",
      "step 11900, training accuracy 1\n",
      "step 12000, training accuracy 1\n",
      "step 12100, training accuracy 1\n",
      "step 12200, training accuracy 1\n",
      "step 12300, training accuracy 1\n",
      "step 12400, training accuracy 0.98\n",
      "step 12500, training accuracy 1\n",
      "step 12600, training accuracy 1\n",
      "step 12700, training accuracy 1\n",
      "step 12800, training accuracy 1\n",
      "step 12900, training accuracy 1\n",
      "step 13000, training accuracy 1\n",
      "step 13100, training accuracy 1\n",
      "step 13200, training accuracy 1\n",
      "step 13300, training accuracy 1\n",
      "step 13400, training accuracy 1\n",
      "step 13500, training accuracy 1\n",
      "step 13600, training accuracy 1\n",
      "step 13700, training accuracy 1\n",
      "step 13800, training accuracy 1\n",
      "step 13900, training accuracy 0.98\n",
      "step 14000, training accuracy 1\n",
      "step 14100, training accuracy 1\n",
      "step 14200, training accuracy 1\n",
      "step 14300, training accuracy 1\n",
      "step 14400, training accuracy 1\n",
      "step 14500, training accuracy 1\n",
      "step 14600, training accuracy 1\n",
      "step 14700, training accuracy 1\n",
      "step 14800, training accuracy 1\n",
      "step 14900, training accuracy 1\n",
      "step 15000, training accuracy 1\n",
      "step 15100, training accuracy 1\n",
      "step 15200, training accuracy 1\n",
      "step 15300, training accuracy 1\n",
      "step 15400, training accuracy 1\n",
      "step 15500, training accuracy 0.98\n",
      "step 15600, training accuracy 1\n",
      "step 15700, training accuracy 1\n",
      "step 15800, training accuracy 1\n",
      "step 15900, training accuracy 1\n",
      "step 16000, training accuracy 1\n",
      "step 16100, training accuracy 0.98\n",
      "step 16200, training accuracy 1\n",
      "step 16300, training accuracy 1\n",
      "step 16400, training accuracy 1\n",
      "step 16500, training accuracy 1\n",
      "step 16600, training accuracy 1\n",
      "step 16700, training accuracy 1\n",
      "step 16800, training accuracy 1\n",
      "step 16900, training accuracy 1\n",
      "step 17000, training accuracy 1\n",
      "step 17100, training accuracy 1\n",
      "step 17200, training accuracy 1\n",
      "step 17300, training accuracy 1\n",
      "step 17400, training accuracy 1\n",
      "step 17500, training accuracy 1\n",
      "step 17600, training accuracy 1\n",
      "step 17700, training accuracy 1\n",
      "step 17800, training accuracy 1\n",
      "step 17900, training accuracy 1\n",
      "step 18000, training accuracy 1\n",
      "step 18100, training accuracy 1\n",
      "step 18200, training accuracy 1\n",
      "step 18300, training accuracy 1\n",
      "step 18400, training accuracy 1\n",
      "step 18500, training accuracy 1\n",
      "step 18600, training accuracy 1\n",
      "step 18700, training accuracy 1\n",
      "step 18800, training accuracy 1\n",
      "step 18900, training accuracy 1\n",
      "step 19000, training accuracy 1\n",
      "step 19100, training accuracy 1\n",
      "step 19200, training accuracy 1\n",
      "step 19300, training accuracy 1\n",
      "step 19400, training accuracy 1\n",
      "step 19500, training accuracy 1\n",
      "step 19600, training accuracy 1\n",
      "step 19700, training accuracy 1\n",
      "step 19800, training accuracy 1\n",
      "step 19900, training accuracy 1\n",
      "test accuracy 0.9935\n"
     ]
    }
   ],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))\n",
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)\n",
    "correct_prediction = tf.equal(tf.argmax(y_conv,1), tf.argmax(y_,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "sess.run(tf.global_variables_initializer())\n",
    "for i in range(20000):\n",
    "  batch_xs, batch_ys = mnist.train.next_batch(50)\n",
    "  if i%100 == 0:\n",
    "    train_accuracy = accuracy.eval(feed_dict={\n",
    "        x:batch_xs, y_: batch_ys, keep_prob: 1.0})\n",
    "    print(\"step %d, training accuracy %g\"%(i, train_accuracy))\n",
    "  train_step.run(feed_dict={x: batch_xs, y_: batch_ys, keep_prob: 0.5})\n",
    "\n",
    "print(\"test accuracy %g\"%accuracy.eval(feed_dict={\n",
    "    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../../MyModel/mnist_conv/-20000'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### 保存模型\n",
    "saver = tf.train.Saver()\n",
    "saver.save(sess, \"../../MyModel/mnist_conv/\", global_step=20000)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}