![](\"images/NST_LOSS.png\")
\n",
"\n",
"**Exercise:** Compute the \"content cost\" using TensorFlow. \n",
"\n",
"**Instructions**: The 3 steps to implement this function are:\n",
"1. Retrieve dimensions from a_G: \n",
" - To retrieve dimensions from a tensor X, use: `X.get_shape().as_list()`\n",
"2. Unroll a_C and a_G as explained in the picture above\n",
" - If you are stuck, take a look at [Hint1](https://www.tensorflow.org/versions/r1.3/api_docs/python/tf/transpose) and [Hint2](https://www.tensorflow.org/versions/r1.2/api_docs/python/tf/reshape).\n",
"3. Compute the content cost:\n",
" - If you are stuck, take a look at [Hint3](https://www.tensorflow.org/api_docs/python/tf/reduce_sum), [Hint4](https://www.tensorflow.org/api_docs/python/tf/square) and [Hint5](https://www.tensorflow.org/api_docs/python/tf/subtract)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# GRADED FUNCTION: compute_content_cost\n",
"\n",
"def compute_content_cost(a_C, a_G):\n",
" \"\"\"\n",
" Computes the content cost\n",
" \n",
" Arguments:\n",
" a_C -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing content of the image C \n",
" a_G -- tensor of dimension (1, n_H, n_W, n_C), hidden layer activations representing content of the image G\n",
" \n",
" Returns: \n",
" J_content -- scalar that you compute using equation 1 above.\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ###\n",
" # Retrieve dimensions from a_G (≈1 line)\n",
" m, n_H, n_W, n_C = a_G.get_shape().as_list()\n",
" \n",
" # Reshape a_C and a_G (≈2 lines)\n",
" a_C_unrolled = tf.transpose(a_C)\n",
" a_G_unrolled = tf.transpose(a_G)\n",
" \n",
" # compute the cost with tensorflow (≈1 line)\n",
" J_content = (1/ (4* n_H * n_W * n_C)) * tf.reduce_sum(tf.pow((a_G_unrolled - a_C_unrolled), 2))\n",
" ### END CODE HERE ###\n",
" \n",
" return J_content"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"J_content = 6.76559\n"
]
}
],
"source": [
"tf.reset_default_graph()\n",
"\n",
"with tf.Session() as test:\n",
" tf.set_random_seed(1)\n",
" a_C = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
" a_G = tf.random_normal([1, 4, 4, 3], mean=1, stddev=4)\n",
" J_content = compute_content_cost(a_C, a_G)\n",
" print(\"J_content = \" + str(J_content.eval()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output**:\n",
"\n",
"| \n", " **J_content**\n", " | \n", "\n", " 6.76559\n", " | \n", "
![](\"images/NST_GM.png\")
\n",
"\n",
"The result is a matrix of dimension $(n_C,n_C)$ where $n_C$ is the number of filters. The value $G_{ij}$ measures how similar the activations of filter $i$ are to the activations of filter $j$. \n",
"\n",
"One important part of the gram matrix is that the diagonal elements such as $G_{ii}$ also measures how active filter $i$ is. For example, suppose filter $i$ is detecting vertical textures in the image. Then $G_{ii}$ measures how common vertical textures are in the image as a whole: If $G_{ii}$ is large, this means that the image has a lot of vertical texture. \n",
"\n",
"By capturing the prevalence of different types of features ($G_{ii}$), as well as how much different features occur together ($G_{ij}$), the Style matrix $G$ measures the style of an image. \n",
"\n",
"**Exercise**:\n",
"Using TensorFlow, implement a function that computes the Gram matrix of a matrix A. The formula is: The gram matrix of A is $G_A = AA^T$. If you are stuck, take a look at [Hint 1](https://www.tensorflow.org/api_docs/python/tf/matmul) and [Hint 2](https://www.tensorflow.org/api_docs/python/tf/transpose)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# GRADED FUNCTION: gram_matrix\n",
"\n",
"def gram_matrix(A):\n",
" \"\"\"\n",
" Argument:\n",
" A -- matrix of shape (n_C, n_H*n_W)\n",
" \n",
" Returns:\n",
" GA -- Gram matrix of A, of shape (n_C, n_C)\n",
" \"\"\"\n",
" \n",
" ### START CODE HERE ### (≈1 line)\n",
" GA = tf.matmul(A, tf.transpose(A))\n",
" ### END CODE HERE ###\n",
" \n",
" return GA"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GA = [[ 6.42230511 -4.42912197 -2.09668207]\n",
" [ -4.42912197 19.46583748 19.56387138]\n",
" [ -2.09668207 19.56387138 20.6864624 ]]\n"
]
}
],
"source": [
"tf.reset_default_graph()\n",
"\n",
"with tf.Session() as test:\n",
" tf.set_random_seed(1)\n",
" A = tf.random_normal([3, 2*1], mean=1, stddev=4)\n",
" GA = gram_matrix(A)\n",
" \n",
" print(\"GA = \" + str(GA.eval()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Expected Output**:\n",
"\n",
"| \n", " **GA**\n", " | \n", "\n",
" [[ 6.42230511 -4.42912197 -2.09668207] \n", " [ -4.42912197 19.46583748 19.56387138] \n", " [ -2.09668207 19.56387138 20.6864624 ]]\n", " | \n",
"
| \n", " **J_style_layer**\n", " | \n", "\n", " 9.19028\n", " | \n", "
| \n", " **J**\n", " | \n", "\n", " 35.34667875478276\n", " | \n", "
| \n", " **Iteration 0 : **\n", " | \n", "\n",
" total cost = 5.05035e+09 \n", " content cost = 7877.67 \n", " style cost = 1.26257e+08\n", " | \n",
"
![](\"images/louvre_generated.png\")
\n",
"\n",
"We didn't want you to wait too long to see an initial result, and so had set the hyperparameters accordingly. To get the best looking results, running the optimization algorithm longer (and perhaps with a smaller learning rate) might work better. After completing and submitting this assignment, we encourage you to come back and play more with this notebook, and see if you can generate even better looking images. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are few other examples:\n",
"\n",
"- The beautiful ruins of the ancient city of Persepolis (Iran) with the style of Van Gogh (The Starry Night)\n",
"![](\"images/perspolis_vangogh.png\")
\n",
"\n",
"- The tomb of Cyrus the great in Pasargadae with the style of a Ceramic Kashi from Ispahan.\n",
"![](\"images/pasargad_kashi.png\")
\n",
"\n",
"- A scientific study of a turbulent fluid with the style of a abstract blue fluid painting.\n",
"![](\"images/circle_abstract.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5 - Test with your own image (Optional/Ungraded)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, you can also rerun the algorithm on your own images! \n",
"\n",
"To do so, go back to part 4 and change the content image and style image with your own pictures. In detail, here's what you should do:\n",
"\n",
"1. Click on \"File -> Open\" in the upper tab of the notebook\n",
"2. Go to \"/images\" and upload your images (requirement: (WIDTH = 300, HEIGHT = 225)), rename them \"my_content.png\" and \"my_style.png\" for example.\n",
"3. Change the code in part (3.4) from :\n",
"```python\n",
"content_image = scipy.misc.imread(\"images/louvre.jpg\")\n",
"style_image = scipy.misc.imread(\"images/claude-monet.jpg\")\n",
"```\n",
"to:\n",
"```python\n",
"content_image = scipy.misc.imread(\"images/my_content.jpg\")\n",
"style_image = scipy.misc.imread(\"images/my_style.jpg\")\n",
"```\n",
"4. Rerun the cells (you may need to restart the Kernel in the upper tab of the notebook).\n",
"\n",
"You can also tune your hyperparameters: \n",
"- Which layers are responsible for representing the style? STYLE_LAYERS\n",
"- How many iterations do you want to run the algorithm? num_iterations\n",
"- What is the relative weighting between content and style? alpha/beta"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6 - Conclusion\n",
"\n",
"Great job on completing this assignment! You are now able to use Neural Style Transfer to generate artistic images. This is also your first time building a model in which the optimization algorithm updates the pixel values rather than the neural network's parameters. Deep learning has many different types of models and this is only one of them! \n",
"\n",
"\n",
"What you should remember:\n",
"- Neural Style Transfer is an algorithm that given a content image C and a style image S can generate an artistic image\n",
"- It uses representations (hidden layer activations) based on a pretrained ConvNet. \n",
"- The content cost function is computed using one hidden layer's activations.\n",
"- The style cost function for one layer is computed using the Gram matrix of that layer's activations. The overall style cost function is obtained using several hidden layers.\n",
"- Optimizing the total cost function results in synthesizing new images. \n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This was the final programming exercise of this course. Congratulations--you've finished all the programming exercises of this course on Convolutional Networks! We hope to also see you in Course 5, on Sequence models! \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### References:\n",
"\n",
"The Neural Style Transfer algorithm was due to Gatys et al. (2015). Harish Narayanan and Github user \"log0\" also have highly readable write-ups from which we drew inspiration. The pre-trained network used in this implementation is a VGG network, which is due to Simonyan and Zisserman (2015). Pre-trained weights were from the work of the MathConvNet team. \n",
"\n",
"- Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, (2015). A Neural Algorithm of Artistic Style (https://arxiv.org/abs/1508.06576) \n",
"- Harish Narayanan, Convolutional neural networks for artistic style transfer. https://harishnarayanan.org/writing/artistic-style-transfer/\n",
"- Log0, TensorFlow Implementation of \"A Neural Algorithm of Artistic Style\". http://www.chioka.in/tensorflow-implementation-neural-algorithm-of-artistic-style\n",
"- Karen Simonyan and Andrew Zisserman (2015). Very deep convolutional networks for large-scale image recognition (https://arxiv.org/pdf/1409.1556.pdf)\n",
"- MatConvNet. http://www.vlfeat.org/matconvnet/pretrained/\n"
]
}
],
"metadata": {
"coursera": {
"course_slug": "convolutional-neural-networks",
"graded_item_id": "owWbQ",
"launcher_item_id": "lEthw"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.0"
}
},
"nbformat": 4,
"nbformat_minor": 1
}