{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "07_PyTorch",
"version": "0.3.2",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"metadata": {
"id": "bOChJSNXtC9g",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# PyTorch"
]
},
{
"metadata": {
"id": "OLIxEDq6VhvZ",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/logo.png\")
\n",
"\n",
"在这节课中，我们将学习PyTorch，它是一个用于构建动态神经网络的学习库。我们将在本课程中了解其基础知识，以及如何创建和使用张量（Tensor），而我们将在下一课中使用它制作模型。\n",
"\n",
"![](\"https://raw.githubusercontent.com/LisonEvf/practicalAI-cn/master/images/pytorch.png\")
"
]
},
{
"metadata": {
"id": "VoMq0eFRvugb",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Tensor 基础知识"
]
},
{
"metadata": {
"id": "0-dXQiLlTIgz",
"colab_type": "code",
"outputId": "699f711d-a3ff-4888-b89c-4f7ce8455228",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 138
}
},
"cell_type": "code",
"source": [
"# 安装PyTorch库\n",
"!pip3 install torch"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting torch\n",
"\u001b[?25l Downloading https://files.pythonhosted.org/packages/7e/60/66415660aa46b23b5e1b72bc762e816736ce8d7260213e22365af51e8f9c/torch-1.0.0-cp36-cp36m-manylinux1_x86_64.whl (591.8MB)\n",
"\u001b[K 100% |████████████████████████████████| 591.8MB 28kB/s \n",
"tcmalloc: large alloc 1073750016 bytes == 0x61f04000 @ 0x7f61a90372a4 0x591a07 0x5b5d56 0x502e9a 0x506859 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x502209 0x502f3d 0x506859 0x504c28 0x502540 0x502f3d 0x507641 0x504c28 0x502540 0x502f3d 0x507641\n",
"\u001b[?25hInstalling collected packages: torch\n",
"Successfully installed torch-1.0.0\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "t6z1cCPTZSwo",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"> 注意：上述安装PyTorch的方式与官网并不一致，因为这里是基于Colab云端安装的，而大家一般是使用pip或者conda来安装PyTorch。具体安装教程参考：https://pytorch.org/"
]
},
{
"metadata": {
"id": "rX7Vs1JxL9wX",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"import numpy as np\n",
"import torch"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "Nv0xryLkKujV",
"colab_type": "code",
"outputId": "8584294d-7022-4194-d0ed-bc152d950efc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 118
}
},
"cell_type": "code",
"source": [
"# 创建一个张量\n",
"x = torch.Tensor(3, 4)\n",
"print(\"Type: {}\".format(x.type()))\n",
"print(\"Size: {}\".format(x.shape))\n",
"print(\"Values: \\n{}\".format(x))"
],
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": [
"Type: torch.FloatTensor\n",
"Size: torch.Size([3, 4])\n",
"Values: \n",
"tensor([[8.1343e-37, 0.0000e+00, 2.8026e-44, 0.0000e+00],\n",
" [ nan, 0.0000e+00, 1.3733e-14, 4.7429e+30],\n",
" [1.9431e-19, 4.7429e+30, 5.0938e-14, 0.0000e+00]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "vnyzY4PHL7c5",
"colab_type": "code",
"outputId": "551e96fb-edab-4a29-f046-2bd17b05df78",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
}
},
"cell_type": "code",
"source": [
"# 创建一个随机张量\n",
"x = torch.randn(2, 3) # torch.randn对应于正态分布，而rand(2,3)对应于均匀分布\n",
"print (x)"
],
"execution_count": 4,
"outputs": [
{
"output_type": "stream",
"text": [
"tensor([[-0.2826, -2.0357, -1.6480],\n",
" [ 0.7468, -1.1572, 0.4160]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "DVwGNeKxMXI8",
"colab_type": "code",
"outputId": "447bec04-dadc-460d-89fc-cc2d06bac5b1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"cell_type": "code",
"source": [
"# 0和1张量\n",
"x = torch.zeros(2, 3)\n",
"print (x)\n",
"x = torch.ones(2, 3)\n",
"print (x)"
],
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"text": [
"tensor([[0., 0., 0.],\n",
" [0., 0., 0.]])\n",
"tensor([[1., 1., 1.],\n",
" [1., 1., 1.]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "BPjHnDmCMXLm",
"colab_type": "code",
"outputId": "e0a0a80e-6c6b-4e07-d9f0-8a83a7283a9f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"cell_type": "code",
"source": [
"# 列表（List） → 张量\n",
"x = torch.Tensor([[1, 2, 3],[4, 5, 6]])\n",
"print(\"Size: {}\".format(x.shape)) \n",
"print(\"Values: \\n{}\".format(x))"
],
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 3])\n",
"Values: \n",
"tensor([[1., 2., 3.],\n",
" [4., 5., 6.]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "mG4-CHkgMXOE",
"colab_type": "code",
"outputId": "4258a024-6f69-42a2-c42a-8747fac8458d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"cell_type": "code",
"source": [
"# NumPy 数组 → 张量\n",
"x = torch.from_numpy(np.random.rand(2, 3))\n",
"print(\"Size: {}\".format(x.shape)) \n",
"print(\"Values: \\n{}\".format(x))"
],
"execution_count": 7,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 3])\n",
"Values: \n",
"tensor([[0.0632, 0.4443, 0.9658],\n",
" [0.4391, 0.9068, 0.6760]], dtype=torch.float64)\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "L8X2-5cqMXRA",
"colab_type": "code",
"outputId": "78351b75-3baa-4aef-83e0-a04bd58213b4",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
}
},
"cell_type": "code",
"source": [
"# 改变张量类型（张量默认为float类型）\n",
"x = torch.Tensor(3, 4)\n",
"print(\"Type: {}\".format(x.type()))\n",
"x = x.long()\n",
"print(\"Type: {}\".format(x.type()))"
],
"execution_count": 8,
"outputs": [
{
"output_type": "stream",
"text": [
"Type: torch.FloatTensor\n",
"Type: torch.LongTensor\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "S2BRPaMvPbe3",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Tensor 运算"
]
},
{
"metadata": {
"id": "Xrn8I76TMXT1",
"colab_type": "code",
"outputId": "009a5444-3b28-4f07-d20e-c0e67111c6ee",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"cell_type": "code",
"source": [
"# 加法\n",
"x = torch.randn(2, 3)\n",
"y = torch.randn(2, 3)\n",
"z = x + y\n",
"print(\"Size: {}\".format(z.shape)) \n",
"print(\"Values: \\n{}\".format(z))"
],
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 3])\n",
"Values: \n",
"tensor([[ 0.7976, 0.3528, 2.3146],\n",
" [-1.6895, 1.9033, -1.0576]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "157fC9WsMXWf",
"colab_type": "code",
"outputId": "ad90d3b9-fbcc-4028-9eba-9371869f710c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"cell_type": "code",
"source": [
"# 向量点积\n",
"x = torch.randn(2, 3)\n",
"y = torch.randn(3, 2)\n",
"z = torch.mm(x, y)\n",
"print(\"Size: {}\".format(z.shape)) \n",
"print(\"Values: \\n{}\".format(z))"
],
"execution_count": 10,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 2])\n",
"Values: \n",
"tensor([[ 0.3005, -2.8694],\n",
" [ 1.5858, -3.1363]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "G6316lAmMXZG",
"colab_type": "code",
"outputId": "d79058de-5148-45d8-bea4-dbe0fe0335e3",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 169
}
},
"cell_type": "code",
"source": [
"# 转置\n",
"x = torch.randn(2, 3)\n",
"print(\"Size: {}\".format(x.shape)) \n",
"print(\"Values: \\n{}\".format(x))\n",
"y = torch.t(x)\n",
"print(\"Size: {}\".format(y.shape)) \n",
"print(\"Values: \\n{}\".format(y))"
],
"execution_count": 11,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 3])\n",
"Values: \n",
"tensor([[ 0.2164, 0.5022, 0.8684],\n",
" [-0.7384, 0.9556, 1.0076]])\n",
"Size: torch.Size([3, 2])\n",
"Values: \n",
"tensor([[ 0.2164, -0.7384],\n",
" [ 0.5022, 0.9556],\n",
" [ 0.8684, 1.0076]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "FCgDCOCjMXcF",
"colab_type": "code",
"outputId": "be2708da-4378-4e4f-e3fa-dee9b99ee764",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 101
}
},
"cell_type": "code",
"source": [
"# Reshape\n",
"z = x.view(3, 2)\n",
"print(\"Size: {}\".format(z.shape)) \n",
"print(\"Values: \\n{}\".format(z))"
],
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([3, 2])\n",
"Values: \n",
"tensor([[ 0.2164, 0.5022],\n",
" [ 0.8684, -0.7384],\n",
" [ 0.9556, 1.0076]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "T3-6nGgvECH9",
"colab_type": "code",
"outputId": "a2bfc523-8188-4575-89ee-646f168c92ec",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 557
}
},
"cell_type": "code",
"source": [
"# reshaping的危险（意外后果）\n",
"x = torch.tensor([\n",
" [[1,1,1,1], [2,2,2,2], [3,3,3,3]],\n",
" [[10,10,10,10], [20,20,20,20], [30,30,30,30]]\n",
"])\n",
"print(\"Size: {}\".format(x.shape)) \n",
"print(\"Values: \\n{}\\n\".format(x))\n",
"a = x.view(x.size(1), -1)\n",
"print(\"Size: {}\".format(a.shape)) \n",
"print(\"Values: \\n{}\\n\".format(a))\n",
"b = x.transpose(0,1).contiguous()\n",
"print(\"Size: {}\".format(b.shape)) \n",
"print(\"Values: \\n{}\\n\".format(b))\n",
"c = b.view(b.size(0), -1)\n",
"print(\"Size: {}\".format(c.shape)) \n",
"print(\"Values: \\n{}\".format(c))"
],
"execution_count": 14,
"outputs": [
{
"output_type": "stream",
"text": [
"Size: torch.Size([2, 3, 4])\n",
"Values: \n",
"tensor([[[ 1, 1, 1, 1],\n",
" [ 2, 2, 2, 2],\n",
" [ 3, 3, 3, 3]],\n",
"\n",
" [[10, 10, 10, 10],\n",
" [20, 20, 20, 20],\n",
" [30, 30, 30, 30]]])\n",
"\n",
"Size: torch.Size([3, 8])\n",
"Values: \n",
"tensor([[ 1, 1, 1, 1, 2, 2, 2, 2],\n",
" [ 3, 3, 3, 3, 10, 10, 10, 10],\n",
" [20, 20, 20, 20, 30, 30, 30, 30]])\n",
"\n",
"Size: torch.Size([3, 2, 4])\n",
"Values: \n",
"tensor([[[ 1, 1, 1, 1],\n",
" [10, 10, 10, 10]],\n",
"\n",
" [[ 2, 2, 2, 2],\n",
" [20, 20, 20, 20]],\n",
"\n",
" [[ 3, 3, 3, 3],\n",
" [30, 30, 30, 30]]])\n",
"\n",
"Size: torch.Size([3, 8])\n",
"Values: \n",
"tensor([[ 1, 1, 1, 1, 10, 10, 10, 10],\n",
" [ 2, 2, 2, 2, 20, 20, 20, 20],\n",
" [ 3, 3, 3, 3, 30, 30, 30, 30]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "hRtG5LShMXew",
"colab_type": "code",
"outputId": "60350550-09c2-4894-e652-e51072f8ef5f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 135
}
},
"cell_type": "code",
"source": [
"# 维度操作\n",
"x = torch.randn(2, 3)\n",
"print(\"Values: \\n{}\".format(x))\n",
"y = torch.sum(x, dim=0) # 为每列添加各行叠加的值\n",
"print(\"Values: \\n{}\".format(y))\n",
"z = torch.sum(x, dim=1) # 为每行添加各列叠加的值\n",
"print(\"Values: \\n{}\".format(z))"
],
"execution_count": 16,
"outputs": [
{
"output_type": "stream",
"text": [
"Values: \n",
"tensor([[-2.2913, 0.1027, -0.3427],\n",
" [ 0.7189, 0.4223, -0.0364]])\n",
"Values: \n",
"tensor([-1.5724, 0.5250, -0.3791])\n",
"Values: \n",
"tensor([-2.5313, 1.1048])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "zI0ZV45PrYmw",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 索引，切片和级联"
]
},
{
"metadata": {
"id": "iM3UFrs0MXhL",
"colab_type": "code",
"outputId": "11a9c76d-ea38-4bf6-e6c3-fb4ccc70feec",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 152
}
},
"cell_type": "code",
"source": [
"x = torch.randn(3, 4)\n",
"print(\"x: \\n{}\".format(x))\n",
"print (\"x[:1]: \\n{}\".format(x[:1]))\n",
"print (\"x[:1, 1:3]: \\n{}\".format(x[:1, 1:3]))"
],
"execution_count": 17,
"outputs": [
{
"output_type": "stream",
"text": [
"x: \n",
"tensor([[ 0.5660, 1.3207, 0.6355, 0.4770],\n",
" [-1.5738, -1.6227, 0.1413, -1.8778],\n",
" [-0.0666, -0.2320, -0.3943, -0.4921]])\n",
"x[:1]: \n",
"tensor([[0.5660, 1.3207, 0.6355, 0.4770]])\n",
"x[:1, 1:3]: \n",
"tensor([[1.3207, 0.6355]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "_tbpwGxcMXj0",
"colab_type": "code",
"outputId": "ebe7ce15-842d-4bff-fba6-e7d8165c91b9",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 152
}
},
"cell_type": "code",
"source": [
"# 选择维度索引\n",
"x = torch.randn(2, 3)\n",
"print(\"Values: \\n{}\".format(x))\n",
"col_indices = torch.LongTensor([0, 2])\n",
"chosen = torch.index_select(x, dim=1, index=col_indices) # 第0和第2列的值\n",
"print(\"Values: \\n{}\".format(chosen)) \n",
"row_indices = torch.LongTensor([0, 1])\n",
"chosen = x[row_indices, col_indices] # 来自（0,0）和（2,1）的值\n",
"print(\"Values: \\n{}\".format(chosen)) "
],
"execution_count": 18,
"outputs": [
{
"output_type": "stream",
"text": [
"Values: \n",
"tensor([[ 1.7797, -1.2580, -0.6378],\n",
" [-1.3466, -0.2311, 0.8040]])\n",
"Values: \n",
"tensor([[ 1.7797, -0.6378],\n",
" [-1.3466, 0.8040]])\n",
"Values: \n",
"tensor([1.7797, 0.8040])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "tMeqSQtuMXmH",
"colab_type": "code",
"outputId": "c5560c19-806d-4678-8339-7ca341174d2c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 152
}
},
"cell_type": "code",
"source": [
"# 级联\n",
"x = torch.randn(2, 3)\n",
"print(\"Values: \\n{}\".format(x))\n",
"y = torch.cat([x, x], dim=0) # 按行堆叠（dim = 1按列堆叠）\n",
"print(\"Values: \\n{}\".format(y))"
],
"execution_count": 20,
"outputs": [
{
"output_type": "stream",
"text": [
"Values: \n",
"tensor([[-0.4772, 0.7716, 2.2474],\n",
" [ 1.3227, -0.4346, 1.1521]])\n",
"Values: \n",
"tensor([[-0.4772, 0.7716, 2.2474],\n",
" [ 1.3227, -0.4346, 1.1521],\n",
" [-0.4772, 0.7716, 2.2474],\n",
" [ 1.3227, -0.4346, 1.1521]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "JqiDuIC-ByvO",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# 梯度"
]
},
{
"metadata": {
"id": "qxpGB7-VL7fs",
"colab_type": "code",
"outputId": "b33ec033-9f90-4911-80c1-730c3a775128",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 152
}
},
"cell_type": "code",
"source": [
"# 带有gradient bookkeeping的张量\n",
"x = torch.rand(3, 4, requires_grad=True)\n",
"y = 3*x + 2\n",
"z = y.mean()\n",
"z.backward() # z是标量\n",
"print(\"Values: \\n{}\".format(x))\n",
"print(\"x.grad: \\n\", x.grad)"
],
"execution_count": 22,
"outputs": [
{
"output_type": "stream",
"text": [
"Values: \n",
"tensor([[0.9651, 0.9546, 0.2183, 0.5656],\n",
" [0.2327, 0.4397, 0.7654, 0.8600],\n",
" [0.5148, 0.8122, 0.1790, 0.7555]], requires_grad=True)\n",
"x.grad: \n",
" tensor([[0.2500, 0.2500, 0.2500, 0.2500],\n",
" [0.2500, 0.2500, 0.2500, 0.2500],\n",
" [0.2500, 0.2500, 0.2500, 0.2500]])\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "uf7htaAMDcRV",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"* $ y = 3x + 2 $\n",
"* $ z = \\sum{y}/N $\n",
"* $ \\frac{\\partial(z)}{\\partial(x)} = \\frac{\\partial(z)}{\\partial(y)} \\frac{\\partial(z)}{\\partial(x)} = \\frac{1}{N} * 3 = \\frac{1}{12} * 3 = 0.25 $"
]
},
{
"metadata": {
"id": "VQtpZh1YD-kz",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# CUDA 张量"
]
},
{
"metadata": {
"id": "E_C3en05L7iT",
"colab_type": "code",
"outputId": "bbe1e507-598c-4089-ee77-2b4cce9d64bc",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"# CUDA可用吗？\n",
"print (torch.cuda.is_available())"
],
"execution_count": 23,
"outputs": [
{
"output_type": "stream",
"text": [
"True\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "za47KWEJ6en2",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"如果上面的代码返回False，那么请转到菜单栏上的`Runtime`→`Change runtime type`并在`Hardware accelerator`下选择`GPU`。"
]
},
{
"metadata": {
"id": "BY2DdN3j6ZxO",
"colab_type": "code",
"outputId": "e637aa7f-d685-4d9a-f453-7f0718d1b009",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"# 创建一个CPU版的0张量\n",
"x = torch.Tensor(3, 4).to(\"cpu\")\n",
"print(\"Type: {}\".format(x.type()))"
],
"execution_count": 25,
"outputs": [
{
"output_type": "stream",
"text": [
"Type: torch.FloatTensor\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "EcmdTggzEFPi",
"colab_type": "code",
"outputId": "20fcc79d-6b4e-4c75-8acf-c5e4c2a76786",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"# 创建一个CUDA版的0张量\n",
"x = torch.Tensor(3, 4).to(\"cuda\")\n",
"print(\"Type: {}\".format(x.type()))"
],
"execution_count": 27,
"outputs": [
{
"output_type": "stream",
"text": [
"Type: torch.cuda.FloatTensor\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "iD-JUbH4hgt7",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
""
],
"execution_count": 0,
"outputs": []
}
]
}