\u001b[1;34m\u001b[0m\n\u001b[0;32m 1\u001b[0m \u001b[1;31m# Example 2 - breaking\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mtorch\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmm\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mB\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mA\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[1;31mRuntimeError\u001b[0m: size mismatch, m1: [3 x 4], m2: [2 x 3] at C:\\Users\\builder\\AppData\\Local\\Temp\\pip-req-build-9msmi1s9\\aten\\src\\TH/generic/THTensorMath.cpp:197"
]
}
],
"source": [
"# Example 2 - breaking\n",
"torch.mm(B, A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example 3.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
If your keyboard has the character '@', then you can write
\n",
"
Transpose
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
When we transpose a matrix we transform rows to columns, and columns to rows.
\n",
"
Example 1.
"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[-0.7157, 1.2350],\n",
" [ 0.0141, -1.2724],\n",
" [ 0.1572, 0.7450]])"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Example 1 \n",
"torch.t(A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example 2.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
Again, you should thoughtful, because $(AB)^T\\neq A^T B^T$, indeed
\n",
"
\n",
"
The correct result is $(AB)^T=B^T A^T$.
\n",
"
Example 3.
"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([[0., 0., 0.],\n",
" [0., 0., 0.],\n",
" [0., 0., 0.]])"
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Example 3\n",
"torch.t(C @ D) - (torch.t(D) @ torch.t(C))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gradient
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
In machine learning we should minimize different type \n",
" of error function.
\n",
"
\n",
" The derivative of a multivariable scalar valued function is a matrix, the so-called \n",
" \n",
" Jacobi matrix.\n",
"
\n",
"
\n",
"
If you want to calculate the derivative, you should give this option \n",
" in the definition of the tensor, using requires_grad=True
\n",
"
In Example 1 we use the vector norm (in fact, the Frobenius norm, which gives the \n",
" Eucledian vector norm in case of vectors).
\n",
"
Example 1.
"
]
},
{
"cell_type": "code",
"execution_count": 122,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A: \n",
" tensor([[-0.3062, -0.5272, 1.2869],\n",
" [-1.8781, -1.9212, 0.1393],\n",
" [-1.0218, 1.2361, -0.1057]], requires_grad=True)\n",
"x: \n",
" tensor([[ 0.5530],\n",
" [-0.6164],\n",
" [ 0.8250]], requires_grad=True)\n",
"b: \n",
" tensor([[-1.0424],\n",
" [-1.3509],\n",
" [ 0.8377]], requires_grad=True)\n",
"y: \n",
" tensor(3.5743, grad_fn=)\n"
]
}
],
"source": [
"# Example 1\n",
"# Create tensors.\n",
"A = torch.randn(3, 3, requires_grad=True)\n",
"x = torch.randn(3, 1, requires_grad=True)\n",
"b = torch.randn(3, 1, requires_grad=True)\n",
"y = torch.norm(A @ x - b, p='fro')\n",
"print(\"A: \\n\", A)\n",
"print(\"x: \\n\", x)\n",
"print(\"b: \\n\", b)\n",
"print(\"y: \\n\", y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
Now we calculate the derivatives $\\dfrac{\\partial y}{\\partial A}$, \n",
" $\\dfrac{\\partial y}{\\partial x}$, $\\dfrac{\\partial y}{\\partial b}$.
\n",
"
To compute the derivatives, we call the .backward \n",
" method on our result $y$.
\n",
"
Example 2.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
Instead of Frobenius norm we can choose any $1\\leq p<\\infty$ norm.\n",
" In the next example we choose $p=1$.
\n",
"
Example 3.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
At this moment the infinity norm $p=\\infty$ does not work.
\n",
"
Eigenvalues, eigenvectors
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
When we work on linear operators (matrices) many times is a must to \n",
" determine their eigenvalues and eigenvectors. \n",
" To do this, we use torch.eig
\n",
"
Example 1.
"
]
},
{
"cell_type": "code",
"execution_count": 111,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"eigenvalue: tensor([1.1316, 0.0000], grad_fn=)\n",
"eigvector: tensor([-0.9053, 0.4609, 0.0952], grad_fn=)\n",
"\n",
"\n",
"eigenvalue: tensor([-0.2300, 0.0000], grad_fn=)\n",
"eigvector: tensor([-0.1106, -0.8415, -0.8702], grad_fn=)\n",
"\n",
"\n",
"eigenvalue: tensor([-0.9825, 0.0000], grad_fn=)\n",
"eigvector: tensor([-0.4100, 0.2818, -0.4834], grad_fn=)\n",
"\n",
"\n"
]
}
],
"source": [
"# Example 1 \n",
"(eigvalues, eigvectors) = torch.eig(A, eigenvectors=True)\n",
"for i in range(3):\n",
" print('eigenvalue: ', eigvalues[i])\n",
" print('eigvector: ', eigvectors[i])\n",
" print('\\n')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example 2.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
\n",
" Here is an example when the tensor has one eigenvalue with three different \n",
" eigenvectors.\n",
"
\n",
"
Example 3.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
Only square matrices can have eigenvalues.
\n",
"
Least square norm
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
Many times we should minimize the quantity $\\Vert AX-B\\Vert_2$, \n",
" where $A,B$ are given matrices, and $X$ is the one we want to calculate. \n",
" We use the torch.lstsq function.\n",
"
\n",
"
Example 1.
"
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"A: \n",
" tensor([[-0.2769, 1.0674, 0.6662, -1.5984],\n",
" [ 0.8249, 1.3140, -0.0192, -0.3846],\n",
" [ 2.4818, 0.5684, -1.7736, 0.6815]])\n",
"B: \n",
" tensor([[ 0.4494],\n",
" [-0.9212],\n",
" [-1.1869]])\n",
"\n",
"\n",
"The solution of the least square problem is: \n",
"\n",
" tensor([[-0.2753],\n",
" [-0.7950],\n",
" [-0.3150],\n",
" [-0.8957]])\n",
"\n",
"\n",
"The error is: tensor(1.8849e-07)\n"
]
}
],
"source": [
"# Example 1\n",
"A = torch.randn(3, 4)\n",
"B = torch.randn(3, 1) # a vector\n",
"print(\"A: \\n\", A)\n",
"print(\"B: \\n\", B)\n",
"print(\"\\n\")\n",
"\n",
"X, _ = torch.lstsq(B, A)\n",
"\n",
"error = torch.norm(A @ X - B, p=2)\n",
"print(\"The solution of the least square problem is: \\n\\n\", X)\n",
"print(\"\\n\")\n",
"print(\"The error is: \", error)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Example 2.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
\n",
" Now consider the previous example, but with $p=1$ norm.\n",
"
\n",
"
\n",
"
The error is greater.
\n",
"
Example 3.
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
If we give matrices of wrong dimensions, we obtain \n",
" a 'size mismatch' error message.
\n",
"
Summary
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"
We considered some basic linear algebraic problems, that PyTorch can \n",
" solve very effectively, and we will use them many times later.\n",
"
Reference Links
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Official documentation for torch.Tensor \n",
" \n",
" https://pytorch.org/docs/stable/tensors.html
\n",
"Official documentation for torch.Autograd \n",
" \n",
" https://pytorch.org/docs/stable/autograd.html
"
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {
"collapsed": true
},
"outputs": [
{
"data": {
"text/html": [
"\n"
],
"text/plain": [
""
]
},
"execution_count": 175,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.core.display import HTML\n",
"import urllib.request\n",
"response = urllib.request.urlopen('https://raw.githubusercontent.com/wesszabo/Pytorch-basics/master/CSS/pytorch_basics.css')\n",
"HTML(response.read().decode(\"utf-8\"))"
]
}
],
"metadata": {
"hide_input": false,
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": true,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 4
}