{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataFrames.jl @ 0.22.2\n",
    "\n",
    "最近Juliaでデータフレームを操作するパッケージ[DataFrames.jl](https://github.com/JuliaData/DataFrames.jl/releases/tag/v0.22.0)のバージョンが少しbump upしました。ただし，この執筆時点での最新バージョンは0.22.2です。\n",
    "\n",
    "そもそもデータフレームとは，基本的に二次元配列の構造で，行列と同様に行と列を持ちます。\n",
    "\n",
    "多言語だとRのdplyrやPythonのpandasなどがありますが，JuliaではDataFrames.jlがデータフレームの操作を実行するためのパッケージとなります。連携するパッケージとしては`DataFramesMeta.jl`や`Query`などがありますが，それらを使わなくても，たいていのことはできる印象です。\n",
    "\n",
    "ここ最近0.21系から0.22にバージョンアップしました。\n",
    "\n",
    "Juliaの公式パッケージはGutHub上で管理されているので，GutHubで変更履歴や，おもな変更点を確認することができます。\n",
    "\n",
    "どうやら今回のおもな変更点はデータフレームの列操作に関する関数`select`/`transform`の改善や，データの持ち方を縦から横に変更する関数`unstack`に関する挙動のようです。\n",
    "\n",
    "これに加えて，いくつかの補助関数(`isapprox`, `empty`)が新たに追加されました。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# using Pkg; Pkg.add(name = \"DataFrames\", version = \"0.22.0\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "using DataFrames;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## 1. 基本操作\n",
    "\n",
    "### 1.1 データフレームをつくる\n",
    "\n",
    "`DataFrame(列名 = 変数)`という作り方がいちばんベーシックな作り方だと思います。その他にも，次のような書き方もできます。\n",
    "\n",
    "1. `DataFrame(行列)`\n",
    "2. `DataFrame(行列, [シンボルor文字列のベクトル])`\n",
    "3. `DataFrame(辞書)`\n",
    "4. `DataFrame(列名 => 要素)`\n",
    "\n",
    "４つ目の書き方はペア`Pair`を利用した書き方です。気をつけなければならないのは，辞書やペアを利用する場合はすべての要素の大きさが同じである必要があります。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 3 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td><td>0.277444</td></tr><tr><th>2</th><td>-0.567185</td><td>1.5837</td><td>-0.36308</td></tr><tr><th>3</th><td>-1.33798</td><td>0.694831</td><td>-0.375632</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & b & c\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 & 0.277444 \\\\\n",
       "\t2 & -0.567185 & 1.5837 & -0.36308 \\\\\n",
       "\t3 & -1.33798 & 0.694831 & -0.375632 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b         \u001b[0m\u001b[1m c         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼─────────────────────────────────\n",
       "   1 │  0.800516  -2.26194    0.277444\n",
       "   2 │ -0.567185   1.5837    -0.36308\n",
       "   3 │ -1.33798    0.694831  -0.375632"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mat = randn(3, 3);\n",
    "A = DataFrame(mat, [:a, :b, :c])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "DimensionMismatch(\"column :a has length 2 and column :b has length 3\")",
     "output_type": "error",
     "traceback": [
      "DimensionMismatch(\"column :a has length 2 and column :b has length 3\")",
      "",
      "Stacktrace:",
      " [1] DataFrame(::Array{Any,1}, ::DataFrames.Index; copycols::Bool) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:181",
      " [2] DataFrame(::Pair{Symbol,UnitRange{Int64}}, ::Vararg{Pair{Symbol,UnitRange{Int64}},N} where N; makeunique::Bool, copycols::Bool) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:218",
      " [3] DataFrame(::Pair{Symbol,UnitRange{Int64}}, ::Vararg{Pair{Symbol,UnitRange{Int64}},N} where N) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:216",
      " [4] top-level scope at In[4]:1",
      " [5] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091"
     ]
    }
   ],
   "source": [
    "DataFrame(:a => 1:2, :b => 2:4) # 長さが違うので，エラーになる。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 1.2 データフレームの要素を取り出す\n",
    "\n",
    "#### 取り出す\n",
    "\n",
    "作ったデータフレームの列要素のアクセスは，データフレームを`A`，取り出したい列名を`a`とすると，\n",
    "\n",
    "- `A.a`\n",
    "- `A.\"a\"`\n",
    "- `A[!, :A]`\n",
    "- `A[!, \"A\"]`\n",
    "- `getproperty(A, :a)`\n",
    "- `getproperty(A, \"a\")`\n",
    "\n",
    "で取り出すことができます。`[]`や`getproperty`を使用する場合，シンボルでも文字列でもどちらでもアクセス可能です（シンボルのほうがわずかに効率がいいらしい）。\n",
    "\n",
    "これらの方法で取り出された配列はコピーされません。コピーされないということは取り出した要素を変化させると，もとのデータフレームも変化するということです。コピーを取り出したい場合は`A[:, :a]`のように，`!`の代わりに`:`を使用します。\n",
    "\n",
    "複数の列にアクセスすることもできます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 2 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td></tr><tr><th>2</th><td>-0.567185</td><td>1.5837</td></tr><tr><th>3</th><td>-1.33798</td><td>0.694831</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 \\\\\n",
       "\t2 & -0.567185 & 1.5837 \\\\\n",
       "\t3 & -1.33798 & 0.694831 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼──────────────────────\n",
       "   1 │  0.800516  -2.26194\n",
       "   2 │ -0.567185   1.5837\n",
       "   3 │ -1.33798    0.694831"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[:, [:a, :b]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 正規表現を使って要素を指定する\n",
    "\n",
    "正規表現を使ってマッチする列を取り出すこともできます。正規表現を使う場合には`r\"\"`をつかって`Regex`を作り出します。\n",
    "\n",
    "正規表現による強力なマッチング機能が利用できるのは，DataFrames.jlの強みだと思います。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 2 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td></tr><tr><th>2</th><td>-0.567185</td><td>1.5837</td></tr><tr><th>3</th><td>-1.33798</td><td>0.694831</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 \\\\\n",
       "\t2 & -0.567185 & 1.5837 \\\\\n",
       "\t3 & -1.33798 & 0.694831 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼──────────────────────\n",
       "   1 │  0.800516  -2.26194\n",
       "   2 │ -0.567185   1.5837\n",
       "   3 │ -1.33798    0.694831"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[:, r\"[ab]\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "サポート関数を使っても，列方向に対する操作ができます。利用できるサポート関数は\n",
    "\n",
    "- `Not()`   選択したシンボルか`Regex`にマッチした列**以外**を取り出す。\n",
    "- `Cols()`  選択したシンボルか`Regex`にマッチした列**だけ**を取り出す。\n",
    "- `All()`   選んでくる条件は`Cols()`と同じ。\n",
    "- `Between(first, last)`    `first`~`last`間の列をすべて取り出す。\n",
    "\n",
    "です。\n",
    "\n",
    "`Cols`と`All()`は機能的には同一ですが，`Cols()`は何も列を選択しないことができるので，こちらのほうができることがひとつ多い印象。\n",
    "\n",
    "**並び替える**\n",
    "\n",
    "これらのサポート関数は，列方向に関する並び替えを行いたいときに役に立ちます。並び替えには`Regex`とシンボル，文字列が使えます。つまり，列選択のときと全く同じです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>c</th><th>b</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 3 columns</p><tr><th>1</th><td>0.800516</td><td>0.277444</td><td>-2.26194</td></tr><tr><th>2</th><td>-0.567185</td><td>-0.36308</td><td>1.5837</td></tr><tr><th>3</th><td>-1.33798</td><td>-0.375632</td><td>0.694831</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & c & b\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & 0.277444 & -2.26194 \\\\\n",
       "\t2 & -0.567185 & -0.36308 & 1.5837 \\\\\n",
       "\t3 & -1.33798 & -0.375632 & 0.694831 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m c         \u001b[0m\u001b[1m b         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼─────────────────────────────────\n",
       "   1 │  0.800516   0.277444  -2.26194\n",
       "   2 │ -0.567185  -0.36308    1.5837\n",
       "   3 │ -1.33798   -0.375632   0.694831"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[!, Cols(r\"a|c\", \"b\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 列を指定して，変更する\n",
    "\n",
    "列の要素を指定する方法は他にもあります。それが`select()`と`transform()`です。しかし，この２種類の関数は，ただ取り出すだけでなく，指定した要素に対して関数を作用させるための関数です。\n",
    "\n",
    "- `select()`, `select!()`   選択した列**だけ**を取り出す\n",
    "- `transform()`, `transform!()` 選択した列**以外も**取り出す。\n",
    "\n",
    "`!`がついている方はin-place関数であり，作用させた変数の内容そのものを破壊的に変更します。\n",
    "\n",
    "`transform`は選択列以外も取り出すので「なんのこっちゃ」となるかもしれませんが，この２つの関数は，ただ列を取り出すだけでなく，取り出した列に対して関数を作用させるための関数であるため，`transform`が重要になります。\n",
    "\n",
    "これらは`Regex`やシンボル，文字列と組み合わせて使うことができるだけでなく，先述したサポート関数と一緒に使うこともできます。\n",
    "\n",
    "**`select()`**\n",
    "\n",
    "マッチした列だけを取り出して，新しいデータフレームを作成します。\n",
    "\n",
    "さらに，単純に取り出すだけでなく，列名を変更したり，取り出した列に関数を作用させて新しい列にすることもできます。`select`はデフォルトでは自動的にコピーを作成する仕様になっていますが，`..., copycols = false`でコピーの作成をしないようにすることもできます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th></tr><tr><th></th><th>Float64</th></tr></thead><tbody><p>3 rows × 1 columns</p><tr><th>1</th><td>0.800516</td></tr><tr><th>2</th><td>-0.567185</td></tr><tr><th>3</th><td>-1.33798</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|c}\n",
       "\t& a\\\\\n",
       "\t\\hline\n",
       "\t& Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 \\\\\n",
       "\t2 & -0.567185 \\\\\n",
       "\t3 & -1.33798 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×1 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\n",
       "─────┼───────────\n",
       "   1 │  0.800516\n",
       "   2 │ -0.567185\n",
       "   3 │ -1.33798"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B = select(A, :a, copycols = true)\n",
    "C = select(A, :a, copycols = false)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "false"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B.a === A.a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "C.a === A.a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`select()`は単に列を取り出すだけでなく，選択した列に対して関数を適用し，それを新たな列として返すという操作もできます。\n",
    "\n",
    "新しい列を作るための書き方は，\n",
    "\n",
    "`select(A, :列名 => 関数 => :新しい列名)`\n",
    "\n",
    "です。関数には`()`でくくることで無名関数も使うことができます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>d</th></tr><tr><th></th><th>Float64</th></tr></thead><tbody><p>3 rows × 1 columns</p><tr><th>1</th><td>1.60103</td></tr><tr><th>2</th><td>-1.13437</td></tr><tr><th>3</th><td>-2.67596</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|c}\n",
       "\t& d\\\\\n",
       "\t\\hline\n",
       "\t& Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1.60103 \\\\\n",
       "\t2 & -1.13437 \\\\\n",
       "\t3 & -2.67596 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×1 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m d        \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64  \u001b[0m\n",
       "─────┼──────────\n",
       "   1 │  1.60103\n",
       "   2 │ -1.13437\n",
       "   3 │ -2.67596"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "select(A, :a => (x -> 2x) => :d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "関数は列の要素に対してまとめててきようされます。そのため行の各要素に関数を作用させたいときはbroadcastを利用します.\n",
    "\n",
    "例えば各要素を`round`したいときは，\n",
    "\n",
    "```\n",
    "select(A, :a => round => :d)\n",
    "```\n",
    "\n",
    "ではなく"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>d</th></tr><tr><th></th><th>Float64</th></tr></thead><tbody><p>3 rows × 1 columns</p><tr><th>1</th><td>1.0</td></tr><tr><th>2</th><td>-1.0</td></tr><tr><th>3</th><td>-1.0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|c}\n",
       "\t& d\\\\\n",
       "\t\\hline\n",
       "\t& Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1.0 \\\\\n",
       "\t2 & -1.0 \\\\\n",
       "\t3 & -1.0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×1 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m d       \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64 \u001b[0m\n",
       "─────┼─────────\n",
       "   1 │     1.0\n",
       "   2 │    -1.0\n",
       "   3 │    -1.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "select(A, :a => (x -> round.(x)) => :d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "と実行する必要があります。\n",
    "\n",
    "**`transform()`**\n",
    "\n",
    "`select()`と同じ記述方法で列を指定しますが，`select()`とは異なり，選択していない列も残して，値を返します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th><th>d</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td><td>0.277444</td><td>1.60103</td></tr><tr><th>2</th><td>-0.567185</td><td>1.5837</td><td>-0.36308</td><td>-1.13437</td></tr><tr><th>3</th><td>-1.33798</td><td>0.694831</td><td>-0.375632</td><td>-2.67596</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& a & b & c & d\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 & 0.277444 & 1.60103 \\\\\n",
       "\t2 & -0.567185 & 1.5837 & -0.36308 & -1.13437 \\\\\n",
       "\t3 & -1.33798 & 0.694831 & -0.375632 & -2.67596 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b         \u001b[0m\u001b[1m c         \u001b[0m\u001b[1m d        \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64   \u001b[0m\u001b[90m Float64  \u001b[0m\n",
       "─────┼───────────────────────────────────────────\n",
       "   1 │  0.800516  -2.26194    0.277444   1.60103\n",
       "   2 │ -0.567185   1.5837    -0.36308   -1.13437\n",
       "   3 │ -1.33798    0.694831  -0.375632  -2.67596"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform(A, :a => (x -> 2x) => :d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 行方向の操作を実行する。\n",
    "\n",
    "`select`や`transform`では基本的に列方向に対して関数を作用させました。しかしデータフレームには行ごとに意味をもたせたデータを入力したいこともあります（たとえばあるクラスのテスト得点の表とか）。\n",
    "\n",
    "そんな操作をするためには`ByRow`を使うと良いです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 英語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼──────────────────────────────\n",
       "   1 │ Kato        20    100     10\n",
       "   2 │ Noguchi     31     23     30\n",
       "   3 │ Yamada      55     78     89"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B = DataFrame(\"名前\" => [\"Kato\", \"Noguchi\", \"Yamada\"], \"国語\" => [20, 31, 55], \"数学\" => [100, 23, 78], \"英語\" => [10, 30, 89])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th><th>合計</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>3 rows × 5 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td><td>130</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td><td>84</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td><td>222</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語 & 合計\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 & 130 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 & 84 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 & 222 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×5 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 合計  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │ Kato        20    100     10    130\n",
       "   2 │ Noguchi     31     23     30     84\n",
       "   3 │ Yamada      55     78     89    222"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform(B, Not(:名前) => ByRow(+) => :合計)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**行方向の取り出し**\n",
    "\n",
    "列方向のデータフレーム操作をみたところで，単純な取り出しの例についても確認しておきましょう。\n",
    "\n",
    "列方向だけでなく，行方向に対しても一部の要素を取り出すことができます。`[,]`のカンマの左側に，ベクトルなどで位置を指定してやることで，取り出せます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td><td>0.277444</td></tr><tr><th>2</th><td>-0.567185</td><td>1.5837</td><td>-0.36308</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & b & c\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 & 0.277444 \\\\\n",
       "\t2 & -0.567185 & 1.5837 & -0.36308 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b        \u001b[0m\u001b[1m c         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64  \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼────────────────────────────────\n",
       "   1 │  0.800516  -2.26194   0.277444\n",
       "   2 │ -0.567185   1.5837   -0.36308"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[1:2, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特定の条件にマッチする行だけを取り出したい場合には，２通りの書き方ができます。\n",
    "\n",
    "- `A[条件, :]`\n",
    "- `filter(:ID -> 関数, A)`\n",
    "\n",
    "`filter()`は結構便利な関数です。次のように派生した書き方ができます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>2 rows × 3 columns</p><tr><th>1</th><td>-0.567185</td><td>1.5837</td><td>-0.36308</td></tr><tr><th>2</th><td>-1.33798</td><td>0.694831</td><td>-0.375632</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & b & c\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & -0.567185 & 1.5837 & -0.36308 \\\\\n",
       "\t2 & -1.33798 & 0.694831 & -0.375632 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a         \u001b[0m\u001b[1m b        \u001b[0m\u001b[1m c         \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64   \u001b[0m\u001b[90m Float64  \u001b[0m\u001b[90m Float64   \u001b[0m\n",
       "─────┼────────────────────────────────\n",
       "   1 │ -0.567185  1.5837    -0.36308\n",
       "   2 │ -1.33798   0.694831  -0.375632"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filter(:a => i -> i < 0, A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.7 `DataFramesRow`には要注意\n",
    "\n",
    "しかし，１行だけ取り出す場合，`DataFrame`から`DataFramesRow`と呼ばれる型に変化します。`DataFrame`の型を維持したままで１行だけ取り出す場合は，`A[[1], :a]`のように，ベクトルで行の要素を指定すればよいです。\n",
    "\n",
    "`DataFramesRow`に対してはbroadcastをすることができません。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p>DataFrameRow (2 columns)</p><table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Float64</th><th>Float64</th></tr></thead><tbody><tr><th>1</th><td>0.800516</td><td>-2.26194</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1mDataFrameRow\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a        \u001b[0m\u001b[1m b        \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64  \u001b[0m\u001b[90m Float64  \u001b[0m\n",
       "─────┼────────────────────\n",
       "   1 │ 0.800516  -2.26194"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[1, [:a, :b]] # DataFramesRow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th></tr><tr><th></th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>1 rows × 2 columns</p><tr><th>1</th><td>0.800516</td><td>-2.26194</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& a & b\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 0.800516 & -2.26194 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m1×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a        \u001b[0m\u001b[1m b        \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Float64  \u001b[0m\u001b[90m Float64  \u001b[0m\n",
       "─────┼────────────────────\n",
       "   1 │ 0.800516  -2.26194"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A[[1], [:a, :b]] # DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "ArgumentError: broadcasting over `DataFrameRow`s is reserved",
     "output_type": "error",
     "traceback": [
      "ArgumentError: broadcasting over `DataFrameRow`s is reserved",
      "",
      "Stacktrace:",
      " [1] broadcastable(::DataFrameRow{DataFrame,DataFrames.SubIndex{DataFrames.Index,Array{Int64,1},Array{Int64,1}}}) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframerow/dataframerow.jl:415",
      " [2] broadcasted(::Function, ::DataFrameRow{DataFrame,DataFrames.SubIndex{DataFrames.Index,Array{Int64,1},Array{Int64,1}}}) at ./broadcast.jl:1255",
      " [3] top-level scope at In[20]:1",
      " [4] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091"
     ]
    }
   ],
   "source": [
    "string.(A[1, [:a, :b]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "このように`DataFramesRow`に対してはbroadcast演算をすることはできません。\n",
    "\n",
    "---\n",
    "\n",
    "## 2. 結合する\n",
    "\n",
    "データフレーム操作で頻繁に行われるのが，データフレームの結合です。単純に行や列の大きさが同じものをくっつけるだけでなく，特定列の要素をキーにマッチさせたりすることができます。\n",
    "\n",
    "こうした操作は`**join(A, B)`系の関数で実行できます。\n",
    "\n",
    "### 2.1 `**join()`の基本的な書き方\n",
    "\n",
    "基本的な使用方法は\n",
    "\n",
    "```\n",
    "**join(A, B, on = :列名)\n",
    "```\n",
    "\n",
    "です。AとBで列名が違っている場合，\n",
    "\n",
    "```\n",
    "**join(A, B, on = :Aの列名 => :Bの列名)\n",
    "```\n",
    "\n",
    "というように，`Pair`を使用します。複数のキーがある場合はシンボルやペアをベクトルとして渡してあげます。\n",
    "\n",
    "```\n",
    "**join(A, B, on = [:列名1, :Aの列名２ => :Bの列名２])\n",
    "```\n",
    "\n",
    "### 2.2 joinの種類\n",
    "\n",
    "- `leftjoin`, `rightjoin` 左右どちらかのデータフレーム(A)を基準としてマッチさせる。基準のデータフレームAにないBの行は無視され，Bの列のうちAに含まれていないものは欠測`missing`として追加される。\n",
    "- `innerjoin` 両方のデータフレームの要素を行に関してすべて残す形でマッチさせる。片方のデータフレームにしかない行も完全に保存される。\n",
    "- `outerjoin` 両方のデータフレームの要素のうち，キーがマッチした行だけ残し，それ以外は無視する。この形では，マッチによる欠測が生じない。\n",
    "- `semijoin` マッチした行だけを保存するが，さらに列に関しては基準となるAの列だけを保存し，Bに関する要素は完全に無視される。\n",
    "- `antijoin` マッチした行**以外**を保存するが，列に関しては基準となるAの列だけが保存される。`semijoin`の逆バージョン。\n",
    "- `crossjoin` 与えたデータフレームらの直積を返す。マッチと言うよりも，全パタンの網羅するための関数？\n",
    "\n",
    "実際の実行例は[パッケージドキュメント](https://dataframes.juliadata.org/stable/man/joins/)に詳しいです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "people = DataFrame(ID = [20, 40], Name = [\"John Doe\", \"Jane Doe\"]);\n",
    "jobs = DataFrame(ID = [20, 40], Job = [\"Lawyer\", \"Doctor\"]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>ID</th><th>Name</th><th>ID_1</th><th>Job</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>String</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>20</td><td>John Doe</td><td>20</td><td>Lawyer</td></tr><tr><th>2</th><td>20</td><td>John Doe</td><td>40</td><td>Doctor</td></tr><tr><th>3</th><td>40</td><td>Jane Doe</td><td>20</td><td>Lawyer</td></tr><tr><th>4</th><td>40</td><td>Jane Doe</td><td>40</td><td>Doctor</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& ID & Name & ID\\_1 & Job\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 20 & John Doe & 20 & Lawyer \\\\\n",
       "\t2 & 20 & John Doe & 40 & Doctor \\\\\n",
       "\t3 & 40 & Jane Doe & 20 & Lawyer \\\\\n",
       "\t4 & 40 & Jane Doe & 40 & Doctor \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m ID    \u001b[0m\u001b[1m Name     \u001b[0m\u001b[1m ID_1  \u001b[0m\u001b[1m Job    \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String   \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m String \u001b[0m\n",
       "─────┼────────────────────────────────\n",
       "   1 │    20  John Doe     20  Lawyer\n",
       "   2 │    20  John Doe     40  Doctor\n",
       "   3 │    40  Jane Doe     20  Lawyer\n",
       "   4 │    40  Jane Doe     40  Doctor"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "crossjoin(people, jobs; makeunique = true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 列方向の結合\n",
    "\n",
    "列名が同じ要素の２つのデータフレームを結合したい場合は`vcat`が便利です。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>ID</th><th>Name</th><th>Job</th></tr><tr><th></th><th>Int64</th><th>String?</th><th>String?</th></tr></thead><tbody><p>4 rows × 3 columns</p><tr><th>1</th><td>20</td><td>John Doe</td><td><em>missing</em></td></tr><tr><th>2</th><td>40</td><td>Jane Doe</td><td><em>missing</em></td></tr><tr><th>3</th><td>20</td><td><em>missing</em></td><td>Lawyer</td></tr><tr><th>4</th><td>40</td><td><em>missing</em></td><td>Doctor</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& ID & Name & Job\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String? & String?\\\\\n",
       "\t\\hline\n",
       "\t1 & 20 & John Doe & \\emph{missing} \\\\\n",
       "\t2 & 40 & Jane Doe & \\emph{missing} \\\\\n",
       "\t3 & 20 & \\emph{missing} & Lawyer \\\\\n",
       "\t4 & 40 & \\emph{missing} & Doctor \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m ID    \u001b[0m\u001b[1m Name     \u001b[0m\u001b[1m Job     \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String?  \u001b[0m\u001b[90m String? \u001b[0m\n",
       "─────┼──────────────────────────\n",
       "   1 │    20  John Doe \u001b[90m missing \u001b[0m\n",
       "   2 │    40  Jane Doe \u001b[90m missing \u001b[0m\n",
       "   3 │    20 \u001b[90m missing  \u001b[0m Lawyer\n",
       "   4 │    40 \u001b[90m missing  \u001b[0m Doctor"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vcat((people, jobs)...; cols = :union)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 データフレームを変形させる\n",
    "\n",
    "データフレームをピボットしたり，変形したりする操作もDataFrames.jlの標準的な関数として提供されています。それが`stack`と`unstack`です。\n",
    "データフレームを縦方向に伸ばす（要素を縦に積み上げる）のが`stack`であり，逆に横方向に伸ばす（積み上がっている要素を横に展開する）のが`unstack`です。\n",
    "\n",
    "もっともかんたんな使用方法は，\n",
    "\n",
    "`stack(A, [縦に伸ばしたい複数列]; variable_name = :列名を要素とする新たな列の名前, value_name = :列の要素を立てに伸ばした新たな列の名前)`\n",
    "\n",
    "です。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 英語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼──────────────────────────────\n",
       "   1 │ Kato        20    100     10\n",
       "   2 │ Noguchi     31     23     30\n",
       "   3 │ Yamada      55     78     89"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>教科</th><th>得点</th></tr><tr><th></th><th>String</th><th>String</th><th>Int64</th></tr></thead><tbody><p>9 rows × 3 columns</p><tr><th>1</th><td>Kato</td><td>国語</td><td>20</td></tr><tr><th>2</th><td>Noguchi</td><td>国語</td><td>31</td></tr><tr><th>3</th><td>Yamada</td><td>国語</td><td>55</td></tr><tr><th>4</th><td>Kato</td><td>数学</td><td>100</td></tr><tr><th>5</th><td>Noguchi</td><td>数学</td><td>23</td></tr><tr><th>6</th><td>Yamada</td><td>数学</td><td>78</td></tr><tr><th>7</th><td>Kato</td><td>英語</td><td>10</td></tr><tr><th>8</th><td>Noguchi</td><td>英語</td><td>30</td></tr><tr><th>9</th><td>Yamada</td><td>英語</td><td>89</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& 名前 & 教科 & 得点\\\\\n",
       "\t\\hline\n",
       "\t& String & String & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 国語 & 20 \\\\\n",
       "\t2 & Noguchi & 国語 & 31 \\\\\n",
       "\t3 & Yamada & 国語 & 55 \\\\\n",
       "\t4 & Kato & 数学 & 100 \\\\\n",
       "\t5 & Noguchi & 数学 & 23 \\\\\n",
       "\t6 & Yamada & 数学 & 78 \\\\\n",
       "\t7 & Kato & 英語 & 10 \\\\\n",
       "\t8 & Noguchi & 英語 & 30 \\\\\n",
       "\t9 & Yamada & 英語 & 89 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m9×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 教科   \u001b[0m\u001b[1m 得点  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼────────────────────────\n",
       "   1 │ Kato     国語       20\n",
       "   2 │ Noguchi  国語       31\n",
       "   3 │ Yamada   国語       55\n",
       "   4 │ Kato     数学      100\n",
       "   5 │ Noguchi  数学       23\n",
       "   6 │ Yamada   数学       78\n",
       "   7 │ Kato     英語       10\n",
       "   8 │ Noguchi  英語       30\n",
       "   9 │ Yamada   英語       89"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "long_B = stack(B, Between(:国語, :英語), variable_name = :教科, value_name = :得点)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`unstack(A, :列名にしたい要素を持つ列名, :横に伸ばしたい要素をもつ列名))`で元のデータフレームに戻せます。\n",
    "\n",
    "※unstackした列は型が勝手に変わってしまうことに注意。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th></tr><tr><th></th><th>String</th><th>Int64?</th><th>Int64?</th><th>Int64?</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64? & Int64? & Int64?\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語   \u001b[0m\u001b[1m 数学   \u001b[0m\u001b[1m 英語   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m Int64? \u001b[0m\u001b[90m Int64? \u001b[0m\u001b[90m Int64? \u001b[0m\n",
       "─────┼─────────────────────────────────\n",
       "   1 │ Kato         20     100      10\n",
       "   2 │ Noguchi      31      23      30\n",
       "   3 │ Yamada       55      78      89"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wide_B = unstack(long_B, :教科, :得点)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Array{Union{Missing, Int64},1}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "typeof(wide_B.国語)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 欠測値を扱う\n",
    "\n",
    "データフレームで扱う変数には様々な理由により観測されない値が入っています。JuliaのDataFrameでは`Missing`という型で，この欠測値を扱います。\n",
    "\n",
    "先程のunstackで変化した型はこの`Missing`を含んだものでした。\n",
    "\n",
    "この`Missing`の型に関連して，次のような関数が準備されています。\n",
    "\n",
    "\n",
    "- `allowmissing` `allowmissing!`列の型を欠測値を認める型`Union{hoge, Missing}`に変える。\n",
    "- `disallowmissing` `disallowmissing!` 列の方を欠測値を認めない型に変える。\n",
    "- `completecases` すべての列，もしくは一部の列で欠測を含んでいない行を検索して`true` or `false`のベクトルを返す。\n",
    "- `dropmissing` `dropmissing!` 欠測を含んでいない行だけを残して，返す。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "┌ Error: Error adding value to column :名前.\n",
      "└ @ DataFrames /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:1237\n"
     ]
    },
    {
     "ename": "LoadError",
     "evalue": "MethodError: Cannot `convert` an object of type Missing to an object of type String\nClosest candidates are:\n  convert(::Type{T}, !Matched::T) where T<:AbstractString at strings/basic.jl:229\n  convert(::Type{T}, !Matched::AbstractString) where T<:AbstractString at strings/basic.jl:230\n  convert(::Type{S}, !Matched::CategoricalArrays.CategoricalValue) where S<:Union{AbstractChar, AbstractString, Number} at /Users/takuizum/.julia/packages/CategoricalArrays/ZjBSI/src/value.jl:73\n  ...",
     "output_type": "error",
     "traceback": [
      "MethodError: Cannot `convert` an object of type Missing to an object of type String\nClosest candidates are:\n  convert(::Type{T}, !Matched::T) where T<:AbstractString at strings/basic.jl:229\n  convert(::Type{T}, !Matched::AbstractString) where T<:AbstractString at strings/basic.jl:230\n  convert(::Type{S}, !Matched::CategoricalArrays.CategoricalValue) where S<:Union{AbstractChar, AbstractString, Number} at /Users/takuizum/.julia/packages/CategoricalArrays/ZjBSI/src/value.jl:73\n  ...",
      "",
      "Stacktrace:",
      " [1] setindex!(::Array{String,1}, ::Missing, ::Int64) at ./array.jl:847",
      " [2] _unsafe_copyto!(::Array{String,1}, ::Int64, ::Array{Missing,1}, ::Int64, ::Int64) at ./array.jl:257",
      " [3] unsafe_copyto! at ./array.jl:311 [inlined]",
      " [4] _copyto_impl! at ./array.jl:335 [inlined]",
      " [5] copyto! at ./array.jl:321 [inlined]",
      " [6] append!(::Array{String,1}, ::Array{Missing,1}) at ./array.jl:977",
      " [7] append!(::DataFrame, ::DataFrame; cols::Symbol, promote::Bool) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:1194",
      " [8] append!(::DataFrame, ::DataFrame) at /Users/takuizum/.julia/packages/DataFrames/yqToF/src/dataframe/dataframe.jl:1131",
      " [9] top-level scope at In[28]:1",
      " [10] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1091"
     ]
    }
   ],
   "source": [
    "append!(B, DataFrame(名前 = missing, 国語 = 0, 数学 = 0, 英語 = 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th></tr><tr><th></th><th>String?</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>4 rows × 4 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td></tr><tr><th>4</th><td><em>missing</em></td><td>0</td><td>0</td><td>0</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語\\\\\n",
       "\t\\hline\n",
       "\t& String? & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 \\\\\n",
       "\t4 & \\emph{missing} & 0 & 0 & 0 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 英語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String? \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼──────────────────────────────\n",
       "   1 │ Kato        20    100     10\n",
       "   2 │ Noguchi     31     23     30\n",
       "   3 │ Yamada      55     78     89\n",
       "   4 │\u001b[90m missing \u001b[0m     0      0      0"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "allowmissing!(B, :名前);\n",
    "append!(B, DataFrame(名前 = missing, 国語 = 0, 数学 = 0, 英語 = 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4-element BitArray{1}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "completecases(B)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>名前</th><th>国語</th><th>数学</th><th>英語</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>3 rows × 4 columns</p><tr><th>1</th><td>Kato</td><td>20</td><td>100</td><td>10</td></tr><tr><th>2</th><td>Noguchi</td><td>31</td><td>23</td><td>30</td></tr><tr><th>3</th><td>Yamada</td><td>55</td><td>78</td><td>89</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccc}\n",
       "\t& 名前 & 国語 & 数学 & 英語\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & Kato & 20 & 100 & 10 \\\\\n",
       "\t2 & Noguchi & 31 & 23 & 30 \\\\\n",
       "\t3 & Yamada & 55 & 78 & 89 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×4 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m 名前    \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 英語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String  \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼──────────────────────────────\n",
       "   1 │ Kato        20    100     10\n",
       "   2 │ Noguchi     31     23     30\n",
       "   3 │ Yamada      55     78     89"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropmissing!(B, :名前)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. グループごとに処理を実行する。\n",
    "\n",
    "`groupby`は特定の列の要素が同じもの同士でグループ化したデータフレームをひとまとめにした`GroupedDataFrame`を返します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>ID</th><th>英語</th><th>数学</th><th>国語</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>9 rows × 5 columns</p><tr><th>1</th><td>1</td><td>QNPLM</td><td>22</td><td>45</td><td>18</td></tr><tr><th>2</th><td>1</td><td>EKWFU</td><td>13</td><td>91</td><td>75</td></tr><tr><th>3</th><td>1</td><td>KMKBG</td><td>47</td><td>84</td><td>82</td></tr><tr><th>4</th><td>2</td><td>DIGUX</td><td>6</td><td>89</td><td>35</td></tr><tr><th>5</th><td>2</td><td>LNOXB</td><td>40</td><td>36</td><td>32</td></tr><tr><th>6</th><td>2</td><td>XGAVU</td><td>10</td><td>62</td><td>81</td></tr><tr><th>7</th><td>3</td><td>PZGSC</td><td>53</td><td>38</td><td>71</td></tr><tr><th>8</th><td>3</td><td>SSNOF</td><td>6</td><td>40</td><td>27</td></tr><tr><th>9</th><td>3</td><td>XIQKF</td><td>84</td><td>91</td><td>28</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& クラス & ID & 英語 & 数学 & 国語\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & QNPLM & 22 & 45 & 18 \\\\\n",
       "\t2 & 1 & EKWFU & 13 & 91 & 75 \\\\\n",
       "\t3 & 1 & KMKBG & 47 & 84 & 82 \\\\\n",
       "\t4 & 2 & DIGUX & 6 & 89 & 35 \\\\\n",
       "\t5 & 2 & LNOXB & 40 & 36 & 32 \\\\\n",
       "\t6 & 2 & XGAVU & 10 & 62 & 81 \\\\\n",
       "\t7 & 3 & PZGSC & 53 & 38 & 71 \\\\\n",
       "\t8 & 3 & SSNOF & 6 & 40 & 27 \\\\\n",
       "\t9 & 3 & XIQKF & 84 & 91 & 28 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m9×5 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m ID     \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 国語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │      1  QNPLM      22     45     18\n",
       "   2 │      1  EKWFU      13     91     75\n",
       "   3 │      1  KMKBG      47     84     82\n",
       "   4 │      2  DIGUX       6     89     35\n",
       "   5 │      2  LNOXB      40     36     32\n",
       "   6 │      2  XGAVU      10     62     81\n",
       "   7 │      3  PZGSC      53     38     71\n",
       "   8 │      3  SSNOF       6     40     27\n",
       "   9 │      3  XIQKF      84     91     28"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Random, Statistics\n",
    "D = DataFrame(クラス = [1, 1, 1, 2, 2 ,2, 3, 3, 3], ID = [randstring('A':'Z', 5) for i in 1:9], 英語 = rand(1:1:100, 9), 数学 = rand(1:1:100, 9), 国語 = rand(1:1:100, 9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><b>GroupedDataFrame with 3 groups based on key: クラス</b></p><p><i>First Group (3 rows): クラス = 1</i></p><table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>ID</th><th>英語</th><th>数学</th><th>国語</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><tr><th>1</th><td>1</td><td>QNPLM</td><td>22</td><td>45</td><td>18</td></tr><tr><th>2</th><td>1</td><td>EKWFU</td><td>13</td><td>91</td><td>75</td></tr><tr><th>3</th><td>1</td><td>KMKBG</td><td>47</td><td>84</td><td>82</td></tr></tbody></table><p>&vellip;</p><p><i>Last Group (3 rows): クラス = 3</i></p><table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>ID</th><th>英語</th><th>数学</th><th>国語</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><tr><th>1</th><td>3</td><td>PZGSC</td><td>53</td><td>38</td><td>71</td></tr><tr><th>2</th><td>3</td><td>SSNOF</td><td>6</td><td>40</td><td>27</td></tr><tr><th>3</th><td>3</td><td>XIQKF</td><td>84</td><td>91</td><td>28</td></tr></tbody></table>"
      ],
      "text/latex": [
       "GroupedDataFrame with 3 groups based on key: クラス\n",
       "\n",
       "First Group (3 rows): クラス = 1\n",
       "\n",
       "\\begin{tabular}{r|ccccc}\n",
       "\t& クラス & ID & 英語 & 数学 & 国語\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & QNPLM & 22 & 45 & 18 \\\\\n",
       "\t2 & 1 & EKWFU & 13 & 91 & 75 \\\\\n",
       "\t3 & 1 & KMKBG & 47 & 84 & 82 \\\\\n",
       "\\end{tabular}\n",
       "\n",
       "$\\dots$\n",
       "\n",
       "Last Group (3 rows): クラス = 3\n",
       "\n",
       "\\begin{tabular}{r|ccccc}\n",
       "\t& クラス & ID & 英語 & 数学 & 国語\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 3 & PZGSC & 53 & 38 & 71 \\\\\n",
       "\t2 & 3 & SSNOF & 6 & 40 & 27 \\\\\n",
       "\t3 & 3 & XIQKF & 84 & 91 & 28 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "GroupedDataFrame with 3 groups based on key: クラス\n",
       "First Group (3 rows): クラス = 1\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m ID     \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 国語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │      1  QNPLM      22     45     18\n",
       "   2 │      1  EKWFU      13     91     75\n",
       "   3 │      1  KMKBG      47     84     82\n",
       "⋮\n",
       "Last Group (3 rows): クラス = 3\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m ID     \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 国語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │      3  PZGSC      53     38     71\n",
       "   2 │      3  SSNOF       6     40     27\n",
       "   3 │      3  XIQKF      84     91     28"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "E = groupby(D, :クラス)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "グループ化すると`SubDataFrame`と呼ばれる亜種を内包した`GroupedDataFrame`に変わります。\n",
    "\n",
    "グループ化したデータフレームに対しては，グループごとに処理を実行することができます。代表的なものが`combine`です。\n",
    "\n",
    "`combine`は`SubDataFrame`の指定した列に対して関数を作用させ，最終的に全グループを同じデータフレームにまとめたものを返します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>クラス英語平均</th><th>クラス英語分散</th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 3 columns</p><tr><th>1</th><td>1</td><td>27.3333</td><td>310.333</td></tr><tr><th>2</th><td>2</td><td>18.6667</td><td>345.333</td></tr><tr><th>3</th><td>3</td><td>47.6667</td><td>1542.33</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& クラス & クラス英語平均 & クラス英語分散\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 27.3333 & 310.333 \\\\\n",
       "\t2 & 2 & 18.6667 & 345.333 \\\\\n",
       "\t3 & 3 & 47.6667 & 1542.33 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m クラス英語平均 \u001b[0m\u001b[1m クラス英語分散 \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m Float64        \u001b[0m\u001b[90m Float64        \u001b[0m\n",
       "─────┼────────────────────────────────────────\n",
       "   1 │      1         27.3333         310.333\n",
       "   2 │      2         18.6667         345.333\n",
       "   3 │      3         47.6667        1542.33"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combine(E, :英語 => mean　=> :クラス英語平均, :英語 => var => :クラス英語分散)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`do`ブロックを使った書き方もできます。この書き方のほうが，一度に複数の処理を走らせたり，一時的に利用する変数が必要なときには便利です（が，速度的には遅いと言われているので，多用しないほうが吉）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46.97249746956562, 41.863586949457584, 61.1639155809768]\n",
      "[43.18378796832533, 61.479936053346805, 45.33627597832786]\n",
      "[51.35803075539493, 39.39038472347713, 59.25158452112794]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>英語平均</th><th>英語分散</th></tr><tr><th></th><th>Int64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>3 rows × 3 columns</p><tr><th>1</th><td>1</td><td>27.3333</td><td>17.6163</td></tr><tr><th>2</th><td>2</td><td>18.6667</td><td>18.5831</td></tr><tr><th>3</th><td>3</td><td>47.6667</td><td>39.2726</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& クラス & 英語平均 & 英語分散\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 27.3333 & 17.6163 \\\\\n",
       "\t2 & 2 & 18.6667 & 18.5831 \\\\\n",
       "\t3 & 3 & 47.6667 & 39.2726 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m 英語平均 \u001b[0m\u001b[1m 英語分散 \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m Float64  \u001b[0m\u001b[90m Float64  \u001b[0m\n",
       "─────┼────────────────────────────\n",
       "   1 │      1   27.3333   17.6163\n",
       "   2 │      2   18.6667   18.5831\n",
       "   3 │      3   47.6667   39.2726"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combine(E) do sdf\n",
    "    m = mean(sdf.英語)\n",
    "    s = std(sdf.英語)\n",
    "    t = @. 10*(sdf.英語 - m)/s + 50\n",
    "    println(t)\n",
    "    (英語平均 = m, 英語分散 = s)\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "combineは１グループあたり１行しか値を含むことができないので，もとのデータフレーム長に合わせる場合は`select`を使うと良いです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>ID</th><th>英語</th><th>数学</th><th>国語</th><th>x1</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>9 rows × 6 columns</p><tr><th>1</th><td>1</td><td>QNPLM</td><td>22</td><td>45</td><td>18</td><td>46.9725</td></tr><tr><th>2</th><td>1</td><td>EKWFU</td><td>13</td><td>91</td><td>75</td><td>41.8636</td></tr><tr><th>3</th><td>1</td><td>KMKBG</td><td>47</td><td>84</td><td>82</td><td>61.1639</td></tr><tr><th>4</th><td>2</td><td>DIGUX</td><td>6</td><td>89</td><td>35</td><td>43.1838</td></tr><tr><th>5</th><td>2</td><td>LNOXB</td><td>40</td><td>36</td><td>32</td><td>61.4799</td></tr><tr><th>6</th><td>2</td><td>XGAVU</td><td>10</td><td>62</td><td>81</td><td>45.3363</td></tr><tr><th>7</th><td>3</td><td>PZGSC</td><td>53</td><td>38</td><td>71</td><td>51.358</td></tr><tr><th>8</th><td>3</td><td>SSNOF</td><td>6</td><td>40</td><td>27</td><td>39.3904</td></tr><tr><th>9</th><td>3</td><td>XIQKF</td><td>84</td><td>91</td><td>28</td><td>59.2516</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccc}\n",
       "\t& クラス & ID & 英語 & 数学 & 国語 & x1\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & Int64 & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & QNPLM & 22 & 45 & 18 & 46.9725 \\\\\n",
       "\t2 & 1 & EKWFU & 13 & 91 & 75 & 41.8636 \\\\\n",
       "\t3 & 1 & KMKBG & 47 & 84 & 82 & 61.1639 \\\\\n",
       "\t4 & 2 & DIGUX & 6 & 89 & 35 & 43.1838 \\\\\n",
       "\t5 & 2 & LNOXB & 40 & 36 & 32 & 61.4799 \\\\\n",
       "\t6 & 2 & XGAVU & 10 & 62 & 81 & 45.3363 \\\\\n",
       "\t7 & 3 & PZGSC & 53 & 38 & 71 & 51.358 \\\\\n",
       "\t8 & 3 & SSNOF & 6 & 40 & 27 & 39.3904 \\\\\n",
       "\t9 & 3 & XIQKF & 84 & 91 & 28 & 59.2516 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m9×6 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m ID     \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 国語  \u001b[0m\u001b[1m x1      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Float64 \u001b[0m\n",
       "─────┼──────────────────────────────────────────────\n",
       "   1 │      1  QNPLM      22     45     18  46.9725\n",
       "   2 │      1  EKWFU      13     91     75  41.8636\n",
       "   3 │      1  KMKBG      47     84     82  61.1639\n",
       "   4 │      2  DIGUX       6     89     35  43.1838\n",
       "   5 │      2  LNOXB      40     36     32  61.4799\n",
       "   6 │      2  XGAVU      10     62     81  45.3363\n",
       "   7 │      3  PZGSC      53     38     71  51.358\n",
       "   8 │      3  SSNOF       6     40     27  39.3904\n",
       "   9 │      3  XIQKF      84     91     28  59.2516"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform(E) do sdf\n",
    "    m = mean(sdf.英語)\n",
    "    s = std(sdf.英語)\n",
    "    t = @. 10*(sdf.英語 - m)/s + 50\n",
    "    t\n",
    "end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. その他の関数\n",
    "\n",
    "- `empty(A)` Aと同じ名前を持つ０行の新しいデータフレームを返す \n",
    "- `describe(A)` Aの各列に対して要約統計量を計算する（前はカテゴリカル変数に対応していたが...） `関数 => :新しい列名`で任意の関数を作用させることもできる。\n",
    "- `isapprox(A, B)`　AとBが任意の誤差の範囲内で一致しているかどうかをたしかめる。broadcastすると列ごとに論理値を返す。列名が一緒でないとエラーを吐く。完全一致なら`isqeual()`。\n",
    "- `sort(A)`, `sort(A, :列名)` データフレーム全体か，列名に関してソートする。`!`をつけるとin-place。\n",
    "- `names(A)`, `propertynames(A)` Aの列名を取得する。`names`で文字列のベクトルを，`propertynames`でシンボルのベクトルを得る。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><th></th><th>Float64</th><th>Float64</th><th>Float64</th></tr></thead><tbody><p>0 rows × 3 columns</p></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & b & c\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64\\\\\n",
       "\t\\hline\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m0×3 DataFrame\u001b[0m"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>variable</th><th>mean</th><th>min</th><th>median</th><th>max</th><th>nmissing</th><th>eltype</th></tr><tr><th></th><th>Symbol</th><th>Union…</th><th>Any</th><th>Union…</th><th>Any</th><th>Int64</th><th>DataType</th></tr></thead><tbody><p>5 rows × 7 columns</p><tr><th>1</th><td>クラス</td><td>2.0</td><td>1</td><td>2.0</td><td>3</td><td>0</td><td>Int64</td></tr><tr><th>2</th><td>ID</td><td></td><td>DIGUX</td><td></td><td>XIQKF</td><td>0</td><td>String</td></tr><tr><th>3</th><td>英語</td><td>31.2222</td><td>6</td><td>22.0</td><td>84</td><td>0</td><td>Int64</td></tr><tr><th>4</th><td>数学</td><td>64.0</td><td>36</td><td>62.0</td><td>91</td><td>0</td><td>Int64</td></tr><tr><th>5</th><td>国語</td><td>49.8889</td><td>18</td><td>35.0</td><td>82</td><td>0</td><td>Int64</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& variable & mean & min & median & max & nmissing & eltype\\\\\n",
       "\t\\hline\n",
       "\t& Symbol & Union… & Any & Union… & Any & Int64 & DataType\\\\\n",
       "\t\\hline\n",
       "\t1 & クラス & 2.0 & 1 & 2.0 & 3 & 0 & Int64 \\\\\n",
       "\t2 & ID &  & DIGUX &  & XIQKF & 0 & String \\\\\n",
       "\t3 & 英語 & 31.2222 & 6 & 22.0 & 84 & 0 & Int64 \\\\\n",
       "\t4 & 数学 & 64.0 & 36 & 62.0 & 91 & 0 & Int64 \\\\\n",
       "\t5 & 国語 & 49.8889 & 18 & 35.0 & 82 & 0 & Int64 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m5×7 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean    \u001b[0m\u001b[1m min   \u001b[0m\u001b[1m median \u001b[0m\u001b[1m max   \u001b[0m\u001b[1m nmissing \u001b[0m\u001b[1m eltype   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Symbol   \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Any   \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Any   \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m DataType \u001b[0m\n",
       "─────┼─────────────────────────────────────────────────────────────\n",
       "   1 │ クラス    2.0      1      2.0     3             0  Int64\n",
       "   2 │ ID       \u001b[90m         \u001b[0m DIGUX \u001b[90m        \u001b[0m XIQKF         0  String\n",
       "   3 │ 英語      31.2222  6      22.0    84            0  Int64\n",
       "   4 │ 数学      64.0     36     62.0    91            0  Int64\n",
       "   5 │ 国語      49.8889  18     35.0    82            0  Int64"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "describe(D)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>a</th><th>b</th><th>c</th></tr><tr><th></th><th>Bool</th><th>Bool</th><th>Bool</th></tr></thead><tbody><p>3 rows × 3 columns</p><tr><th>1</th><td>1</td><td>1</td><td>1</td></tr><tr><th>2</th><td>1</td><td>1</td><td>1</td></tr><tr><th>3</th><td>1</td><td>1</td><td>1</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& a & b & c\\\\\n",
       "\t\\hline\n",
       "\t& Bool & Bool & Bool\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & 1 & 1 \\\\\n",
       "\t2 & 1 & 1 & 1 \\\\\n",
       "\t3 & 1 & 1 & 1 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m3×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m a    \u001b[0m\u001b[1m b    \u001b[0m\u001b[1m c    \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Bool \u001b[0m\u001b[90m Bool \u001b[0m\u001b[90m Bool \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │ true  true  true\n",
       "   2 │ true  true  true\n",
       "   3 │ true  true  true"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isequal.(A, A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>クラス</th><th>ID</th><th>英語</th><th>数学</th><th>国語</th></tr><tr><th></th><th>Int64</th><th>String</th><th>Int64</th><th>Int64</th><th>Int64</th></tr></thead><tbody><p>9 rows × 5 columns</p><tr><th>1</th><td>1</td><td>QNPLM</td><td>22</td><td>45</td><td>18</td></tr><tr><th>2</th><td>3</td><td>SSNOF</td><td>6</td><td>40</td><td>27</td></tr><tr><th>3</th><td>3</td><td>XIQKF</td><td>84</td><td>91</td><td>28</td></tr><tr><th>4</th><td>2</td><td>LNOXB</td><td>40</td><td>36</td><td>32</td></tr><tr><th>5</th><td>2</td><td>DIGUX</td><td>6</td><td>89</td><td>35</td></tr><tr><th>6</th><td>3</td><td>PZGSC</td><td>53</td><td>38</td><td>71</td></tr><tr><th>7</th><td>1</td><td>EKWFU</td><td>13</td><td>91</td><td>75</td></tr><tr><th>8</th><td>2</td><td>XGAVU</td><td>10</td><td>62</td><td>81</td></tr><tr><th>9</th><td>1</td><td>KMKBG</td><td>47</td><td>84</td><td>82</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& クラス & ID & 英語 & 数学 & 国語\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String & Int64 & Int64 & Int64\\\\\n",
       "\t\\hline\n",
       "\t1 & 1 & QNPLM & 22 & 45 & 18 \\\\\n",
       "\t2 & 3 & SSNOF & 6 & 40 & 27 \\\\\n",
       "\t3 & 3 & XIQKF & 84 & 91 & 28 \\\\\n",
       "\t4 & 2 & LNOXB & 40 & 36 & 32 \\\\\n",
       "\t5 & 2 & DIGUX & 6 & 89 & 35 \\\\\n",
       "\t6 & 3 & PZGSC & 53 & 38 & 71 \\\\\n",
       "\t7 & 1 & EKWFU & 13 & 91 & 75 \\\\\n",
       "\t8 & 2 & XGAVU & 10 & 62 & 81 \\\\\n",
       "\t9 & 1 & KMKBG & 47 & 84 & 82 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m9×5 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m クラス \u001b[0m\u001b[1m ID     \u001b[0m\u001b[1m 英語  \u001b[0m\u001b[1m 数学  \u001b[0m\u001b[1m 国語  \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64  \u001b[0m\u001b[90m String \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\u001b[90m Int64 \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │      1  QNPLM      22     45     18\n",
       "   2 │      3  SSNOF       6     40     27\n",
       "   3 │      3  XIQKF      84     91     28\n",
       "   4 │      2  LNOXB      40     36     32\n",
       "   5 │      2  DIGUX       6     89     35\n",
       "   6 │      3  PZGSC      53     38     71\n",
       "   7 │      1  EKWFU      13     91     75\n",
       "   8 │      2  XGAVU      10     62     81\n",
       "   9 │      1  KMKBG      47     84     82"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sort(D, :国語)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{String,1}:\n",
       " \"a\"\n",
       " \"b\"\n",
       " \"c\""
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3-element Array{Symbol,1}:\n",
       " :a\n",
       " :b\n",
       " :c"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "propertynames(A)"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.5.3",
   "language": "julia",
   "name": "julia-1.5"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}