{ "cells": [ { "cell_type": "code", "execution_count": 3, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# %load /Users/facai/Study/book_notes/preconfig.py\n", "%matplotlib inline\n", "\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "from IPython.display import SVG" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "逻辑回归在spark中的实现简介\n", "=======================" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "分析用的代码版本信息：\n", "\n", "```bash\n", "~/W/g/spark ❯❯❯ git log -n 1\n", "commit d9ad78908f6189719cec69d34557f1a750d2e6af\n", "Author: Wenchen Fan \n", "Date: Fri May 26 15:01:28 2017 +0800\n", "\n", " [SPARK-20868][CORE] UnsafeShuffleWriter should verify the position after FileChannel.transferTo\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 0. 总纲\n", "\n", "下图是ml包中逻辑回归的构成情况：" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": false }, "outputs": [ { "data": { "image/svg+xml": [ "LogisticRegression+train()ProbabilisticClassifierInstanceInstrumentationMultivariateOnlineSummarizerMultiClassSummarizerMetadataUtils+getNumClasses()LogisticCostFun+calculate()DiffFunction+calculate()LogisticAggregator+gradient+loss+add()+merge()LBFGSQWLQNCachedDiffFunctionFirstOrderMinimizer+State+iterations()" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "SVG(\"./res/spark_ml_lr.svg\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "可以看到，逻辑回归是比较简单的，在它的`train`函数里，除开左侧的几个辅助类：\n", "\n", "+ Instance: 封装数据\n", "+ MetadataUtils: 数据信息\n", "+ Instrumentation: 日志 \n", "+ Multi*Summarizer: 统计\n", "\n", "主要就是做两件事：\n", "\n", "+ 构造损失函数 => costFun: DiffFunction\n", "+ 创建寻优算子 => optimizer: FirstOrderMinizer \n", " ml里两种算子都是拟牛顿法，理论上比SGD迭代更少，收敛更快。其中QWLQN是LBFGS的变种，可使用L1正则。\n", " \n", "接下来，我们就将精力放在这两件事的实现上。这里寻优算子主要是根据正则确定的，而损失函数会由二分类和多分类而有所变化，下面一一叙迖述。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. 寻优算子\n", "\n", "jupyter的markdown，无法正确处理`$`取值语法，所以做了点小变动。\n", "\n", "```scala\n", " 645 val optimizer = if (elasticNetParam == 0.0 || regParam == 0.0) {\n", " 646 // +-- 4 lines: if (lowerBounds != null && upperBounds != null) {--------------------------------------\n", " 650 new BreezeLBFGS[BDV[Double]](maxIter, 10, tol)\n", " 651 }\n", " 652 } else {\n", " 653 val standardizationParam = standardization\n", " 654 def regParamL1Fun = (index: Int) => {\n", " 655 // +-- 2 lines: Remove the L1 penalization on the intercept--------------------------------------------\n", " 657 if (isIntercept) {\n", " 658 0.0\n", " 659 } else {\n", " 660 if (standardizationParam) {\n", " 661 regParamL1\n", " 662 } else {\n", " 663 val featureIndex = index / numCoefficientSets\n", " 664 // +-- 5 lines: If `standardization` is false, we still standardize the data---------------------------\n", " 669 if (featuresStd(featureIndex) != 0.0) {\n", " 670 regParamL1 / featuresStd(featureIndex)\n", " 671 } else {\n", " 672 0.0\n", " 673 }\n", " 674 }\n", " 675 }\n", " 676 }\n", " 677 new BreezeOWLQN[Int, BDV[Double]](maxIter, 10, regParamL1Fun, $(tol))\n", " 678 }\n", "```\n", "\n", "可以看到，逻辑很简单：如果不用正则，或只用L2，就用LBFGS算子；如果用到L1正则，就用QWLQN算子。其中下半代码均是在折算合适的L1正则值。\n", "\n", "因为QWLQN会自己处理L1正则，所以在接下来的损失函数计算中，我们只考虑L2正则，而不管L1。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. 损失函数\n", "#### 2.1 二分类\n", "\n", "预测公式：$f(x) = \\frac1{1 + e^{w^T x}}$\n", "\n", "[损失函数](http://spark.apache.org/docs/latest/mllib-linear-methods.html#logistic-regression)定义是：\n", "\n", "\\begin{equation}\n", "L(w;x,y) = \\log(1+e^{-y w^T x}) + r_2 \\cdot \\frac{1}{2} w^T w + r_1 \\cdot \\|w\\|\n", "\\end{equation}\n", "\n", "[导数是](http://spark.apache.org/docs/latest/mllib-linear-methods.html#loss-functions)：\n", "\n", "\\begin{align}\n", " \\frac{\\partial L}{\\partial w} &= -y \\left(1-\\frac1{1+e^{-y w^T x}} \\right) \\cdot x + r_2 w \\pm r_1 \\\\\n", " &= \\left ( \\frac{y}{1+e^{-y w^T x}} - y \\right ) \\cdot x + r_2 w \\pm r_1 \\\\\n", " \\text{因为$y$只有1和-1两值，可简化为} \\\\\n", " &= \\left ( \\frac{1}{1+e^{-w^T x}} - y \\right ) \\cdot x + r_2 w \\pm r_1 \\\\\n", " &= \\left ( f(x) - y \\right ) \\cdot x + r_2 w \\pm r_1\n", "\\end{align}\n", "\n", "好，我们先看没有正则的计算，在LogisticAggregator类里：\n", "\n", "```scala\n", "1670 /** Update gradient and loss using binary loss function. */\n", "1671 private def binaryUpdateInPlace(\n", "1672 features: Vector,\n", "1673 weight: Double,\n", "1674 label: Double): Unit = {\n", "1675 +-- 4 lines: val localFeaturesStd = bcFeaturesStd.value----------\n", "1679 val margin = - {\n", "1680 var sum = 0.0\n", "1681 features.foreachActive { (index, value) =>\n", "1682 if (localFeaturesStd(index) != 0.0 && value != 0.0) {\n", "1683 sum += localCoefficients(index) * value / localFeaturesStd(index)\n", "1684 }\n", "1685 }\n", "1686 if (fitIntercept) sum += localCoefficients(numFeaturesPlusIntercept - 1)\n", "1687 sum\n", "1688 }\n", "1689\n", "1690 val multiplier = weight * (1.0 / (1.0 + math.exp(margin)) - label)\n", "1691\n", "1692 features.foreachActive { (index, value) =>\n", "1693 if (localFeaturesStd(index) != 0.0 && value != 0.0) {\n", "1694 localGradientArray(index) += multiplier * value / localFeaturesStd(index)\n", "1695 }\n", "1696 }\n", "1697\n", "1698 if (fitIntercept) {\n", "1699 localGradientArray(numFeaturesPlusIntercept - 1) += multiplier\n", "1700 }\n", "1701\n", "1702 if (label > 0) {\n", "1703 // The following is equivalent to log(1 + exp(margin)) but more numerically stable.\n", "1704 lossSum += weight * MLUtils.log1pExp(margin)\n", "1705 } else {\n", "1706 lossSum += weight * (MLUtils.log1pExp(margin) - margin)\n", "1707 }\n", "1708 }\n", "```\n", "\n", "其中，\n", "+ margin = $-w^T x$ \n", " 注意：这里用的$x / \\operatorname{std}(x)$，相当于归一化，统一量纲。很奇怪，没有同时移动坐标，我不清楚是否合理。\n", "+ multiplier = $\\frac1{1 + e^{w^T x}} - y$ = $f(x) - y$\n", "+ localGradientArray = $(f(x) - y) x$\n", "+ lossSum = $\\log(1+e^{-y w^T x})$。注意：因为margin计算时是$y=1$，所以1706L，对$y=-1$做了变换。数学技巧比较简单：\n", "\n", "\\begin{align}\n", " log(1 + e^x) - x &= log(1 + e^x) - log(e^x) \\\\\n", " &= log(\\frac{1 + e^x}{e^x}) \\\\\n", " &= log(1 + e^{-x})\n", "\\end{align}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "再在损失函数和偏导里，均加上L2的部份，代码在LogisticCostFun类的calculate方法里：\n", "\n", "```scala\n", "1877 override def calculate(coefficients: BDV[Double]): (Double, BDV[Double]) = {\n", "1878 // +-- 6 lines: val coeffs = Vectors.fromBreeze(coefficients)------\n", "1884\n", "1885 val logisticAggregator = {\n", "1886 // +-- 3 lines: val seqOp = (c: LogisticAggregator, instance: Instance) =>\n", "1889 instances.treeAggregate(\n", "1890 new LogisticAggregator(bcCoeffs, bcFeaturesStd, numClasses, fitIntercept,\n", "1891 multinomial)\n", "1892 )(seqOp, combOp, aggregationDepth)\n", "1893 }\n", "1894\n", "1895 val totalGradientMatrix = logisticAggregator.gradient\n", "1896 val coefMatrix = new DenseMatrix(numCoefficientSets, numFeaturesPlusIntercept, coeffs.toArray)\n", "1897 // regVal is the sum of coefficients squares excluding intercept for L2 regularization.\n", "1898 val regVal = if (regParamL2 == 0.0) {\n", "1899 0.0\n", "1900 } else {\n", "1901 var sum = 0.0\n", "1902 coefMatrix.foreachActive { case (classIndex, featureIndex, value) =>\n", "1903 // We do not apply regularization to the intercepts\n", "1904 val isIntercept = fitIntercept && (featureIndex == numFeatures)\n", "1905 if (!isIntercept) {\n", "1906 // +-- 2 lines: The following code will compute the loss of the regularization; also---\n", "1908 sum += {\n", "1909 if (standardization) {\n", "1910 val gradValue = totalGradientMatrix(classIndex, featureIndex)\n", "1911 totalGradientMatrix.update(classIndex, featureIndex, gradValue + regParamL2 * value)\n", "1912 value * value\n", "1913 // +-- 14 lines: } else {------------------\n", "1927 }\n", "1928 }\n", "1929 }\n", "1930 }\n", "1931 0.5 * regParamL2 * sum\n", "1932 }\n", "1933 // +-- 2 lines: bcCoeffs.destroy(blocking = false)--------\n", "1935 (logisticAggregator.loss + regVal, new BDV(totalGradientMatrix.toArray))\n", "1936 }\n", "1\n", "```\n", "\n", "其中，1912L和1931L是加L2正则$r_2 \\cdot \\frac{1}{2}w^T w$；1911L是加L2的偏导$r_2 \\cdot w$。因为有额外的分支处理归一的情况，分支较多。同时，损失和偏导混在一起算，代码有点混杂。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "如此，就有了二分类的损失函数和偏导数。\n", "\n", "```scala\n", " 601 val costFun = new LogisticCostFun(instances, numClasses, fitIntercept,\n", " 602 standardization, bcFeaturesStd, regParamL2, multinomial = isMultinomial,\n", " 603 aggregationDepth)\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 2.2 多分类" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "spark在这里用的是softmax函数来代替logit函数，是比较有意思的解决方案。因为在LogisticAggregator类，已经详细地注释了推导关键过程，所以我就直接搬运过来，稍微作点附注，再把代码和公式对应起来就好。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "LogisticAggregator computes the gradient and loss for binary or multinomial logistic (softmax)\n", "loss function, as used in classification for instances in sparse or dense vector in an online\n", "fashion.\n", " \n", "Two LogisticAggregators can be merged together to have a summary of loss and gradient of\n", "the corresponding joint dataset.\n", " \n", "For improving the convergence rate during the optimization process and also to prevent against\n", "features with very large variances exerting an overly large influence during model training,\n", "packages like R's GLMNET perform the scaling to unit variance and remove the mean in order to\n", "reduce the condition number. The model is then trained in this scaled space, but returns the\n", "coefficients in the original scale. See page 9 in\n", "http://cran.r-project.org/web/packages/glmnet/glmnet.pdf\n", " \n", "However, we don't want to apply the [[org.apache.spark.ml.feature.StandardScaler]] on the\n", "training dataset, and then cache the standardized dataset since it will create a lot of overhead.\n", "As a result, we perform the scaling implicitly when we compute the objective function (though\n", "we do not subtract the mean).\n", " \n", "Note that there is a difference between multinomial (softmax) and binary loss. The binary case\n", "uses one outcome class as a \"pivot\" and regresses the other class against the pivot. In the\n", "multinomial case, the softmax loss function is used to model each class probability\n", "independently. Using softmax loss produces `K` sets of coefficients, while using a pivot class\n", "produces `K - 1` sets of coefficients (a single coefficient vector in the binary case). In the\n", "binary case, we can say that the coefficients are shared between the positive and negative\n", "classes. When regularization is applied, multinomial (softmax) loss will produce a result\n", "different from binary loss since the positive and negative don't share the coefficients while the\n", "binary regression shares the coefficients between positive and negative.\n", " \n", "The following is a mathematical derivation for the multinomial (softmax) loss.\n", " \n", "The probability of the multinomial outcome $y$ taking on any of the K possible outcomes is:\n", " \n", "
\n", " $$\n", " P(y_i=0|\\vec{x}_i, \\beta) = \\frac{e^{\\vec{x}_i^T \\vec{\\beta}_0}}{\\sum_{k=0}^{K-1}\n", " e^{\\vec{x}_i^T \\vec{\\beta}_k}} \\\\\n", " P(y_i=1|\\vec{x}_i, \\beta) = \\frac{e^{\\vec{x}_i^T \\vec{\\beta}_1}}{\\sum_{k=0}^{K-1}\n", " e^{\\vec{x}_i^T \\vec{\\beta}_k}}\\\\\n", " P(y_i=K-1|\\vec{x}_i, \\beta) = \\frac{e^{\\vec{x}_i^T \\vec{\\beta}_{K-1}}\\,}{\\sum_{k=0}^{K-1}\n", " e^{\\vec{x}_i^T \\vec{\\beta}_k}}\n", " $$\n", "
\n", " \n", "The model coefficients $\\beta = (\\beta_0, \\beta_1, \\beta_2, ..., \\beta_{K-1})$ become a matrix\n", "which has dimension of $K \\times (N+1)$ if the intercepts are added. If the intercepts are not\n", "added, the dimension will be $K \\times N$.\n", " \n", "Note that the coefficients in the model above lack identifiability. That is, any constant scalar\n", "can be added to all of the coefficients and the probabilities remain the same.\n", " \n", "
\n", " $$\n", " \\begin{align}\n", " \\frac{e^{\\vec{x}_i^T \\left(\\vec{\\beta}_0 + \\vec{c}\\right)}}{\\sum_{k=0}^{K-1}\n", " e^{\\vec{x}_i^T \\left(\\vec{\\beta}_k + \\vec{c}\\right)}}\n", " = \\frac{e^{\\vec{x}_i^T \\vec{\\beta}_0}e^{\\vec{x}_i^T \\vec{c}}\\,}{e^{\\vec{x}_i^T \\vec{c}}\n", " \\sum_{k=0}^{K-1} e^{\\vec{x}_i^T \\vec{\\beta}_k}}\n", " = \\frac{e^{\\vec{x}_i^T \\vec{\\beta}_0}}{\\sum_{k=0}^{K-1} e^{\\vec{x}_i^T \\vec{\\beta}_k}}\n", " \\end{align}\n", " $$\n", "
\n", " \n", "However, when regularization is added to the loss function, the coefficients are indeed\n", "identifiable because there is only one set of coefficients which minimizes the regularization\n", "term. When no regularization is applied, we choose the coefficients with the minimum L2\n", "penalty for consistency and reproducibility. For further discussion see:\n", " \n", "Friedman, et al. \"Regularization Paths for Generalized Linear Models via Coordinate Descent\"\n", " \n", "The loss of objective function for a single instance of data (we do not include the\n", "regularization term here for simplicity) can be written as\n", " \n", "
\n", " $$\n", " \\begin{align}\n", " \\ell\\left(\\beta, x_i\\right) &= -log{P\\left(y_i \\middle| \\vec{x}_i, \\beta\\right)} \\\\\n", " &= log\\left(\\sum_{k=0}^{K-1}e^{\\vec{x}_i^T \\vec{\\beta}_k}\\right) - \\vec{x}_i^T \\vec{\\beta}_y\\\\\n", " &= log\\left(\\sum_{k=0}^{K-1} e^{margins_k}\\right) - margins_y\n", " \\end{align}\n", " $$\n", "
\n", " \n", "where ${margins}_k = \\vec{x}_i^T \\vec{\\beta}_k$.\n", " \n", "For optimization, we have to calculate the first derivative of the loss function, and a simple\n", "calculation shows that\n", " \n", "
\n", " $$\n", " \\begin{align}\n", " \\frac{\\partial \\ell(\\beta, \\vec{x}_i, w_i)}{\\partial \\beta_{j, k}}\n", " &= x_{i,j} \\cdot w_i \\cdot \\left(\\frac{e^{\\vec{x}_i \\cdot \\vec{\\beta}_k}}{\\sum_{k'=0}^{K-1}\n", " e^{\\vec{x}_i \\cdot \\vec{\\beta}_{k'}}\\,} - I_{y=k}\\right) \\\\\n", " &= x_{i, j} \\cdot w_i \\cdot multiplier_k\n", " \\end{align}\n", " $$\n", "
\n", " \n", "where $w_i$ is the sample weight, $I_{y=k}$ is an indicator function\n", " \n", "
\n", " $$\n", " I_{y=k} = \\begin{cases}\n", " 1 & y = k \\\\\n", " 0 & else\n", " \\end{cases}\n", " $$\n", "
\n", " \n", "and\n", " \n", "
\n", " $$\n", " multiplier_k = \\left(\\frac{e^{\\vec{x}_i \\cdot \\vec{\\beta}_k}}{\\sum_{k=0}^{K-1}\n", " e^{\\vec{x}_i \\cdot \\vec{\\beta}_k}} - I_{y=k}\\right)\n", " $$\n", "
\n", "\n", "$\\exp(709.78)$超出Double上限。\n", "\n", "If any of margins is larger than 709.78, the numerical computation of multiplier and loss\n", "function will suffer from arithmetic overflow. This issue occurs when there are outliers in\n", "data which are far away from the hyperplane, and this will cause the failing of training once\n", "infinity is introduced. Note that this is only a concern when max(margins) > 0.\n", " \n", "Fortunately, when max(margins) = maxMargin > 0, the loss function and the multiplier can\n", "easily be rewritten into the following equivalent numerically stable formula.\n", " \n", "这里变换非常简单，将括号打开，用指数和对数规则依次套用。 \n", " \n", "
\n", " $$\n", " \\ell\\left(\\beta, x\\right) = log\\left(\\sum_{k=0}^{K-1} e^{margins_k - maxMargin}\\right) -\n", " margins_{y} + maxMargin\n", " $$\n", "
\n", " \n", "Note that each term, $(margins_k - maxMargin)$ in the exponential is no greater than zero; as a\n", "result, overflow will not happen with this formula.\n", " \n", "For $multiplier$, a similar trick can be applied as the following,\n", " \n", "
\n", " $$\n", " multiplier_k = \\left(\\frac{e^{\\vec{x}_i \\cdot \\vec{\\beta}_k - maxMargin}}{\\sum_{k'=0}^{K-1}\n", " e^{\\vec{x}_i \\cdot \\vec{\\beta}_{k'} - maxMargin}} - I_{y=k}\\right)\n", " $$\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```scala\n", "1711 private def multinomialUpdateInPlace(\n", "1712 // +-- 12 lines: features: Vector,----------\n", "1724 // marginOfLabel is margins(label) in the formula\n", "1725 var marginOfLabel = 0.0\n", "1726 var maxMargin = Double.NegativeInfinity\n", "1727\n", "1728 val margins = new Array[Double](numClasses)\n", "1729 features.foreachActive { (index, value) =>\n", "1730 // +-- 2 lines: val stdValue = value / localFeaturesStd(index)-------\n", "1732 while (j < numClasses) {\n", "1733 margins(j) += localCoefficients(index * numClasses + j) * stdValue\n", "1734 // +-- 4 lines: j += 1-----------------\n", "1738 while (i < numClasses) {\n", "1739 if (fitIntercept) {\n", "1740 margins(i) += localCoefficients(numClasses * numFeatures + i)\n", "1741 }\n", "1742 if (i == label.toInt) marginOfLabel = margins(i)\n", "1743 if (margins(i) > maxMargin) {\n", "1744 maxMargin = margins(i)\n", "1745 }\n", "1746 i += 1\n", "1747 }\n", "1748 // +-- 6 lines: *---------------------\n", "1754 val multipliers = new Array[Double](numClasses)\n", "1755 val sum = {\n", "1756 var temp = 0.0\n", "1757 var i = 0\n", "1758 while (i < numClasses) {\n", "1759 if (maxMargin > 0) margins(i) -= maxMargin\n", "1760 val exp = math.exp(margins(i))\n", "1761 temp += exp\n", "1762 multipliers(i) = exp\n", "1763 i += 1\n", "1764 }\n", "1765 temp\n", "1766 }\n", "1767\n", "1768 margins.indices.foreach { i =>\n", "1769 multipliers(i) = multipliers(i) / sum - (if (label == i) 1.0 else 0.0)\n", "1770 }\n", "1771 features.foreachActive { (index, value) =>\n", "1772 if (localFeaturesStd(index) != 0.0 && value != 0.0) {\n", "1773 val stdValue = value / localFeaturesStd(index)\n", "1774 var j = 0\n", "1775 while (j < numClasses) {\n", "1776 localGradientArray(index * numClasses + j) +=\n", "1777 weight * multipliers(j) * stdValue\n", "1778 j += 1\n", "1779 }\n", "1780 }\n", "1781 }\n", "1782 if (fitIntercept) {\n", "1783 var i = 0\n", "1784 while (i < numClasses) {\n", "1785 localGradientArray(numFeatures * numClasses + i) += weight * multipliers(i)\n", "1786 i += 1\n", "1787 }\n", "1788 }\n", "1789\n", "1790 val loss = if (maxMargin > 0) {\n", "1791 math.log(sum) - marginOfLabel + maxMargin\n", "1792 } else {\n", "1793 math.log(sum) - marginOfLabel\n", "1794 }\n", "1795 lossSum += weight * loss\n", "1796 }\n", "```\n", "\n", "+ 1728L-1733L，在计算margins = $x \\beta$。1738L的循环是找出maxMargin和标签对应的marginOfLabel，因为后面公式要用到。\n", "+ 1754L-1770L，计算了multipliers。我个人很不喜欢这种一个循环做两件事，且出口不同的风格。\n", "+ 1771L-1788L，计算导数localGradientArray = $x_{i, j} \\cdot w_i \\cdot \\operatorname{multiplier}_k$。\n", "+ 1790L-1795L，根据最大margin是否大于0，计算损失值loss。注意1759L也有针对做修正。\n", "\n", "公式较复杂，但代码挺简单的。为了效率，有的地方写得不太好看。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. 小结\n", "\n", "spark-ml里逻辑回归支持样本加权，二分类和多分类。寻优算子是相对优秀的拟牛顿算法，多分类是softmax。总体而言，功能完整够用，实现也比较优秀。但有的代码，个人认为像面条，冗余，不够清减。" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 0 }