{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# %load /Users/facai/Study/book_notes/preconfig.py\n", "%matplotlib inline\n", "\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "sns.set(color_codes=True)\n", "sns.set(font='SimHei', font_scale=2.5)\n", "\n", "from IPython.display import Image" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "TensorFlow 2.0资料汇总\n", "============\n", "\n", "[RFC management](https://github.com/tensorflow/community/projects/1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [RFC: Variables in TensorFlow 2.0 #11](https://github.com/tensorflow/community/pull/11)\n", "\n", "+ drawbacks:\n", " 1. impossible-to-reason-about semantics => RefVariable -> ResourceVariable\n", " 2. reliance on global scopes, and reliance on global collections. => keras, object oriented\n", "\n", "+ main changes:\n", " 1. `get_variable` => `tf.Variable` + scoped factory function\n", " 2. remove `variable_scope` => `name_scope`\n", " - graph.variable_scope_stack => module-global weak dict[graph] = variable_scope_stack\n", " 3. custom_getters: `variable_creator_scope`\n", " 4. tf.assign* will be removed.\n", " \n", "+ feedback:\n", " 1. scope, too magic?\n", " 2. slice: `a[1].assign(5)`\n", " 3. restore from proto (graph def) or other python object: say, tf.Variable.from_proto.\n", " 4. scope和Variable搭配，粒度太粗: 比如ResourceVariableScope下面，能创建PartitionVariable么? 参数不一致下，必须感知，如Resource和Partition。\n", " 5. tf.make_template: 依赖于variable_scope来共享variable，机制要重写？\n", "\n", "+ TODO:\n", " 0. check scoped constructor functions: like `variable_scope.variable`\n", " ```python\n", " # 封装到variable_scope.variable_createor_scope:\n", " with ops.get_default_graph()._variable_creator_scope(custom_creator):\n", " # 封装进tf.Variable:\n", " previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)\n", " for getter in ops.get_default_graph()._variable_creator_stack: # pylint: disable=protected-access\n", " previous_getter = _make_getter(getter, previous_getter)\n", " ```\n", " 1. check implementation: PartitionedVariable, MirroredVariable, _UnreadVariable, CastVariable, etc\n", " 猜测：\n", " + subclass要override __init__方法\n", " + sublcass要搭配对应的creator function\n", " + 使用时要显示配置variable creator scope\n", " ```python\n", " with variable_creator_scope(custom_creator_function):\n", " tf.Variable(args)\n", " ```\n", " 参考RefVariable, ResourceVariable查看\n", " 2. check `tf.Variable(*args, **kwargs)`, use `__init__` or `__call__`.\n", " => metaclass behavior\n", " 3. check low-level implement of RefVariable, ResourceVariable\n", " - 确保读写顺序\n", " - Optimizer don't support Tensor: RefVariable\n", " 4. Checkpointable class and API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: New tf.print #14 ](https://github.com/tensorflow/community/pull/14)\n", "\n", "1. `tf.Print` => `tf.print`, or `tf.strings.format`\n", " - `print`在python 2里是statement，不能重载\n", "2. identity op => depensencies\n", " ```python\n", " t = xxx\n", " with tf.control_dependencies([tf.print(xxxx)]):\n", " t_2 = 2 * t\n", " ```\n", "3. support log level.\n", "4. stateful => ensure order, and not being pruned.\n", "5. support nested structure.\n", "\n", "problem:\n", "+ c++ standard out / error => notebook kernel's console output\n", "+ device: cpu:0 by default (avoid GPU crash)\n", "+ 使用仍然不如常见的print直觉" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: Unify RNN interface #15 ](https://github.com/tensorflow/community/pull/15/)\n", "\n", "Unify TF RNN and Keras RNN: port functionalities from TF RNN to Keras.\n", "\n", "+ 统一API：RNN Layer; RNN Cell\n", " 1. gate order: 统一， [IFCO](https://github.com/keras-team/keras/blob/f534cb0c459ce2a8384e3bc0375eb1cbf80edb56/keras/layers/recurrent.py#L1899), ICFO\n", "+ RNN Cell in tf.contrib.rnn: 只迁移少部份\n", "+ NVidia CuDNN: 性能好，但功能受限制\n", " ```python\n", " if activation == 'tan' and dropout == 0 and use_bias == True: \n", " self.could_use_cudnn = True\n", " else:\n", " self.could_use_cudnn = False\n", " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: TensorFlow API symbols and namespaces #16 ](https://github.com/tensorflow/community/pull/16)\n", "\n", "+ namespace, endpoint\n", "+ @tf_export: TODO: 弄清机制\n", " ```python\n", " @tf_export('keras.metrics.mean_squared_error',\n", " 'keras.metrics.mse',\n", " 'keras.metrics.MSE',\n", " 'keras.losses.mean_squared_error',\n", " 'keras.losses.mse',\n", " 'keras.losses.MSE')\n", " def mean_squared_error(y_true, y_pred):\n", " return K.mean(math_ops.square(y_pred - y_true), axis=-1)\n", " ```\n", "+ additional namespaces\n", " - tf.losses => tf.keras.losses\n", " - tf.metrics => tf.keras.metrics\n", " - tf.layers => tf.keras.layers\n", "+ deprecated namespaces\n", " - tf.logging\n", " - tf.manip" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: Deprecate Collections #17 ](https://github.com/tensorflow/community/pull/17)\n", "\n", "常用三种用法：\n", "1. 搜集变量或者op => 各自追踪\n", "2. 利用它来序列化数据：SaveModel\n", "3. 利用它来维持状态：SharedEmbeddingColumns => 全局变量\n", "\n", "queue => tf.data\n", "\n", "variable\n", "\n", "```python\n", "class VariableTracker(object):\n", " def __init__(self):\n", " self.variables = []\n", "\n", " def variable_tracker(self, next_creator, **kwargs):\n", " v = next_creator(**kwargs)\n", " self.variables.append(v)\n", " return v\n", "\n", "VariableTracker tracker\n", "with tf.variable_creator_scope(tracker.variable_tracker):\n", " ...\n", " a = tf.Variable(0)\n", " ...\n", "\n", "assert tracker.variables == [a]\n", "```\n", "\n", "update\n", "\n", "```python\n", "def model_fn(features, labels, mode, params, config):\n", " logits = ...\n", " batch_norm = tf.BatchNormalization(momentum=0.99)\n", " logits = batch_norm(logits)\n", "\n", " train_op = …\n", " train_op = tf.group(train_op, *batch_norm.updates)\n", " return tf.EstimatorSpec(..., train_op=train_op, ...)\n", "```\n", "\n", "Table: 还未确定\n", "- keras: track_table\n", "- table_creator_scope\n", "- tf.data: iterator, 初始化耗时serving\n", "\n", "Summary, Condition => V2 design" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: Sunset tf.contrib #18 ](https://github.com/tensorflow/community/pull/18)\n", "\n", "处理：\n", "1. moving to core: symbols should be prefixed with `experimental`. \n", "2. moving to a seperate repository\n", " + tensorflow/addons: layer, metric, loss, optimizer, op or kernel\n", " \n", " 人员组成\n", " - Google\n", " - Martin Wicke: tf owner\n", " - Edd Wilder-James: administrative questions\n", " - Karmel Allison: TensorFlow technical contact\n", " - Scott Zhu: contrib.recurrent and contrib.rnn\n", " - Goldie Gadde\n", " - Community\n", " - Sean Morgan: project lead\n", " - Armando Fandango\n", " - Andreas Madsen: sparsemax\n", " - Facai Yan\n", " - Srivatsa P\n", " \n", " 10月底会有第一次kickoff\n", " + tensorflow/network\n", " + tensorflow/scientific\n", "3. deleting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [ RFC: Functions, not sessions in 2.0 #20 ](https://github.com/tensorflow/community/pull/20)\n", "\n", "+ tf.contrib.eager.defun 或者 Defun class\n", "\n", "obejctive:\n", "+ graph + session => function\n", "+ state: tf object == python object\n", "+ export: Checkpoint\n", "+ eager execution by default\n", "\n", "```python\n", "import tensorflow as tf\n", "\n", "@tf.function\n", "def compute_z1(x, y):\n", " return tf.add(x, y)\n", "\n", "@tf.function\n", "def compute_z0(x):\n", " return compute_z1(x, tf.square(x))\n", "\n", "z0 = compute_z0(2.)\n", "z1 = compute_z1(2., 2.)\n", "```\n", "\n", "机制：@tf.function: function => class: trace_cache_key 缓存graph\n", "\n", "特性：\n", "1. For `W`, `b`, and `c`, the lifetime of the Python objects and the runtime state are tied together.\n", " ```python\n", " W = tf.Variable(\n", " tf.glorot_uniform_initializer()(\n", " (10, 10)))\n", " b = tf.Variable(tf.zeros(10))\n", " c = tf.Variable(0)\n", " \n", " @tf.function\n", " def f(x):\n", " c.assign_add(1)\n", " return tf.matmul(x, W) + b\n", " \n", " print(f(make_input_value())\n", " assert int(c) == 1\n", " ```\n", "2. automatically insert control dependencies to ensure stateful operations follow graph construction order.\n", " - the intention here is to avoid observable differences from program order\n", " ```python\n", " a = tf.Variable(1.0)\n", " b = tf.Variable(1.0)\n", " @tf.function\n", " def f():\n", " a.assign(2.0)\n", " b.assign(3.0)\n", " return a + b\n", " print(f())\n", " ```\n", "3. cache graph:\n", " - input argument: type, shape, dtype\n", " - \"context\": eg: device\n", " - Too many traces\n", " ```python\n", " @tf.function\n", " def f(x):\n", " return tf.square(x)\n", " \n", " f(tf.constant(1, dtype=tf.int32))\n", " f(tf.constant(1.0, dtype=tf.float32))\n", " f(2.0) # use tf.constant instead.\n", " f(3.0)\n", " ```\n", " 1. gc + weak reference\n", " 2. input sigatures\n", " ```\n", " @tf.function(input_signature=((tf.float32, [None]))\n", " def f(x): return tf.add(x, 1.)\n", " ```\n", " 3. log an error: call / total >= threshold\n", "4. class\n", " ```python\n", " class ScalarModel(object):\n", " def __init__(self):\n", " self.v = tf.Variable(0)\n", " \n", " @tf.function\n", " def increment(self, amount):\n", " self.v.assign_add(amount)\n", " ```\n", "5. tf.compat.v1.wrap_function: Transitioning from 1.x\n", "6. Serialization: Exporting SavedModel/GraphDefs\n", " - `graph = f.graph_function((tf.float32, (None, 10)).graph`\n", "7. Derived/Related Graphs\n", " ```python\n", " @tf.function\n", " def f(x):\n", " return tf.square(x)\n", " \n", " @tf.function\n", " def g(x):\n", " return tf.square(f(x))\n", " \n", " g(2.0) # 16.0\n", " ```\n", "8. Distributed Execution: details will be discussed separately\n", " \n", " \n", "限制：\n", "1. 只能初始化变量一次（后面是reuse):\n", " - weak references to these created variables\n", " - local tf.Variable in function: not supported yet => investigated independently.\n", "2. Python control flow: data-dependent control flow, not supported => [AutoGraph](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/python/autograph)\n", " ```python\n", " def f(x, y):\n", " if tf.equal(y, 0.0):\n", " return y\n", " return x / y\n", " ```\n", "3. summaries\n", "4. Supporting structured inputs?\n", "\n", "其他：\n", "1. eager executution: symbolic Tensor inside function, while concrete Tensor object outside function.\n", "\n", "feedback:\n", "1. 引入太多复杂性\n", "\n", "TODO:\n", "1. 查看Defun class实现细节" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.2" } }, "nbformat": 4, "nbformat_minor": 2 }