{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Docker提交\n", "\n", "本次竞赛的Docker提交大致可以分为两小块：\n", "\n", "1. 线下文件准备好：包括DockerFile，代码，预测的代码；\n", "2. Build同时pull提交\n", "\n", "如果之前没有提交过docker，可以根据这篇教程熟悉一下：https://tianchi.aliyun.com/forum/postDetail?spm=5176.12586969.1002.9.51df4127FoZKeL&postId=165595 \n", "\n", "\n", "## 线下文件准备\n", "\n", "### Requirement \n", "\n", "- 运行代码所依赖的python库，缺什么就把需要装的文件放在requirement下面" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "numpy\n", "tensorflow==2.2.0 " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 运行的代码\n", "\n", "#### 放在code下面即可" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import tensorflow as tf\n", "import tensorflow.keras.backend as K\n", "from tensorflow.keras.layers import *\n", "from tensorflow.keras.models import *\n", "from tensorflow.keras.optimizers import *\n", "from tensorflow.keras.callbacks import *\n", "from tensorflow.keras.layers import Input \n", "import numpy as np\n", "import os\n", "import zipfile\n", "\n", "def RMSE(y_true, y_pred):\n", " return tf.sqrt(tf.reduce_mean(tf.square(y_true - y_pred)))\n", "\n", "def build_model(): \n", " inp = Input(shape=(12,24,72,4)) \n", " \n", " x_4 = Dense(1, activation='relu')(inp) \n", " x_3 = Dense(1, activation='relu')(tf.reshape(x_4,[-1,12,24,72]))\n", " x_2 = Dense(1, activation='relu')(tf.reshape(x_3,[-1,12,24]))\n", " x_1 = Dense(1, activation='relu')(tf.reshape(x_2,[-1,12]))\n", " \n", " x = Dense(64, activation='relu')(x_1) \n", " x = Dropout(0.25)(x) \n", " x = Dense(32, activation='relu')(x) \n", " x = Dropout(0.25)(x) \n", " output = Dense(24, activation='linear')(x) \n", " model = Model(inputs=inp, outputs=output)\n", "\n", " adam = tf.optimizers.Adam(lr=1e-3,beta_1=0.99,beta_2 = 0.99) \n", " model.compile(optimizer=adam, loss=RMSE)\n", "\n", " return model \n", "\n", "model = build_model()\n", "model.load_weights('./user_data/model_data/model_mlp_baseline.h5')\n", "\n", "test_path = './tcdata/enso_round1_test_20210201/'\n", "\n", "### 1. 测试数据读取\n", "files = os.listdir(test_path)\n", "test_feas_dict = {}\n", "for file in files:\n", " test_feas_dict[file] = np.load(test_path + file)\n", " \n", "### 2. 结果预测\n", "test_predicts_dict = {}\n", "for file_name,val in test_feas_dict.items():\n", " test_predicts_dict[file_name] = model.predict(val).reshape(-1,)\n", "# test_predicts_dict[file_name] = model.predict(val.reshape([-1,12])[0,:])\n", "\n", "### 3.存储预测结果\n", "for file_name,val in test_predicts_dict.items(): \n", " np.save('./result/' + file_name,val)\n", "\n", "#打包目录为zip文件（未压缩）\n", "def make_zip(source_dir='./result/', output_filename = 'result.zip'):\n", " zipf = zipfile.ZipFile(output_filename, 'w')\n", " pre_len = len(os.path.dirname(source_dir))\n", " source_dirs = os.walk(source_dir)\n", " print(source_dirs)\n", " for parent, dirnames, filenames in source_dirs:\n", " print(parent, dirnames)\n", " for filename in filenames:\n", " if '.npy' not in filename:\n", " continue\n", " pathfile = os.path.join(parent, filename)\n", " arcname = pathfile[pre_len:].strip(os.path.sep) #相对路径\n", " zipf.write(pathfile, arcname)\n", " zipf.close()\n", "make_zip() " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### run.sh\n", "\n", "- 运行预测的代码" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#!/bin/sh\n", "CURDIR=\"`dirname $0`\" #获取此脚本所在目录\n", "echo $CURDIR\n", "cd $CURDIR #切换到该脚本所在目录\n", "python /code/mlp_predict.py" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### DockerFile" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Base Images\n", "## 从天池基础镜像构建 \n", "FROM registry.cn-shanghai.aliyuncs.com/tcc-public/tensorflow:latest-cuda10.0-py3\n", "\n", "## 把当前文件夹里的文件构建到镜像的根目录下（.后面有空格，不能直接跟/）\n", "ADD . /\n", "\n", "## 指定默认工作目录为根目录（需要把run.sh和生成的结果文件都放在该文件夹下，提交后才能运行）\n", "WORKDIR /\n", "\n", "## Install Requirements（requirements.txt包含python包的版本）\n", "## 这里使用清华镜像加速安装\n", "RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple --upgrade pip\n", "RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt\n", "#RUN pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt\n", "\n", "## 镜像启动后统一执行 sh run.sh\n", "CMD [\"sh\", \"run.sh\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 其它\n", "\n", "- 按照官方要求把所需的文件全部按要求准备好即可。" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 线上提交\n", "\n", "在所有的文件都准备之后，下面一步就是进行线上的提交，这里又分为三块。\n", "\n", "1. 按照要求进行线上配置\n", "2. 进行build和pull；\n", "3. 提交" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 按照要求进行线上配置\n", "\n", "\"first_step\"\n", "\n", "\\\\ \n", "\n", "\"first_step\"\n", "\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 进行build和pull；\n", "\n", "\n", "\"first_step\"\n", "\\\\\n", "\"first_step\"\n", "\\\\\n", "\"first_step\"\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#### 1.登录\n", "sudo docker login --username=\"自己的用户名\" registry.cn-shenzhen.aliyuncs.com\n", "#### 2.build\n", "docker build registry.cn-shenzhen.aliyuncs.com/ai_earth_baseline/test_ai_earth_submit:1.0 .\n", "#### 3.push\n", "docker push registry.cn-shenzhen.aliyuncs.com/ai_earth_baseline/test_ai_earth_submit:1.0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 提交\n", "\n", "\n", "\"first_step\"\n", "\n", "根据自己的不同进行提交即可，如果不出意外，等待一会儿，线上跑完了就会有结果了。\n", "\n", "\"first_step\"\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.6" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "288px" }, "toc_section_display": true, "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 4 }