{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"# What's under your hood?\n",
"\n",
"This tutorial is not about math methods and models.\n",
"I'll focus on the environment for learning ML\n",
"\n",
"Learning process is a hard way. And there are many factors that got influence on this process. Your mood often make big sense.
\n",
"And in self-study programms motivation and attitude - your main friends. \n",
"## Let's start. \n",
"Now there will be a bit of philosophy, and I ask you don't scroll to the **Technical part**. There will be some big ideas.\n",
"\n",
"Folks that came over Tutorials in this session are already definitely strong guys and knows a lot.\n",
"\n",
"I think that this notebook (maybe an article) will be more useful for the next session patients. А lot of tutorials from the previous course played an important role for me.\n",
"\n",
"Why do I think that this course is the best of the best? Two things. First - excellent presentation of the material. And the second is - mechanism of motivation. It is awesome. \n",
"**COMPETITIONS**. The competitive process gives a rise to the best decisions and motivates the participants. So it was with me.\n",
"\n",
"But, what if the new feature for regression no longer come to mind\n",
"
\n",
"
\n",
"\n",
"
\n",
"
\n",
"And Medium-competition has already killed the kernel on Google Colab for several times. Your computer becomes a brick?\n",
"\n",
"
\n",
"
\n",
"At the same time, a guy (or girl) from the other end of the world successfully hits the baselines and goes to the top 1%. So come on a piece of iron! Let's try to tune some parameters of the model! \n",
"\n",
"Restarting. Oh sh*t...I will go to sleep, Tomorrow I will optimize:\n",
" - [ ] code\n",
" - [ ] calculation\n",
" - [ ] hardware\n",
" - [ ] brain \n",
"\n",
"*Forget it*\n",
"\n",
" - Try out a new approach (*hello n-gramm in td-idf*). \n",
" - Look if there will be a gain in cross-validation by adding Alice’s time spent on YouTube. \n",
" \n",
" Both of them takes time, which directly (almost always) depends on the hardware which you launching your model.\n",
"\n",
"If you also discovered the world of gradient boostings \n",
"\n",
" \n",
"## Options\n",
"\n",
"I have an elderly laptop with i5 and 4 GB RAM. My brain is faster than him sometimes.\n",
"\n",
"I was annoyed by his long freezes on training of the model or the memory error in challenge of beating the baselines in this course.\n",
"\n",
"### What I can do:\n",
"\n",
"\t\n",
"\n",
" 1. **The first variant:**\n",
"\n",
"\tBuy a new hardware. What we have there, oh yes:\n",
"\n",
"\t\n",
"\n",
" - I7-8700k 400 \\$\n",
" - 1080TI 1000 \\$\n",
" - 64 GB RAM = in about 500 \\$\n",
" - motherboard\n",
" - ssd disk and so on...\n",
"\n",
"\t\tStop-stop, I'm just learning ...\n",
"\n",
"2. **Option 2:** \n",
"\n",
"\tСode or computing optimization. A good choice, the right groundwork for the future.\n",
" But deadlines-work-study-family... where to find time for all of this?\n",
"\tPlus, I spend a lot of time in transport (metro-train-transfer) and I want to spend my time effectively. \n",
" In addition, good idea often comes in the most unexpected places and I want to immediately test it.\n",
"
\n",
"
\n",
"3. **From the above, a third option appears** - take a strong hardware in the cloud. And connect with it from any device, always having a powerful beast under the hood. *\"A rom dom dom\" NFS Underground OST.* \n",
"\n",
"\n",
"## Google Compute Engine.\n",
"\n",
"In fact, there are many companies whom offer resources for cloud computing. An important aspect for me is that the service must by free of charge, flexy and completeness of the opportunities provided. Therefore, I've choosed Google.\n",
"\n",
"On light computations you can work with Colab or Kaggle Kernels. But there is limit of the memory.\n",
"
\n",
"
\n",
"\n",
"\n",
"There may be several reasons. The most often is a lack of computing resources (memory). And the most annoying thing is that all cells now need to be restarted. When you save the resulting objects in txt or pickle you are the winner. But only practice makes perfect. At the beginning of the learning path, such restarts of the kernel can, in the first place, enrage. Secondly, it takes a lot of time.\n",
"\n",
"For now we going to use of a nice gift from Google in the amount of *300\\$.*\n",
"\n",
"# Technical part\n",
"What you need to start:\n",
"\n",
" 1. **Register account** *(interesting, how many of you, dear readers, doesn’t have it yet?)*\n",
"\n",
"There is nothing complicated, everything is standard. Mail and phone confirmation. Specify a real phone number it will be used for billing confirmation.\n",
"\n",
" 2. **Let's go on** [https://console.cloud.google.com/freetrial/signup](https://console.cloud.google.com/freetrial/signup)\n",
"\n",
"You'll see that they give us 300\\$ and 12 months. I note that my account decreased by only 10.53 \\$ during this course from Professor Yorko and his colleagues.\n",
"You can safely confirm the card (system will block $ 1 on it for one hour in average)\n",
"\n",
"3. **Jump to the** [https://console.cloud.google.com/compute/](https://console.cloud.google.com/compute/)\n",
"\n",
"- #### Create a project.\n",
"\n",
"- #### The most interesting part starts now.\n",
"\n",
"Open the hardware shop (or rental).\n",
"
\n",
"
\n",
"***Fill the power?!***\n",
"\n",
"- #### Let's create new instance!\n",
"Take more powerful. 8 CPU and 52 GB RAM.\n",
"\n",
"\tI think there is no point for my instance to take the power of the GPU. Because neural networks in this course are not considered. And if you want - there is a free GPU in Google Colab. But some of the boosting libraries are now supports GPU. Therefore, keep a close eye on the changes and choose the parameters of the instance based on your needs.\n",
"\n",
"*I felt an urgent need for additional resources when I started using XGBOOST, the documentation for it says that there is support for the GPU, but when I started on the Colab GPUs, it began to kill the kernels.*\n",
"\n",
"- #### Boot Disk change to Ubuntu 16.04. But it is for your choice\n",
"![enter image description here](https://lh3.googleusercontent.com/O81SHvcQlhYKQcx8K4iXBLgvvrUWHkUMY-uY3NJDVETDTnN1YpApo8twOlwQs_0oRoDaakFYL4mT \"500\")\n",
"\n",
"- #### Configure the firewall and remove the checkbox from \"Disks\" tab.\n",
"\n",
"\n",
"- #### Press a create button - and voila - we have a strong hardware to ride with.\n",
"\n",
"*But wait a minute. You'll need to set a static IP adress:*\n",
"\n",
"- #### Go to the VPC Network - Extrenal IP Addresses. Change it to Static.\n",
"\n",
"\n",
"\n",
"- #### Ok. Next step - tune Firewall settings - Firewall rules. Add one new.\n",
"\n",
"\n",
"\n",
"- #### Connect with instance by SSH.\n",
"\n",
"\n",
"- #### Let's install Anaconda\n",
"\n",
"\n",
"> wget http://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh\n",
"> \n",
"> bash Anaconda3-5.0.1-Linux-x86_64.sh\n",
"\n",
"Read the rules, review and run the installation.\n",
"- Now we will make the *config*, and open a port for tcp connection.\n",
"\n",
" \n",
"> jupyter notebook --generate-config\n",
"> \n",
"> nano ~/.jupyter/jupyter_notebook_config.py\n",
"\n",
"- Add this lines. Or search and uncomment existing.\n",
"\n",
"> c.NotebookApp.ip = '*'\n",
"> \n",
"> c.NotebookApp.open_browser = False\n",
"> \n",
"> c.NotebookApp.port = 8000\n",
"\n",
"- Lunch Jupyter\n",
"\n",
"> Jupyter notebook --ip=0.0.0.0\n",
"\n",
"- To see the usual Jupyter main page - launch the browser and go to:\n",
"\n",
"> https://\"StaticIP\":\"Port\"/?token=YOUR_TOKEN\n",
">\n",
"\n",
"\"StaticIP\" - External IP of instance\n",
"
\n",
"\"Port\"= 8000 (tcp port that we allowed in Firewall )\n",
"
\n",
"Token- you'll see it in terminal\n",
"
\n",
"\n",
"## You are in business.\n",
"\n",
"All the necessary libraries can be installed by run the following command.\n",
"\n",
"> ! pip install lightGBM\n",
"\n",
"Let's see what's under the hood:\n",
"\n",
"> !cat /proc/cpuinfo\n",
"> \n",
"\n",
"There are 8 pcs like this\n",
"> \n",
"> !cat /proc/meminfo\n",
"\n",
"\n",
"*Not bad, it's time to ride on*\n",
"\n",
"\n",
"\n",
"## Kaggel API\n",
"\n",
"- Make dir for token:\n",
"\n",
"> !mkdir -p ~/.kaggle\n",
"> \n",
"- Take the json from here:\n",
"https://www.kaggle.com/\"YourAccountName\"/account\n",
"![\n",
"](https://lh3.googleusercontent.com/ey8NCFmjKvBy9GpyXonQT4WO7vStWdkwsd3wq1O_2hn8cHTLmrqy0xHg4y_KmArxHfrdkrofqQWf \"700\")\n",
"\n",
"\n",
"- And put it here:\n",
"\n",
"\n",
"- Copy to the \"kaggle\" directory:\n",
"\n",
"> !cp kaggle.json ~/.kaggle/\n",
"\n",
"*Now you can download/send files from Kaggle. And it very fast*\n",
"\n",
" - Download the Medium Competition dataset\n",
"*Make new dir for Medium*\n",
"\n",
"> !mkdir Medium\n",
"\n",
" - Change os dir for easy access to files\n",
"\n",
"> import os\n",
"> os.chdir('Medium')\n",
"\n",
" - Start downloading\n",
"\n",
"> !kaggle competitions download -c how-good-is-your-medium-article.\n",
"\n",
"## Now about the speed.\n",
"\n",
"The same dataset, the same code. Look at the time:\n",
" #### Google Cloud\t \n",
"\n",
"  \n",
"\n",
"#### My Laptop  \n",
"*Fine.*\n",
"\n",
"## Git\n",
"\n",
"- By the way you can easily clone your repositories from github.\n",
"\n",
"> !git clone\n",
"> [https://github.com/mikhailsergeevi4/crm.git](https://github.com/mikhailsergeevi4/crm.git)\n",
"\n",
"- Commit\n",
"\n",
"- And push them back.\n",
"\n",
"> !git push origin master\n",
"\n",
"There are many other useful tricks about Google Cloud and Colab. But I think it's enough for this time\n",
"\n",
"As a result, you get a powerful machine for study ML.\n",
"\n",
"Learn, or work, or make presentations of your notebook from any device.\n",
"\n",
"Presented 300$ is enough to complete the course. I've used instances mostly for high resource operations. For simple things, there is Google Colab (with a free video card, by the way)\n",
"\n",
"**IMPORTANT: don't forget to stop the instance when you finished. This machine will quickly devour the allocated budget!**\n",
""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}