{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Trying out SWAN\n", "\n", "Why does it open notebooks in the same tab? How do I get back to the home directory?\n", "\n", "Workaround is to right click on notebooks and open in a new tab, but it's very annoying.\n", "\n", "When I create a new notebook it seems to open ok in a new tab.\n", "\n", "Also, WHY IS IT SO SLOW?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What packages are here?" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "AGILe==1.4.1\n", "astroid==1.3.6\n", "backports.lzma==0.0.6\n", "backports.shutil-get-terminal-size==1.0.0\n", "backports.ssl-match-hostname==3.4.0.2\n", "bleach==2.0.0\n", "certifi==14.5.14\n", "cmmnbuild-dep-manager==2.1.2\n", "configparser==3.5.0\n", "CouchDB==1.0.1\n", "coverage==4.4.1\n", "cx-Oracle==5.1.1\n", "cycler==0.10.0\n", "Cython==0.23.4\n", "decorator==4.0.9\n", "dill==0.2.5\n", "Django==1.9.12\n", "elasticsearch==2.3.0\n", "entrypoints==0.2.2\n", "funcsigs==1.0.2\n", "future==0.15.2\n", "Genshi==0.7\n", "glite-wms-wmproxy-api-python==3.4.0\n", "gosam===2.0.4-12f4de9\n", "grpcio==1.2.0\n", "h5py==2.6.0\n", "hepdata-converter==0.1.22\n", "hepdata-validator==0.1.8\n", "hspy==1.6.4\n", "html5lib==0.999999999\n", "ipykernel==4.6.1\n", "ipython==5.4.1\n", "ipython-genutils==0.1.0\n", "ipywidgets==5.2.2\n", "Jinja2==2.7.3\n", "joblib==0.10.0\n", "JPype1==0.6.2\n", "jsonschema==2.4.0\n", "jupyter==1.0.0\n", "jupyter-client==4.3.0\n", "jupyter-console==5.0.0\n", "jupyter-contrib-core==0.3.0\n", "jupyter-contrib-nbextensions==0.2.1\n", "jupyter-core==4.2.0\n", "jupyter-nbextensions-configurator==0.2.2\n", "Keras==1.1.0\n", "LHAPDF==6.2.0\n", "logilab-common==1.0.1\n", "lxml==2.3\n", "M2Crypto==0.25.1\n", "MarkupSafe==0.23\n", "matplotlib==1.5.1\n", "messaging==1.0\n", "metakernel==0.13.1\n", "mistune==0.5.1\n", "mock==2.0.0\n", "multiprocess==0.70.4\n", "nbconvert==5.1.1\n", "nbformat==4.0.1\n", "networkx==1.11\n", "nose==1.1.2\n", "notebook==4.2.1\n", "numexpr==2.4.3\n", "numpy==1.11.0\n", "pandas==0.23.4\n", "pandocfilters==1.4.1\n", "pathlib2==2.1.0\n", "pathos==0.2.0\n", "patsy==0.4.1\n", "pbr==2.0.0\n", "pexpect==4.2.0\n", "pickleshare==0.7.2\n", "pox==0.2.2\n", "ppft==1.6.4.6\n", "prettytable==0.7.2\n", "prompt-toolkit==1.0.3\n", "protobuf==3.2.0\n", "psutil==4.3.1\n", "ptyprocess==0.5.1\n", "py==1.4.8\n", "py2neo==1.4.6\n", "py4j==0.10.3\n", "pydot==1.2.3\n", "pydot-ng==1.0.0\n", "Pygments==2.0.2\n", "PyHEADTAIL==1.12.1\n", "pyjapc==1.4.1\n", "pylint==1.4.4\n", "pyparsing==2.1.8\n", "pystan==2.16.0.0\n", "pytest==2.2.4\n", "python-dateutil==2.8.0\n", "python-gitlab==0.20\n", "pytimber==2.4.2\n", "pytz==2015.4\n", "PyYAML==3.11\n", "pyzmq==14.5.0\n", "qmtest==2.4.1\n", "qtconsole==4.2.1\n", "requests==2.10.0\n", "rivet==2.5.4\n", "root-numpy==4.6.0\n", "rootpy==0.8.1\n", "rpy2==2.8.0\n", "scikit-learn==0.18.1\n", "scipy==0.18.1\n", "seaborn==0.7.1\n", "simplegeneric==0.8.1\n", "simplejson==3.8.2\n", "singledispatch==3.4.0.3\n", "six==1.10.0\n", "SQLAlchemy==0.7.7\n", "statsmodels==0.8.0\n", "stomp.py==3.1.3\n", "sympy==0.7.6\n", "tensorflow==1.0.0\n", "terminado==0.5\n", "testpath==0.3.1\n", "Theano==0.8.2\n", "tornado==4.0.2\n", "traitlets==4.2.2\n", "troveharvester==0.1.13\n", "typing==3.5.2.2\n", "unicodecsv==0.14.1\n", "urllib3==1.14\n", "wcwidth==0.1.7\n", "webencodings==0.5.1\n", "widgetsnbextension==1.2.6\n", "yoda==1.6.7\n", "\u001b[33mYou are using pip version 8.1.1, however version 19.0.3 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "!pip freeze" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, so it looks like all the SciPy type packages are there as you would expect — including Keras and Tensorflow. But no image processing, no BeautifulSoup (but lxml is there), no Altair/Vega for visualisations..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Can I install new packages?\n", "\n", "I can't access a terminal, but pip works. Note the `--user` to install into my own account.\n", "\n", "Note that these additions persist between sessions, ie. no need to `pip` install them more than once. Of course that also means that you can't have different versions, as there's doesn't seem to be a way to have virtual environments." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting troveharvester\n", "Requirement already satisfied (use --upgrade to upgrade): requests in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from troveharvester)\n", "Collecting unicodecsv (from troveharvester)\n", "Installing collected packages: unicodecsv, troveharvester\n", "Successfully installed troveharvester unicodecsv\n", "\u001b[33mYou are using pip version 8.1.1, however version 18.1 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "!pip install --user troveharvester" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: troveharvester [-h] {start,restart,report} ...\n", "\n", "positional arguments:\n", " {start,restart,report}\n", " start Start a new harvest\n", " restart Restart an unfinished harvest\n", " report Report on a harvest\n", "\n", "optional arguments:\n", " -h, --help show this help message and exit\n" ] } ], "source": [ "# The two dashes '--' are necessary to identify the arguments\n", "%run -m troveharvester -- -h" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Can I use git?\n", "\n", "Again, no terminal access, but you seem to able to use shell commands ok." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: git [--version] [--exec-path[=GIT_EXEC_PATH]] [--html-path]\r\n", " [-p|--paginate|--no-pager] [--no-replace-objects]\r\n", " [--bare] [--git-dir=GIT_DIR] [--work-tree=GIT_WORK_TREE]\r\n", " [--help] COMMAND [ARGS]\r\n", "\r\n", "The most commonly used git commands are:\r\n", " add Add file contents to the index\r\n", " bisect Find by binary search the change that introduced a bug\r\n", " branch List, create, or delete branches\r\n", " checkout Checkout a branch or paths to the working tree\r\n", " clone Clone a repository into a new directory\r\n", " commit Record changes to the repository\r\n", " diff Show changes between commits, commit and working tree, etc\r\n", " fetch Download objects and refs from another repository\r\n", " grep Print lines matching a pattern\r\n", " init Create an empty git repository or reinitialize an existing one\r\n", " log Show commit logs\r\n", " merge Join two or more development histories together\r\n", " mv Move or rename a file, a directory, or a symlink\r\n", " pull Fetch from and merge with another repository or a local branch\r\n", " push Update remote refs along with associated objects\r\n", " rebase Forward-port local commits to the updated upstream head\r\n", " reset Reset current HEAD to the specified state\r\n", " rm Remove files from the working tree and from the index\r\n", " show Show various types of objects\r\n", " status Show the working tree status\r\n", " tag Create, list, delete or verify a tag object signed with GPG\r\n", "\r\n", "See 'git help COMMAND' for more information on a specific command.\r\n" ] } ], "source": [ "!git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Git cloning a GitHub repository seems to work as expected." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "error: could not commit config file /webdav/trove-newspaper-harvester/.git/config\n", "Initialized empty Git repository in /webdav/trove-newspaper-harvester/.git/\n", "error: could not commit config file /webdav/trove-newspaper-harvester/.git/config\n", "error: could not commit config file /webdav/trove-newspaper-harvester/.git/config\n", "remote: Enumerating objects: 22, done.\u001b[K\n", "remote: Counting objects: 100% (22/22), done.\u001b[K\n", "remote: Compressing objects: 100% (15/15), done.\u001b[K\n", "remote: Total 34 (delta 10), reused 17 (delta 7), pack-reused 12\u001b[K\n", "Unpacking objects: 100% (34/34), done.\n" ] } ], "source": [ "!git clone https://github.com/GLAM-Workbench/trove-newspaper-harvester.git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installing from `requirements.txt`\n", "\n", "So a typical thing I'd want to do is clone a repository and then install all the necessary packages from a `requirements.txt` file." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/webdav/trove-newspaper-harvester\n" ] } ], "source": [ "cd trove-newspaper-harvester" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting altair==2.2.2 (from -r requirements.txt (line 1))\n", " Downloading https://files.pythonhosted.org/packages/a4/02/d0ad006918024f8c1509a7f76e94e5e3bdb102ed58da160d3dcfb621fc33/altair-2.2.2-py2.py3-none-any.whl (387kB)\n", "\u001b[K 100% |████████████████████████████████| 389kB 2.7MB/s eta 0:00:01\n", "\u001b[?25hCollecting ipyleaflet==0.9.0 (from -r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/c4/fe/01839b1cdcc4bf8c8424f3f1ddf50f9e4fbee2cb33b52645a4d2be28432c/ipyleaflet-0.9.0-py2.py3-none-any.whl (906kB)\n", "\u001b[K 100% |████████████████████████████████| 911kB 1.2MB/s eta 0:00:01\n", "\u001b[?25hCollecting pandas==0.23.4 (from -r requirements.txt (line 3))\n", " Downloading https://files.pythonhosted.org/packages/5d/d4/6e9c56a561f1d27407bf29318ca43f36ccaa289271b805a30034eb3a8ec4/pandas-0.23.4-cp35-cp35m-manylinux1_x86_64.whl (8.7MB)\n", "\u001b[K 100% |████████████████████████████████| 8.7MB 168kB/s eta 0:00:01\n", "\u001b[?25hCollecting textblob==0.15.1 (from -r requirements.txt (line 4))\n", " Downloading https://files.pythonhosted.org/packages/11/18/7f55c8be6d68ddc4036ffda5382ca51e23a1075987f708b9123712091af1/textblob-0.15.1-py2.py3-none-any.whl (631kB)\n", "\u001b[K 100% |████████████████████████████████| 634kB 1.8MB/s eta 0:00:01\n", "\u001b[?25hCollecting troveharvester==0.2.2 (from -r requirements.txt (line 5))\n", " Downloading https://files.pythonhosted.org/packages/00/12/57284df3a8555ce2e227058017a631b6ea142be698a92e32a58bdb3a299a/troveharvester-0.2.2-py3-none-any.whl\n", "Collecting vega==1.4.0 (from -r requirements.txt (line 6))\n", " Downloading https://files.pythonhosted.org/packages/0a/ce/1ab1bf943f9b80f9e87d807a29337daff93cb64c7080b37633f4655b8c21/vega-1.4.0.tar.gz (1.1MB)\n", "\u001b[K 100% |████████████████████████████████| 1.1MB 1.1MB/s eta 0:00:01\n", "\u001b[?25hCollecting wordcloud==1.5.0 (from -r requirements.txt (line 7))\n", " Downloading https://files.pythonhosted.org/packages/5e/b7/c16286efa3d442d6983b3842f982502c00306c1a4c719c41fb00d6017c77/wordcloud-1.5.0-cp35-cp35m-manylinux1_x86_64.whl (357kB)\n", "\u001b[K 100% |████████████████████████████████| 358kB 2.9MB/s eta 0:00:01\n", "\u001b[?25hCollecting scipy==1.1.0 (from -r requirements.txt (line 8))\n", " Downloading https://files.pythonhosted.org/packages/cd/32/5196b64476bd41d596a8aba43506e2403e019c90e1a3dfc21d51b83db5a6/scipy-1.1.0-cp35-cp35m-manylinux1_x86_64.whl (33.1MB)\n", "\u001b[K 100% |████████████████████████████████| 33.1MB 48kB/s eta 0:00:01\n", "\u001b[?25hCollecting scikit-learn==0.19.2 (from -r requirements.txt (line 9))\n", " Downloading https://files.pythonhosted.org/packages/b6/e2/a1e254a4a4598588d4fe88b45ab88a226c289ecfd0f6c90474eb6a9ea6b3/scikit_learn-0.19.2-cp35-cp35m-manylinux1_x86_64.whl (4.9MB)\n", "\u001b[K 100% |████████████████████████████████| 4.9MB 323kB/s eta 0:00:01\n", "\u001b[?25hCollecting matplotlib==2.2.3 (from -r requirements.txt (line 10))\n", " Downloading https://files.pythonhosted.org/packages/de/af/6258db9b26313dd7ad70dba30a60bec62bf030a44208d4cb62966206666f/matplotlib-2.2.3-cp35-cp35m-manylinux1_x86_64.whl (12.6MB)\n", "\u001b[K 100% |████████████████████████████████| 12.6MB 129kB/s eta 0:00:01 50% |████████████████▎ | 6.4MB 52.2MB/s eta 0:00:01\n", "\u001b[?25hCollecting tqdm==4.29.1 (from -r requirements.txt (line 11))\n", " Downloading https://files.pythonhosted.org/packages/ed/d6/3458d39cf4978f4ece846295e83daf5ece710ab0a4106774f7f7b3a68697/tqdm-4.29.1-py2.py3-none-any.whl (46kB)\n", "\u001b[K 100% |████████████████████████████████| 51kB 5.0MB/s eta 0:00:01\n", "\u001b[?25hCollecting arrow==0.13.0 (from -r requirements.txt (line 12))\n", " Downloading https://files.pythonhosted.org/packages/5d/c7/468bb95a10fb8ddb5f3f80e1aef06b78f64d6e5df958c39672f80581381f/arrow-0.13.0.tar.gz (92kB)\n", "\u001b[K 100% |████████████████████████████████| 102kB 6.2MB/s ta 0:00:01\n", "\u001b[?25hCollecting appmode==0.4.0 (from -r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/9d/c2/544a4cd86892b17f00b0006be323026d3b36e28c0eccabc68c3e9e1fb16b/appmode-0.4.0.tar.gz\n", "Requirement already satisfied (use --upgrade to upgrade): six in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from altair==2.2.2->-r requirements.txt (line 1))\n", "Requirement already satisfied (use --upgrade to upgrade): typing in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from altair==2.2.2->-r requirements.txt (line 1))\n", "Requirement already satisfied (use --upgrade to upgrade): jsonschema in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from altair==2.2.2->-r requirements.txt (line 1))\n", "Collecting toolz (from altair==2.2.2->-r requirements.txt (line 1))\n", " Downloading https://files.pythonhosted.org/packages/14/d0/a73c15bbeda3d2e7b381a36afb0d9cd770a9f4adc5d1532691013ba881db/toolz-0.9.0.tar.gz (45kB)\n", "\u001b[K 100% |████████████████████████████████| 51kB 4.8MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): numpy in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from altair==2.2.2->-r requirements.txt (line 1))\n", "Requirement already satisfied (use --upgrade to upgrade): entrypoints in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from altair==2.2.2->-r requirements.txt (line 1))\n", "Collecting xarray<0.11,>=0.10 (from ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/d2/40/e3d9a6b679a7b4ba80e8cdf0b0226809ef454773f7b79ef1b2bd1898bd48/xarray-0.10.9-py2.py3-none-any.whl (473kB)\n", "\u001b[K 100% |████████████████████████████████| 481kB 2.3MB/s eta 0:00:01\n", "\u001b[?25hCollecting ipywidgets<8,>=7.0.0 (from ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/30/9a/a008c7b1183fac9e52066d80a379b3c64eab535bd9d86cdc29a0b766fd82/ipywidgets-7.4.2-py2.py3-none-any.whl (111kB)\n", "\u001b[K 100% |████████████████████████████████| 112kB 5.9MB/s eta 0:00:01\n", "\u001b[?25hCollecting traittypes<3,>=0.2.1 (from ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/9c/d1/8d5bd662703cc1764d986f6908a608777305946fa634d34c470cd4a1e729/traittypes-0.2.1-py2.py3-none-any.whl\n", "Collecting python-dateutil>=2.5.0 (from pandas==0.23.4->-r requirements.txt (line 3))\n", " Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)\n", "\u001b[K 100% |████████████████████████████████| 235kB 2.9MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): pytz>=2011k in /cvmfs/sft.cern.ch/lcg/releases/pytz/2015.4-e9c1e/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages/pytz-2015.4-py3.5.egg (from pandas==0.23.4->-r requirements.txt (line 3))\n", "Collecting nltk>=3.1 (from textblob==0.15.1->-r requirements.txt (line 4))\n", " Downloading https://files.pythonhosted.org/packages/6f/ed/9c755d357d33bc1931e157f537721efb5b88d2c583fe593cc09603076cc3/nltk-3.4.zip (1.4MB)\n", "\u001b[K 100% |████████████████████████████████| 1.4MB 823kB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): unicodecsv in /webdav/.local/lib/python3.5/site-packages (from troveharvester==0.2.2->-r requirements.txt (line 5))\n", "Requirement already satisfied (use --upgrade to upgrade): requests in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from troveharvester==0.2.2->-r requirements.txt (line 5))\n", "Collecting pillow (from wordcloud==1.5.0->-r requirements.txt (line 7))\n", " Downloading https://files.pythonhosted.org/packages/8b/e9/5c47710fe383f0582da668302a80a6355fe15c2ce2dde89b50fe34acefa6/Pillow-5.4.1-cp35-cp35m-manylinux1_x86_64.whl (2.0MB)\n", "\u001b[K 100% |████████████████████████████████| 2.0MB 576kB/s eta 0:00:01\n", "\u001b[?25hCollecting kiwisolver>=1.0.1 (from matplotlib==2.2.3->-r requirements.txt (line 10))\n", " Downloading https://files.pythonhosted.org/packages/7e/31/d6fedd4fb2c94755cd101191e581af30e1650ccce7a35bddb7930fed6574/kiwisolver-1.0.1-cp35-cp35m-manylinux1_x86_64.whl (949kB)\n", "\u001b[K 100% |████████████████████████████████| 952kB 1.1MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from matplotlib==2.2.3->-r requirements.txt (line 10))\n", "Requirement already satisfied (use --upgrade to upgrade): cycler>=0.10 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from matplotlib==2.2.3->-r requirements.txt (line 10))\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Collecting notebook>=5 (from appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/0a/d8/4e9521354ed3d730ba6d8a5af440b66c73245ef46be706e51bead71afc21/notebook-5.7.6-py2.py3-none-any.whl (9.0MB)\n", "\u001b[K 100% |████████████████████████████████| 9.0MB 168kB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): ipython>=4.0.0; python_version >= \"3.3\" in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", "Requirement already satisfied (use --upgrade to upgrade): ipykernel>=4.5.1 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", "Collecting nbformat>=4.2.0 (from ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/da/27/9a654d2b6cc1eaa517d1c5a4405166c7f6d72f04f6e7eea41855fe808a46/nbformat-4.4.0-py2.py3-none-any.whl (155kB)\n", "\u001b[K 100% |████████████████████████████████| 163kB 5.1MB/s eta 0:00:01\n", "\u001b[?25hCollecting traitlets>=4.3.1 (from ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl (74kB)\n", "\u001b[K 100% |████████████████████████████████| 81kB 6.1MB/s eta 0:00:01\n", "\u001b[?25hCollecting widgetsnbextension~=3.4.0 (from ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", " Downloading https://files.pythonhosted.org/packages/8a/81/35789a3952afb48238289171728072d26d6e76649ddc8b3588657a2d78c1/widgetsnbextension-3.4.2-py2.py3-none-any.whl (2.2MB)\n", "\u001b[K 100% |████████████████████████████████| 2.2MB 498kB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): singledispatch in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nltk>=3.1->textblob==0.15.1->-r requirements.txt (line 4))\n", "Requirement already satisfied (use --upgrade to upgrade): setuptools in /cvmfs/sft.cern.ch/lcg/releases/setuptools/36.0.1-8a405/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib==2.2.3->-r requirements.txt (line 10))\n", "Collecting jupyter-client>=5.2.0 (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/3b/c3/3043fe9ffd140d03c9d091a056794ccdc427c56ec19b8eea74f9ea0a498f/jupyter_client-5.2.4-py2.py3-none-any.whl (89kB)\n", "\u001b[K 100% |████████████████████████████████| 92kB 6.5MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): nbconvert in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Collecting terminado>=0.8.1 (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/2e/20/a26211a24425923d46e1213b376a6ee60dc30bcdf1b0c345e2c3769deb1c/terminado-0.8.1-py2.py3-none-any.whl\n", "Collecting pyzmq>=17 (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/39/7a/9b98aabbde03fc4c5c38079ddec1331af3846d80c0b73bdb691a21adee9c/pyzmq-18.0.1-cp35-cp35m-manylinux1_x86_64.whl (1.0MB)\n", "\u001b[K 100% |████████████████████████████████| 1.1MB 1.1MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): jinja2 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Collecting jupyter-core>=4.4.0 (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/1d/44/065d2d7bae7bebc06f1dd70d23c36da8c50c0f08b4236716743d706762a8/jupyter_core-4.4.0-py2.py3-none-any.whl (126kB)\n", "\u001b[K 100% |████████████████████████████████| 133kB 6.0MB/s eta 0:00:01\n", "\u001b[?25hCollecting Send2Trash (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/49/46/c3dc27481d1cc57b9385aff41c474ceb7714f7935b1247194adae45db714/Send2Trash-1.5.0-py3-none-any.whl\n", "Requirement already satisfied (use --upgrade to upgrade): ipython-genutils in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Collecting prometheus-client (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/4c/bd/b42db3ec90ffc6be805aad09c1cea4bb13a620d0cd4b21aaa44d13541d71/prometheus_client-0.6.0.tar.gz\n", "Collecting tornado<7,>=4.1 (from notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", " Downloading https://files.pythonhosted.org/packages/b1/7f/8109821ff9df1bf3519169e34646705c32ac13be6a4d51a79ed57f47686e/tornado-6.0.1.tar.gz (481kB)\n", "\u001b[K 100% |████████████████████████████████| 491kB 2.3MB/s eta 0:00:01\n", "\u001b[?25hRequirement already satisfied (use --upgrade to upgrade): decorator in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from traitlets>=4.3.1->ipywidgets<8,>=7.0.0->ipyleaflet==0.9.0->-r requirements.txt (line 2))\n", "Requirement already satisfied (use --upgrade to upgrade): mistune!=0.6 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): pygments in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): bleach in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): pandocfilters>=1.4.1 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): testpath in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): ptyprocess; os_name != \"nt\" in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from terminado>=0.8.1->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): markupsafe in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from jinja2->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): html5lib>=0.99999999 in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from bleach->nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Requirement already satisfied (use --upgrade to upgrade): webencodings in /cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages (from html5lib>=0.99999999->bleach->nbconvert->notebook>=5->appmode==0.4.0->-r requirements.txt (line 13))\n", "Building wheels for collected packages: vega, arrow, appmode, toolz, nltk, prometheus-client, tornado\n", " Running setup.py bdist_wheel for vega ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/db/57/85/e0285100a71f35c73d7b3975a8899894a73c92068cd4cc1dea\n", " Running setup.py bdist_wheel for arrow ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/1e/d5/30/cace6155de216c74f70b1b866ce6ec2c9bbc60be88073a1fcd\n", " Running setup.py bdist_wheel for appmode ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/c6/d0/7b/f36cfd6c017396336f95f324c147c87ee5b46d9c34fded3f6e\n", " Running setup.py bdist_wheel for toolz ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/f4/0c/f6/ce6b2d1aa459ee97cc3c0f82236302bd62d89c86c700219463\n", " Running setup.py bdist_wheel for nltk ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/4b/c8/24/b2343664bcceb7147efeb21c0b23703a05b23fcfeaceaa2a1e\n", " Running setup.py bdist_wheel for prometheus-client ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/4b/04/b8/3709c73e7453f311ebd46ad581b89642543213f995e2659b9e\n", " Running setup.py bdist_wheel for tornado ... \u001b[?25ldone\n", "\u001b[?25h Stored in directory: /tmp/user/.cache/pip/wheels/09/a9/78/41e23ffcb122e8f4b02537f4480a276ec1ad760dd492373961\n", "Successfully built vega arrow appmode toolz nltk prometheus-client tornado\n", "Installing collected packages: python-dateutil, pandas, toolz, altair, xarray, traitlets, jupyter-core, nbformat, tornado, pyzmq, jupyter-client, terminado, Send2Trash, prometheus-client, notebook, widgetsnbextension, ipywidgets, traittypes, ipyleaflet, nltk, textblob, tqdm, arrow, troveharvester, vega, pillow, wordcloud, scipy, scikit-learn, kiwisolver, matplotlib, appmode\n" ] } ], "source": [ "!pip install --user -r requirements.txt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok, so this seemed to work ok, though it was VERY VERY SLOW.\n", "\n", "Note that it updated some already available packages such as Pandas, let's check that I'm using the updated version." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.19.0\n", "CPU times: user 418 ms, sys: 136 ms, total: 554 ms\n", "Wall time: 22.6 s\n" ] } ], "source": [ "%%time\n", "import pandas as pd\n", "print(pd.__version__)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mYou are using pip version 8.1.1, however version 19.0.3 is available.\r\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n", "pandas==0.24.1\r\n" ] } ], "source": [ "!pip freeze | grep pandas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's weird. According to `pip` we've installed 24.1, but when we import it's only getting 19.0. How do you upgrade installed packages?" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": true }, "outputs": [], "source": [ "!export PYTHONPATH=$CERNBOX_HOME/.local/lib/python3.5/site-packages:$PYTHONPATH" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import os\n", "try:\n", " user_paths = os.environ['PYTHONPATH'].split(os.pathsep)\n", "except KeyError:\n", " user_paths = []" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/native/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/CMake/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/glib-2.0/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/plugin/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/texinfo/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python3.5/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/visualvm/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/whizard/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/CLHEP-2.3.4.4/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python2.6/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/cmake/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/py/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/archive/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/SHERPA-MC/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/ThePEG/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/JupyROOT/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/graphviz/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/gio/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/libffi-3.2.1/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/accsoft/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/trace_rlog/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/amd64/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/valgrind/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/ruby/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/JsMVA/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/VBFNLO/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/ext/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/__pycache__/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/cairo/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/libxslt-plugins/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/pkgconfig/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/python2.7/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/gle/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/missioncontrol/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/php/site-packages',\n", " '/cvmfs/sft.cern.ch/lcg/views/LCG_91python3/x86_64-slc6-gcc62-opt/lib/lib/site-packages',\n", " '']" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "user_paths" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/webdav\r\n" ] } ], "source": [ "!pwd" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already up-to-date: pandas in ./.local/lib/python3.5/site-packages\n", "Requirement already up-to-date: numpy>=1.12.0 in ./.local/lib/python3.5/site-packages (from pandas)\n", "Requirement already up-to-date: python-dateutil>=2.5.0 in ./.local/lib/python3.5/site-packages (from pandas)\n", "Collecting pytz>=2011k (from pandas)\n", " Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl\n", "Requirement already up-to-date: six>=1.5 in ./.local/lib/python3.5/site-packages (from python-dateutil>=2.5.0->pandas)\n", "Installing collected packages: pytz\n" ] } ], "source": [ "!pip install --upgrade --user pandas" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied (use --upgrade to upgrade): pillow in ./.local/lib/python3.5/site-packages\n", "\u001b[33mYou are using pip version 8.1.1, however version 19.0.3 is available.\n", "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n" ] } ], "source": [ "!pip install --user pillow" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.19.0\n", "CPU times: user 364 µs, sys: 91 µs, total: 455 µs\n", "Wall time: 338 µs\n" ] } ], "source": [ "%%time\n", "import pandas as pd\n", "print(pd.__version__)" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ASIO files Shared datasets\t trying_out_swan.ipynb welcome.txt\n", "lost+found test\t\t Untitled1.ipynb\n", "Shared\t trove-newspaper-harvester Untitled.ipynb\n", "CPU times: user 10.8 ms, sys: 7.01 ms, total: 17.8 ms\n", "Wall time: 566 ms\n" ] } ], "source": [ "%time !ls" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/webdav\r\n" ] } ], "source": [ "!pwd" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": true }, "outputs": [], "source": [ "import cv2" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "img = cv2.imread('Shared/ANU-Library/Sydney Stock Exchange 1901-1950/AU NBAC N193-001/N193-001_0001.tif')" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(6646, 5328, 3)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "img.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting tqdm\n", " Using cached https://files.pythonhosted.org/packages/6c/4b/c38b5144cf167c4f52288517436ccafefe9dc01b8d1c190e18a6b154cd4a/tqdm-4.31.1-py2.py3-none-any.whl\n", "Installing collected packages: tqdm\n" ] } ], "source": [ "!pip install --user tqdm" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 2 }