{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. A Brief Introduction to Unix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[1. A Brief Introduction to Unix](#1.-A-Brief-Introduction-to-Unix) \n", "[2. Making and breaking things in Unix](#2.-Making-and-breaking-things-in-Unix) \n", "[3. Some more Unix and associated tools](#3.-Some-more-Unix-and-associated-tools) \n", "[4. Summary of all commands](#4.-Summary-of-all-commands)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.1 In this section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you will be introduced to python notebooks and the unix operating system.\n", "\n", "You should have an understanding of:\n", "\n", "1. Unix as an operating system\n", "2. How to change your password (`passwd`).\n", "3. Directory structures in unix\n", "4. Relative and absolute pathnames\n", "5. The role and meaning of the characters `~` `.` and `..` (*twiddle*, *dot* and *dot-dot*)\n", "6. The unix commands `pwd`, `cd`, and `ls`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Commands in this section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\[[`cd`](#cd)\\] : change directory \n", "\\[[`.` ](#dot)\\]: dot (current level) \n", "\\[[`..`](#dotdot)\\] : dot dot (up one level) \n", "\\[[`ls`](#passwd)\\] : change password \n", "\\[[`passwd`](#ls)\\] : list \n", "\\[[`pwd` ](#pwd)\\]: print working directory \n", "\\[[`~` ](#twiddle)\\]: tilde (twiddle) - home " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Unix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "This course will be taught mainly under a Unix operating system. The operating system allows you access to the running of the computer (and associated devices on the network), and enables you to be able to control what you want the computer to do. \n", "\n", "There is some overhead to learning this (like learning a new language), but we will introduce you to the basic operations you will need in this session.\n", "\n", "There are many online tutorials on unix. A good place to start backup material and some more features for the material we will cover today is [software-carpentry.org](http://software-carpentry.org/v3/shell01.html)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2.1 Using these notes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The interface for these notes is done using something called *ipython notebooks*, that we will return more to later.\n", "\n", "Not surprisingly, this is mainly aimed at running commands in the python programming language.\n", "\n", "In this section of the notes, we will be using *unix* commands, where we are working directly with the operating system (rather than in the python language and environment that is on top of the underlying system). \n", "\n", "One really convenient feature of ipython notebooks is that you, the student, can simply download then (using the `Download notebook` button in the corner of the browser display) and then run them on your local computer. When you have downloaded a notebook, you can use it by typing:\n", "\n", "`berlin% ipython notebook` \n", "\n", "which will open a web browser with an interface tou your notebooks.\n", "\n", "You *can* execute ('run') each block of code by selecting the code block (click on it) and simply hitting the return key. This conveniently works with basic unix commands as well as python commands.\n", "\n", "As an example of some commands we will use later" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/geogg122\n" ] } ], "source": [ "cd ~" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "u'/home/geogg122'" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These two commands (`cd ~` and `pwd`) are *unix* commands (that we happen to have run here through the ipython interpreter). You can, as we have suggested, simply download the notebook and hit `` to run these, but to get used to using the underlying operatoring system, you will want to open a *terminal* or *shell tool* (or `xterm` in some cases). This is a 'window' which has a *command line prompt* at which you can type and execute unix commands.\n", "\n", "It will look something like this:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](files/images/terminal.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where, in this case the command line prompt is `pallas% `, where `pallas` happens to be the name of one of the computers we might use. \n", "\n", "You type the commands at the prompt, then hit `` (i.e. the return key) to execute (run) the command.\n", "\n", "When you open a new 'terminal', you will usually be *located* in your *home directory*. We will discuss this further below.\n", "\n", "Let's check that now, and introduce the command `pwd` (print working directory) that will print (in the terminal) where we currently are working." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](files/images/terminal1.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, whilst you *can* simply hit `` in the code blocks here, we will want you to run the unix commands in todays session in a terminal.\n", "\n", "**Open a terminal now, and run the command `pwd` to get used to this.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.3 What is Unix?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "UNIX is a multi-user operating system: that is, a suite of programs which run on a computer and allows interface to the hardware and software available. It allows many users to share a powerful machine and all the available resources, each user running their own processes simultaneously. This true multi-tasking aspect of UNIX allows a far greater flexibility than is possible with other operating systems, even more recent multi-tasking versions of popular PC operating systems (note that the linux is simply a ‘flavour’ of UNIX & that the Mac operating system OS-X has a good deal of UNIX system within it). The advantages of a truly flexible multi-tasking operating system has been demonstrated with the popularity of UNIX in the scientific, engineering and financial communities, along with the rise of Linux and OS-X.\n", "\n", "X11 is the windowing system that you will generally use to interface with the operating system. A key aspect of this has always been the idea of remote graphical user interfaces via X11, which allows a user to run and visualise processes on different Unix machines. You are not limited to accessing the computer you are sat in front of, but can easily make wider use of local or external resources. You will find this useful as you can log on to the UCL Geography Unix computers from outside UCL (from pretty well any other operating system) and (with appropriate X11 software such as cygwin or exceed, if you are working from a Windows computer, or directly from any Unix/linux computer os OS-X machine) run windowing sessions the same as if you were physically at the UCL computer (given sufficient bandwidth)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3.1 Workstations and networks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Workstations that run UNIX are not like the way most people use PCs running Windows... At any time of the day or night a UNIX machine is running literally hundreds of processes simultaneously: Some are long-term user processes (such as research experiments) placed at low priority to avoid affecting other users, which may run for weeks, but most are concerned with the second by second activities of maintaining the network.\n", "\n", "Physically, the network consists of 'wires' (ethernet cables, phone lines, fibre-optic cables etc.) or wireless connections (locally and externally) running between computers both at local and remote sites. At the local level (within a University Dept. for example) the wires literally connect every machine to every other, either in a loop, or in a star-like arrangement with machines linked around central communications hubs. Such arrangements of connections, machines and attached peripheral devices is known as a Local Area Network or LAN. Larger arrangements of connections, for instance between departments in a university rather than individual computers, are known as Wide Area Networks or WANs.\n", "\n", "It is through this ability to transparently network computers together that UNIX really gains its power. Very often the data a user processes on one computer is actually stored on another (a paradigm that has (indirectly) led to the development of the Internet and World Wide Web). Similarly it is commonplace to process data on one machine whilst physically sitting in quite a different location (even on a different continent!). The limiting idea of the machine that you are sitting in front of being the machine you are using for processing (as in the PC-based computing environment) is very much obsolete in the case of UNIX computing. Using the network, it is also possible to break large processes into a number of smaller jobs and distribute them across multiple machines simultaneously, thereby reducing execution times dramatically. In a properly set-up UNIX LAN network, the storage of a users data on a remote machine, or their working on a remote machine should be a relatively transparent operation. Such concepts of networking are not confined to UNIX machines, although UNIX (in all its varieties) is far and away the most common from of operating system used for wide area networking.\n", "\n", "Because a UNIX machine never stops processing, the are **never switched off** by anyone other than the system manager or a relevant member of staff. Simply switching off as you might with a PC can cause irreparable damage to a UNIX machine and/or the data stored on it - remember, your data/project could be on the disk you just killed, so don’t do it! If you think that there is something wrong with a machine, do not attempt to fix it yourself: call or email the system manager or ask a knowledgeable person for advice. Under no circumstances should you attempt to reboot a workstation yourself, as unless shut down properly damage to data, both yours and that of others is likely to result.\n", "\n", "Many workstations you will come across have a simple power-saving device: the screen turns itself off if the keyboard hasn’t been used for a few minutes. To turn the screen back on, simply press a key on the keyboard such as shift, ctrl or the space bar. Often the monitor will have been turned off by the last user to save power (and to stop the lab heating up!) - in most cases, the monitors have a switch or button located on the lower right front of the monitor. If the monitor is on, a small green LED will light up. This is the only switch you should ever touch on the workstations. If it’s not on the front of the monitor, please leave it alone." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3.2 Accounts and passwords" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### passwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The UNIX prompt is a sequence of characters that the part of the operating system (OS) that you interact with, known as the *shell*, places at the start of each line when it is expecting you to enter a command. Often this prompt will be formed by the name of the machine to which you are logged in, followed by a `%` sign. For instance:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "berlin%" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "which will indicate that you are logged on through a window (which you might call a terminal or a shell though these are subtly different things).\n", "\n", "The nature of the prompt will vary from system to system, but the Geography machines have a prompt as above. When you become experienced you can customise the prompt to your own liking. The cursor (a small solid rectangle or underscore symbol) placed after the prompt tells you where your current typing position is (clicking in the window with the mouse doesn’t mover the cursor - it always remians at the end of the current line you’re on). The current line with the prompt is therefore known as the command line. Depending on the particular shell you are using and the sort of keyboard you have, you may be able to use up and down arrows to quickly go back through commands that you have typed at that prompt (in that shell session) (i.e. your history)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first time you log in, you will be given a default password (this is the same for all students).\n", "\n", "You should make sure that you change your password as soon as possible, but this needs to be something you will remember (as with all passwords) and secure enough that it can't easily be guessed.\n", "You could possibkly use the same password that you have for your ucl system account (or a variant of it). UCL will force you to change it every 3-6 months, so make sure youy also change your password on the system here. If you forget your password, you will need to see or email the system manager to get them to reset it (it is encrypted so they can't know what it is, but they can reset it).\n", "\n", "To change your password, type `passwd` at the command prompt and follow the instructions:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "berlin% passwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3.3 Logging out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you have finished your session at the computer, you must log off. To achieve this from a remote PC/Mac connection, simply type logout or exit at the prompt. From within a windowing session at the machine itself, select the logout option from your root menu (see the section on the windowing system for more details), and then confirm the logout by clicking on yes. If you leave yourself logged on then other people may come along and use your account, access your email etc. Whilst in a University environment this is not necessarily a problem - the next user will probably come along and log you out so that they can log on themselves it is important to log out so that other people don’t waste time wondering whether you have finished, or have just popped out for a few minutes. Don’t lock your workstation - this is anti-social behaviour as it prevents anyone else using it, and if we see it we’ll log you out anyway, and probably take all your pocket money away. If you leave for more than a few minutes, log yourself out. One of the advantages of a UNIX system is that you can leave a job running in the ‘background’ (don’t worry, we’ll get to that later) which means it’ll keep running for as long as you want even after you log out." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.4 The UNIX file system" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All information in UNIX, whether it is a program, data or text, is stored in files. These files are maintained by the operating system on hard disks (usually), and read into the computer’s memory when required. Files may be grouped together in directories (equivalent to folders), and these directories may themselves contain other directories and/or files. In fact, directories are really a special kind of file, but the user perceives the whole structure as forming a hierarchy of files and directories. This hierarchy is known as the filesystem. When UNIX computers are networked, the filesystem is not contained within one single machine, but spans the entire network. Each file and directory within the network filesystem is addressable via its own unique name - its filename, or directory name, and to the user the fact that the filesystem straddles multiple machines and hard disks goes largely unnoticed.\n", "\n", "The filesystem may be visualised as the roots of a tree. At the very top level of the filesystem on each individual machine resides the root directory, denoted / (slash).\n", "\n", "Beneath this directory lie the other directories containing files and further directories, including data and references to data stored elsewhere on the network." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "![](images/diagramSystem1.png)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.4.1 Absolute and relative path names" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### dotdot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have already mentioned that every file or directory in the filesystem is uniquely addressable by its filename or directory name. In fact a file’s full name is a description of the path from the root of the file system to the file itself. For instance, the directory `plewis` under the `home` directory has the full name:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "/home/plewis" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that as well as being the root of the filesystem, `/` is also used as a directory separator for full filenames and paths (note that is the opposite of the Windows/DOS directory separator `\\`). Here the directory `plewis` is reached by starting at the root of the system, `/`, going into the `home` directory. A directory contained within another directory is known as a sub-directory of its parent, or container, directory. The `/home/` section of the full filename is known as the absolute path(name) to the file - absolute because it starts at the root of the system. With this nomenclature we are able to move around the filesystem. \n", "\n", "Similarly, relative pathnames are permissible, describing the route to another path or directory from the current directory: the file `country.dat` in the directory `/home/plewis` may be addressed either absolutely as:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "/home/plewis/country.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "or relative to e.g. the directory `/home/mdisney` as:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "../plewis/country.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where `..` is a special symbol meaning ‘up one directory level’. An analogy is the address of your next door neighbour’s house. If you wanted to tell a friend what next door’s address is, you could give them the house number, the street, the city, the post code, and even the country. That’s the absolute path name to your neighbour’s house. Of course it’s much easier to specify “one house up from mine” or “number 18” etc. This is the relative path (relative to yours)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### dot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since `..` (dot-dot) means 'up one level' in unix, we can also note at this point that the symbol `.` (dot) means 'the current directory', so:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "./country.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "would refer to a file called `country.dat` that is in the current directory." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.5 Negotiating the file system" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each user in a UNIX network system has specific area of the filesystem belonging to them known as their home directory in which they are initially placed when they log on. Armed with a basic knowledge of the structure of the filesystem, each user is free to explore and visit almost any area of the system unless it has been specifically protected by its owner." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A good place to start is to be aware of *where you are* in the current shell.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you opened your shell, you should normally be in your *home* directory, which will be something like `/home/fbloggs` on our system (if your username were fbloggs) (or on a mac, under OS X, it might be `/Users/fbloggs`).\n", "\n", "The command to tell you where you are (in this shell) on the system is `pwd` (print working directory).\n", "\n", "Try this for yourself now **in a shell** and confirm that this is the case. Make sure to hit the `return` key to execute the command." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "u'/home/geogg122'" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### twiddle" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The home directory is a special place ... it's the part of the system that you 'own' and have control over.\n", "\n", "There is a special symbol in unix (formally, the *tilde* symbol `~`, though we often call it *twiddle*). In '*unix speak*' then we might refer to the home directory of the user `plewis` and '*twiddle plewis*', which we would write as `~plewis`.\n", "\n", "If you want to refer to your *own* home directory, you can shortcut this to just '*twiddle*', i.e. `~`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### cd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, we know how to find out where we are. How do we get somewhere else on the system?\n", "\n", "To change your working directory to somewhere else, use the command `cd` (change directory).\n", "\n", "*Let's use that now (in a shell) to change our working directory to the root of the file system.*" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/\n" ] } ], "source": [ "cd /" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "which will (once you hit the return key to execute the command) change your location, your ‘working directory’ to the root directory `/`. \n", "\n", "A few things to pay attention to when first coming across this:\n", "\n", "* first, there is space (‘white space’ as we call it) between the command cd and the ‘argument’ `/`. This is one or more spaces or tab characters so that the shell can understand that you want to run the command cd and give it some extra information (where to change directory to in this case), rather than, for example typing `cd/` in which case the shell will interpret `cd/` as the command you are trying to run;\n", "* second, if you think about what you want to achieve with the command, (change directory to somewhere) it should be quite apparent that apart from the command, you also need to give the shell an indication of where you want to go (`/` here) so you (normally) will have to type cd somewhere where somewhere is where you want to go. When you are first using these commands, pausing and thinking about what you want to achive is particularly important. Later, it will generally become second nature (as when you learn any new language).\n", "\n", "Now, if you type e.g.:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/bin\n" ] } ], "source": [ "cd usr/bin" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You will change directory to the subdirectory `bin` of the subdirectory `usr` of `/`, which is the same as doing (using an absolute pathname):" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/bin\n" ] } ], "source": [ "cd /usr/bin" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "These look similar at first glance, but the second importantly has a `/` at the front of the directory name, and so is an absolute pathname, whereas the first does not, so it is a relative pathname.\n", "\n", "The idea should be simple and intuitive once you get the idea of / being a separator and / being the top of the directory tree (the root directory)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also practice using the symbol `.` (dot) at this point ... what does the following command do? (hint: check which directory you are in, run the command below, then chack again to see where you are)." ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/bin\n" ] } ], "source": [ "cd ." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The unix command to give a listing of files or directories is `ls` (list). e.g.:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/usr/local\n" ] } ], "source": [ "cd /usr/local" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbin\u001b[0m/ \u001b[01;34metc\u001b[0m/ \u001b[01;34mgrads-2.0.2\u001b[0m/ \u001b[01;34mlib64\u001b[0m/ \u001b[01;34mMRTSwath\u001b[0m/ \u001b[01;34mshare\u001b[0m/\r\n", "\u001b[01;36mcuda\u001b[0m@ \u001b[01;34mexelis\u001b[0m/ \u001b[01;34minclude\u001b[0m/ \u001b[01;34mlibexec\u001b[0m/ \u001b[01;34mpanoply-4.6.0\u001b[0m/ \u001b[01;34msrc\u001b[0m/\r\n", "\u001b[01;34mcuda-7.5\u001b[0m/ \u001b[01;34mgames\u001b[0m/ \u001b[01;34mlib\u001b[0m/ \u001b[01;34mmatlab_r2016a\u001b[0m/ \u001b[01;34msbin\u001b[0m/ \u001b[01;34mxconv\u001b[0m/\r\n" ] } ], "source": [ "ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tells us that in the system directory `/usr/local/` there are (on this particular computer) a set of directories called `bin`, `etc` and so on.\n", "\n", "We could have a look to see what is in those direcories e.g.:" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34macresso\u001b[0m/ \u001b[01;34mapplications\u001b[0m/ \u001b[01;34minfo\u001b[0m/ \u001b[01;34mmacrovision\u001b[0m/ \u001b[01;34mman\u001b[0m/\r\n" ] } ], "source": [ "ls /usr/local/share" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "so we find some more sub-directories.\n", "\n", "Note that just typing:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbin\u001b[0m/ \u001b[01;34metc\u001b[0m/ \u001b[01;34mgrads-2.0.2\u001b[0m/ \u001b[01;34mlib64\u001b[0m/ \u001b[01;34mMRTSwath\u001b[0m/ \u001b[01;34mshare\u001b[0m/\r\n", "\u001b[01;36mcuda\u001b[0m@ \u001b[01;34mexelis\u001b[0m/ \u001b[01;34minclude\u001b[0m/ \u001b[01;34mlibexec\u001b[0m/ \u001b[01;34mpanoply-4.6.0\u001b[0m/ \u001b[01;34msrc\u001b[0m/\r\n", "\u001b[01;34mcuda-7.5\u001b[0m/ \u001b[01;34mgames\u001b[0m/ \u001b[01;34mlib\u001b[0m/ \u001b[01;34mmatlab_r2016a\u001b[0m/ \u001b[01;34msbin\u001b[0m/ \u001b[01;34mxconv\u001b[0m/\r\n" ] } ], "source": [ "ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "has the same effect as typing:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbin\u001b[0m/ \u001b[01;34metc\u001b[0m/ \u001b[01;34mgrads-2.0.2\u001b[0m/ \u001b[01;34mlib64\u001b[0m/ \u001b[01;34mMRTSwath\u001b[0m/ \u001b[01;34mshare\u001b[0m/\r\n", "\u001b[01;36mcuda\u001b[0m@ \u001b[01;34mexelis\u001b[0m/ \u001b[01;34minclude\u001b[0m/ \u001b[01;34mlibexec\u001b[0m/ \u001b[01;34mpanoply-4.6.0\u001b[0m/ \u001b[01;34msrc\u001b[0m/\r\n", "\u001b[01;34mcuda-7.5\u001b[0m/ \u001b[01;34mgames\u001b[0m/ \u001b[01;34mlib\u001b[0m/ \u001b[01;34mmatlab_r2016a\u001b[0m/ \u001b[01;34msbin\u001b[0m/ \u001b[01;34mxconv\u001b[0m/\r\n" ] } ], "source": [ "ls ." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.6 Exercise Unix-1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. Change directory back to your home. Check where you are when you get there. \n", "2. Use an appropriate unix command to find out which files and directories you have in your home.\n", "2. Change directory to your neighbour's home (you will need to ask them their username). Check where you are when you get there, and see what files they have.\n", "3. Repeat these two exercises, (a) using `~`; (b) using absolute pathnames; (c) using relative path names, perhaps throwing in the symbol `.` to make sure you understand that." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.7 Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you have been introduced to the unix file system and some basic unix commands for navigation around the system." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### commands in this section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\[[`cd`](#cd)\\] : change directory \n", "\\[[`.` ](#dot)\\]: dot (current level) \n", "\\[[`..`](#dotdot)\\] : dot dot (up one level) \n", "\\[[`ls`](#passwd)\\] : change password \n", "\\[[`passwd`](#ls)\\] : list \n", "\\[[`pwd` ](#pwd)\\]: print working directory \n", "\\[[`~` ](#twiddle)\\]: tilde (twiddle) - home " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 2. Making and breaking things in Unix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.0 In this section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you will be introduced to some new unix commands to create, copy, move and remove directories and files, as well as some concepts to start to give you more control of your environment.\n", "\n", "\\[[`chmod`](#chmod)\\] : change mode \n", "\\[[`cp`](#cp)\\] : copy \n", "\\[[`df`](#df)\\] : disk free \n", "\\[[`du`](#du)\\] : disk usage \n", "\\[[`ls -l`](#ls--l)\\] : long listing \n", "\\[[`mkdir`](#mkdir)\\] : make directory \n", "\\[[`mv`](#mv)\\] : move \n", "\\[[`quota`](#quota)\\] : personal disk quota \n", "\\[[`rm`](#rm)\\] : remove \n", "\\[[`rmdir`](#rmdir)\\] : remove (empty) directory \n", "\\[[`ssh`](#ssh)\\] : secure shell \n", "\\[[`*?`](#Wildcards)\\] : wildcards \n", "\n", "If you run through this exercise multiple times, you will probably want to delete the directories `~/DATA/bar`, `~/DATA/foo` and `~/DATA/testCp` and their contents before starting:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -rf ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -rf ~/DATA/testCp" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -rf ~/DATA/bar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.1 Making and removing directories" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### mkdir" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The command `mkdir` (make directory) is used to create ('make') a directory. You will want to do this to organise your files (e.g. put all of the files for a practical in one place). \n", "\n", "In the unix system you have here, there is (should be) a data directory (called `Data` or `DATA`). This is a large storage disk that you can use to put large exercises and data. Note that this disk is not so regularly backed up as your home directory, but it is much larger (and does not have a quota, as your home area does).\n", "\n", "Lets first move to the data directory:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/data/store01/data_dirs/staff/geogg122\n" ] } ], "source": [ "cd ~/DATA" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "and check where we are:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "collapsed": false }, "outputs": [ { "data": { "text/plain": [ "u'/data/store01/data_dirs/staff/geogg122'" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pwd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's create a directory that we might call foo, and move (i.e. change directory) into it:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir foo" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/data/store01/data_dirs/staff/geogg122/foo\n" ] } ], "source": [ "cd foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "or alternatively, to go straight there:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/data/store01/data_dirs/staff/geogg122/foo\n" ] } ], "source": [ "cd ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "collapsed": false }, "outputs": [], "source": [ "ls" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There should be no files in this new directory, so ls will tell us that.\n", "\n", "It is instructive to see what happens if we try to create a directory that already exists:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "mkdir: cannot create directory ‘/home/geogg122/DATA/foo’: File exists\r\n" ] } ], "source": [ "mkdir ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, the command *complains* that the file already exists.\n", "\n", "If we want to avoid this behaviour and create it is it doesn't exist or just leave it if it already does, we can use the *command line option* `-p` for `mkdir`:" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Command line options (that are normally preceeded by a `-` character or `--` in some cases) change the behaviour of a command." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### rmdir" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now suppose we don't want the directory any more. \n", "\n", "To remove an *empty* directory, we use the command `rmdir` (remove directory).\n", "\n", "Its not a good idea to try to remove the directory we are in, so let's `cd` home first:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/geogg122\n" ] } ], "source": [ "cd ~" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rmdir ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;36m/home/geogg122/DATA\u001b[0m@\r\n" ] } ], "source": [ "ls ~/DATA" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the command `ls` we should be able to confirm that the directory has gone. \n", "Again, it is instructive to see what happens if things 'go wrong', so let's create a directory that is not empty, and try to use rmdir to delete it:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p ~/DATA/foo/bar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that another use of the `-p` option in `mkdir` is to allow us to create a hierarchy of directories at one shot, so now, in the directory `~/DATA/foo` we have a sub-directory `~/DATA/foo/bar':" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbar\u001b[0m/\r\n" ] } ], "source": [ "ls ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "rmdir: failed to remove ‘/home/geogg122/DATA/foo’: Directory not empty\r\n" ] } ], "source": [ "rmdir ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, `rmdir` complains that the directory is not empty, and so doesn't delete it. We will see below that we use another command in this case." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.2 Wildcards" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Very often, we don't want to have to refer to the *full* file or directory name. Equally, we might often want to refer to multiple filenames or directories.\n", "\n", "To do this in unix, we use *wildcards*. The wildcard symbols are: `*` and `?`.\n", "\n", "`*` is interpreted as *zero or more characters*\n", "`?` is a single (wildcard) character\n", "\n", "We use these to form *patterns* of filenames.\n", "\n", "So, if we wanted to get a listing of all of the files in the directory `~plewis/msc` that end with the suffix `.dat` we would type:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/plewis/msc/atm.dat /home/plewis/msc/helloWorld.dat\r\n", "/home/plewis/msc/country.dat /home/plewis/msc/landsat.dat\r\n", "/home/plewis/msc/dem.dat /home/plewis/msc/listing.dat\r\n", "/home/plewis/msc/forest.dat /home/plewis/msc/max.dat\r\n", "/home/plewis/msc/head.dat /home/plewis/msc/points.dat\r\n", "/home/plewis/msc/header.dat /home/plewis/msc/popden.dat\r\n", "/home/plewis/msc/hello.dat\r\n" ] } ], "source": [ "ls ~plewis/msc/*.dat" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/plewis/geog/log.dat /home/plewis/msc/landsat.dat\r\n", "/home/plewis/harwood/lookAngles.dat /home/plewis/msc/listing.dat\r\n", "/home/plewis/home/log.dat /home/plewis/plewis/log.dat\r\n", "/home/plewis/krugerSims/light.default.dat /home/plewis/p/log.dat\r\n", "/home/plewis/krugerSims/light.lidar.dat /home/plewis/public_html/log.dat\r\n" ] } ], "source": [ "ls ~plewis/*/l*.dat" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/plewis/msc/listing.dat\r\n" ] } ], "source": [ "ls ~plewis/m??/l*n?.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.3 Copying, moving and deleting files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### cp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The command for copying a file is `cp` (copy), e.g.:" ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cp -n ~plewis/msc/hello.dat ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, we created the directory ~/Data/foo, then we used the command cp to copy a file from ~plewis/msc/hello.dat to the directory ~/DATA/foo.\n", "\n", "Usually you will be able to simply type:\n", "\n", "cp ~plewis/msc/hello.dat ~/DATA/foo\n", "\n", "i.e. without the -n option we have put here, but on some systems / setups you will find that cp might complain if one or more of the files you are trying to copy to already exist (this is a safety mechanism to stop you overwriting something you might not have meant to!).\n", "\n", "The -n option makes cp copy the files, but not overwrite an existing file.\n", "\n", "Note that if cp detects an attempt to copy a file to itself, the copy will fail.\n", "\n", "**Check that the file has been correctly copied using an appropriate command.** \n", "\n", "Note the *whitespace* between the command `cp` and the two *arguments* `~plewis/msc/hello.dat` and `~/Data/foo`, as well as the use of the `-n` option here.\n", "\n", "A common mistake people make when first using this command is not giving two (actually, two or more) arguments, but if you think about what information you would need to give to the computer to copy a file from somewhere to somewhere else, you will soon learn that you need at least two arguments.\n", "\n", "We can copy more than one file at a time using wildcard characters. \n", "\n", "To explore this, lets get a listing of all of the files that end with `.dat` but start with `h` in the directory `~plewis/msc`:" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/plewis/msc/head.dat /home/plewis/msc/hello.dat\r\n", "/home/plewis/msc/header.dat /home/plewis/msc/helloWorld.dat\r\n" ] } ], "source": [ "ls ~plewis/msc/h*.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To copy multiple files to the same place then, we could simply list the names:" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cp -n ~plewis/msc/hello.dat ~plewis/msc/helloWorld.dat ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this case, the command has three arguments. The final argument (`~/DATA/foo`) is interpreted as the place we want to copy the files *to*, and everything before that is a list of files we want to copy there\n", "\n", "If we knew we'd want all of the files `h*.dat`, then we can more simply type:" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cp -n ~plewis/msc/h*.dat ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, after you have done this, check to see what files are in the directory `~/DATA/foo`." ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbar\u001b[0m/ head.dat header.dat hello.dat helloWorld.dat\r\n" ] } ], "source": [ "ls ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use the option -R to do a recursive copy, i.e. copy a directory and everything below it, e.g.:" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p ~/DATA/testCp" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cp -Rf ~plewis/msc ~/DATA/testCp" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;36mmsc\u001b[0m@\r\n" ] } ], "source": [ "ls ~/DATA/testCp" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;36m/home/geogg122/DATA/testCp/msc\u001b[0m@\r\n" ] } ], "source": [ "ls ~/DATA/testCp/msc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### rm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The command `rm` (remove) is used to remove (delete) a file. For new users, we normally set the default behaviour of this (through something called an *alias*) to `rm -i` which then prompts the user about whether they *really* wanted to delete that file.\n", "\n", "The opposite of this (i.e. force a delete) is `rm -f` that we will illustrate here." ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -f ~/DATA/foo/header.dat" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbar\u001b[0m/ head.dat hello.dat helloWorld.dat\r\n" ] } ], "source": [ "ls ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "so we see the file has gone from that directory.\n", "\n", "*Normally*, especially when you are just starting to use unix, you should probably avoid using the `-f` option to `rm`.\n", "\n", "As an exercise, use `rm` and wildcards to delete all of the files in `~/DATA/foo` that start with `hel` and end in `.dat`, and confirm that what you wanted to happen actually has." ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -f ~/DATA/foo/hel*.dat" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mbar\u001b[0m/ head.dat\r\n" ] } ], "source": [ "ls ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use the option `-R` to make `rm` do a hierarchical ('recursive') deletion, e.g. delete everything from some directory downwards.\n", "\n", "This can be rather a dangerous command to use, as you might (if you are not careful) delete everything on your system, but of course it is of great practical use.\n", "\n", "But we will use it here to delete the directory `~/DATA/foo` and all of its contents." ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "collapsed": false }, "outputs": [], "source": [ "rm -Rf ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[0m\u001b[01;34mDownloads\u001b[0m/ \u001b[01;34mgeogg122.20161004\u001b[0m/ \u001b[01;34mtestCp\u001b[0m/\r\n", "\u001b[01;34mgeogg122\u001b[0m/ geogg122.tar.Z UCL_Kubuntu.ova\r\n" ] } ], "source": [ "ls ~/DATA/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "so now its gone." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### mv" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we want to *move* files or directories or *rename* them, then you should use the `mv` (move) command.\n", "\n", "Let's make a new directories `~/DATA/bar`, `~/DATA/foo` and copy some files into `~/DATA/bar`:" ] }, { "cell_type": "code", "execution_count": 51, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mkdir -p ~/DATA/bar ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 52, "metadata": { "collapsed": false }, "outputs": [], "source": [ "cp -n ~plewis/msc/h*dat ~/DATA/bar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should check that the files you expect to see in `~/DATA/bar` are there:" ] }, { "cell_type": "code", "execution_count": 53, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "head.dat header.dat hello.dat helloWorld.dat\r\n" ] } ], "source": [ "ls ~/DATA/bar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, let's move the files `hello.dat` and `helloWorld.dat` into a directory `~/DATA/foo` (using wildcards):" ] }, { "cell_type": "code", "execution_count": 54, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mv -n ~/DATA/bar/hello*dat ~/DATA/foo" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/home/geogg122/DATA/bar:\r\n", "head.dat header.dat\r\n", "\r\n", "/home/geogg122/DATA/foo:\r\n", "hello.dat helloWorld.dat\r\n" ] } ], "source": [ "ls ~/DATA/bar ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To use `mv` to rename a file is in effect the same as moving it to a file of a different name:" ] }, { "cell_type": "code", "execution_count": 56, "metadata": { "collapsed": false }, "outputs": [], "source": [ "mv ~/DATA/bar/head.dat ~/DATA/bar/tail.dat" ] }, { "cell_type": "code", "execution_count": 57, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "header.dat tail.dat\r\n" ] } ], "source": [ "ls ~/DATA/bar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.4 Getting more control" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You should now have some understanding of the main unix commands for dealing with directories and files. \n", "\n", "We will now introduce a few concepts that give you more control over what you are doing on the system.\n", "\n", "**NOTE**: Although in the examples above, we have been able to type some basic system commands into the python, we can't do this for some of the commands we will use below. A more general interface to unix (shell) commands from a python prompt involves putting an exclamation mark (`!`, known as *'bang'*) in front of the command. In these notes then, where you see *bang* (`!`) in front of a command, you leave that out when using the command at the unix prompt. e.g.:" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello.dat helloWorld.dat\r\n" ] } ], "source": [ "!ls ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also run shell commands by invoking the `bash` shell:" ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello.dat\n", "helloWorld.dat\n" ] } ], "source": [ "%%bash\n", "ls ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### quota" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We mentioned above that your home directories have a quota associated with them. This means that when you try to go over that quota, you will no longer be able to write to your home area.\n", "\n", "This can be a cause of confusion with new students, in particular because when you log into to a windowing session on the unix computers, several files tend to be written to your user area on startup. So, if you are over your quota, you will find that your attempt to login fails. This can be confusing because you may just imagine that the system is 'broken', or perhaps you have typed the wrong password. These are things that *can* happen of course, but by far the most common reason for your being unable to log in to a windowing session (and probably somethiong that will happen to most of you at some point over the year) is that you have gone over your quota.\n", "\n", "The unix command to check your quota is, helpfully, `quota`, though you would usually use the `-v` option:" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Disk quotas for user geogg122 (uid 4112): \n", " Filesystem blocks quota limit grace files quota limit grace\n", "disco:/RAID_I/rsu_raid_0\n", " 0 0 0 0 0 0 \n" ] } ], "source": [ "%%bash\n", "quota -v" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you happen to be on a system where no quota is set (e.g. your own computer), this command will tell you that." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ssh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's all very well to be able to find out if you are over your quota, but how can you do that if you can't log in?\n", "\n", "Well, in that case, you need to find another terminal somewhere (e.g. 'borrow' a window on another user's display, with their permission) and you can check it from there.\n", "\n", "One way to do this is to use the command `ssh` which you can use to spawn a process on a remote machine.\n", "\n", "For example, you have tried to log in to a computer called `berlin.geog.ucl.ac.uk` but the login fails. You then go to a friend who is logged in to another computer and ask if you can check your quota from there. You open a new terminal, and use `ssh` (secure shell):\n", "\n", "```bash\n", "ssh plewis@berlin.geog.ucl.ac.uk \"quota -v\"\n", "```\n", "\n", "which will then prompt you for your password and return the result of running the command `quota -v` on the computer `berlin.geog.ucl.ac.uk` as the user `plewis` (obviously, you replace `plewis` by your user name).\n", "\n", "Alternatively, if you just type:\n", "\n", "```bash\n", "ssh plewis@berlin.geog.ucl.ac.uk\n", "```\n", "\n", "Then this will start an interactive session as the user `plewis` (or rather, your username) on that computer and you should then be able to check your quota and fix any issues you have.\n", "\n", "We will see some further use of `ssh` later on, but knowing that you can use it to run a remote process is very useful." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you are using large datasets (that you may easily be doing in remote sensing), then you should have some awareness of how much space there is on the disk. If a disk gets full or nearly full, and you attempt to write to it, you may get unexpected results. You may also waste a lot of time.\n", "\n", "The command `df` (disk free) will also tell you how much disk space there is, and how much is used. It is generally of value to use the `-h` option that gives the results in 'human readable' form e.g.:" ] }, { "cell_type": "code", "execution_count": 61, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Filesystem Size Used Avail Use% Mounted on\n", "store01:/store 22T 22T 545G 98% /data/store01\n" ] } ], "source": [ "%%bash\n", "df -h ~/DATA" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output of this command is also of value to you if you need to know the actual device name of some disk area. For example, what you see on the file system as e.g. `~/DATA` may be physically on a device called `/dev/disk0s2`. Mostly, you won't need to know that level of detail about the system, but occasionally, and as you become more expert in using unix, you will so it is good to have these things at the back of your mind." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### du" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "To check how much disk space *you are using*, use the command `du` (disk usage). Often, you would use this with the option `-s` which produces a summary. Again, it is often of value to use the `-h` option that gives the results in 'human readable' form e.g.:" ] }, { "cell_type": "code", "execution_count": 62, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0\t/home/geogg122/DATA\n" ] } ], "source": [ "%%bash\n", "du -sh ~/DATA" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This summary disk usage information tells us (here in human readbale form) how much disk space is used in the directory `~/DATA` and its subdirectories." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ls -l" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have come across the command `ls` earlier as the command to give a listing of directory contents.\n", "\n", "A useful option for the command `ls` that gives you a *long* (verbose) listing of files or directories is `ls -l` (long listing).\n", "\n", "We will first make move the directory `~/DATA/bar` into the directory `~/DATA/foo`, then look at a long listing of what there is in `~/DATA/foo`:" ] }, { "cell_type": "code", "execution_count": 63, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 8\n", "drwxr-xr-x. 2 geogg122 Group_2015 38 Oct 5 08:34 bar\n", "-rw-r--r--. 1 geogg122 Group_2015 14 Oct 5 08:34 hello.dat\n", "-rw-r--r--. 1 geogg122 Group_2015 13 Oct 5 08:34 helloWorld.dat\n" ] } ], "source": [ "%%bash\n", "mv ~/DATA/bar ~/DATA/foo\n", "ls -l ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The long listing displays information on the owner of a file (`plewis` here), the size of the file (e.g. ~/DATA/foo/hello.dat is 14 bytes), and when it was last modified (26 Sep 17:05). The first field that you see (e.g. `drwxr-xr-x`) is a series of codes that tells you about what type of file it is (d in the first element means that it is a directory) then there are three sets of three elements that may be set to `rwx` or unset as in `---`. These refer to read `r`, write `w` and execute `x` permissions for this file (or directory), so that if you see e.g. `rw-` that means that read and write permission are set but not execute. Note that directories have the `x` bit set if you are to be able to see into that directory, but you will not generally alter that.\n", "\n", "The three groupings refer to *user* `u`, *group* `g` and *other* `o`.\n", "\n", "A useful additional option to `ls` is the `-h` option, that gives file sizes in 'human readable' format (i.e. something easier to understand):" ] }, { "cell_type": "code", "execution_count": 64, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 8.0K\n", "drwxr-xr-x. 2 geogg122 Group_2015 38 Oct 5 08:34 bar\n", "-rw-r--r--. 1 geogg122 Group_2015 14 Oct 5 08:34 hello.dat\n", "-rw-r--r--. 1 geogg122 Group_2015 13 Oct 5 08:34 helloWorld.dat\n" ] } ], "source": [ "%%bash\n", "ls -lh ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where the `B` at the end of the size field tells us the size is in bytes (or `K` for Kilobytes, `M` for Megabytes, `G` for Gigabytes etc)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### chmod" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, using `ls -l` give us information on what the file (/directory) permissions are. By default this will tend to be a sensible but open `-rw-r--r--` for files, i.e. read permission for the user, groups and the rest of the world but only write permission for the user themselves. For directories, the default is generally similar, but the `x` bit is set as well.\n", "\n", "We can manipulate the file permission settings, e.g. to remove read permission for some secure piece of work we are doing, or to open up write permission for other users on some shared piece of work, using the command `chmod` (change mode).\n", "\n", "If you have followed the material above, you should have a file called `hello.dat` in the directory `~/DATA/foo` (check that this is the case) and we will make a new sub-directory in there called `foobar` from which we will *remove* all write permissions. " ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "total 8\n", "drwxr-xr-x. 2 geogg122 Group_2015 38 Oct 5 08:34 bar\n", "drwxr-xr-x. 2 geogg122 Group_2015 6 Oct 5 08:35 foobar\n", "-rw-r--r--. 1 geogg122 Group_2015 14 Oct 5 08:34 hello.dat\n", "-rw-r--r--. 1 geogg122 Group_2015 13 Oct 5 08:34 helloWorld.dat\n", "total 8\n", "drwxr-xr-x. 2 geogg122 Group_2015 38 Oct 5 08:34 bar\n", "dr-xr-xr-x. 2 geogg122 Group_2015 6 Oct 5 08:35 foobar\n", "-rw-r--r--. 1 geogg122 Group_2015 14 Oct 5 08:34 hello.dat\n", "-rw-r--r--. 1 geogg122 Group_2015 13 Oct 5 08:34 helloWorld.dat\n" ] } ], "source": [ "%%bash\n", "mkdir -p ~/DATA/foo/foobar\n", "ls -l ~/DATA/foo\n", "chmod uog-w ~/DATA/foo/foobar\n", "ls -l ~/DATA/foo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "From which we see that the `w` bit on the directory `~/DATA/foo/foobar` has been removed (unset). The `uog` part refers to *user*, *group* and *other*. The `-` part means *remove* and the `w` part refers to the write bit.\n", "\n", "So, if we now try to copy or move a file into the `bar` directory, we would expect it to fail:" ] }, { "cell_type": "code", "execution_count": 66, "metadata": { "collapsed": false }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "cp: cannot create regular file ‘/home/geogg122/DATA/foo/foobar/hello.dat’: Permission denied\n" ] } ], "source": [ "%%bash\n", "cp -f ~/DATA/foo/hello.dat ~/DATA/foo/foobar" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.5 Exercise" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Go through the notes above, making sure you understand how to create and remove files and directories and how to move around the file system. That is the *minimum* you will need to start with. When you are doing this, **don't** just blindly type the commands given above, vary the file and directory names and make sure you appreciate what each command you type is doing (otherwise you won't learn this, I'm afraid).\n", "\n", "You should pay some attention to the notes on `quota` and related disk space/usage commands, as it is really quite likely that you will hit such problems at some time in the year, and you really ought to be able to fix such problems yourself. Remember, if you can't log on, the most likely reasons are: (i) you typed something wrong (so, check carefully); (ii) if you can't get in at the desktop, then probably you have gone over your quota. The most likely reason for going over quota is that you have put too many large files in your home area, instead of putting them in your *DATA* area as you are supposed to. To fix that, log in as given above and delete the files from your home area until you are below quota, *or* (better) move them into the *DATA* area (or better still, put them in the *DATA* area in the first place and avoid this problem)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2.6 Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you should have learned how to deal with copying, moving files and directories and related issues such as file permissions.\n", "You should also have gained some understanding of how to control what is going on in the unix environment a little more. \n", "\n", "#### Commands in this section \n", "\n", "\\[[`chmod`](#chmod)\\] : change mode \n", "\\[[`cp`](#cp)\\] : copy \n", "\\[[`df`](#df)\\] : disk free \n", "\\[[`du`](#du)\\] : disk usage \n", "\\[[`ls -l`](#ls--l)\\] : long listing \n", "\\[[`mkdir`](#mkdir)\\] : make directory \n", "\\[[`mv`](#mv)\\] : move \n", "\\[[`quota`](#quota)\\] : personal disk quota \n", "\\[[`rm`](#rm)\\] : remove \n", "\\[[`rmdir`](#rmdir)\\] : remove (empty) directory \n", "\\[[`ssh`](#ssh)\\] : secure shell \n", "\\[[`*?`](#Wildcards)\\] : wildcards " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3. Some more Unix and associated tools" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.1 UNIX Command Structure, Data Flow and File Manipulation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The previous sections dealt with the basic tools of Unix. Now, we will look at a few useful tools you have access to and some slightly more advanced concepts that again enable you to do more with the computer. \n", "\n", "Before starting this section, let's create a directory in your `DATA` area called `unix`:" ] }, { "cell_type": "code", "execution_count": 67, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "mkdir -p ~/DATA/unix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Remember that commands below that start with `!` (bang) should not have this symbol when you type them at the unix prompt: it is only because we are going through a python interpreter in these notes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1.1 Data Flow : stdin and stdout" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### streams" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Within unix all information is stored in files. Programs are 'told' (by the operating system) to read from and write to these files either by redirecting their input and output, or by modifying their action through the use of command line arguments or options. The channels through which the data flows to or from are known as *streams*. \n", "\n", "In the special case where data flows directly from one program into another, these channels are known as *pipes*.\n", "\n", "By default, with no options or arguments to specify otherwise, most commands will read their input form the keyboard and write their output to the screen. The data channels which attach the keyboard and the screen to the program are known as the standard input (stdin) and the standard output (stdout) respectively.\n", "\n", "Stdin and stdout can be redirected to and from files rather than the keyboard or screen using the unix `<` (“read from\") and `>` (“write to”) symbols." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### echo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The *shell* command `echo` will display text that you type after the command in the terminal (by default), e.g.:" ] }, { "cell_type": "code", "execution_count": 68, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "my home is /home/geogg122 and my name is geogg122\n" ] } ], "source": [ "%%bash\n", "echo \"my home is $HOME and my name is $USER\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, `HOME` is an *environment variable* that is passed through to programs to give contextual information about a users's environment (i.e. how things are set up). \n", "\n", "`USER` is another environment variable that gives your username ('login' or 'account' name). \n", "\n", "In unix shells, we refer to the *value* of an environment variable with a `$` symbol. Environment variables tend to be in upper case (capital letters) whereas another type of variable, shell variables (that refer only to a particular shell) are in lower case (small letters).\n", "\n", "Often, we would put quotes around the text:" ] }, { "cell_type": "code", "execution_count": 69, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello world geogg122\n" ] } ], "source": [ "%%bash\n", "echo \"hello world $USER\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This makes it a bit neater to see what we are doing, but is not strictly necessary in this case. \n", "\n", "There are different quotes that we use in unix. The double quote `\"` allows shell variables to be interpreted in the *string* enclosed by the quotes, but a single quote `'` does not." ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello world $USER\n" ] } ], "source": [ "%%bash\n", "echo 'hello world $USER'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### stdout" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So, when we use the unix command `echo`, the *result* of running that command appears in the terminal we are using.\n", "\n", "In fact, it is directed to the standard output channel `stdout`, which allows us to redirect it, e.g. to a file:" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "echo \"hello world\" > ~/DATA/unix/hello.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, nothing should have appeared in the terminal ... the text resulting from the `echo` command went instead to the file `~/DATA/unix/hello.dat`. \n", "\n", "We can check to see how big this file is to see if that makes sense:" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-rw-r--r--. 1 geogg122 Group_2015 12 Oct 5 08:35 /home/geogg122/DATA/unix/hello.dat\n" ] } ], "source": [ "%%bash\n", "ls -lh ~/DATA/unix/hello.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that the file *exists* and is of size 12 bytes. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### wc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We could count up the number of characters in `\"hello world\"` and see that it has 11 (including the space). We might infer from that that the file representation uses 1 byte per character for text, which is in fact the case. The *encoding* used is normally the [ASCII character set](http://en.wikipedia.org/wiki/ASCII).\n", "\n", "Why is the file 12 bytes though? The answer is that there is an additional character at the end of the file that tells the operating system that this is the end of the file. Not surprisingly, this is called the end of file marker or more properly, end of transmission `EOT`. This is represented as by the character `^D` (`control D`) in the [ASCII character set](http://en.wikipedia.org/wiki/ASCII).\n", "\n", "Sometimes it's too much effort to count up the characters in a string or a file. In unix, we can use the command `wc` (word count):" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 1 2 12\n" ] } ], "source": [ "%%bash\n", "wc < ~/DATA/unix/hello.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, this displays the number of *lines* (`1` here), *words* (`2` here) and *bytes* (`12` here) in the file.\n", "\n", "Useful modifcations of behaviour are e.g.:" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1\n" ] } ], "source": [ "%%bash\n", "wc -l < ~/DATA/unix/hello.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "which reports only the number of lines (similarly `wc -w` for words or `wc -c` for characters)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### stdin" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You may have noticed that we used the `stdin` symbol `<` here, which redirects the contents of the file `~/DATA/unix/hello.dat` *into* the command `wc`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### pipe" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One powerful concept in unix is the idea of *pipes*. This allows us to take the `stdout` resulting from one program and direct it to the `stdin` of another. \n", "\n", "This means we can e.g. avoid writing files out where all we want to do with the information is to pass it to another program.\n", "\n", "So, for the examples above, we could write:" ] }, { "cell_type": "code", "execution_count": 75, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "12\n" ] } ], "source": [ "%%bash\n", "echo \"hello world\" | wc -c" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where `|` is the `pipe` symbol." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sed" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This sort of idea is very useful if a sequence of programs (unix commands) that you want to run essentially work as 'filters'. \n", "\n", "The command `sed` (stream editor) can be very useful in this regard for dealing with text. The syntax of `sed` can be a little awkward for users to begin with, but we will just show a few examples here." ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "welcome geogg122 to using a computer\n", "welcome geogg122 to using unix\n" ] } ], "source": [ "%%bash\n", "echo \"welcome $USER to using a computer\"\n", "echo \"welcome $USER to using a computer\" | sed 's/a computer/unix/'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `s` part of the `sed` command means 'substitute'. It then changes an occurrence of the string `a computer` into the string `unix`.\n", "\n", "If you wanted to change *all* occurrences of the string, use a `g` at the end, e.g.:" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "welcome geogg122 to using unix. How do you like using a computer?\n", "welcome geogg122 to using unix. How do you like using unix?\n" ] } ], "source": [ "%%bash\n", "echo \"welcome $USER to using a computer. How do you like using a computer?\" | sed 's/a computer/unix/'\n", "echo \"welcome $USER to using a computer. How do you like using a computer?\" | sed 's/a computer/unix/g'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### awk" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You are not limited to a single pipe of course, but can pipe through multiple commands.\n", "\n", "To illustrate this, we introduce a new command awk that is a pattern scanning and interpretation language. You can use awk (or variants such as nawk or gawk) for many things but it is particularly useful for doing operations on columns of text or data that are in ASCII (i.e. text) format. For example:" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "4\n" ] } ], "source": [ "%%bash\n", "echo \"1 2 3 4 5\" | awk '{print $1+$3}'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "will output the sum of numbers in columns 1 and 3. To use it below, we can note here that the term `$0` refers to the complete input string and that there is a reasonably rich syntax for performing operations on columsn of data." ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 2 3 4 5\n", "the sum is 15\n", "one 2 3 4 5\n", "one two three four five, once I caught a fish alive\n" ] } ], "source": [ "%%bash\n", "echo \"1 2 3 4 5\"\n", "echo \"1 2 3 4 5\" | awk '{for(i=1;i<=NF;i++)sum+=$i} END{print \"the sum is\",sum}'\n", "echo \"1 2 3 4 5\" | sed 's/1/one/'\n", "echo \"1 2 3 4 5\" | sed 's/1/one/' | sed 's/2/two/' | sed 's/3/three/' | sed 's/4/four/' \\\n", " | sed 's/5/five/' | awk '{print $0 \", once I caught a fish alive\"}'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### cat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The unix command `cat` (concatenate) is used for a variety of purposes, but normally you will use it to *concatenate* (join together) files.\n", "\n", "If you specify a filename as `-` then the information of `stdin` is used, e.g.:" ] }, { "cell_type": "code", "execution_count": 80, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "hello\n" ] } ], "source": [ "%%bash\n", "echo \"hello\" | cat -" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's create some files of one line and join them together." ] }, { "cell_type": "code", "execution_count": 83, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "echo \"hello $USER\" > ~/DATA/unix/hello.dat\n", "echo \"welcome to the world of unix\" > ~/DATA/unix/hello2.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### date" ] }, { "cell_type": "code", "execution_count": 84, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Wed 5 Oct 08:35:46 BST 2016\n", "hello geogg122\n", "welcome to the world of unix\n" ] } ], "source": [ "%%bash\n", "date | cat - ~/DATA/unix/hello.dat ~/DATA/unix/hello2.dat " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we ran the command `date` that prints the current date (time) to `stdout`, concatenated the files `~/DATA/unix/hello.dat` and `~/DATA/unix/hello2.dat` to the end of this and sent the result to `stdout` (the terminal).\n", "\n", "If we wanted to send this to another file, we would simply redirect the output:" ] }, { "cell_type": "code", "execution_count": 85, "metadata": { "collapsed": false }, "outputs": [], "source": [ "%%bash\n", "date | cat - ~/DATA/unix/hello.dat ~/DATA/unix/hello2.dat > ~/DATA/unix/helloWorld.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also use the command `cat` to simply put the contents of its `stdin` channel to `stdout`, which is one way to display the contents of an ASCII file at the terminal." ] }, { "cell_type": "code", "execution_count": 86, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Wed 5 Oct 08:35:52 BST 2016\n", "hello geogg122\n", "welcome to the world of unix\n" ] } ], "source": [ "%%bash\n", "cat < ~/DATA/unix/helloWorld.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### appending" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When we redirect `stdout` using the `>` symbol, this will create a file if it doesn't already exist, but will overwrite one if it does exist:" ] }, { "cell_type": "code", "execution_count": 87, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Now you see me\n", "now you don't\n" ] } ], "source": [ "%%bash\n", "echo \"Now you see me\" > ~/DATA/unix/test.dat\n", "cat ~/DATA/unix/test.dat\n", "echo \"now you don't\" > ~/DATA/unix/test.dat\n", "cat ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to *append* to a file, rather than writing over the contents, you can use the `>>` symbol:" ] }, { "cell_type": "code", "execution_count": 88, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Now you see me\n", "now you still see me\n" ] } ], "source": [ "%%bash\n", "echo \"Now you see me\" > ~/DATA/unix/test.dat\n", "echo \"now you still see me\" >> ~/DATA/unix/test.dat\n", "cat ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "and you can continue adding lines in this way:" ] }, { "cell_type": "code", "execution_count": 89, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Now you see me\n", "now you still see me\n", "ad nauseam\n" ] } ], "source": [ "%%bash\n", "echo \"ad nauseam\" >> ~/DATA/unix/test.dat\n", "cat ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### more" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the text file you are looking at has a lot of lines, it can scroll off the screen, which can be a bit frustrating.\n", "\n", "The unix command `more` lets you view a file one page at a time.\n", "\n", "First, let's create a long file:\n", "\n" ] }, { "cell_type": "code", "execution_count": 90, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "this is the start of file\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "this is the end of file\n" ] } ], "source": [ "%%bash\n", "echo \"this is the start of file\" > ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test.dat >> ~/DATA/unix/test2.dat\n", "\n", "echo \"this is the end of file\" >> ~/DATA/unix/test2.dat\n", "cat ~/DATA/unix/test2.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is probably long enough to scroll off your terminal.\n", "\n", "Instead of using `cat` to view such a file, use `more`.\n", "\n", "You can use the `space bar` to go down one page at a time, or `return` for a line at a time (also `b` to go back, `:3` to go to line 3, `/end` to search for the string `end`, etc.)" ] }, { "cell_type": "code", "execution_count": 91, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "::::::::::::::\n", "/home/geogg122/DATA/unix/test2.dat\n", "::::::::::::::\n", "this is the start of file\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "this is the end of file\n" ] } ], "source": [ "%%bash\n", "more ~/DATA/unix/test2.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### less" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In some ways better that `more` is `less`. Try that on the file now." ] }, { "cell_type": "code", "execution_count": 92, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "this is the start of file\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "Now you see me\n", "now you still see me\n", "ad nauseam\n", "this is the end of file\n" ] } ], "source": [ "%%bash\n", "less ~/DATA/unix/test2.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### grep" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another very useful utility is grep ('globally search a regular expression and print'). You will commonly use this to find lines in a file that match some pattern. E.g.:" ] }, { "cell_type": "code", "execution_count": 93, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Now you see me\n", "now you still see me\n" ] } ], "source": [ "%%bash\n", "grep see < ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We told `grep` to return (i.e. put on `stdout`) lines of input that contain the string `see` in this case.\n", "\n", "By default, the pattern we search for is case sensitive (so, `Now` is different from `now`).\n", "\n", "We can use the `-v` option to ignore the case (i.e. whether it is lower case or upper case):" ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "now you still see me\n" ] } ], "source": [ "%%bash\n", "grep now < ~/DATA/unix/test.dat" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Now you see me\n", "now you still see me\n" ] } ], "source": [ "%%bash\n", "grep -i now < ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Usefully, we can also return lines that *don't* match the pattern using the `-v` option:" ] }, { "cell_type": "code", "execution_count": 96, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ad nauseam\n" ] } ], "source": [ "%%bash\n", "grep -v see < ~/DATA/unix/test.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### stderr" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Some commands make use of the `stderr` channel for reporting information to the shell (or e.g. a log file) whilst not interfering with information going to `stdout`.\n", "\n", "To redirect the `stderr` channel to a file, use `>&` in `csh` or `tcsh` but `2>` in [`bash`](http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html) and similar shells." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.2 Creating and editing text files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2.1 Creating text files with cat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A common thing you will want to do is create or edit a text file (ASCII text file).\n", "\n", "We have seen that you can redirect the standard output of unix commands to a file, so if the commands output ASCII, this is obviously one way you can create a file.\n", "\n", "We have also mentioned that the default input for `stdin` is what you type at the keyboard, so we can use that to create some text in a file.\n", "\n", "We can use the command `cat` for this.\n", "\n", "If you type `cat` at the prompt:\n", "\n", "```%berlin cat```\n", "\n", "then what you type after that will be read into `cat` and sent to `stdout`.\n", "\n", "Try that now. \n", "\n", "To end the input (end the file) we type a `^D` character (the `control` and `D` keys together).\n", "\n", "for example, type or copy and paste some text such as:\n", "\n", "```%berlin cat\n", "Gallia est omnis divisa in partes tres; unam\n", "partem incolunt Belgae, aliam Aquitani, tertiam\n", "qui ipsorum lingua Celtae, nostra Galli, appellantur.\n", "^D ```\n", "\n", "You should find that the lines you type are 'echoed' to the terminal when you hit `return` at the end of each line. This is because you are sending information to `cat` from the keyboard (visualised in the terminal so you can see what you are typing) on `stdin` and the output is also going to the terminal on `stdout`.\n", "\n", "If you wanted to type information that will go into a file, you can redirect the output of `cat` to a file:\n", "\n", "```%berlin cat > ~/DATA/unix/gettysburg.dat\n", "The world will little note, \n", "nor long remember what we say here, \n", "but it can never forget what they did here.\n", "^D ```\n", "\n", "This will now be stored in the file `~/DATA/unix/gettysburg.dat`.\n", "\n", "That's all very well, and often you will use this methbod to create some text in a file, but if you make a mistake, you will find that you can't easily edit it, which can be rather inconvenient." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2.2 text editing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are many text editors available for unix, and what you use day-to-day will depend a lot on personal preferences.\n", "\n", "A short tutorial introducing some of these is available [here](https://github.com/profLewis/geogg122/blob/master/Text%20processing.pdf?raw=true).\n", "\n", "Most of these will open a new 'window', which will then have buttons and menus and other convenient gizmos, much like a word processor. Although you obviously *can* type and `process` `words` in such tools, you should remember that these are *not* really the same as word processors as the aim is to type and manipulate text represented as ASCII characters (i.e. not in MS word or rtf format or whatever). That said, some texteditors *can* store files in formats other than ASCII (e.g. rtf), and also some word processing formats are simply ASCII text representations.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### vi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In these notes and the associated lecture, we will introduce you to one of the **most basic** unix text editors, `vi` (which is similar to its varaint `vim` that you will sometimes come across).\n", "\n", "Whilst there is some learning overhead on this, two very good reasons for knowing the bare bones of this are:\n", "\n", "1. it is available on *any* unix system\n", "2. you use it through a terminal, so it doesn't require any windowing system\n", "\n", "This latter point could become important to you e.g. if you had broken or corrupted your unix installation or e.g. if you were working remotely over a connection that was slow.\n", "\n", "It is worthwhile then learning these basics,but you can also follow a good short tutorial on this called [vi for smarties](http://jerrywang.net/vi/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### vi commands \n", "`:w` - write \n", "`:q` - quit \n", "`:w somethingElse.dat` - write to file `somethingElse.dat` \n", "`:wq` - write and quit \n", "`:q!` - force quit (without saving) \n", "`:u` - undo \n", "`:100` - go to line 100 (etc)\n", "\n", "##### escape (and general panic button)\n", "`ESC` - exit insert mode (escape key)\n", "\n", "##### navigation (or use arrow keys)\n", "`h` - left \n", "`j` - down \n", "`k` - up \n", "`l` - right \n", "\n", "##### other\n", "`/` - search for (regular expression), e.g. `/here` \n", "`n` - next (regular expression) \n", "\n", "##### insert/delete\n", "`i` - insert (before). N.B. puts you into insert mode, until you hit `ESC`. \n", "`a` - insert (after) (i.e. append) N.B. puts you into insert mode, until you hit `ESC`.\n", "`x` - delete current character \n", "`10x` - delete next 10 characters (etc.) \n", "`dw` - delete word (so, `10dw` etc) \n", "`dd` - delete line (do `10dd` etc.) \n", "`J` - delete end of line (so bring next line up)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2.3 Exercise" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Use vi now to edit a file you have created, e.g.\n", "\n", "```berlin% vi ~/DATA/unix/gettysburg.dat```\n", "\n", "Practice adding some more lines, changing the words etc., and save your edited file to `~/DATA/unix/myGettysburg.dat`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.3 Process control" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3.1 Foreground and background" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### foreground" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When you start a process that opens another window, you may have noticed that no prompt was returned and that any subsequent typing in the parent window was ignored until you exited the command (or it finished some other way). This was because tool you started has taken over input to and output from the parent window, or more accurately, the shell running the window. In this state the program you started is said to be running in the foreground - that is no further processes can be started from that window/shell until textedit relinquishes control.\n", "\n", "As an example, try:\n", "\n", "```berlin% display ~plewis/msc/sar.jpg```\n", "\n", "This should open a window on your computer and display a SAR image (of Barton Bendish in Norfolk).\n", "\n", "![](files/images/sar.jpg)\n", "\n", "The problem you will find is that if you go back to the terminal you ran the command from, you will now longer be able to run any other commands.\n", "\n", "Try typing `ls` for example, your terminal will *seem* to be 'stuck':\n", "\n", "``berlin% display ~plewis/msc/sar.jpg\n", "ls``\n", "\n", "The reason for this is that the terminal will not accept any further commands until you quit the current process (or job, we might call it), which is `xv` here.\n", "\n", "Quit `display` by using the menu.\n", "\n", "Now, any commands that you had stacked up will run ... which probably isn't what you wanted to happen.\n", "\n", "When we run a process in this way, we say that it is in the *foreground*." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### background" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes then, we want to run a job in the *background*. This effectively means that the process will run independently of the shell, which means that the shell prompt will be freed up for further commands. \n", "\n", "We do this by putting an ampersand (`&`) at the end of the command:\n", "\n", "```berlin% display ~plewis/msc/sar.jpg &\n", "berlin%```\n", "\n", "And now you should see the command line prompt appear again (so you can nbow type `ls` if thats what you wanted to do).\n", "\n", "You should also have notices something else appearing which would be something like:\n", "\n", "```[1] 3568```\n", "\n", "The [n] indicates this is the nth job to be started in this shell, and n is the unique job number by which the process may be referenced within that shell (window). The second number on the line (3568) is known as the process ID (PID) of the program. This is the number by which the central processor of the workstation refers to the job. This number is unique to the program you have started, and may be used in any window to refer to this process, as will be see." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### bg" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If the process you wish to place in the background is already running, it can be stopped using `^Z` (`Control Z`). \n", "\n", "The prompt will then return, and the job is placed in the background using the `bg` command (note: all *you* type here (after typing `display ~plewis/msc/sar.jpg`) is `^Z` and `bg`):\n", "\n", "```bash\n", "berlin% display ~plewis/msc/sar.jpg \n", "^Z\n", "\n", "Suspended \n", "berlin% bg \n", "[1] 3568 \n", "berlin%``` \n", "\n", "so display is now running in the background, and the prompt should appear again so you can type new commands." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.4 Killing a job running in the foreground" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Normally, you can terminate a job running in the foreground with `^C` (`control` and `c`). If that does not work, try `^Z` as above, followed by `jobs` to see the number of the job to be killed, then kill it (see below)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.4.1 Job control within the Shell" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### fg" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Within its parent window, a job may be referred to in certain commands a `%n`, where `n` is the number that was given in square brackets when the process was started.\n", "\n", "The job may be brought back into the foreground using:\n", "\n", "```berlin% fg %n``` \n", "\n", "no prompt will be returned until either the job finishes or is placed back into the background." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### kill" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The job may be killled using:\n", "\n", "```berlin% kill %n``` \n", "\n", "or \n", "\n", "```berlin% kill -9 %n```\n", "\n", "if it refuses to die the first time.\n", "\n", "The priority of the job may be changed to let other people share the workstation:\n", "\n", "```berlin% renice +19 %n\n", "0: old priority 0, new priority 19```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### jobs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A list of jobs currently running within the shell may be obtained using the `jobs` command. On some systems, `jobs -l` will give the most useful information. e.g.:\n", "\n", "```bash\n", "berlin% xv ~plewis/msc/sar.jpg\n", "^Z \n", "\n", "Suspended \n", "berlin% jobs -l \n", "[1] + 9524 Suspended xv ~plewis/msc/sar.jpg```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.4.2 Killing or changing the priority of a job from outside its parent shell" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you made a note of the `PID` of each process when it started, then `kill` and `renice` can be used as before, but using the `PID` instead of job number:\n", "\n", "```berlin% renice +19 3568\n", "3568 : old priority 0, new priority 19\n", "berlin% kill 3568```\n", "\n", "If you didn’t note the PID, there are two ways to find it:\n", "\n", "* run the `top` command in another window and look at the process in the display\n", "* use `ps` (e.g. `ps auxww`)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### top" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`top` is a program used to display the processes currently using the most CPU (processing capacity) on a workstation. Typing `top` in a shelltool should result in a display similar to:\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "last pid : 3609; load averages : 0.00, 0.00, 0.00 03:47:19\n", "\n", "49 processes : 48 sleeping, 1 running\n", " \n", "Cpu states : % user, % nice, % system, % idle\n", " \n", "Memory : 11984K available, 11812K in use, 172K free, 2572K locked\n", " \n", "PID Username PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND\n", " \n", "3598 plewis 15 19 125K 99K run 9:20 12:49% 0.95% xv\n", "3609 plewis 26 0 225K 472K run 10:29 7.69% 0.22% textedit \n", "3607 plewis 36 0 422K 432K sleep 0.00 3.23% 0.12% csh```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Look for your program in the far right column, and get its PID from the far left column (in this format). Typing `i` may reveal more processes if yours is not shown. Typing `u` and then your username will show only your jobs. Jobs can be killed (`k`) and reniced (`r`) from inside `top` - see `?`? for more details." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`ps` is similar to `top`, but just lists the processes rather than providing a continuously updated display: \n", "\n", "\n", "```berlin% ps auxww | less\n", "USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND\n", "root 1 0.0 0.0 10368 696 ? Ss Sep13 0:04 init [5] \n", "rpcuser 4034 0.0 0.0 10180 796 ? Ss Sep13 0:00 rpc.statd``` \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### nice" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Whenever a job that will either use a lot of the machines processing power (such as an image processing command) or take a long time to run is started, it must be ‘nice’d up’. That is, the priority of the job must be set so that other users can make use of the workstation whilst your job is running. This is achieved using either the `renice` command, as above, or preferably by using the `nice` command as the job is started:\n", "\n", "```berlin% nice +19 xv ~plewis/msc/sar.jpg &\n", "[2] 3610\n", "```\n", "\n", "This starts `xv` in the background, and sets its priority to allow fair use of the machine by other people. In fact, xv is not a process that would normally need to be niced, as, unless image processing operations are being performed it usually takes virtually no processing power. It is used here simply as an example." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.5 remote access" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ping" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`ping` sends some test packets to a remote computer and reports on what is returned. Normally, e.g. to check the status of a machine called `socrates.ucl.ac.uk` you just type:\n", "\n", "```berlin% ping socrates.ucl.ac.uk``` \n", "\n", "which will report e.g.:\n", "\n", "```PING socrates.ucl.ac.uk (144.82.110.1) 56(84) bytes of data.```\n", "\n", "```64 bytes from socrates.ucl.ac.uk (144.82.110.1): icmp_seq=1 ttl=252 time=0.527 ms```\n", "\n", "```64 bytes from socrates.ucl.ac.uk (144.82.110.1): icmp_seq=2 ttl=252 time=0.342 ms```\n", "\n", "```64 bytes from socrates.ucl.ac.uk (144.82.110.1): icmp_seq=3 ttl=252 time=0.355 ms```\n", "\n", "```--- socrates.ucl.ac.uk ping statistics ---```\n", "\n", "```3 packets transmitted, 3 received, 0% packet loss, time 2001ms```\n", "\n", "```rtt min/avg/max/mdev = 0.342/0.408/0.527/0.084 ms```\n", "\n", "Use `^C` to quit `ping`.\n", "\n", "If nothing happens for some time, the machine is probably not available (because it is down or the network in unreachable). You can wait for a timeout, or use `^C` (`Control` and `C`) at the same time) to quit the command. To limit the call to a certain number of attempts, use:\n", "\n", "```berlin% ping -c 10 socrates.ucl.ac.uk```\n", "\n", "If the response is:\n", "\n", "```--- socrates.ucl.ac.uk ping statistics ---\n", "10 packets transmitted, 0 packets received, 100.0% packet loss```\n", "\n", "Then you can’t currently access the machine.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### sftp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Secure file transfer using `ssh` protocols. If working from a terminal (shell) and you want to transfer files from your local machine to a machine e.g. `shankly.geog.ucl.ac.uk`, `cd` to where the files are on the local machine, then type:\n", "\n", "```berlin% sftp shankly.geog.ucl.ac.uk```\n", "\n", "at which point you will normally be prompted for a password. If the username on the local and remote machines are different, you should use:\n", "\n", "```berlin% sftp plewis@shankly.geog.ucl.ac.uk```\n", "\n", "You can then use the [limited command set](http://kb.iu.edu/data/akqg.html) within `sftp` to change directory etc. and copy the files to or from the remote machine. A typical session then would be something along the lines of:\n", "\n", "```sftp> cd whereIWantToPutTheData \n", "sftp> put theFileIWantToTransfer.dat```\n", "\n", "to copy the local file `theFileIWantToTransfer.dat` into the directory `whereIWantToPutTheData` on the remote machine. Similarly to pull a file from the remote machine, use `get` rather than `put`.\n", "\n", "Use `exit` to exit the `sftp` session." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### scp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes, it is convenient to use `scp`, a secure copy command that you can use over digfferent machines on the network when they do not have common disk access. The syntax is of the form:\n", "\n", "```scp plewis@shankly.geog.ucl.ac.uk:///home/plewis/msc/hello.dat ~/DATA/helloLewis.dat```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### ssh" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Securely connect (‘log in’) to a shell on a remote computer. To set up an interactive session, from a terminal or `ssh` client type:\n", "\n", "```home% ssh plewis@shankly.geog.ucl.ac.uk```\n", "\n", "and respond to any confirmation request (the first time you connect two machines) and the request for your password.\n", "\n", "If you wish to run a session that uses X11, you need to set X11 tunneling:\n", "\n", "```home% ssh -X plewis@shankly.geog.ucl.ac.uk```\n", "\n", "on most unix/linux machines, or:\n", "\n", "```home% ssh -Y ucfaabc@shankly.geog.ucl.ac.uk``` \n", "\n", "from OS X.\n", "\n", "From a Windows machine, you should be able to connect using a client such as `putty`, but this does not directly allow `X11` sessions. To do that, you need some `X11` software (on the Windows machine) such as `exceed`. You can purchase that software, or at no cost, use a WTS session (see above) to start up [exceed](http://faq.rutgers.edu/?q=node/836) and then use e.g. `putty` (or tools in exceed) with `X11` forwarding. \n", "\n", "There are several alternatives to this from a Windows machine, one of which is to set up a unix environment within windows using `cygwin` but that might be too complex for novice users. \n", "\n", "Another is to make your Windows computer dual boot and install a linux system such as `ubuntu` on it. \n", "\n", "Yet another is to run linux as a virtual machine on top of Windows (or the other way around!). \n", "\n", "Another alternative is `Xming` that will run X11 on top of windows. \n", "\n", "Please note that whilst there are all of these potential solutions, we do not endorse or provide support for any particular solution. If you want to be able to do this and think it would be useful, try it out yourself and use the internet for resources to help you solve any problems you come across. If you have particular problems you think are connected with the UCL end of things, you can try asking the Geography System Manager, or ISD (if it is a UCL issue), or at a push, Professor Lewis in his office hours.\n", "\n", "**Important**: Note that you cannot use `ssh` (or `sftp`) to connect to *any* of the UCL Geography computers, as we only have a small number open to the UCL firewall (such as `shankly.geog.ucl.ac.uk`) and the machines we do have open are not ones you would normally want to do any processing on (or at least, we don’t want everyone processing on these gateway machines, as noone else will be able to get in). \n", "\n", "You will normally have to use `ssh` to get on to one of these firewall gateway machines, then `ssh` (normally `ssh -X` if you want to maintain X11 port forwarding) from there to a machine (e.g. one in the classroom) to do any processing. This might seem a little ‘roundabout’, but you will soon get used to it and it is like this for good security reasons.\n", "\n", "N.B. if your username is different on the two machines, use e.g. `ssh -X ucfaabc@socrates.ucl.ac.uk` or `ssh -X socrates.ucl.ac.uk -l ucfaabc` to give the correct username for the remote machine (the one you are trying to log on to), otherwise you can just use `ssh -X socrates.ucl.ac.uk`.\n", "\n", "Once you (think you have) established a remote session with X11, you should test it out, trying to open a simple application such as `xclock` or `xv` or `xeyes` from the remote machine onto your local machine.\n", "\n", "If you are sure you are on a secure machine, you can set things up so that you don’t have to keep typing the password when you connect (see here) but remember that anybody else will be able to connect those machines without a password.\n", "\n", "Finally, if what you want to do with `ssh` is not to run an interactive session, but to run a process on the remote machine (e.g. for some parallel processing), then you would normally use `ssh -f` for this, with the option `-X` (or `-Y` for OS X) if the processing requires `X11`. A simple example would be, get a listing of your home directory:\n", "\n", "```home% ssh -f ucfaabc@shankly.geog.ucl.ac.uk \"ls -l /home/plewis\"```\n", "\n", "This will execute the sequence of commands in quotes on the remote machine, then, when completed, terminate the session (that is what the `-f` flag does). If you want to ssh from that machine onto another (to run a process) that is a little more involved.\n", "\n", "One way to do this would be e.g.:\n", "\n", "```home% ssh -f shankly.geog.ucl.ac.uk 'ssh -f berlin.geog.ucl.ac.uk \"uname -a;pwd; cd /home/plewis ; ls -l\"'```\n", "\n", "which will `ssh` onto `shankly.geog.ucl.ac.uk` and from there run an `ssh` process onto `berlin` and then run the requested sequence of commands. Note that in this case you will have needed to previously ssh'd from `shankly` to `berlin` to have responded to the authentification request. This is a littler convoluted, especially the use of quotes here but is it feasible and sometimes useful to do this. You are much more likely to make use of this sort of command directly from within the UCL Geography system, in which case you do not need multiple levels of `ssh` calls to get through the firewall." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.6 Exercise" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This final section covers a bit more than the basics of unix, and in that sense, is a bit more than what we consider *critical* for you in this class. If you *do* learn these few other commands though, you should (hopefully) feel empowered to do more with your computers and get some real control over whats going on (rather than just clicking a button or sliding a slider).\n", "\n", "As a follow up to this section, you could try to see if you can access the unix system from outside of the lab and copy files to and from another computer.\n", "\n", "You might also like to look at further command line options for some of these commands. Mostly, there is a ``--help`` option (e.g. `ls --help`) that you can use. You can also access manual pages (type: `man ls`) or on some systems info pages (type: `info ls`). This will give you more depth to your understanding of what you can do with these commands. \n", "\n", "Getting to 'guru' status in unix will take you some time, but the more you explore and try things on the operating system, the more you will learn. If you are worried about breaking things, or just want to explore more, buy a cheap computer such as a [raspberry pi](http://www.raspberrypi.org/) (around £25), install a free version of linux on it, and play!\n", "\n", "Finally, one other skill you might like to develop is to learn to write some shell programs (e.g. in bash). This can be a very useful thing to be able to do as you essentially make your own unix program by combining others. We will learn such skills using python in subsequent sessions, but there is value to at least learning the basics in a lower level shell language (e.g. [bash](http://www.tldp.org/LDP/Bash-Beginners-Guide/html/) or [csh](http://www-cs.canisius.edu/ONLINESTUFF/UNIX/shellprogramming.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3.7 Summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this section, you should have learned a wider set of commands to control what is going on in the unix environment a little more. You may not remember all of these to start with, but can use this page as a reference as you explore what you can do with the operating system.\n", "\n", "#### Commands in this section \n", "\n", "\\[[awk](#awk)\\] : pattern scanning and interpretation language \n", "\\[[>>](#appending)\\] : appending to file \n", "\\[[bg](#background)\\] : background process \n", "\\[[cat](#cat)\\] : concatenate \n", "\\[[date](#date)\\] : date \n", "\\[[echo](#echo)\\] : echo \n", "\\[[fg](#fg)\\] : foreground \n", "\\[[grep](#grep)\\] : globally search a regular expression and print \n", "\\[[jobs](#jobs)\\] : process jobs \n", "\\[[kill](#kill)\\] : kill a process \n", "\\[[less](#less)\\] : less \n", "\\[[more](#more)\\] : more \n", "\\[[nice](#nice)\\] : nice a process \n", "\\[[ping](#ping)\\] : ping \n", "\\[[|](#pipe)\\] : pipe \n", "\\[[ps](#ps)\\] : process show \n", "\\[[sed](#sed)\\] : stream editor \n", "\\[[scp](#sfscptp)\\] : secure copy \n", "\\[[sftp](#sftp)\\] : secure file transfer protocol \n", "\\[[ssh](#ssh)\\] : secure shell \n", "\\[[>&](#stderr)\\] : standard error `>&` or `2>` \n", "\\[[<](#stdin)\\] : standard input `<` \n", "\\[[>](#stdout)\\] : standard output `>` \n", "\\[[streams](#streams)\\] : streams \n", "\\[[top](#top)\\] : top processes \n", "\\[[vi](#vi)\\] : vi editor \n", "\\[[wc](#wc)\\] : wc \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 4. Summary of all commands" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\\[[awk](#awk)\\] : pattern scanning and interpretation language \n", "\\[[>>](#appending)\\] : appending to file \n", "\\[[bg](#background)\\] : background process \n", "\\[[cat](#cat)\\] : concatenate \n", "\\[[cd](#cd)\\] : change directory \n", "\\[[chmod](#chmod)\\] : change mode \n", "\\[[cp](#cp)\\] : copy \n", "\\[[date](#date)\\] : date \n", "\\[[df](#df)\\] : disk free \n", "\\[[du](#du)\\] : disk usage \n", "\\[[. ](#dot)\\]: dot (current level) \n", "\\[[..](#dotdot)\\] : dot dot (up one level) \n", "\\[[echo](#echo)\\] : echo \n", "\\[[fg](#fg)\\] : foreground \n", "\\[[grep](#grep)\\] : globally search a regular expression and print \n", "\\[[jobs](#jobs)\\] : process jobs \n", "\\[[kill](#kill)\\] : kill a process \n", "\\[[less](#less)\\] : less \n", "\\[[ls](#passwd)\\] : list \n", "\\[[ls -l](#ls--l)\\] : long listing \n", "\\[[more](#more)\\] : more \n", "\\[[mkdir](#mkdir)\\] : make directory \n", "\\[[mv](#mv)\\] : move \n", "\\[[nice](#nice)\\] : nice a process \n", "\\[[passwd](#ls)\\] : change password \n", "\\[[ping](#ping)\\] : ping \n", "\\[[|](#pipe)\\] : pipe \n", "\\[[ps](#ps)\\] : process show \n", "\\[[pwd](#pwd)\\]: print working directory \n", "\\[[quota](#quota)\\] : personal disk quota \n", "\\[[rm](#rm)\\] : remove \n", "\\[[rmdir](#rmdir)\\] : remove (empty) directory \n", "\\[[sed](#sed)\\] : stream editor \n", "\\[[scp](#sfscptp)\\] : secure copy \n", "\\[[sftp](#sftp)\\] : secure file transfer protocol \n", "\\[[ssh](#ssh)\\] : secure shell \n", "\\[[>&](#stderr)\\] : standard error `>&` or `2>` \n", "\\[[<](#stdin)\\] : standard input `<` \n", "\\[[>](#stdout)\\] : standard output `>` \n", "\\[[streams](#streams)\\] : streams \n", "\\[[top](#top)\\] : top processes \n", "\\[[~](#twiddle)\\]: tilde (twiddle) - home \n", "\\[[vi](#vi)\\] : vi editor \n", "\\[[wc](#wc)\\] : wc \n", "\\[[*?](#Wildcards)\\] : wildcards \n", "\n", " \n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.11" } }, "nbformat": 4, "nbformat_minor": 0 }