{ "metadata": { "name": "", "signature": "sha256:89586ce495c37933625a3aafc3988c1d6a65c6c63d58b61d6b238d41f52b14a0" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Helpful Tips for using the mosaicking pipeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Most of the program's run time is spent on downloading and uploading imaging data. So the program could be significantly sped up if the investigator already has the survey data avaliable locally." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- If a mosacing is conducted on the whole-catalog, the job will probably take a long time (~1 week for SDSS on RC3). You can use __nohup__ to run the job on a remote machine to log off without terminating the process:\n", " nohup python bulk_run.py\n", "Other options such as __qsub__, __at__, __batch__ may also be used.\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- It is okay to run different procedures on different surveys and different parts of the sky simultaneously as long as the imaging bands do not have overlapping bands to prevent file-already-exist errors" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Place @ inside rc3.txt (RC3 Data file) to denote where rc3_Catalog methods should start. This often comes in handy when debugging. \n", "To run several (or single image), a quick and easy way is to place @ in rc3.txt and just run the on the survey that you want (SDSS: bulk_run.py ; DSS: bulk_run_dss.py; 2MASS: bulk_run_twomass.py), then terminate the process when finish running on the objects." ] }, { "cell_type": "heading", "level": 5, "metadata": {}, "source": [ "Example:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Inside rc3.txt, " ] }, { "cell_type": "raw", "metadata": {}, "source": [ "99.9008333333 20.7444444444 0.0303283477042 19385 \n", "99.925 65.4611111111 0.0235422924575 19390 \n", "99.9579166667 -38.2452777778 0.0296379902266 19391 \n", "@\n", "100.02875 60.0827777778 0.0491868205428 19397 \n", "100.0625 -72.4558333333 0.0246518065188 19400 \n", "100.134166667 50.105 0.0152001806863 19409 \n", "100.180833333 -58.5255555556 0.0491868205428 19413 " ] }, { "cell_type": "code", "collapsed": false, "input": [ "cd ../pipeline/" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 4 }, { "cell_type": "code", "collapsed": false, "input": [ "from rc3Catalog import RC3Catalog\n", "from sdss import SDSS" ], "language": "python", "metadata": {}, "outputs": [], "prompt_number": 5 }, { "cell_type": "markdown", "metadata": {}, "source": [ "Mosaicking starts from PGC19397: " ] }, { "cell_type": "code", "collapsed": false, "input": [ "rc3_cat = RC3Catalog()\n", "rc3_cat.mosaicAll(SDSS())" ], "language": "python", "metadata": {}, "outputs": [ { "output_type": "stream", "stream": "stdout", "text": [ "Working on PGC19397, at(100.02875 , 60.0827777778)\n", "------------------mosaic_band----------------------\n", "Now mosaic_band on 19397\n", "(100.02875, 60.0827777778, 0.0737802308142)\n", "Working on PGC19400, at(100.0625 , -72.4558333333)" ] }, { "output_type": "stream", "stream": "stdout", "text": [ "\n", "------------------mosaic_band----------------------\n", "Now mosaic_band on 19400\n", "(100.0625, -72.4558333333, 0.0369777097782)\n", "Working on PGC19409, at(100.134166667 , 50.105)" ] } ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Sometimes the program may crash because of Server-side errors or bad Network Connection: " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "for SDSS:" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "[['PGC 71234', '350.619333', '29.180944'], [' \\r\\n'], [' Description: An application error occurred on the server. The current cust\n", "\n", "om error settings for this application prevent the details o\n", "\n", "f the application error from being viewed remotely (for security reasons). It could', ' however', ' be viewed by browser\n", "\n", "s running on the local server machine.\\r\u2019]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "for DSS: " ] }, { "cell_type": "raw", "metadata": {}, "source": [ " File \"/usr/local/lib/python2.7/dist-packages/astroquery/vizier/core.py\", line 367, in query_region_async\n", " timeout=self.TIMEOUT)\n", " File \"/usr/local/lib/python2.7/dist-packages/astroquery/query.py\", line 172, in request\n", " response = query.request(self.__session, self.cache_location)\n", " File \"/usr/local/lib/python2.7/dist-packages/astroquery/query.py\", line 72, in request\n", " timeout=self.timeout))\n", " File \"/usr/lib/python2.7/dist-packages/requests/sessions.py\", line 383, in request\n", "\n", " File \"/usr/lib/python2.7/dist-packages/requests/adapters.py\", line 387, in send\n", "\n", " raise Timeout(e)\n", "requests.exceptions.Timeout: HTTPConnectionPool(host='vizier.u-strasbg.fr', port=80): Read timed out. (read timeout=60)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In these cases, restart the job starting from where you left off,using the trick described above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- When conducting the [post-processing procedures](http://nbviewer.ipython.org/github/ProfessorBrunner/rc3-pipeline/blob/master/Documentation/Post-processing%20procedures.ipynb),it is usually safer to save a backup copy of the whole data product file, so that you can always start from this step instead of taking a week or more to regenerate the data product." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Network and downloading speed will be much slower if you are running multiple mosaicing jobs (ex. bulk_run_dss.py and bulk_run.py) on the same machine." ] } ], "metadata": {} } ] }