{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scanners"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<IB connected to 127.0.0.1:7497 clientId=17>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from ib_insync import *\n",
    "util.startLoop()  \n",
    "\n",
    "ib = IB()\n",
    "ib.connect('127.0.0.1', 7497, clientId=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Scanner\n",
    "\n",
    "To create a scanner, create a `ScannerSubscription` to submit to the `reqScannerData` method. For any scanner to work, at least these three fields must be filled: `instrument` (the what), `locationCode` (the where), and `scanCode` (the ranking).\n",
    "\n",
    "For example, to find the top ranked US stock percentage gainers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50 results, first one:\n",
      "ScanData(rank=0, contractDetails=ContractDetails(contract=Contract(secType='STK', conId=326089336, symbol='HCCHR', exchange='SMART', currency='USD', localSymbol='HCCHR', tradingClass='SCM'), marketName='SCM', minTick=0.0, orderTypes='', validExchanges='', priceMagnifier=0, underConId=0, longName='', contractMonth='', industry='', category='', subcategory='', timeZoneId='', tradingHours='', liquidHours='', evRule='', evMultiplier=0, mdSizeMultiplier=0, aggGroup=0, underSymbol='', underSecType='', marketRuleIds='', secIdList=[], realExpirationDate='', lastTradeTime='', stockType='', cusip='', ratings='', descAppend='', bondType='', couponType='', callable=False, putable=False, coupon=0, convertible=False, maturity='', issueDate='', nextOptionDate='', nextOptionType='', nextOptionPartial=False, notes=''), distance='', benchmark='', projection='', legsStr='')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error 162, reqId 3: Historical Market Data Service error message:API scanner subscription cancelled: 3\n"
     ]
    }
   ],
   "source": [
    "sub = ScannerSubscription(\n",
    "    instrument='STK', \n",
    "    locationCode='STK.US.MAJOR', \n",
    "    scanCode='TOP_PERC_GAIN')\n",
    "\n",
    "scanData = ib.reqScannerData(sub)\n",
    "\n",
    "print(f'{len(scanData)} results, first one:')\n",
    "print(scanData[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The displayed error 162 can be ignored*\n",
    "\n",
    "The scanner returns a list of contract details, without current market data (this can be obtained via seperate market data requests).\n",
    "\n",
    "## Filtering scanner results, the old way\n",
    "\n",
    "The `ScannerSubscription` object has addional parameters that can be set to filter the results, such as `abovePrice`, `aboveVolume`, `marketCapBelow` or `spRatingAbove`. \n",
    "\n",
    "For example, to reuse the previous `sub` and query only for stocks with a price above 200 dollar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['MKTX', 'TPL', 'BIO B', 'TSLA', 'COKE', 'SHOP', 'RNG', 'TYL', 'BIO', 'ALX', 'MCO', 'ANSS', 'AMT', 'BH A', 'SEB', 'GHC', 'MSG', 'ICUI']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error 162, reqId 4: Historical Market Data Service error message:API scanner subscription cancelled: 4\n"
     ]
    }
   ],
   "source": [
    "sub.abovePrice = 200\n",
    "scanData = ib.reqScannerData(sub)\n",
    "\n",
    "symbols = [sd.contractDetails.contract.symbol for sd in scanData]\n",
    "print(symbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filtering, the new way\n",
    "\n",
    "In the new way there is a truly vast number of parameters available to use for filtering.\n",
    "These new scanner parameters map directly to the options available through the TWS \"Advanced Market Scanner.\" The parameters\n",
    "are dynamically available from a huge XML document that is returned by ``reqScannerParameters``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1708609 bytes\n"
     ]
    }
   ],
   "source": [
    "xml = ib.reqScannerParameters()\n",
    "\n",
    "print(len(xml), 'bytes')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the XML in a web browser:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path = 'scanner_parameters.xml'\n",
    "with open(path, 'w') as f:\n",
    "    f.write(xml)\n",
    "\n",
    "import webbrowser\n",
    "webbrowser.open(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1144 tags, showing first 100:\n",
      "['priceAbove', 'priceBelow', 'usdPriceAbove', 'usdPriceBelow', 'volumeAbove', 'usdVolumeAbove', 'usdVolumeBelow', 'avgVolumeAbove', 'avgVolumeBelow', 'avgUsdVolumeAbove', 'avgUsdVolumeBelow', 'ihNumSharesInsiderAbove', 'ihNumSharesInsiderBelow', 'ihInsiderOfFloatPercAbove', 'ihInsiderOfFloatPercBelow', 'iiNumSharesInstitutionalAbove', 'iiNumSharesInstitutionalBelow', 'iiInstitutionalOfFloatPercAbove', 'iiInstitutionalOfFloatPercBelow', 'marketCapAbove1e6', 'marketCapBelow1e6', 'moodyRatingAbove', 'moodyRatingBelow', 'spRatingAbove', 'spRatingBelow', 'ratingsRelation', 'bondCreditRating', 'maturityDateAbove', 'maturityDateBelow', 'currencyLike', 'excludeConvertible', 'couponRateAbove', 'couponRateBelow', 'optVolumeAbove', 'optVolumeBelow', 'avgOptVolumeAbove', 'optVolumePCRatioAbove', 'optVolumePCRatioBelow', 'impVolatAbove', 'impVolatBelow', 'impVolatOverHistAbove', 'impVolatOverHistBelow', 'imbalanceAbove', 'imbalanceBelow', 'displayImbalanceAdvRatioAbove', 'displayImbalanceAdvRatioBelow', 'regulatoryImbalanceAbove', 'regulatoryImbalanceBelow', 'displayRegulatoryImbAdvRatioAbove', 'displayRegulatoryImbAdvRatioBelow', 'avgRatingAbove', 'avgRatingBelow', 'numRatingsAbove', 'numRatingsBelow', 'avgPriceTargetAbove', 'avgPriceTargetBelow', 'numPriceTargetsAbove', 'numPriceTargetsBelow', 'avgAnalystTarget2PriceRatioAbove', 'avgAnalystTarget2PriceRatioBelow', 'stkTypes', 'hasOptionsIs', 'leadFutOnly', 'dividendFrdAbove', 'dividendFrdBelow', 'dividendYieldFrdAbove', 'dividendYieldFrdBelow', 'dividendNextDateAbove', 'dividendNextDateBelow', 'dividendNextAmountAbove', 'dividendNextAmountBelow', 'histDividendFrdAbove', 'histDividendFrdBelow', 'histDividendFrdYieldAbove', 'histDividendFrdYieldBelow', 'minGrowthRate', 'maxGrowthRate', 'minPeRatio', 'maxPeRatio', 'minQuickRatio', 'maxQuickRatio', 'minRetnOnEq', 'maxRetnOnEq', 'minPrice2Bk', 'maxPrice2Bk', 'minPrice2TanBk', 'maxPrice2TanBk', 'firstTradeDateAbove', 'firstTradeDateBelow', 'changePercAbove', 'changePercBelow', 'afterHoursChangePercAbove', 'afterHoursChangePercBelow', 'changeOpenPercAbove', 'changeOpenPercBelow', 'openGapPercAbove', 'openGapPercBelow', 'priceRangeAbove', 'priceRangeBelow', 'tradeCountAbove']\n"
     ]
    }
   ],
   "source": [
    "# parse XML document\n",
    "import xml.etree.ElementTree as ET\n",
    "tree = ET.fromstring(xml)\n",
    "\n",
    "# find all tags that are available for filtering\n",
    "tags = [elem.text for elem in tree.findall('.//AbstractField/code')]\n",
    "print(len(tags), 'tags, showing first 100:')\n",
    "print(tags[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how ``abovePrice`` is now called ``priceAbove``...\n",
    "\n",
    "Using three of these filter tags, let's perform a query to find all US stocks that went up 20% and have a current price between 5 and 50 dollar, sorted by percentage gain:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['BSGM', 'ARNC', 'FTAI', 'ALIN PRA', 'MDLX', 'CNX', 'IFRX']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Error 162, reqId 5: Historical Market Data Service error message:API scanner subscription cancelled: 5\n"
     ]
    }
   ],
   "source": [
    "sub = ScannerSubscription(\n",
    "    instrument='STK',\n",
    "    locationCode='STK.US.MAJOR',\n",
    "    scanCode='TOP_PERC_GAIN')\n",
    "\n",
    "tagValues = [\n",
    "    TagValue(\"changePercAbove\", \"20\"),\n",
    "    TagValue('priceAbove', 5),\n",
    "    TagValue('priceBelow', 50)]\n",
    "\n",
    "# the tagValues are given as 3rd argument; the 2nd argument must always be an empty list\n",
    "# (IB has not documented the 2nd argument and it's not clear what it does)\n",
    "scanData = ib.reqScannerData(sub, [], tagValues)\n",
    "\n",
    "symbols = [sd.contractDetails.contract.symbol for sd in scanData]\n",
    "print(symbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any scanner query that TWS can do can alse be done through the API. The `scanCode` parameter maps directly to the \"Parameter\" window in the TWS \"Advanced Market Scanner.\" We can verify this by printing out the `scanCode` values available:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "531 scan codes, showing the ones starting with \"TOP\":\n",
      "['TOP_PERC_GAIN', 'TOP_PERC_LOSE', 'TOP_TRADE_COUNT', 'TOP_TRADE_RATE', 'TOP_PRICE_RANGE', 'TOP_VOLUME_RATE', 'TOP_OPEN_PERC_GAIN', 'TOP_OPEN_PERC_LOSE', 'TOP_AFTER_HOURS_PERC_GAIN', 'TOP_AFTER_HOURS_PERC_LOSE', 'TOP_OPT_IMP_VOLAT_GAIN', 'TOP_OPT_IMP_VOLAT_LOSE', 'TOP_STOCK_BUY_IMBALANCE_ADV_RATIO', 'TOP_STOCK_SELL_IMBALANCE_ADV_RATIO', 'TOP_STOCK_BUY_REG_IMBALANCE_ADV_RATIO', 'TOP_STOCK_SELL_REG_IMBALANCE_ADV_RATIO', 'TOP_PERC_GAIN', 'TOP_PERC_LOSE', 'TOP_TRADE_RATE', 'TOP_PERC_GAIN']\n"
     ]
    }
   ],
   "source": [
    "scanCodes = [e.text for e in tree.findall('.//scanCode')]\n",
    "\n",
    "print(len(scanCodes), 'scan codes, showing the ones starting with \"TOP\":')\n",
    "print([sc for sc in scanCodes if sc.startswith('TOP')])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Queries are not limited to stocks. To get a list of all supported instruments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'FUT.HK', 'SSF.NA', 'IND.EU', 'PORTSTK', 'ETF.EQ.US', 'STOCK.ME', 'BOND.GOVT.NON-US', 'FUT.EU', 'FUT.NA', 'BOND', 'IND.HK', 'IND.US', 'WAR.EU', 'FUT.US', 'STOCK.NA', 'BOND.AGNCY', 'SSF.HK', 'BOND.GOVT', 'STK', 'SSF.EU', 'BOND.CD', 'BOND.MUNI', 'SSF.US', 'NATCOMB', 'Global', 'SLB.US', 'STOCK.EU', 'STOCK.HK', 'ETF.FI.US'}\n"
     ]
    }
   ],
   "source": [
    "instrumentTypes = set(e.text for e in tree.findall('.//Instrument/type'))\n",
    "print(instrumentTypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find all location codes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['STK.US', 'STK.US.MAJOR', 'STK.NYSE', 'STK.AMEX', 'STK.ARCA', 'STK.NASDAQ', 'STK.NASDAQ.NMS', 'STK.NASDAQ.SCM', 'STK.BATS', 'STK.US.MINOR', 'STK.PINK', 'STK.OTCBB', 'ETF.EQ.US', 'ETF.EQ.US.MAJOR', 'ETF.EQ.ARCA', 'ETF.EQ.NASDAQ.NMS', 'ETF.EQ.BATS', 'ETF.FI.US', 'ETF.FI.US.MAJOR', 'ETF.FI.ARCA', 'ETF.FI.NASDAQ.NMS', 'ETF.FI.BATS', 'FUT.US', 'FUT.GLOBEX', 'FUT.ECBOT', 'FUT.IPE', 'FUT.NYBOT', 'FUT.NYMEX', 'FUT.NYSELIFFE', 'FUT.CFE', 'FUT.CFECRYPTO', 'FUT.CMECRYPTO', 'FUT.ICECRYPTO', 'IND.US', 'SSF.US', 'SSF.ONE', 'BOND.WW', 'BOND.US', 'BOND.EU.EURONEXT', 'BOND.CD.US', 'BOND.AGNCY.US', 'BOND.GOVT.US', 'BOND.MUNI.US', 'BOND.GOVT.NON-US', 'BOND.GOVT.US.NON-US', 'BOND.GOVT.EU.EURONEXT', 'BOND.GOVT.HK.SEHK', 'SLB.PREBORROW', 'STK.NA', 'STK.NA.CANADA', 'STK.NA.TSE', 'STK.NA.VENTURE', 'STK.NA.MEXI', 'FUT.NA', 'FUT.NA.CDE', 'FUT.NA.MEXDER', 'SSF.NA', 'SSF.NA.MEXDER', 'STK.EU', 'STK.EU.VSE', 'STK.EU.SBF', 'STK.EU.IBIS', 'STK.EU.IBIS-XETRA', 'STK.EU.IBIS-EUSTARS', 'STK.EU.IBIS-USSTARS', 'STK.EU.IBIS-ETF', 'STK.EU.IBIS-NEWX', 'STK.EU.BVME', 'STK.EU.AEB', 'STK.EU.BM', 'STK.EU.SFB', 'STK.EU.SWISS', 'STK.EU.VIRTX', 'STK.EU.EBS', 'STK.EU.LSE', 'STK.EU.OTHER', 'STK.EU.PRA', 'STK.EU.CPH', 'STK.EU.HEX', 'STK.EU.OSE', 'STK.EU.BVL', 'STK.EU.MOEX', 'FUT.EU', 'FUT.EU.BELFOX', 'FUT.EU.DTB', 'FUT.EU.FTA', 'FUT.EU.EDXNO', 'FUT.EU.IDEM', 'FUT.EU.UK', 'FUT.EU.ICEEU', 'FUT.EU.LMEOTC', 'FUT.EU.MEFFRV', 'FUT.EU.MONEP', 'FUT.EU.OMS', 'FUT.EU.SOFFEX', 'IND.EU', 'IND.EU.BELFOX', 'IND.EU.DTB', 'IND.EU.FTA', 'IND.EU.ICEEU', 'IND.EU.MONEP', 'SSF.EU', 'SSF.EU.DTB', 'SSF.EU.EDXNO', 'SSF.EU.IDEM', 'SSF.EU.ICEEU', 'SSF.EU.MEFFRV', 'SSF.EU.OMS', 'SSF.EU.SOFFEX', 'WAR.EU.ALL', 'STK.ME', 'STK.ME.TASE', 'STK.HK', 'STK.HK.TSE_JPN', 'STK.HK.SEHK', 'STK.HK.SEHKNTL', 'STK.HK.SEHKSZSE', 'STK.HK.ASX', 'STK.HK.NSE', 'STK.HK.SGX', 'FUT.HK', 'FUT.HK.HKFE', 'FUT.HK.KSE', 'FUT.HK.NSE', 'FUT.HK.OSE_JPN', 'FUT.HK.SGX', 'FUT.HK.SNFE', 'IND.HK', 'IND.HK.HKFE', 'IND.HK.KSE', 'IND.HK.NSE', 'IND.HK.OSE_JPN', 'IND.HK.SGX', 'IND.HK.SNFE', 'SSF.HK', 'SSF.HK.HKFE', 'SSF.HK.KSE', 'SSF.HK.NSE', 'SSF.HK.SGX', 'NATCOMB', 'NATCOMB.OPT.US', 'NATCOMB.OPT.AMEX', 'NATCOMB.OPT.CBOE', 'NATCOMB.OPT.ISE', 'NATCOMB.OPT.PHLX', 'NATCOMB.OPT.PSE', 'NATCOMB.GLOBEX', 'BACKTEST']\n"
     ]
    }
   ],
   "source": [
    "locationCodes = [e.text for e in tree.findall('.//locationCode')]\n",
    "print(locationCodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "ib.disconnect()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}