# Fail2Ban configuration file # # Regexp to catch known spambots and software alike. Please verify # that it is your intent to block IPs which were driven by # above mentioned bots. [Definition] badbotscustom = TrackBack/1\.02|sogou music spider badbots = 173\.212soso|192\.comagent|1noonbot|1on1searchbot|360Spider|3de_search2|3d_search|3g bot|3gse|404checker|404enemy|50\.nu|80legs|A1 sitemap generator|A1 website download|A6-indexer|Aasp|Abachobot|Abonti|Abotemailsearch|Aboundex|Aboutusbot|Accmonitor compliance server|Accoon|Achulkov\.net page walker|Acme\.spider|Acoonbot|Acquia-crawler|Activetouristbot|Acunetix|Adamm bot|Adbeat_bot|ADmantX|Adminshop\.com|Ad muncher|Advanced email extractor|Aesop_com_spiderman|Aespider|AfD-Verbotsverfahren|Af knowledge now verity spider|Aggregator\:vocus|Ah-ha\.com crawler|Ahrefs|AhrefsBot|AIBOT|Aidu|AiHitBot|Aipbot|Aisiid|Aitcsrobot/1\.1|Ajsitemap|Akamai-sitesnapshot|Alexawebsearchplatform|Alexfdownload|Alexibot|Alkalinebot|All acronyms bot|Alligator|AllSubmitter|AlphaBot|Alpha search agent|Amerla search bot|Amfibibot|Ampmppc\.com|Amznkassocbot|Anarchie|Anemone|Anonymous|Anotherbot|Answerbot|Answerbus|Answerchase prove|Antbot|Antibot|Antisantyworm|Antro\.net|Aonde-spider|Apexoo|Aport|Appengine-google|Appid\: s~stremor-crawler-|Aqua_products|Arabot|Arachmo|Arachnophilia|Aria equalizer|Arianna\.libero\.it|Arikus_spider|Artavisbot|Artera|Art-online\.com|Asaha search engine turkey|Ask|Aspider|ASPSeek|Asterias|Astrofind|Athenusbot|Atlocalbot|Atomic_email_hunter|Attach|Attrakt|Attributor|Augurfind|Auresys|Autobaron crawler|Autoemailspider|Autowebdir|Avsearch-|Axfeedsbot|Axonize-bot|Ayna|B2w|BackDoorBot|Backlink-Ceck|Backlink-check|BacklinkCrawler|Backrub|BackStreet|Backstreet browser|BackWeb|Badass|Bandit|Barkrowler|BatchFTP|Battleztar Bazinga|Baypup|BBBike|BDCbot|BDFetch|Becomebot|Becomejpbot|Beetlebot|Bender|Besserscheitern-crawl|BetaBot|Bigado\.com|Big brother|Bigcliquebot|Big data|Bigfoot|Biglotron|Bilbo|Bilgibetabot|Bilgibot|Bintellibot|Bitacle|Bitlybot|Bitvouseragent|Bizbot|Bizworks retriever|Blackbird|Blackboard|Black\.hole|Black Hole|Blackmask\.net search engine|BlackWidow|Bladder fusion|Blaiz-bee|Blexbot|Blinkx|Blitzbot|Blog conversation project|Blogmyway|Blogpulselive|Blogrefsbot|Blogscope|Blogslive|Bloobybot|Blow|BlowFish|Blt|Bnf\.fr_bot|Boaconstrictor|Boardreader|Boia\.org|Boia-scan-agent|Boi_crawl_00|Boitho|Bolt|Bookmark buddy bookmark checker|Bookmark search tool|Bosug|BotALot|Bot apoena|Botrighthere|Botswana|Bottybot|Bpbot|Braintime_search|Brandprotect|Brandwatch|Brokenlinkcheck\.com|Browseremulator|Browsermob|Bruinbot|Bsearchr&d|Bspider|Btbot|Btsearch|Bubing|Buddy|Buibui|Buildcms crawler|BuiltBotTough|BuiltWith|Bullseye|Bumblebee|BunnySlippers|Buscadorclarin|Buscaplus robi|Butterfly|Buyhawaiibot|Buzzbot|BuzzSumo|Byindia|Byspider|Byteserver|Bzbot|Cacheblaster|Caddbot|Cafi|Calculon|Camcrawler|Camelstampede|Canon-webrecord|Careerbot|Cataguru|Catchbot|CATExplorador|Cazoodle|CCBot|Ccgcrawl|Ccubee|Cd-preload|Cegbfeieh|Ce-preload|Cerberian drtrs|Cert figleafbot|Cfetch|Cfnetwork|Chameleon|Charlotte|Checkbot|Check&get|Checklinks|CheeseBot|Chemiede-nodebot|CherryPicker|Chilkat|ChinaClaw|Chlooe|Chromeframe|Cipinetbot|Cis455crawler|Citeseerxbot|Cizilla|Clariabot|Claritybot|Climate ark|Climateark spider|Cliqzbot|Cloud mapping|Clshttp|Clushbot|Coast scan engine|Coast webmaster pro|Coccoc|Coccocbot-web|Cogentbot|Cognitiveseo|Collapsarweb|Collector|Colocrossing|Combine|Com\.plumanalytics|Connectsearch|Conpilot|Contentsmartz|Contextad bot|Contype|Cookienet|Coolbot|Coolcheck|Copernic|Copier|CopyRightCheck|Copyscape|Core-project|Cosmos|Covario-ids|Cowbot-|Cowdog bot|Crabbybot|Craftbot|C r a w l 3 r|Crawl_application|Crawler@|Crawler43\.ejupiter\.com|Crawler4j|Crawler\.feedback|Crawler_for_infomine|Crawler\.kpricorn\.org|crawl\.sogou\.com|Crawly|CrazyWebCrawler|Creativecommons|Crescent|Cs-crawler|Cse html validator|CSHttp|Cshttpclient|Cuasarbot|Culsearch|Curious|Custo|Cvaulev|Cyberdog|Cybernavi_webget|Cyberpatrol sitecat webbot|Cyberspyder|Cydralspider|D1garabicengine|DatabaseDriverMysqli|DataCha0s|Datafountains|Dataparksearch|Dataprovider\.com|Datascape robot|Dataspearspiderbot|Dataspider|Dattatec\.com|Daumoa|DBLBot|Dcpbot|Declumbot|Deepindex|Deepnet crawler|Deeptrawl|Dejan|Del\.icio\.us-thumbnails|Deltascan|Delvubot|Demandbase-bot|Demon|Der gro§e bildersauger|Der große bildersauger|Deusu|Devil|Dfs-fetch|Diagem|Diamond|Dibot|Didaxusbot|Digext|Digger|Digincore|Digi-rssbot|Digitalarchivesbot|DigitalPebble|Digout4u|DIIbot|Dillo|Dirbuster|Dir_snatch\.exe|Disco|Discobot|Discoverybot|Distilled-reputation-monitor|DittoSpyder|Djangotraineebot|Dkimrepbot|Dmoz downloader|DnyzBot|Docomo|Dof-verify|DomainAppender|DomainCrawler|Domainscan|DomainSigmaCrawler|DomainStatsBot|Domainwatcher bot|Dotbot|Dotspotsbot|Dow jones searchbot|Download|Download Wonder|Doy|Dragonfly|Drip|Drone|Dtaagent|DTS Agent|Dtsearchspider|Dumbot|Dwaar|Dxseeker|Eah|Earth platform indexer|Earth science educator robot|EasyDL|Ebingbong|Ec2linkfinder|Ecairn-grabber|ECatch|ECCP/1\.0|Echoosebot|Ecxi|Edisterbot|Edugovsearch|Egothor|Eidetica\.com|EirGrabber|Elblindo the blind bot|Elisabot|Ellerdalebot|Emailcollector|Email extractor|Emailleach|Emailsiphon|EMail Siphon|Emailwolf|EMail Wolf|Emeraldshield|Empas_robot|Enabot|Endeca|Enigmabot|Enswer neuro bot|Enter user-agent|Entitycubebot|EroCrawler|E-societyrobot|Estylesearch|Esyndicat bot|Eurosoft-bot|Evaal|Evc-batch|Eventware|Everest-vulcan inc\.|Evil|Exabot|Exactsearch|Exactseek|Exooba|Exploder|Express WebPictures|ExtLinksBot|Extractor|ExtractorPro|Extreme Picture Finder|Eyenetie|EyeNetIE|Ezooms|Ez-robot|Factbot|Fairad client|Falcon|Fastbot crawler|Fastbot\.de crawler|Fast data search document retriever|Fast esp|Fast-search-engine|Fatbot|Favcollector|Faviconizer|Favorites sweeper|F-bot test pilot|FDM|Fdse robot|Fedcontractorbot|Fembot|FemtosearchBot|Fetch api request|Fetch_ici|Fgcrawler|FHscan|Filangy|Filehound|Fimap|Findanisp\.com_isp_finder|Findlinks|Findweb|Firebat|Firefox/7\.0|Firstgov\.gov search|Flaming attackbot|Flamingo_searchengine|Flashcapture|FlashGet|Flickysearchbot|Fluffy the spider|Flunky|Focused_crawler|Followsite|Foobot|Fooooo_web_video_crawl|Fopper|Formulafinderbot|Forschungsportal|Francis|Fr_crawler|Freeuploader|Freewebmonitoring sitechecker|Freshcrawler|Freshdownload|Freshlinks\.exe|Friendfeedbot|Frodo\.at|Froggle|FrontPage|Froola bot|Full_breadth_crawler|Fu-nbi|Funnelback|Furlbot|FyberSpider|Fyrebot|G10-bot|Gaisbot|GalaxyBot|Gazz|Gbplugin|Generate_infomine_category_classifiers|Genevabot|Geniebot|Genieo|Geomaxenginebot|Geometabot|Geonabot|Geovisu|GermCrawler|Gethtmlcontents|Getintent|Getleft|GetRight|Getsmart|Geturl\.rexx|GetWeb|Giant|Gigablast|Gigablastopensource|Gigabot|G-i-g-a-b-o-t|Girafabot|Gleamebot|Gnome-vfs|Go-Ahead-Got-It|Goforitbot|Goforit\.com|Go-http-client|Gold crawler|Goldfire server|Golem|Goodjelly|Gordon-college-google-mini|Goroam|Goseebot|Gotit|Govbot|GoZilla|Go!Zilla|Gpu p2p crawler|Grabber|GrabNet|Grafula|GrapeFX|Grapeshot|Grbot|Greenyogi|GridBot|Gromit|Grub|Gslfbot|GT\:\:WWW|Gulliver|Gulperbot|Gurujibot|Gvcbot\.com|Gvc business crawler|Gvc crawler|Gvc search bot|Gvc web crawler|Gvc weblink crawler|Gvc world links|Haansoft|HaosouSpider|Happyfunbot|Harvest|Hatena antenna|Havij|Hawler|Hcat|Hclsreport-crawler|Hd nutch agent|Header_test_client|HEADMasterSEO|Healia|Helix|Here will be link to crawler site|Heritrix|Hiscan|Hisoftware accmonitor server|Hisoftware accverify|Hitcrawler|Hivabot|Hloader|Hmsebot|HMView|Hoge|Holmes|Homepagesearch|Hooblybot-image|Hoowwwer|Hostcrawler|Hsft - link scanner|Hsft - lvu scanner|Hslide|Ht\://check|Htdig|Html link validator|HTMLparser|Httplib|HTTP\:\:Lite|HTTrack|Huaweisymantecspider|Hul-wax|Humanlinks|HybridBot|Hyperestraier|Hyperix|Iblog|Ibuena|Icab|Icds-ingestion|Ichiro|Icopyright conductor|IDBot|IDG/UK|Id-search|Ieautodiscovery|Iecheck|Ihwebchecker|Iiitbot|Iim_405|IlseBot|Iltrovatore|Imagebot|Image Fetch|Image-fetcher|Imagefortress|Imageshereimagesthereimageseverywhere|Image stripper|Image Sucker|Imagevisu|Imds_monitor|Imo-google-robot-intelink|Inagist\.com url crawler|IndeedBot|Indexer|Industry cortex webcrawler|Indylabs_marius|Indy Library|Inelabot|Inet32 ctrl|Inetbot|Infolink|Infomine|InfoNaviRobot|Informant|Info seeker|Infoseek sidewinder|InfoTekies|Infousabot|Ingrid|Inktomi|Insightscollector|Insightsworksbot|Inspirebot|Instabid|Insumascout|Intelix|Intelliseek|InterGET|Internetlinkagent|Internet Ninja|Internet radio crawler|InternetSeer|InternetVista monitor|Interseek|Ioi|Ipadd bot|Ips-agent|Ipselonbot|Ip-web-crawler\.com|Iria|IRLbot|Iron33|Isara|Isearch|Isilox|Iskanie|IstellaBot|Its-learning crawler|Iu_csci_b659_class_crawler|Ivia|Jadynave|JamesBOT|Jbot|Jbrofuzz|Jemmathetourist|JennyBot|Jetbot|Jetbrains omea pro|JetCar|JikeSpider|Jim|Jobo|Jobspider_ba|Joc|JOC Web Spider|Joedog|Joomla|Jorgee|Joyscapebot|Jspyda|Junut bot|JustView|Jyxobot|Kakclebot|Kalooga|Katatudo-spider|Kbeta1|Keepni web site monitor|Kenjin\.spider|Kenjin Spider|Keybot translation-search-machine|Keywenbot|Keyword\.density|Keyword Density|Kinjabot|Kitenga-crawler-bot|Kiwistatus|Kmbot-|Kmccrew bot search|Knight|Knowitall|Knowledge\.com|Knowledge engine|Koepabot|Koninklijke|Korniki|Kozmosbot|Krowler|Ksbot|K\.s\.bot|Kuloko-bot|Kulturarw3|Kummhttp|Kurzor|Kyluka crawl|Labhoo|Labourunions411|Lachesis|Lament|Lamerexterminator|Lanshanbot|Lapozzbot|Larbin|Lbot|Leaptag|LeechFTP|LeechGet|Letscrawl\.com|LexiBot|Lexxebot|Lftp|Libcrawl|Libiviacore|Libw|LibWeb|Libwhisker|Lightspeedsystems|Likse|Linguee bot|Linkalarm|Linkbot|Linkcheck by siteimprove\.com|Linkchecker|Link_checker|Link checker|Linkcheck scanner|Linkdexbot|Linkdex\.com|LinkextractorPro|Linklint|Linklooker|Linkman|LinkpadBot|LinkScan|LinksManager|Linksmanager\.com_bot|Links sql|Linksweeper|Link validator|LinkWalker|LinqiaMetadataDownloaderBot|LinqiaRSSBot|LinqiaScrapeBot|Lipperhey|Litefinder|Litemage_walker|Litlrbot|Little grabber at skanktale\.com|Livelapbot|Lm harvester|Lmqueuebot|Lmspider|LNSpiderguy|Loadtimebot|Localcombot|Locust|Lolongbot|Lookbot|Lsearch|Lssbot|Lt scotland checklink|Ltx71|L\.webis|LWP|LWP\:\:Simple|Lwp-trivial|Lycos_spider|Lydia entity spider|Lynnbot|Lytranslate|Magnet|Mag-Net|Magpie-crawler|Magus bot|Mail\.ru|Mail\.RU_Bot|Mainseek_bot|Majestic12|Mammoth|Map robot|MarkMonitor|MarkWatch|Masagool|Masidani_bot_|Masscan|Mass Downloader|Mata\.hari|Mata Hari|Matentzn at cs dot man dot ac dot uk|MauiBot|Maxamine\.com|Maxomobot|Mcbot|Meanpathbot|Mediawords|Medrabbit|MegaIndex\.ru|Megite|Memacbot|Memo|Mendeleybot|Mercator-|Mercuryboard_user_agent_sql_injection\.nasl|Metacarta|Metaeuro web search|Metager2|Metagloss|Metal crawler|Metaquerier|Metaspider|Metaspinner|Metauri|Mfcrawler|MFC_Tear_Sample|Mfhttpscan|Microsoft Data Access|Microsoft URL Control|MIDown tool|MIIxpc|Minibot|Minirank|Mini-robot|Mirror|Missigua locator|Mister\.pix|Mister PiX|Miva|MJ12bot|Mnogosearch|Mod_accessibility|Moduna\.com|Moget|Mojeek|Monkeycrawl|Morfeus Fucking Scanner|Moses|Mowserbot|Mqbot|Mr\.4x3|Mse360|MSFrontPage|MSIECrawler|Msindianwebcrawl|Msmobot|Msnptc|Msrabot|Msrbot|MS Web Services Client Protocol|Mt-soft|Muhstik-scan|Multitext|Musobot|Myapp|Mycompanybot|Mycrawler|Myengines-us-bot|Myfamilybot|My-heritrix-crawler|My_little_searchengine_project|Myra|Nabot|Najdi\.si|Nambu|Name Intelligence|Nameprotect|Nasa search|Natchcvs|Natweb-bad-link-mailer|Naver|Navroad|NearSite|Nec-meshexplorer|Needle|Neosciocrawler|Nerdbynature\.bot|Nerdybot|Nerima-crawl-|Nessus|Nestreader|NetAnts|Netcarta cyberpilot pro|Netcraft|NetEstate NE Crawler|Netexperts|Netid\.com bot|NetLyzer|NetMechanic|Netprospector|Netresearchserver|Netseer|Netshift|Netsongbot|Netsparker|NetSpider|Netsrcherp|Nettrack|Net\:\:trackback|Net Vampire|Netvibes|NetZIP|Newmedhunt|News bot|Newsgatherer|Newsgroupreporter|News_search_app|Newstrovebot|NextGenSearchBot|Nextthing\.org|Nibbler|Nicebot|NICErsPRO|Niki-bot|Nikto|NimbleCrawler|Nimbus-1|Ninetowns|Ninja|Njuicebot|Nlese|Nmap|Nogate|Norbert the spider|Noteworthybot|NPbot|Nrcan intranet crawler|Nsdl_search_bot|Nuggetize\.com bot|Nusearch spider|Nutch|Nu_tch|Nwspider|Nymesis|Nys-crawler|Objectssearch|OBot|Obvius external linkcheck|Ocelli|Octopus|Odp entries t_st|Oegp|Offline\.explorer|Offline Explorer|Offline Navigator|Ogspider|Omiexplorer_bot|Omniexplorer|Omnifind|Omniweb|Onetszukaj|Online link validator|Oozbot|Openbot|Openfind|Openintelligencedata|Openisearch|OpenLinkProfiler|Openlink virtuoso rdf crawler|Opensearchserver_bot|OpenVAS|Opidig|Optidiscover|Oracle secure enterprise search|Oracle ultra search|OrangeBot|OrangeSpider|Orisbot|Ornl_crawler|Ornl_mercury|Osis-project\.jp|Oso|OutclicksBot|OutfoxBot|Outfoxmelonbot|Owler-bot|Owsbot|Ozelot|P3p client|PageAnalyzer|Page Analyzer|Pagebiteshyperbot|Pagebull|Pagedown|Pagefetcher|PageGrabber|Pagepeeker|Pagerank monitor|Page scorer|PageScorer|Page_verifier|Pamsnbot\.htm|Panopy bot|Panscient|Pansophica|Papa Foto|Paperlibot|Parasite|Parsijoo|Pathtraq|Pattern|Patwebbot|Pavuk|Paxleframework|Pbbot|PcBrowser|Pcore-http|Pd-crawler|PECL\:\:HTTP|Penthesila|PeoplePal|Perform_crawl|Perman|Personal ultimate crawler|PHPCrawl|Phpdig|Php version tracker|Picosearch|Picscout|Picsearch|PictureFinder|Pieno robot|Pimonster|Pi-Monster|Pipbot|Pipeliner|Pita|Pixfinder|Pixray|Piyushbot|Planetwork bot search|PleaseCrawl|Plucker|Plukkie|Plumanalytics|Plumtree|Pockey|Pocohttp|POE-Component-Client-HTTP|Pogodak\.ba|Pogodak\.co\.yu|Poirot|Polybot|Pompos|Poodle predictor|Popscreenbot|Postpost|Privacyfinder|Probethenet|Projectwf-java-test-crawler|ProPowerBot|ProWebWalker|Proxem websearch|Proximic|Proxy crawler|Psbot|Pss-bot|Psycheclone|Pub-crawler|Pucl|Pulsebot|Pump|Pwebot|PxBroker|PyCurl|Qeavis agent|Qfkbot|Qualidade|Qualidator\.com bot|Quepasacreep|Queryn\.metasearch|Queryn metasearch|QueryN Metasearch|Quest\.durato|Quick-Crawler|Quintura-crw|Qunarbot|Qwantify|Qweerybot|Qweery_robot\.txt_checkbot|R2ibot|R6_commentreader|R6_feedfetcher|R6_votereader|Rabot|Radian6|Radiation retriever|Rampybot|RankActive|RankFlex|RankingBot|RankingBot2|Rankivabot|Rankur|RankurBot|Rational sitecheck|Rcstartbot|RealDownload|Reaper|RebelMouse|Rebi-shoveler|Recorder|Redbot|Redcarpet|RedesScrapy|ReGet|RepoMonkey|Research robot|Riddler|Riight|Ripper|Risenetbot|Riverglassscanner|Robopal|Robosourcer|Robotek|Robozilla|RocketCrawler|Roger|Rogerbot|Rome client|Rondello|Rotondo|Roverbot|Rpt-httpclient|Rtgibot|Rufusbot|Runnk online rss reader|Runnk rss aggregator|S2bot|Safaribookmarkchecker|Safednsbot|Safetynet robot|Saladspoon|SalesIntelligent|Sapienti|Sapphireweb|SBIder|Sbl-bot|ScanAlert|Scanbot|Scan\.lol|Scfcrawler|Scich|Scientificcommons\.org|Scollspider|Scooperbot|Scooter|Scoutjet|Scrapebox|Scrapy|Scrawltest|Screaming|ScreenerBot|Scrubby|Scspider|Scumbot|Searcharoo\.net|Searchblox|Searchbot|Search-channel|Searchengine|Search-engine-studio|Searchestate|Searchhippo\.com|Searchit-bot|Search\.kumkie\.com|Searchmarking|Searchmarks|Searchmee|SearchmetricsBot|Searchmining|Searchnowbot|Searchpreview|Search publisher|Searchspider\.com|Search\.updated\.com|Search\.usgs\.gov|Search x-bot|Searqubot|Seb spider|Seekbot|Seeker\.lookseek\.com|Seeqbot|Seeqpod-vertical-crawler|Selflinkchecker|Semager|Semanticdiscovery|Semantifire|Semisearch|Semrush|SemrushBot|Seoengworldbot|SEOkicks|SEOlyticsCrawler|Seomoz|SEOprofiler|Seoscanners|SEOstats|Sexsearcher|Seznam|SeznamBot|Shablastbot|Shadowwebanalyzer|Shareaza|Shelob|Sherlock|Shim-crawler|Shodan|Shopsalad|Shopwiki|Showlinks|Showyoubot|Siclab|Silk|Simplepie|Siphon|SISTRIX|Sitebeam|Sitebot|Sitecheck|SiteExplorer|Sitefinder|Siteguardbot|Siteimprove|SiteLockSpider|Siteorbiter|SiteSnagger|SiteSucker|Site Sucker|Sitesweeper|Sitevigil|Sitexpert|Skimbot|Skimwordsbot|Skreemrbot|Skywalker|Slackbot-LinkExpanding|Sleipnir|Slow-crawler|SlySearch|Smart-crawler|SmartDownload|Smarte bot|Smartwit\.com|SMTBot|Snake|Snapbot|Snap\.com beta crawler|Snappreviewbot|Snappy|Snookit|Snooper|Snoopy|SocialRankIOBot|Societyrobot|Socscibot|Soft411 directory|Sogou|sogouspider|Sogou web spider|Sohu agent|Sohu-search|Sokitomi crawl|Solbot|Sondeur|Sootle|Sosospider|Sottopop|Spacebison|Space bison|SpaceBison|Space fung|Spammen|SpankBot|Spanner|Spatineo monitor controller|Spatineo serval controller|Spatineo serval getmapbot|Spbot|Special_archiver|Speedy|Sphere scout|Sphider|Spiderengine|Spiderku|Spiderman|Spider\.terranautic\.net|Spinn3r|Spinne|Sportcrew-bot|Sproose|SputnikBot|Spyder3\.microsys\.com|Sqlmap|Sqlworm|Squidclamav_redirector|Squid-prefetch|Sq webscanner|Sqworm|Srevbot|Sslbot|Ssm agent|S~stremor-crawler|Stackrambler|Stardownloader|Statbot|Statcrawler|Statedept-crawler|Steeler|Stegmann-bot|Stero|Stripper|Stumbler|Suchclip|Sucker|Sucuri|Sumeetbot|Sumitbot|Summizebot|Summizefeedreader|Sunrise xp|SuperBot|SuperHTTP|Superlumin downloader|Superpagesbot|Supremesearch\.net|Supybot|Surdotlybot|Surf|Surfbot|SurveyBot|Suzuran|Swebot|Swiftbot|Swish-e|Sygolbot|Synapticwalker|Syntryx ant scout chassis pheromone|Sysscan|Systemsearch-robot|Szukacz|T0PHackTeam|T8Abot|Tailrank|TAkeOut|Talkro web-shot|Tamu_crawler|Tapuzbot|Tarantula|Targetblaster\.com|Targetyournews\.com bot|Tausdatabot|Taxinomiabot|Teamsoft wininet component|Tecomi bot|Teezirbot|Teleport|TeleportPro|Telesoft|Telesphoreo|Telesphorep|Teradex mapper|Teragram_crawler|Terrawizbot|Testbot|Testing of bot|Textbot|Thatrobotsite\.com|The dyslexalizer|The\.intraformant|The Intraformant|TheNomad|Theophrastus|Theusefulbot|Thumbbot|Thumbnail\.cz robot|Thumbshots-de-bot|T-h-u-n-d-e-r-s-t-o-n-e|Tigerbot|Tighttwatbot|TightTwatBot|Tineye|Titan|Toata|Tocrawl|To-dress_ru_bot_|To-night-bot|Topicalizer|Topicblogs|Toplistbot|Topserver php|Topyx-crawler|Touche|Tourlentascanner|Toweyabot|Tpsystem|Traazi|Tracemyfile|Transgenikbot|Travelbot|Travellazerbot|Travel-search|Treezy|Trendiction|Trendictionbot|Trendiction\.com|Trendiction\.de|Trex|Tridentspider|Trovator|True_Robot|Tscholarsbot|Tsm translation-search-machine|Tswebbot|Tulipchain|Turingos|Turnitin|TurnitinBot|Tutorgigbot|Tweetedtimes bot|Tweetmemebot|TwengaBot|Twice|Twikle|Twinuffbot|Twisted pagegetter|Twitturls|Twitturly|Tygobot|Tygoprowler|Typhoeus|Uberbot|Ucb-nutch|Udmsearch|Ufam-crawler-|Ultraseek|Unchaos|Unidentified|UnisterBot|Unitek uniengine|Universalsearch|Unwindfetchor|Uoftdb_experiment|Updated|Urlappendbot|Urlblaze|Urlchecker|Url-checker|Urlck|Url control|Urldispatcher|Url_gather|Urlspiderpro|URLy\.Warning|URLy Warning|Usaf afkn k2spider|Usasearch|U\.s\. government printing office|Uss-cosmix|Usyd-nlp-spider|Vacobot|Vacuum|Vadixbot|Vagabondo|Validator|Valkyrie|VB Project|Vbseo|VCI|Vci webviewer vci webviewer win32|Verbstarbot|VeriCiteCrawler|Verifactrola|Verity-url-gateway|Vermut|Versus crawler|Versus\.integis\.ch|Viasarchivinginformation\.html|VidibleScraper|Vipr|Virus_detector|Virus-detector|Virusdie|Visbot|Vishal for clia|Visweb|Vital search'n urchin|Vlad|Vlsearch|Vmbot|Vocusbot|VoidEYE|Voil|Voilabot|Voltron|Vortex|Voyager|Vspider|W3c_unicorn|W3c-webcon|W3search|Wacbot|Wallpapers/3\.0|WallpapersHD|Wanadoo|WASALive-Bot|Wastrix|Water conserve portal|Water conserve spider|Watzbot|Wauuu|Wavefire|Waypath|Wazzup|Wbdbot|WBSearchBot|Web-agent|Webalta|Webaltbot|WebAuto|Web Auto|WebBandit|Webbot|Webbul-bot|Webcapture|Web ceo online robot|Webcheck|Webclipping\.com|WebCollage|Web Collage|WebCopier|Webcopy|Webcorp|Webcrawler|Web crawler|Webcrawl\.net|Webdatacentrebot|WEBDAV|Web downloader|Webdownloader for x|Webdup|Webemailextrac|WebEnhancer|Web Enhancer|WebFetch|Web Fetch|WebFuck|Web Fuck|Webgather|Webgobbler|WebGo IS|Web\.image\.collector|Web image collector|WebImageCollector|Webimages|Webinator-search2|Webinator-wbi|Webindex|Weblayers|WebLeacher|Weblexbot|Weblinker|Web link validator|Weblyzard|Web magnet|Webmastercoffee|Webmasterworld extractor|WebmasterWorldForumBot|Webmeup-crawler|Webminer|Webmoose|Webot|WebPix|Web Pix|WebReaper|Webripper|WebSauger|Web Sauger|Webscan|Websearchbench|Webshag|Website|Web site downloader|WebsiteExtractor|WebsiteQuester|Website Quester|Web-sniffer|Webspear|Websphinx|Webspider|Webster|WebStripper|WebSucker|Web Sucker|Webtrafficexpress|Webtrends link analyzer|Webvac|Webwalk|Webwasher|Webwatch|WebWhacker|Webxm|WebZIP|Weddings\.info|Wenbin|Wepa|Wep search|Werelatebot|WeSEE|Whack|Whacker|Whatweb|Whirlpool web engine|Who\.is Bot|Whowhere robot|Widow|Wikiabot|Wikio|Wikiwix-bot-|Winhttp|WinHTTrack|Wire|Wisebot|WiseGuys Robot|WISENutbot|Wish-la|Wish-project|Wisponbot|Wmcai-robot|Wminer|Wmsbot|Wonderbot|Woobot|Woriobot|Worldshop|Worqmada|Wotbox|Wprecon|WPScan|Wume_crawler|Www collector|Www-collector-e|WWW-Collector-E|Www-mechanize|WWW-Mechanize|WWW\:\:Mechanize|WWWOFFLE|Wwwrobot|Wwwster|Wwwwanderer|Wwwxref|Wysigot|X09Mozilla|X22Mozilla|Xaldon|Xaldon_WebSpider|Xaldon WebSpider|X-clawler|X-crawler|Xenu|Xerka metabot|Xerka webbot|Xget|Xirq|Xmarksfetch|Xpymep1\.exe|Xqrobot|Yacybot|Yacy\.net|Yanga worldsearch bot|Yarienavoir\.net|Yasaklibot|Yats crawler|Ybot|Yebolbot|Yellowjacket|Yeti|Y!j|Yolinkbot|Yooglifetchagent|Yoono|Yottacars_bot|YoudaoBot|Yourls|Z-add link checker|Zade|Zagrebin|Zao|Zauba|Zauba\.io|Zedzo\.validate|Zermelo|Zeus|Zgrab|Zibber-v|Zimeno|Zing-bottabo|Zipppbot|Zitebot|ZmEu|Zongbot|Zoomspider|Zotag search|Zsebot|Zuibot|ZumBot|ZyBorg|Zyte failregex = ^ .*(?:%(badbots)s|%(badbotscustom)s).*$ ignoreregex = datepattern = ^[^\[]*\[({DATE}) {^LN-BEG} # DEV Notes: # List of bad bots fetched from https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list # Generated on Mon Oct 15 17:34:39 PDT 2018 manually. # # Author: Yaroslav Halchenko, Alex Gurrola