--- - user_agent: monitoring360bot/1.1 bot: name: 360 Monitoring category: Site Monitor url: https://www.360monitoring.io producer: name: Plesk International GmbH url: https://www.plesk.com - user_agent: 360Spider-Image bot: name: 360Spider-Image category: Search bot url: https://www.so.com/help/help_3_2.html producer: name: Online Media Group, Inc. url: "" - user_agent: 360Spider-Video bot: name: 360Spider-Video category: Search bot url: https://www.so.com/help/help_3_2.html producer: name: Online Media Group, Inc. url: "" - user_agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider bot: name: 360Spider category: Search bot url: https://www.so.com/help/help_3_2.html producer: name: Online Media Group, Inc. url: "" - user_agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider bot: name: 360Spider category: Search bot url: https://www.so.com/help/help_3_2.html producer: name: Online Media Group, Inc. url: "" - user_agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider bot: name: 360Spider category: Search bot url: https://www.so.com/help/help_3_2.html producer: name: Online Media Group, Inc. url: "" - user_agent: 'ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com' bot: name: ADMantX - user_agent: ias-va/3.1 (+https://www.admantx.com/service-fetcher.html) bot: name: ADmantX Service Fetcher category: Service bot url: https://www.admantx.com/service-fetcher.html - user_agent: Aboundex/0.3 (http://www.aboundex.com/crawler/) bot: name: Aboundexbot category: Search bot url: http://www.aboundex.com/crawler/ producer: name: Aboundex.com url: http://www.aboundex.com - user_agent: Mozilla/5.0 (compatible; AcoonBot/4.11.1; +http://www.acoon.de/robot.asp) bot: name: Acoon category: Search bot url: http://www.acoon.de/robot.asp producer: name: Acoon GmbH url: http://www.acoon.de - user_agent: AddThis.com robot tech.support@clearspring.com bot: name: AddThis.com category: Social Media Agent url: "" producer: name: Clearspring Technologies, Inc. url: http://www.clearspring.com - user_agent: ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com) bot: name: Alexa Crawler category: Search bot url: https://support.alexa.com/hc/en-us/sections/200100794-Crawlers producer: name: Alexa Internet url: https://www.alexa.com - user_agent: Mozilla/5.0 (compatible; alexa site audit/1.0; http://www.alexa.com/help/webmasters; ) bot: name: Alexa Site Audit category: Site Monitor url: https://support.alexa.com/hc/en-us/articles/200450194 producer: name: Alexa Internet url: https://www.alexa.com - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) bot: name: Amazonbot category: AI Search Crawler url: https://developer.amazon.com/amazonbot producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Amazon Route 53 Health Check Service; ref:xxxxxxxx-xxxx-xxxxxxxxx-xxxxxxxxxxxx; report http://amzn.to/xxxxxxx bot: name: Amazon Route53 Health Check category: Service Agent producer: name: Amazon Web Services url: https://aws.amazon.com/ - user_agent: Amazon-Route53-Health-Check-Service (ref b0eb04d5-cb5e-40e7-839b-558e52fc3f0d; report http://amzn.to/1vsZADi) bot: name: Amazon Route53 Health Check category: Service Agent producer: name: Amazon Web Services url: https://aws.amazon.com/ - user_agent: AmorankSpider/0.1; +http://amorank.com/webcrawler.html bot: name: Amorank Spider category: Crawler url: http://amorank.com/webcrawler.html producer: name: Amorank url: http://www.amorank.com - user_agent: 'Curious George - www.analyticsseo.com/crawler' bot: name: Analytics SEO Crawler category: Crawler url: http://www.analyticsseo.com/crawler producer: name: Analytics SEO url: http://www.analyticsseo.com - user_agent: ApacheBench/2.3 bot: name: ApacheBench category: Benchmark url: https://httpd.apache.org/docs/2.4/programs/ab.html producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10 _1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot) bot: name: Applebot category: AI Search Crawler url: https://support.apple.com/en-us/119829 producer: name: Apple Inc url: https://www.apple.com/ - user_agent: AppSignalBot/1.0 (+https://appsignal.com) bot: name: AppSignalBot category: Site Monitor url: https://docs.appsignal.com/uptime-monitoring/ producer: name: AppSignal url: https://appsignal.com/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1) bot: name: Applebot category: AI Search Crawler url: https://support.apple.com/en-us/119829 producer: name: Apple Inc url: https://www.apple.com/ - user_agent: Applebot-Extended bot: name: Applebot-Extended category: AI Data Scraper url: https://support.apple.com/en-us/119829 producer: name: Apple Inc url: https://www.apple.com/ - user_agent: iTMS bot: name: iTMS category: Crawler url: https://support.apple.com/en-us/119829 producer: name: Apple Inc url: https://www.apple.com/ - user_agent: Arachni/v1.5.1 bot: name: Arachni category: Security Checker url: https://www.arachni-scanner.com/ producer: name: Sarosys LLC url: https://www.sarosys.com/ - user_agent: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) bot: name: Ask Jeeves category: Search bot url: "" producer: name: Ask Jeeves Inc. url: http://www.ask.com - user_agent: Mozilla/5.0 (compatible;AspiegelBot) bot: name: AspiegelBot category: Crawler url: https://aspiegel.com/ producer: name: Huawei url: https://www.huawei.com/ - user_agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot) bot: name: AspiegelBot category: Crawler url: https://aspiegel.com/ producer: name: Huawei url: https://www.huawei.com/ - user_agent: AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com) bot: name: Awario category: Feed Fetcher url: https://awario.com/bots.html producer: name: TechFusion Ltd. url: https://www.techfusion.com.cy/ - user_agent: AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com) bot: name: Awario category: Search bot url: https://awario.com/bots.html producer: name: TechFusion Ltd. url: https://www.techfusion.com.cy/ - user_agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) bot: name: BLEXBot Crawler category: Crawler url: http://webmeup-crawler.com producer: name: WebMeUp url: http://webmeup.com - user_agent: BUbiNG (+http://law.di.unimi.it/BUbiNG.html) bot: name: BUbiNG category: Crawler url: http://law.di.unimi.it/BUbiNG.html producer: name: The Laboratory for Web Algorithmics (LAW) url: 'http://law.di.unimi.it/software.php#buging' - user_agent: Backlink-Check.de (+http://www.backlink-check.de/bot.html) bot: name: Backlink-Check.de category: Crawler url: http://www.backlink-check.de/bot.html producer: name: Mediagreen Medienservice url: http://www.backlink-check.de - user_agent: BacklinkCrawler (http://www.backlinktest.com/crawler.html) bot: name: BacklinkCrawler category: Crawler url: http://www.backlinktest.com/crawler.html producer: name: 2.0Promotion GbR url: http://www.backlinktest.com - user_agent: Baiduspider+(+http://www.baidu.com/search/spider.htm) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Baiduspider-image+(+http://www.baidu.com/search/spider.htm) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; baidu Transcoder;) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Mozilla/5.0 (Linux;u;Android 2.3.7;zh-cn;) AppleWebKit/533.1 (KHTML,like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; +http://www.baidu.com/search/spider.html) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8;baidu Transcoder) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729) bot: name: Baidu Spider category: Search bot url: http://www.baidu.com/search/spider.htm producer: name: Baidu url: http://www.baidu.com - user_agent: Barkrowler/0.9 (+http://www.exensa.com/crawl) bot: name: Barkrowler category: Crawler url: http://www.exensa.com/crawl - user_agent: Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers) bot: name: BazQux Reader url: https://bazqux.com/fetcher category: Feed Fetcher producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm) bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: msnbot/1.1 (+http://search.msn.com/msnbot.htm) bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: msnbot/2.0b (+http://search.msn.com/msnbot.htm) bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: msnbot/2.0b v1394720947.6703 bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: MSNBot/Nutch-1.5.1 bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: bitlybot/3.0 bot: name: BitlyBot category: Crawler url: https://bitly.com producer: name: Bitly, Inc. url: https://bitly.com - user_agent: Blackbox Exporter/0.24.0 bot: name: Blackbox Exporter category: Site Monitor url: https://github.com/prometheus/blackbox_exporter producer: name: Prometheus url: https://prometheus.io/ - user_agent: Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot) bot: name: Blekkobot category: Search bot url: http://blekko.com/about/blekkobot producer: name: Blekko url: http://blekko.com - user_agent: Bloglovin/1.0 (http://www.bloglovin.com; 1 subscribers) bot: name: Bloglovin url: http://www.bloglovin.com category: Feed Fetcher producer: name: "" url: "" - user_agent: Blogtrottr/2.0 bot: name: Blogtrottr url: "" category: Feed Fetcher producer: name: Blogtrottr Ltd url: https://blogtrottr.com/ - user_agent: BoardReader Favicon Fetcher /1.0 info@boardreader.com bot: name: BoardReader category: Search bot url: https://boardreader.com/ producer: name: Effyis Inc url: https://boardreader.com/ - user_agent: BoardReader Blog Indexer(http://boardreader.com) bot: name: BoardReader Blog Indexer category: Crawler producer: name: BoardReader url: https://boardreader.com/ - user_agent: Mozilla/5.0 (compatible; BountiiBot/1.1; +http://bountii.com/contact.php) bot: name: Bountii Bot category: Search bot url: http://bountii.com/contact.php producer: name: Bountii Inc. url: http://bountii.com - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me) bot: name: BrandVerity category: Crawler url: https://www.brandverity.com/why-is-brandverity-visiting-me producer: name: BrandVerity, Inc. url: https://www.brandverity.com/ - user_agent: Browsershots bot: name: Browsershots category: Service Agent url: http://browsershots.org/faq producer: name: Browsershots.org url: http://browsershots.org - user_agent: Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html) bot: name: Buck category: Search bot url: https://hypefactors.com/ producer: name: Hypefactors A/S url: https://hypefactors.com/ - user_agent: Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 bot: name: Butterfly Robot category: Search bot url: http://labs.topsy.com/butterfly producer: name: Topsy Labs url: http://labs.topsy.com - user_agent: Mozilla/5.0 (Macintosh; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8 bot: name: Butterfly Robot category: Search bot url: http://labs.topsy.com/butterfly producer: name: Topsy Labs url: http://labs.topsy.com - user_agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.5668.1446 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com bot: name: Bytespider category: Search bot url: https://bytedance.com/ producer: name: ByteDance Ltd. url: https://bytedance.com/ - user_agent: CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html) bot: name: CATExplorador category: Search bot url: https://fundacio.cat/ca/domini/ producer: name: Fundació puntCAT url: https://fundacio.cat/ca/domini/ - user_agent: Mozilla/5.0 (compatible; CareerBot/1.1; +http://www.career-x.de/bot.html) bot: name: CareerBot category: Crawler url: http://www.career-x.de/bot.html producer: name: career-x GmbH url: http://www.career-x.de - user_agent: Castro 2, Episode Duration Lookup bot: name: Castro 2 category: Service Agent url: http://supertop.co/castro/ producer: name: Supertop url: http://supertop.co - user_agent: Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot) bot: name: Cliqzbot category: Crawler url: http://cliqz.com/company/cliqzbot producer: name: 10betterpages GmbH url: http://cliqz.com - user_agent: Mozilla/5.0 (compatible; Cloudflare-AMP/1.0; +https://amp.cloudflare.com/doc/fetcher.html) AppleWebKit/534.34 bot: name: CloudFlare AMP Fetcher category: Crawler url: https://amp.cloudflare.com/doc/fetcher.html producer: name: CloudFlare url: http://www.cloudflare.com - user_agent: Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34 bot: name: CloudFlare Always Online category: Site Monitor url: https://www.cloudflare.com/always-online producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Cloudflare-SSLDetector bot: name: Cloudflare SSL Detector category: Site Monitor url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Cloudflare Custom Hostname Verification bot: name: Cloudflare Custom Hostname Verification category: Service Agent url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: 'Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: )' bot: name: Cloudflare Traffic Manager category: Site Monitor url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/) bot: name: Cloudflare Security Insights category: Site Monitor url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: collectd/5.5.1 bot: name: Collectd url: https://collectd.org/ category: Site Monitor producer: name: Collectd url: https://collectd.org/ - user_agent: CommaFeed/1.0 (http://www.commafeed.com) bot: name: CommaFeed url: http://www.commafeed.com category: Feed Fetcher producer: name: "" url: "" - user_agent: coccocbot-web/1.0 (+http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot-ads/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot-fast/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot-shopping/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Mozilla/5.0 (compatible; coccocbot/1.0; +http://help.coccoc.com/searchengine) bot: name: Cốc Cốc Bot url: https://help.coccoc.com/en/search-engine/coccoc-robots category: Search bot producer: name: Cốc Cốc url: https://coccoc.com/ - user_agent: Datadog Agent/5.10.1 bot: name: Datadog Agent url: https://github.com/DataDog/dd-agent category: Site Monitor producer: name: Datadog url: https://www.datadoghq.com/ - user_agent: Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 bot: name: Datanyze url: "" category: Crawler producer: name: Datanyze url: https://www.datanyze.com - user_agent: Mozilla/5.0 (compatible; Dataprovider/6.92; +https://www.dataprovider.com/) bot: name: Dataprovider category: Crawler url: "" producer: name: Dataprovider B.V. url: https://www.dataprovider.com/ - user_agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0; bot: name: Daum category: Search bot url: http://tab.search.daum.net/aboutWebSearch_en.html producer: name: Daum Communications Corp. url: http://www.kakaocorp.com/main - user_agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1 bot: name: Daum category: Search bot url: http://tab.search.daum.net/aboutWebSearch_en.html producer: name: Daum Communications Corp. url: http://www.kakaocorp.com/main - user_agent: Mozilla/5.0 (compatible; Dazoobot/0.1; +http://dazoo.fr) bot: name: Dazoobot category: Search bot url: "" producer: name: DAZOO.FR url: http://dazoo.fr - user_agent: Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html) bot: name: Discobot category: Search bot url: http://discoveryengine.com/discobot.html producer: name: Discovery Engine url: http://discoveryengine.com - user_agent: 'Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com' bot: name: Domain Re-Animator Bot category: Crawler url: "" producer: name: Domain Re-Animator, LLC url: http://domainreanimator.com - user_agent: support@domainreanimator.com bot: name: Domain Re-Animator Bot category: Crawler url: "" producer: name: Domain Re-Animator, LLC url: http://domainreanimator.com - user_agent: Mozilla/5.0 (compatible; Domains Project/1.1.0; +https://domainsproject.org) bot: name: Domains Project category: Crawler url: https://domainsproject.org - user_agent: dotbot bot: name: DotBot category: Crawler url: http://www.opensiteexplorer.org/dotbot producer: name: SEOmoz, Inc. url: http://moz.com/ - user_agent: DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) bot: name: DuckDuckBot category: Search bot url: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ producer: name: DuckDuckGo url: https://duckduckgo.com/ - user_agent: Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com) bot: name: DuckDuckBot category: Search bot url: https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/ producer: name: DuckDuckGo url: https://duckduckgo.com/ - user_agent: DuckAssistBot/1.1; (+http://duckduckgo.com/duckassistbot.html) bot: name: DuckAssistBot category: AI Assistant url: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/ producer: name: DuckDuckGo url: https://duckduckgo.com/ - user_agent: EMail Exractor bot: name: EMail Exractor category: Crawler url: "" producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html) bot: name: Easou Spider category: Search bot url: http://www.easou.com/search/spider.html producer: name: easou ICP url: http://www.easou.com - user_agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us; EasouSpider; +http://www.easou.com/search/spider.html) bot: name: Easou Spider category: Search bot url: http://www.easou.com/search/spider.html producer: name: easou ICP url: http://www.easou.com - user_agent: Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/) bot: name: Embedly category: Crawler url: https://support.embed.ly/hc/en-us producer: name: A Medium, Corp. url: https://medium.com/ - user_agent: Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot) bot: name: ExaBot category: Crawler url: http://www.exabot.com/go/robot producer: name: Dassault Systèmes url: http://www.3ds.com - user_agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) bot: name: ExaBot category: Crawler url: http://www.exabot.com/go/robot producer: name: Dassault Systèmes url: http://www.3ds.com - user_agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) bot: name: ExaBot category: Crawler url: http://www.exabot.com/go/robot producer: name: Dassault Systèmes url: http://www.3ds.com - user_agent: ExactSeek Crawler (nutch 1.4)/Nutch-1.4 (ExactSeek Crawler; http://www.exactseek.com) bot: name: ExactSeek Crawler category: Search bot url: http://www.exactseek.com producer: name: Jayde Online, Inc. url: http://www.jaydeonlineinc.com - user_agent: Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com) bot: name: Ezooms category: Crawler url: "" producer: name: SEOmoz, Inc. url: http://moz.com/ - user_agent: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) bot: name: Facebook Crawler category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: facebookexternalua bot: name: Facebook Crawler category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: facebookplatform/1.0 (+http://developers.facebook.com) bot: name: Facebook Crawler category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: Feed Wrangler/1.0 (3 subscribers; feed-id=248559; http://feedwrangler.net; Allow like Gecko) bot: name: Feed Wrangler url: https://feedwrangler.net/ category: Feed Fetcher producer: name: 'David Smith & Developing Perspective, LLC' url: https://david-smith.org - user_agent: FeedBurner/1.0 (http://www.FeedBurner.com) bot: name: FeedBurner url: http://www.feedburner.com category: Feed Fetcher producer: name: "" url: "" - user_agent: 'Feedbin - 9 subscribers' bot: name: Feedbin url: http://feedbin.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google) bot: name: Feedly url: http://www.feedly.com category: Feed Fetcher producer: name: "" url: "" - user_agent: FeedlyApp/1.0 (http://www.feedly.com) bot: name: Feedly url: http://www.feedly.com category: Feed Fetcher producer: name: "" url: "" - user_agent: FeedlyBot/1.0 (http://feedly.com) bot: name: Feedly url: http://www.feedly.com category: Feed Fetcher producer: name: "" url: "" - user_agent: MetaFeedly/1.0 (http://www.feedly.com) bot: name: Feedly url: http://www.feedly.com category: Feed Fetcher producer: name: "" url: "" - user_agent: Feedspot http://www.feedspot.com bot: name: Feedspot url: http://www.feedspot.com category: Feed Fetcher producer: name: "" url: "" - user_agent: Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko) bot: name: Fever url: http://feedafever.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com) bot: name: Findxbot category: Crawler url: http://www.findxbot.com - user_agent: Mozilla/5.0 (compatible; FlipboardProxy/1.2; +http://flipboard.com/browserproxy) bot: name: Flipboard url: http://flipboard.com/browserproxy category: Feed Fetcher producer: name: Flipboard url: http://flipboard.com/ - user_agent: Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy) bot: name: Flipboard url: http://flipboard.com/browserproxy category: Feed Fetcher producer: name: Flipboard url: http://flipboard.com/ - user_agent: freshrss/0.8-dev (Linux; http://freshrss.org) SimplePie/1.4-dev-FreshRSS bot: name: FreshRSS category: Feed Fetcher url: https://freshrss.org/ - user_agent: FreshRSS/1.12.0 (Linux; https://freshrss.org) bot: name: FreshRSS category: Feed Fetcher url: https://freshrss.org/ - user_agent: Mozilla/5.0 (X11; Linux x86_64; GTmetrix https://gtmetrix.com/) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36 bot: name: GTmetrix category: Crawler url: https://gtmetrix.com/ producer: name: Carbon60 Operating Co. Ltd. url: https://www.carbon60.com/ - user_agent: 7Siters/1.07 (+https://7ooo.ru/siters/) bot: name: Generic Bot - user_agent: COMODO SSL Checker bot: name: Generic Bot - user_agent: help@dataminr.com bot: name: Generic Bot - user_agent: 'HybridBot (hybrid.ru/about. If our bot caused problems please contact us. Contact email: m.lyashkov@targetix.net)' bot: name: Generic Bot - user_agent: 'Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)' bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; +http://tweetedtimes.com) bot: name: TweetedTimes Bot category: Crawler url: https://tweetedtimes.com/ producer: name: TweetedTimes url: https://tweetedtimes.com/ - user_agent: Mozilla/5.0 (compatible; Shareaholicbot/2.0; +http://www.shareaholic.com/bot) bot: name: Steve Bot category: Crawler url: https://www.shareaholic.com/steve producer: name: Shareaholic, Inc. url: https://www.shareaholic.com/ - user_agent: Mozilla/5.0 (compatible; Shareaholic/1.0; +http://www.shareaholic.com/steve) bot: name: Steve Bot category: Crawler url: https://www.shareaholic.com/steve producer: name: Shareaholic, Inc. url: https://www.shareaholic.com/ - user_agent: Mozilla/5.0 (compatible; TrendsmapResolver/0.1) bot: name: Trendsmap category: Crawler url: https://www.trendsmap.com/ producer: name: Trendsmap Pty Ltd url: https://www.trendsmap.com/ - user_agent: niki-bot bot: name: Generic Bot - user_agent: PHPCrawl bot: name: Generic Bot - user_agent: robots bot: name: Generic Bot - user_agent: SeopultContentAnalyzer/1.0 bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) bot: name: Genieo Web filter category: "" url: http://www.genieo.com/webfilter.html producer: name: Genieo url: http://www.genieo.com - user_agent: GigablastOpenSource/1.0 bot: name: Gigablast category: Search bot url: https://github.com/gigablast/open-source-search-engine producer: name: Matt Wells url: http://www.gigablast.com/faq.html - user_agent: Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/) bot: name: Gluten Free Crawler category: Crawler url: http://glutenfreepleasure.com/ producer: name: "" url: "" - user_agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy) bot: name: Gmail Image Proxy category: Crawler url: "" producer: name: Google Inc. url: https://www.google.com/ - user_agent: DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/) bot: name: Goo category: Search bot url: http://search.goo.ne.jp/option/use/sub4/sub4-1 producer: name: NTT Resonant url: http://goo.ne.jp - user_agent: ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/) bot: name: Goo category: Search bot url: http://search.goo.ne.jp/option/use/sub4/sub4-1 producer: name: NTT Resonant url: http://goo.ne.jp - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon bot: name: Google Favicon category: Crawler - user_agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0_1 like Mac OS X; en-us) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Version/4.0.5 Mobile/8A306 Safari/6531.22.7 bot: name: Google PageSpeed Insights category: Site Monitor url: http://developers.google.com/speed/pagespeed/insights/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1 google_partner_monitoring FWSzVTDDBz14547302713138T bot: name: Google Partner Monitoring category: Site Monitor url: "" producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Search Console) Chrome/41.0.2272.118 Safari/537.36 bot: name: Google Search Console category: Crawler url: https://search.google.com/search-console/about producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool) bot: name: Google Structured Data Testing Tool category: Validator url: https://search.google.com/structured-data/testing-tool producer: name: Google Inc. url: https://www.google.com/ - user_agent: AdsBot-Google (+http://www.google.com/adsbot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 19 subscribers; feed-id=13965549748850348809) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 2 subscribers; feed-id=17860707833818568603) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 375 subscribers; feed-id=15381863289700640853) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-AdWords-Express bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-Adwords-Instant (+http://www.google.com/adsbot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-speakr bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot (gocrawl v0.4) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot-Image/1.0 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot-News (2.3.3, ruby 1.9.3 (2013-11-22)) bot: name: Googlebot News category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot-Video/1.0 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot/2.1 (http://www.googlebot.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Googlebot/Nutch-1.7 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: GoogleProducer; (+http://goo.gl/7y4SX) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mediapartners-Google bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; Google-Youtube-Links) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (en-US) AppleWebKit/537.36 (KHTML, like Gecko; Google-Assess) Chrome/34.0.1847.116 Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (en-us) AppleWebKit/537.36(KHTML, like Gecko; Google-Adwords-DisplayAds-WebRender;) Chrome/41.0.2272.118Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Google-Publisher-Plugin) Chrome/27.0.1453 Mobile Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-AMPHTML) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943) bot: name: Google-Read-Aloud category: Crawler url: https://developers.google.com/crawling/docs/crawlers-fetchers/read-aloud-user-agent producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36 bot: name: DuplexWeb-Google category: Crawler url: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Safari/537.36 (compatible; Google-HotelAdsVerifier/2.0) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Shopping-Quality +http://www.google.com/merchants/tos/extend/DE/tos.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Grammarly/1.0 (http://www.grammarly.com) bot: name: Grammarly category: Service bot url: https://www.grammarly.com - user_agent: Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +https://www.grapeshot.com/crawler/) bot: name: Grapeshot category: Crawler url: https://www.grapeshot.com/crawler producer: name: Grapeshot url: https://www.grapeshot.com - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; GrapeshotCrawler/2.0; +https://www.grapeshot.com/crawler/) bot: name: Grapeshot category: Crawler url: https://www.grapeshot.com/crawler producer: name: Grapeshot url: https://www.grapeshot.com - user_agent: HTTPMon/1.0b (http://www.httpmon.com) bot: name: HTTPMon category: Site Monitor url: http://www.httpmon.com producer: name: towards GmbH url: http://www.towards.ch/ - user_agent: 'flieder - neofonie heritrix/1.14.3 (+http://spider.neofonie.de)' bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/1.14.4 +http://webarhive.nlc.gov.cn) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.dla-marbach.de) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: mozilla/5.0 (compatible; heritrix/1.3.0 http://archive.crawler.org) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.baidu.com) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.run4dom.com) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20130207.001528 +http://webarchiv.cz/kontakty/) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; worio bot heritrix/1.10.0 +http://worio.com) bot: name: Heritrix category: Crawler url: https://webarchive.jira.com/wiki/display/Heritrix/Heritrix producer: name: The Internet Archive url: https://archive.org - user_agent: Heurekabot-Feed/1.0 (+https://sluzby.heureka.cz/napoveda/heurekabot/) bot: name: Heureka Feed category: Crawler url: https://sluzby.heureka.cz/napoveda/heurekabot/ producer: name: Heureka.cz, a.s. url: https://www.heureka.cz/ - user_agent: HubPages V0.2.2 (http://hubpages.com/help/crawlingpolicy) bot: name: HubPages category: Crawler url: https://hubpages.com/help/crawlingpolicy producer: name: HubPages, Inc. url: https://discover.hubpages.com/ - user_agent: HubSpot Website Grader (web-crawlers@hubspot.com) bot: name: HubSpot category: Crawler producer: name: HubSpot Inc. url: https://www.hubspot.com - user_agent: ICC-Crawler/2.0 (Mozilla-compatible; ; http://www.nict.go.jp/en/univ-com/plan/crawl.html) bot: name: ICC-Crawler category: Crawler url: http://www.nict.go.jp/en/univ-com/plan/crawl.html producer: name: "" url: "" - user_agent: IDG/IT (http://spaziodati.eu/) bot: name: IDG category: Crawler url: https://www.spaziodati.eu/ producer: name: SpazioDati S.r.l. url: https://www.spaziodati.eu/ - user_agent: iisbot/1.0 (+http://www.iis.net/iisbot.html) bot: name: IIS Site Analysis category: Crawler url: http://www.iis.net/iisbot.html producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: IP-Guide.com Crawler/1.0 (https://ip-guide.com) bot: name: IP-Guide Crawler category: Crawler url: "" producer: name: "" url: https://ip-guide.com - user_agent: Mozilla/5.0 (compatible; ips-agent) bot: name: IPS Agent category: Crawler producer: name: VeriSign, Inc url: http://www.verisign.com/ - user_agent: Kaspersky Lab CFR link resolver cfradmins@kaspersky.com bot: name: Kaspersky category: Security Checker url: https://www.kaspersky.com/ producer: name: AO Kaspersky Lab url: https://www.kaspersky.com/ - user_agent: kouio.com RSS reader bot: name: Kouio url: http://kouio.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: 'kouio.com RSS reader - 6 subscribers' bot: name: Kouio url: http://kouio.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html) bot: name: LCC category: Search bot url: https://corpora.uni-leipzig.de/crawler_faq.html producer: name: Universität Leipzig url: https://www.uni-leipzig.de/ - user_agent: 'ltx71 - (http://ltx71.com/)' bot: name: LTX71 category: Security Checker url: https://ltx71.com/ producer: name: "" url: "" - user_agent: larbin_2.6.3 larbin2.6.3@unspecified.mail bot: name: Larbin web crawler category: Crawler url: http://larbin.sourceforge.net producer: name: "" url: "" - user_agent: "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" bot: name: "Let's Encrypt Validation" category: Service Agent url: https://letsencrypt.org/how-it-works/ producer: name: "Let's Encrypt" url: https://letsencrypt.org - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/61.0.3116.0 Mobile Safari/537.36 Chrome-Lighthouse bot: name: Lighthouse category: Site Monitor url: https://developers.google.com/web/tools/lighthouse producer: name: Lighthouse url: https://developers.google.com/web/tools/lighthouse - user_agent: Mozilla/5.0 (compatible; LastModBot/0.1; +https://last-modified.com/en) bot: name: LastMod Bot category: Site Monitor url: https://last-modified.com/en/about producer: name: "" url: https://last-modified.com/en - user_agent: linkdex.com/v2.0 and linkdex.com/v2.1 bot: name: Linkdex Bot category: Search bot url: http://www.linkdex.com/bots producer: name: Mojeek Ltd. url: http://www.mojeek.com - user_agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/) bot: name: Linkdex Bot category: Search bot url: http://www.linkdex.com/bots producer: name: Mojeek Ltd. url: http://www.mojeek.com - user_agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; linkdexbot-mobile/2.1; +http://www.linkdex.com/about/bots/) bot: name: Linkdex Bot category: Search bot url: http://www.linkdex.com/bots producer: name: Mojeek Ltd. url: http://www.mojeek.com - user_agent: LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) bot: name: LinkedIn Bot category: Social Media Agent url: http://www.linkedin.com producer: name: LinkedIn url: http://www.linkedin.com - user_agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) bot: name: MJ12 Bot category: Search bot url: http://majestic12.co.uk/bot.php producer: name: Majestic-12 url: http://majestic12.co.uk - user_agent: magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) bot: name: Magpie-Crawler category: Social Media Agent url: http://www.brandwatch.com/magpie-crawler/ producer: name: Brandwatch url: http://www.brandwatch.com - user_agent: MagpieRSS/0.72 (+http://magpierss.sf.net) bot: name: MagpieRSS url: http://magpierss.sourceforge.net/ category: Feed Parser producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) bot: name: Mail.Ru Bot category: Search bot url: http://help.mail.ru/webmaster/indexing/robots/types_robots producer: name: Mail.Ru Group url: http://corp.mail.ru - user_agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Fast/2.0; +http://go.mail.ru/help/robots) bot: name: Mail.Ru Bot category: Search bot url: http://help.mail.ru/webmaster/indexing/robots/types_robots producer: name: Mail.Ru Group url: http://corp.mail.ru - user_agent: http.rb/2.2.2 (Mastodon/1.6.1; +https://mathtod.online/) bot: name: Mastodon Bot category: Social Media Agent - user_agent: http.rb/3.2.0 (Mastodon/2.4.3; +https://uwu.social/) bot: name: Mastodon Bot category: Social Media Agent - user_agent: Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html) bot: name: Meanpath Bot category: Search bot url: http://www.meanpath.com/meanpathbot.html producer: name: Meanpath url: http://www.meanpath.com - user_agent: MetaInspector/5.4.0 (+https://github.com/jaimeiniesta/metainspector) bot: name: MetaInspector category: Crawler url: https://github.com/jaimeiniesta/metainspector - user_agent: Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler) bot: name: MetaJobBot category: Crawler url: http://www.metajob.at/the/crawler producer: name: MetaJob url: http://www.metajob.at - user_agent: Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com) bot: name: MixRank Bot category: Crawler url: http://mixrank.com producer: name: Online Media Group, Inc. url: "" - user_agent: Mnogosearch-3.1.21 bot: name: Mnogosearch category: Search bot url: http://www.mnogosearch.org/ producer: name: Lavtech.Com Corp. url: "" - user_agent: Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html) bot: name: MojeekBot category: Search bot url: http://www.mojeek.com/bot.html producer: name: Mojeek Ltd. url: http://www.mojeek.com - user_agent: 'Mozilla/5.0 (compatible; www.monitor.us - free monitoring service; http://www.monitor.us)' bot: name: Monitor.Us category: Site Monitor url: http://www.monitor.us producer: name: Monitor.Us url: http://www.monitor.us - user_agent: munin/2.0.30-1 (libwww-perl/6.15) bot: name: Munin category: Site Monitor url: http://munin-monitoring.org/ producer: name: Munin url: http://munin-monitoring.org/ - user_agent: munin/http_loadtime bot: name: Munin category: Site Monitor url: http://munin-monitoring.org/ producer: name: Munin url: http://munin-monitoring.org/ - user_agent: nlcrawler/1.0 (+http://northernlight.com/) bot: name: NLCrawler category: Crawler url: "" producer: name: Northern Light url: http://northernlight.com - user_agent: check_http/v1.5 (nagios-plugins 1.5) bot: name: Nagios check_http category: Site Monitor url: https://nagios.org producer: name: Nagios Plugins Development Team url: https://nagios.org - user_agent: NalezenCzBot/1.0 (http://www.nalezen.cz/about-crawler) bot: name: NalezenCzBot category: Crawler url: http://www.nalezen.cz/about-crawler producer: name: Jaroslav Kuboš url: "" - user_agent: NetLyzer FastProbe bot: name: NetLyzer FastProbe - user_agent: Mozilla/4.0 (compatible; Netcraft Web Server Survey) bot: name: Netcraft Survey Bot category: Search bot url: "" producer: name: Netcraft url: http://www.netcraft.com - user_agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com) bot: name: Netcraft Survey Bot category: Search bot url: "" producer: name: Netcraft url: http://www.netcraft.com - user_agent: 'Netcraft SSL Server Survey - contact info@netcraft.com' bot: name: Netcraft Survey Bot category: Search bot url: "" producer: name: Netcraft url: http://www.netcraft.com - user_agent: Netvibes (http://www.netvibes.com) bot: name: Netvibes url: http://www.netvibes.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: 'Netvibes (http://www.netvibes.com/; 8 subscribers; feedID: 2244192)' bot: name: Netvibes url: http://www.netvibes.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: 'NewsBlur Favicon Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)' bot: name: NewsBlur url: http://www.newsblur.com category: Feed Fetcher producer: name: "" url: "" - user_agent: 'NewsBlur Feed Fetcher - 7 subscribers - http://www.newsblur.com/site/1948420/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)' bot: name: NewsBlur url: http://www.newsblur.com category: Feed Fetcher producer: name: "" url: "" - user_agent: NewsBlur Feed Finder (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3) bot: name: NewsBlur url: http://www.newsblur.com category: Feed Fetcher producer: name: "" url: "" - user_agent: 'NewsBlur Page Fetcher - 7 subscribers - http://www.newsblur.com/site/3966817/analytics-piwik (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)' bot: name: NewsBlur url: http://www.newsblur.com category: Feed Fetcher producer: name: "" url: "" - user_agent: NewsGatorOnline/2.0 (http://www.newsgator.com; 2 subscribers) bot: name: NewsGator url: http://www.newsgator.com category: Feed Fetcher producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html) bot: name: Nmap category: Security Checker url: https://nmap.org/book/nse.html producer: name: Nmap url: https://nmap.org/ - user_agent: 5acd734a-1ed3-4a17-ad9d-0b09ae7bc2d2/Nutch-1.7 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: CB/Nutch-1.7 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Mozilla/5.0 (compatible; summers;)/Nutch-1.7 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Firefox/3.6.8/Nutch-1.7 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: My Nutch Spider/Nutch-1.6 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: MySpider/Nutch-2.2 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Nutch12/Nutch-1.2 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: NutchSpider/Nutch-1.4 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: spider/Nutch-1.5.1 (spider; http://www.xxx.com) bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Spider/Nutch-2.3-SNAPSHOT (Webcrawler) bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: testnutch/Nutch-1.8 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Your Nutch Spider/Nutch-2.2.1 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: your sipder name/Nutch-1.7 bot: name: Nutch-based Bot category: Crawler url: https://nutch.apache.org producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Nuzzel bot: name: Nuzzel category: Crawler producer: name: Nuzzel url: https://www.nuzzel.com/ - user_agent: NodePing bot: name: NodePing category: Site Monitor url: https://nodeping.com producer: name: NodePing url: https://nodeping.com - user_agent: Octopus 1.0.2 bot: name: Octopus - user_agent: omgili/0.5 +http://omgili.com bot: name: Omgilibot category: Search bot url: https://webz.io/blog/api/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/ producer: name: Webz.io Ltd url: https://webz.io/ - user_agent: omgilibot/0.3 +http://www.omgili.com/Crawler.html bot: name: Omgilibot category: Search bot url: https://webz.io/blog/api/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/ producer: name: Webz.io Ltd url: https://webz.io/ - user_agent: OnlineOrNot.com_bot_1.0_(https://onlineornot.com) bot: name: OnlineOrNot Bot category: Site Monitor url: https://onlineornot.com/website-monitoring producer: name: OnlineOrNot url: https://onlineornot.com - user_agent: Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot ) bot: name: OpenLinkProfiler category: Crawler url: http://openlinkprofiler.org/bot producer: name: Axandra GmbH url: http://www.axandra.com - user_agent: OpenWebSpider v0.1.4 (http://www.openwebspider.org/) bot: name: OpenWebSpider category: Crawler url: http://www.openwebspider.org producer: name: OpenWebSpider Lab url: http://lab.openwebspider.org - user_agent: Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html) bot: name: Openindex Spider category: Search bot url: http://www.openindex.io/en/webmasters/spider.html producer: name: Openindex B.V. url: http://www.openindex.io - user_agent: Mozilla/5.0 (compatible; OrangeBot-Collector/2.0; support.orangebot@orange.com) bot: name: Orange Bot category: Search bot url: http://lemoteur.orange.fr producer: name: Orange url: http://www.orange.fr - user_agent: Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com) bot: name: Orange Bot category: Search bot url: http://lemoteur.orange.fr producer: name: Orange url: http://www.orange.fr - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) bot: name: Orange Bot category: Search bot url: http://lemoteur.orange.fr producer: name: Orange url: http://www.orange.fr - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1;fr;rv:1.8.1) VoilaBotCollector BETA 0.1 (http://www.voila.com/) bot: name: Orange Bot category: Search bot url: http://lemoteur.orange.fr producer: name: Orange url: http://www.orange.fr - user_agent: Mozilla/5.0 (Java) outbrain bot: name: Outbrain category: Crawler url: "" producer: name: Outbrain url: http://www.outbrain.com/ - user_agent: Mozilla/5.0 (compatible; phpservermon/3.1.1; +http://www.phpservermonitor.org) bot: name: PHP Server Monitor category: Site Monitor url: https://github.com/phpservermon/phpservermon producer: name: PHP Server Monitor url: http://www.phpservermonitor.org/ - user_agent: Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com); Windows) bot: name: PRTG Network Monitor category: Network Monitor url: https://www.paessler.com/prtg producer: name: Paessler AG url: https://www.paessler.com - user_agent: Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li) bot: name: PaperLiBot category: Search bot url: http://support.paper.li/entries/20023257-what-is-paper-li producer: name: Smallrivers SA url: http://www.paper.li - user_agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot) bot: name: Petal Bot category: Crawler url: https://aspiegel.com/petalbot - user_agent: phantomas/1.11.0 (PhantomJS/1.9.8; linux x64) bot: name: Phantomas category: Site Monitor url: https://github.com/macbre/phantomas - user_agent: psbot-page (+http://www.picsearch.com/bot.html) bot: name: Picsearch bot category: Search bot url: http://www.picsearch.com/bot.html producer: name: Picsearch url: http://www.picsearch.com - user_agent: psbot/0.1 (+http://www.picsearch.com/bot.html) bot: name: Picsearch bot category: Search bot url: http://www.picsearch.com/bot.html producer: name: Picsearch url: http://www.picsearch.com - user_agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/) bot: name: Pingdom Bot category: Site Monitor url: "" producer: name: Pingdom AB url: https://www.pingdom.com - user_agent: Mozilla/5.0 (compatible; Pinterestbot/1.0; http://www.pinterest.com/bot.html) bot: name: Pinterest url: https://help.pinterest.com/en/business/article/pinterest-crawler category: Crawler producer: name: Pinterest url: https://www.pinterest.com/ - user_agent: Pinterest/0.2 (+http://www.pinterest.com/) bot: name: Pinterest url: https://help.pinterest.com/en/business/article/pinterest-crawler category: Crawler producer: name: Pinterest url: https://www.pinterest.com/ - user_agent: 'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Pinterestbot/1.0; https://www.pinterest.com/bot.html)' bot: name: Pinterest url: https://help.pinterest.com/en/business/article/pinterest-crawler category: Crawler producer: name: Pinterest url: https://www.pinterest.com/ - user_agent: PocketParser/2.0 (+https://getpocket.com/pocketparser_ua) bot: name: Pocket category: Read-it-later Service url: https://getpocket.com/pocketparser_ua producer: name: Read It Later, Inc. url: https://getpocket.com/ - user_agent: PocketImageCache/1.0 bot: name: Pocket category: Read-it-later Service url: https://getpocket.com/pocketparser_ua producer: name: Read It Later, Inc. url: https://getpocket.com/ - user_agent: PritTorrent/1.0 bot: name: PritTorrent category: Crawler url: https://github.com/astro/prittorrent producer: name: Bitlove url: http://bitlove.org/ - user_agent: QuerySeekerSpider ( http://queryseeker.com/bot.html ) bot: name: QuerySeekerSpider category: Crawler url: http://queryseeker.com/bot.html producer: name: QueryEye Inc. url: http://queryeye.com - user_agent: Quora Link Preview/1.0 (http://www.quora.com) bot: name: Quora Link Preview category: Crawler url: "" producer: name: Quora url: http://www.quora.com - user_agent: Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/) bot: name: Qwantbot category: Crawler url: https://help.qwant.com/bot/ producer: name: Qwant Corporation url: https://www.qwant.com/ - user_agent: Mozilla/5.0 (compatible; Qwantify-prod34997/1.0; +https://help.qwant.com/bot/) bot: name: Qwantbot category: Crawler url: https://help.qwant.com/bot/ producer: name: Qwant Corporation url: https://www.qwant.com/ - user_agent: Mozilla/5.0 (compatible; Qwantbot-prod12345/1.0; +Qwantbot@qwant.com) bot: name: Qwantbot category: Crawler url: https://help.qwant.com/bot/ producer: name: Qwant Corporation url: https://www.qwant.com/ - user_agent: Mozilla/5.0 (compatible; Qwantbot-news/2.0; +https://www.qwant.com/) bot: name: Qwantbot category: Crawler url: https://help.qwant.com/bot/ producer: name: Qwant Corporation url: https://www.qwant.com/ - user_agent: Mozilla/5.0 (compatible; Qwantbot-dev12345/1.0; +Qwantbot@qwant.com) bot: name: Qwantbot category: Crawler url: https://help.qwant.com/bot/ producer: name: Qwant Corporation url: https://www.qwant.com/ - user_agent: ROI Hunter; https://api-dev.roihunter.com bot: name: ROI Hunter category: Crawler url: "" producer: name: Roihunter a.s. url: http://roihunter.com/ - user_agent: RSSRadio (Push Notification Scanner;support@dorada.co.uk) bot: name: RSSRadio Bot - user_agent: Rainmeter WebParser plugin bot: name: Rainmeter category: Crawler url: https://www.rainmeter.net - user_agent: RamblerMail/6.0 (incompatible; ImageProxy/6.0) bot: name: RamblerMail Image Proxy category: Crawler url: "" producer: name: 'Rambler&Co' url: https://rambler-co.ru/ - user_agent: Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback) bot: name: Reddit Bot category: Social Media Agent url: http://www.reddit.com/feedback producer: name: reddit inc. url: http://www.reddit.com - user_agent: Robozilla/1.0 bot: name: Robozilla category: Crawler - user_agent: Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot) bot: name: Rogerbot category: Crawler url: http://moz.com/help/pro/what-is-rogerbot- producer: name: SEOmoz, Inc. url: http://moz.com/ - user_agent: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) bot: name: Rogerbot category: Crawler url: http://moz.com/help/pro/what-is-rogerbot- producer: name: SEOmoz, Inc. url: http://moz.com/ - user_agent: rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com) bot: name: Rogerbot category: Crawler url: http://moz.com/help/pro/what-is-rogerbot- producer: name: SEOmoz, Inc. url: http://moz.com/ - user_agent: SEOENGWorldBot/1.0 (+http://www.seoengine.com/seoengbot.htm) bot: name: SEOENGBot category: Crawler url: http://www.seoengine.com/seoengbot.htm producer: name: SEO Engine url: http://www.seoengine.com - user_agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html) bot: name: SEOkicks category: Crawler url: https://www.seokicks.de/robot.html producer: name: SEOkicks url: https://www.seokicks.de/ - user_agent: Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) bot: name: SISTRIX Crawler category: Crawler url: http://crawler.sistrix.net producer: name: SISTRIX GmbH url: http://www.sistrix.de - user_agent: Mozilla/5.0 (compatible; Optimizer) bot: name: SISTRIX Optimizer category: Crawler url: https://optimizer.sistrix.com producer: name: SISTRIX GmbH url: http://www.sistrix.de - user_agent: Mozilla/5.0 (compatible; SISTRIX Optimizer) bot: name: SISTRIX Optimizer category: Crawler url: https://optimizer.sistrix.com producer: name: SISTRIX GmbH url: http://www.sistrix.de - user_agent: Mozilla/5.0 (compatible; SISTRIX Optimizer; Uptime; +https://www.sistrix.com/faq/uptime) bot: name: SISTRIX Optimizer category: Crawler url: https://optimizer.sistrix.com producer: name: SISTRIX GmbH url: http://www.sistrix.de - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot) bot: name: SMTBot category: Search bot url: https://www.similartech.com/smtbot producer: name: SimilarTech Ltd. url: https://www.similartech.com/ - user_agent: SSL Labs (https://www.ssllabs.com/about/assessment.html) bot: name: SSL Labs category: Validator url: https://www.ssllabs.com/about/assessment.html producer: name: SSL Labs url: https://www.ssllabs.com/about/assessment.html - user_agent: SafeDNSBot (https://www.safedns.com/searchbot) bot: name: SafeDNSBot category: Crawler url: https://www.safedns.com/searchbot producer: name: SafeDNS, Inc. url: https://www.safedns.com/ - user_agent: Scrapy/1.0.3.post6+g2d688cd (+http://scrapy.org) bot: name: Scrapy category: Crawler url: http://scrapy.org - user_agent: Screaming Frog SEO Spider/2.22 bot: name: Screaming Frog SEO Spider category: Crawler url: http://www.screamingfrog.co.uk/seo-spider producer: name: Screaming Frog Ltd url: http://www.screamingfrog.co.uk - user_agent: ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com) bot: name: ScreenerBot category: Crawler url: http://www.screenerbot.com producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler) bot: name: Semantic Scholar Bot category: Crawler url: https://www.semanticscholar.org/crawler - user_agent: Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html) bot: name: SemrushBot category: Crawler url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: SensikaBot/x.33 (+http://sensika.com) bot: name: Sensika Bot category: "" url: "" producer: name: Sensika url: http://sensika.com - user_agent: sentry/8.6.0 (https://getsentry.com) bot: name: Sentry Bot producer: name: Sentry url: https://sentry.io - user_agent: Seobility bot: name: Seobility category: Crawler url: 'https://www.seobility.net/en/faq/?category=crawling#!aboutourbot' - user_agent: Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net) bot: name: Seoscanners.net category: Crawler url: "" - user_agent: SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot) bot: name: Serendeputy Bot category: Crawler url: https://serendeputy.com/about/serendeputy-bot - user_agent: Server Density Service Monitoring v2 bot: name: Server Density - user_agent: Mozilla/5.0 (compatible; SeznamBot/3.1-test1; +http://fulltext.sblog.cz/) bot: name: Seznam Bot category: Search bot url: http://www.mapy.cz/cz/seznambot.html producer: name: Seznam.cz, a.s. url: http://www.seznam.cz/ - user_agent: Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://fulltext.sblog.cz/) bot: name: Seznam Bot category: Search bot url: http://www.mapy.cz/cz/seznambot.html producer: name: Seznam.cz, a.s. url: http://www.seznam.cz/ - user_agent: Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/) bot: name: Seznam Bot category: Search bot url: http://www.mapy.cz/cz/seznambot.html producer: name: Seznam.cz, a.s. url: http://www.seznam.cz/ - user_agent: SeznamBot/3.0 (+http://fulltext.sblog.cz/) bot: name: Seznam Bot category: Search bot url: http://www.mapy.cz/cz/seznambot.html producer: name: Seznam.cz, a.s. url: http://www.seznam.cz/ - user_agent: Mozilla/5.0 SeznamEmailProxy/2.0.174 bot: name: Seznam Email Proxy category: Crawler url: "" producer: name: Seznam.cz, a.s. url: http://www.seznam.cz/ - user_agent: Seznam-Zbozi-robot/3.0 bot: name: Seznam Zbozi.cz category: Crawler url: "" producer: name: Seznam.cz, a.s. url: https://www.zbozi.cz/ - user_agent: Mozilla/5.0 (ShopAlike; LadenZeile) FeedBot bot: name: ShopAlike category: Crawler url: "" producer: name: Visual Meta url: https://www.shopalike.cz/ - user_agent: ShopWiki/1.0 ( +http://www.shopwiki.com/wiki/Help:Bot) bot: name: ShopWiki category: Search tools url: http://www.shopwiki.com/wiki/Help:Bot producer: name: ShopWiki Corp. url: http://www.shopwiki.com - user_agent: shopify-partner-homepage-scraper bot: name: Shopify Partner category: Crawler url: https://www.shopify.com/partners producer: name: Shopify url: https://www.shopify.com/ - user_agent: SilverReader/1.0; http://silverreader.com bot: name: SilverReader url: http://silverreader.com category: Feed Fetcher producer: name: "" url: "" - user_agent: SimplePie/1.2.1-dev (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20130514092120 bot: name: SimplePie url: http://www.simplepie.org category: Feed Parser producer: name: "" url: "" - user_agent: SimplePie/1.3.1 (Feed Parser; http://simplepie.org; Allow like Gecko) Build/20121030175911 bot: name: SimplePie url: http://www.simplepie.org category: Feed Parser producer: name: "" url: "" - user_agent: Site24x7 bot: name: Site24x7 Website Monitoring category: Site Monitor url: https://www.site24x7.com/site24x7-faq.html producer: name: Site24x7 url: https://www.site24x7.com - user_agent: SiteSucker for macOS/2.10.5 bot: name: SiteSucker category: Crawler url: http://ricks-apps.com/osx/sitesucker/ - user_agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: sixy.ch/1.0 bot: name: Sixy.ch category: Site Monitor url: http://sixy.ch producer: name: Manuel Kasper url: https://neon1.net/ - user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 bot: name: Skype URI Preview category: Service Agent url: "" producer: name: Skype Communications S.à.r.l. url: https://www.skype.com - user_agent: Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots) bot: name: Slackbot category: Crawler url: https://api.slack.com/robots producer: name: Slack Technologies url: http://slack.com - user_agent: 'AppEngine-Google; (+http://code.google.com/appengine; appid: s~snapchat-proxy)' bot: name: Snapchat Proxy category: Crawler url: https://www.snapchat.com producer: name: Snapchat Inc. url: https://www.snapchat.com - user_agent: New-Sogou-Spider/1.0 (compatible; MSIE 5.5; Windows 98) bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Sogou inst spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Sosospider+(+http://help.soso.com/webspider.htm) bot: name: Soso Spider category: Search bot url: http://help.soso.com/webspider.htm producer: name: Tencent Holdings url: http://www.soso.com - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Sparkler/0.2.0-SNAPSHOT bot: name: Sparkler category: Crawler url: https://github.com/USCDataScience/sparkler - user_agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.19; aggregator:Spinn3r (Spinn3r 3.1); http://spinn3r.com/robot) Gecko/2010040121 Firefox/3.0.19 bot: name: Spinn3r category: Crawler url: http://spinn3r.com/robot producer: name: Tailrank Inc url: http://spinn3r.com - user_agent: Spotify/1.0 bot: name: Spotify producer: name: Spotify url: https://www.spotify.com - user_agent: Mozilla/5.0 (compatible; SputnikBot/2.2) bot: name: Sputnik Bot category: Crawler url: "" - user_agent: Mozilla/5.0 (compatible; SputnikImageBot/2.2) bot: name: Sputnik Image Bot category: Crawler url: "" - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/536.30.1 (KHTML, like Gecko) (compatible; Startpagina-Linkchecker/1.0; +https://www.startpagina.nl/linkchecker) bot: name: Startpagina Linkchecker category: Search bot url: https://www.startpagina.nl/linkchecker producer: name: Startpagina B.V. url: https://www.startpagina.nl/ - user_agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/98 Safari/537.4 (StatusCake) bot: name: StatusCake category: Site Monitor url: https://www.statuscake.com producer: name: StatusCake url: https://www.statuscake.com - user_agent: 'Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch!' bot: name: Superfeedr Bot category: Feed Fetcher url: "" producer: name: Superfeedr url: https://superfeedr.com/ - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools) bot: name: Survey Bot category: Search bot url: http://www.domaintools.com/webmasters/surveybot.php producer: name: Domain Tools url: http://www.domaintools.com - user_agent: TLSProbe/1.0 (+https://scan.trustnet.venafi.com/) bot: name: TLSProbe url: https://scan.trustnet.venafi.com/ category: Security search bot producer: name: Venafi TrustNet url: https://www.venafi.com - user_agent: Tarmot Gezgin/1.0 (compatible; TarmotGezgin/1.1; +http://www.tarmot.com/gezgin) bot: name: Tarmot Gezgin url: http://www.tarmot.com/gezgin/ category: Search bot - user_agent: TelegramBot (like TwitterBot) bot: name: TelegramBot url: https://telegram.org/blog/bot-revolution - user_agent: The Knowledge AI bot: name: The Knowledge AI category: Crawler - user_agent: TinEye-bot/0.02 (see http://www.tineye.com/crawler.html) bot: name: TinEye Crawler category: Search bot url: http://www.tineye.com/crawler.html producer: name: Idée Inc. url: http://ideeinc.com - user_agent: Tiny Tiny RSS/1.10 (http://tt-rss.org/) bot: name: Tiny Tiny RSS url: http://tt-rss.org category: Feed Fetcher producer: name: "" url: "" - user_agent: Tiny Tiny RSS/1.11.4c63934 (http://tt-rss.org/) bot: name: Tiny Tiny RSS url: http://tt-rss.org category: Feed Fetcher producer: name: "" url: "" - user_agent: Mozilla/5.0 (compatible; tracemyfile/1.0) bot: name: TraceMyFile category: Search bot url: https://www.tracemyfile.com/ producer: name: Idee Inc. url: http://ideeinc.com/ - user_agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11 bot: name: Trendiction Bot category: Crawler url: http://www.trendiction.de/bot producer: name: Talkwalker Inc. url: http://www.talkwalker.com - user_agent: TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html) bot: name: TurnitinBot category: Crawler url: https://www.turnitin.com/robot/crawlerinfo.html producer: name: iParadigms, LLC. url: https://www.turnitin.com/ - user_agent: Mozilla/5.0 (compatible; TweetedTimes Bot/1.0; +http://tweetedtimes.com) bot: name: TweetedTimes Bot category: Crawler url: https://tweetedtimes.com/ producer: name: TweetedTimes url: https://tweetedtimes.com/ - user_agent: TweetedTimes Bot/1.0 (Mozilla/5.0 Compatible, +http://tweetedtimes.com) bot: name: TweetedTimes Bot category: Crawler url: https://tweetedtimes.com/ producer: name: TweetedTimes url: https://tweetedtimes.com/ - user_agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/) bot: name: Tweetmeme Bot category: Crawler url: http://tweetmeme.com/ producer: name: Mediasift url: "" - user_agent: Mozilla/5.0 (compatible; Twingly Recon; twingly.com) bot: name: Twingly Recon category: Crawler url: https://app.twingly.com/public-docs/crawler producer: name: Twingly url: https://www.twingly.com - user_agent: Twitterbot/1.0 bot: name: Twitterbot category: Social Media Agent url: https://dev.twitter.com/docs/cards/getting-started producer: name: Twitter url: http://www.twitter.com - user_agent: Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html) bot: name: URLAppendBot category: Crawler url: http://www.profound.net/urlappendbot.html producer: name: Profound Networks url: http://www.profound.net - user_agent: 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36 (via secureurl.fwdcdn.com - mail.ukr.net proxy)' bot: name: UkrNet Mail Proxy category: Crawler url: "" producer: name: UkrNet Ltd url: https://www.ukr.net/ - user_agent: UniversalFeedParser/5.2.1 +https://code.google.com/p/feedparser/ bot: name: UniversalFeedParser category: Feed Fetcher url: https://github.com/kurtmckee/feedparser producer: name: Kurt McKee url: https://github.com/kurtmckee - user_agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/) bot: name: UptimeRobot category: Site Monitor url: https://uptimerobot.com/ producer: name: Uptime Robot url: https://uptimerobot.com/ - user_agent: Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot) bot: name: Uptimebot category: Site Monitor url: https://uptime.com/uptime-bot producer: name: Uptime url: https://uptime.com/ - user_agent: Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share) bot: name: VK Share Button category: Crawler url: https://dev.vk.com/en/widgets/share producer: name: VK url: https://vk.com/ - user_agent: 'Mozilla/4.0 (compatible; Vagabondo/4.0; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)' bot: name: Vagabondo category: Crawler url: "" producer: name: WiseGuys url: http://www.wise-guys.nl/ - user_agent: Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) bot: name: Velen Public Web Crawler category: Crawler url: https://hunter.io/robot - user_agent: Vercelbot (+https://vercel.com) bot: name: Vercel Bot category: Service bot url: https://vercel.com - user_agent: Mozilla/5.0 (compatible; VSMCrawler; http://www.visualsitemapper.com/crawler/) bot: name: Visual Site Mapper Crawler category: Crawler url: http://www.visualsitemapper.com/crawler producer: name: Alentum Software Ltd. url: http://www.alentum.com - user_agent: 'Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See )' bot: name: W3C CSS Validator category: Validator url: http://jigsaw.w3.org/css-validator producer: name: W3C url: http://www.w3.org - user_agent: W3C_I18n-Checker/1.0 (http://validator.w3.org/services) bot: name: W3C I18N Checker category: Validator url: http://validator.w3.org/i18n-checker producer: name: W3C url: http://www.w3.org - user_agent: 'W3C-checklink/4.0 [4.4] libwww-perl/5.803' bot: name: W3C Link Checker category: Validator url: http://validator.w3.org/checklink producer: name: W3C url: http://www.w3.org - user_agent: W3C-checklink/4.81 libwww-perl/5.836 bot: name: W3C Link Checker category: Validator url: http://validator.w3.org/checklink producer: name: W3C url: http://www.w3.org - user_agent: Validator.nu/LV http://validator.w3.org/services bot: name: W3C Markup Validation Service category: Validator url: http://validator.w3.org/services producer: name: W3C url: http://www.w3.org - user_agent: W3C_Validator/1.3 http://validator.w3.org/services bot: name: W3C Markup Validation Service category: Validator url: http://validator.w3.org/services producer: name: W3C url: http://www.w3.org - user_agent: W3C_Validator/1.767 bot: name: W3C Markup Validation Service category: Validator url: http://validator.w3.org/services producer: name: W3C url: http://www.w3.org - user_agent: W3C-mobileOK/DDC-1.0 (see http://www.w3.org/2006/07/mobileok-ddc) bot: name: W3C MobileOK Checker category: Validator url: http://validator.w3.org/mobile producer: name: W3C url: http://www.w3.org - user_agent: W3C_Unicorn/1.0 (http://validator.w3.org/services) bot: name: W3C Unified Validator category: Validator url: http://validator.w3.org/unicorn producer: name: W3C url: http://www.w3.org - user_agent: Mozilla/5.0 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer) bot: name: Wappalyzer url: https://github.com/AliasIO/Wappalyzer producer: name: AliasIO url: https://github.com/AliasIO - user_agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/) bot: name: WeSEE:Search category: Search bot url: http://www.wesee.com/bot producer: name: WeSEE Ltd url: http://www.wesee.com - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Moto G (4) Build/MPJ24.139-64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Mobile Safari/537.36 PTST/180829.190838 bot: name: WebPageTest category: Site Monitor url: https://www.webpagetest.org - user_agent: websitepulse checker/1.1 (compatible; MSIE 5.5; Netscape 4.75; Linux) bot: name: WebSitePulse category: Site Monitor url: http://badcheese.com/crawler.html producer: name: WebSitePulse url: http://www.websitepulse.com/ - user_agent: WebbCrawler 1.0 ( http://badcheese.com/crawler.html ) bot: name: WebbCrawler category: Crawler url: http://badcheese.com/crawler.html producer: name: Steve Webb url: http://badcheese.com - user_agent: weborama-fetcher (+http://www.weborama.com) bot: name: Weborama category: Search bot url: https://weborama.com/ producer: name: Weborama SA url: https://weborama.com/ - user_agent: WikiDo/1.1 (http://wikido.com; crawler@wikido.com) bot: name: WikiDo category: Search bot url: https://www.wikido.com/ producer: name: Fotolitografie Fiorentine di Becchi Antonio s.n.c. url: https://www.wikido.com/ - user_agent: Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/) bot: name: WooRank category: Search bot url: https://www.woorank.com/ producer: name: WooRank sprl url: https://www.woorank.com/ - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1 (compatible; woorankreview/2.0; +https://www.woorank.com/) bot: name: WooRank category: Search bot url: https://www.woorank.com/ producer: name: WooRank sprl url: https://www.woorank.com/ - user_agent: WordPress/4.7.2; https://example.com bot: name: WordPress category: Service Agent url: https://wordpress.org/ producer: name: Wordpress.org url: https://wordpress.org/ - user_agent: Wotbox/2.01 (+http://www.wotbox.com/bot/) bot: name: Wotbox category: Search bot url: http://www.wotbox.com/bot/ producer: name: Wotbox url: http://www.wotbox.com - user_agent: XenForo/2.x (https://www.example.com) bot: name: XenForo category: Service Agent url: https://xenforo.com/ producer: name: XenForo Ltd. url: https://xenforo.com/ - user_agent: yacybot (freeworld/global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_25; Europe/en) http://yacy.net/bot.html bot: name: YaCy category: Search bot url: http://yacy.net/bot.html producer: name: YaCy url: http://yacy.net - user_agent: yacybot (freeworld/global; x86 Windows XP 5.1; java 1.7.0_21; GMT+04:00/ru) http://yacy.net/bot.html bot: name: YaCy category: Search bot url: http://yacy.net/bot.html producer: name: YaCy url: http://yacy.net - user_agent: Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html) bot: name: Yahoo Gemini category: Crawler url: https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Y!J-BRW/1.0 (https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716) bot: name: Yahoo! Japan BRW category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Mozilla/5.0 (compatible; Y!J-WSC/1.0; +https://yahoo.jp/3BSZgF) bot: name: Yahoo! Japan WSC category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html) bot: name: Yahoo! Link Preview category: Crawler url: https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Yahoo:LinkExpander:Slingstone bot: name: Yahoo! Link Preview category: Crawler url: https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) NOT Firefox/3.5 bot: name: Yahoo! Slurp category: Search bot url: http://help.yahoo.com/ysearch/slurp producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) bot: name: Yahoo! Slurp category: Search bot url: http://help.yahoo.com/ysearch/slurp producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) bot: name: Yahoo! Slurp category: Search bot url: http://help.yahoo.com/ysearch/slurp producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Mozilla/5.0 (compatible; Linux i686; Yandex.Gazeta Bot/1.0; +http://gazeta.yandex.ru) bot: name: Yandex.Gazeta Bot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YaDirectFetcher/1.0; +http://yandex.com/bots) bot: name: YaDirectFetcher category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexAntivirus/2.0; +http://yandex.com/bots) bot: name: YandexAntivirus category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; B; +http://yandex.com/bots) bot: name: YandexBlogs category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) bot: name: YandexBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots) bot: name: YandexDirect category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots) bot: name: YandexFavicons category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots) bot: name: YandexImageResizer category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) bot: name: YandexImages category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots) bot: name: YandexMetrika category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexMobileScreenShotBot/1.0; +http://yandex.com/bots) bot: name: YandexMobileScreenShotBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexNews/3.0; +http://yandex.com/bots) bot: name: YandexNews category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexNewslinks; +http://yandex.com/bots) bot: name: YandexNewslinks category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexOntoDB/1.0; +http://yandex.com/bots) bot: name: YandexOntoDB category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexOntoDBAPI/1.0; +http://yandex.com/bots) bot: name: YandexOntoDBAPI category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexPartner/3.0; +http://yandex.com/bots) bot: name: YandexPartner category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexRCA/1.0; +http://yandex.com/bots) bot: name: YandexRCA category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots) bot: name: YandexSearchShop category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexTracker/1.0; +http://yandex.com/bots) bot: name: YandexTracker category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexTurbo/1.0; +http://yandex.com/bots) bot: name: YandexTurbo category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexVerticals/1.0; http://yandex.com/bots) bot: name: YandexVerticals category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; NaverJapan/1.0; +http://corp.naver.jp/) bot: name: Yeti/Naverbot category: Search bot url: http://help.naver.com/robots/ producer: name: Naver url: http://www.naver.com - user_agent: Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/) bot: name: Yeti/Naverbot category: Search bot url: http://help.naver.com/robots/ producer: name: Naver url: http://www.naver.com - user_agent: Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) bot: name: Youdao Bot category: Search bot url: http://www.youdao.com/help/webmaster/spider producer: name: NetEase, Inc. url: http://corp.163.com - user_agent: YOURLS v1.5.1 +http://yourls.org/ (running on http://fhort.com) bot: name: Yourls category: Crawler url: http://yourls.org - user_agent: Mozilla/5.0 (compatible; YRSpider; +http://www.yunrang.com/yrspider.html) bot: name: Yunyun Bot category: Search bot url: http://www.yunyun.com/SiteInfo.php?r=about producer: name: YunYun url: http://www.yunyun.com - user_agent: Mozilla/5.0 (compatible; YRSpider; +http://www.yunyun.com/SiteInfo.php?r=about) bot: name: Yunyun Bot category: Search bot url: http://www.yunyun.com/SiteInfo.php?r=about producer: name: YunYun url: http://www.yunyun.com - user_agent: Mozilla/5.0 (compatible; YYSpider; +http://www.yunyun.com/spider.html) bot: name: Yunyun Bot category: Search bot url: http://www.yunyun.com/SiteInfo.php?r=about producer: name: YunYun url: http://www.yunyun.com - user_agent: Zao/0.1 (http://www.kototol.org/zao) bot: name: Zao category: Crawler - user_agent: zelist.ro feed parser (+http://www.zelist.ro) bot: name: Ze List url: https://www.zelist.ro/ category: Feed Fetcher producer: name: Treeworks SRL url: https://www.tree.ro/ - user_agent: Zookabot/2.5;++http://zookabot.com bot: name: Zookabot category: Crawler url: http://zookabot.com producer: name: Hwacha ApS url: http://hwacha.dk - user_agent: Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry) bot: name: ZumBot category: Search bot url: http://help.zum.com/inquiry producer: name: ZUM internet url: http://www.zuminternet.com/ - user_agent: ZumBot/1.0 (ZUM Search; http://help.zum.com/inquiry) bot: name: ZumBot category: Search bot url: http://help.zum.com/inquiry producer: name: ZUM internet url: http://www.zuminternet.com/ - user_agent: AhrefsBot.Feeds v0.1; http://ahrefs.com/ bot: name: aHrefs Bot category: Crawler url: https://ahrefs.com/robot producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/robot - user_agent: Mozilla/5.0 (compatible; AhrefsBot/3.1; +http://ahrefs.com/robot/ bot: name: aHrefs Bot category: Crawler url: https://ahrefs.com/robot producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/robot - user_agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) bot: name: aHrefs Bot category: Crawler url: https://ahrefs.com/robot producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/robot - user_agent: 'Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) AppEngine-Google; (+http://code.google.com/appengine; appid: s~proxyfile1-hrd)' bot: name: aHrefs Bot category: Crawler url: https://ahrefs.com/robot producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/robot - user_agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) bot: name: aHrefs Bot category: Crawler url: https://ahrefs.com/robot producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/robot - user_agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; archive.org_bot/3.3.0 +http://pandora.nla.gov.au/crawl.html) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; heritrix/3.1.2-SNAPSHOT-20131029-0036 +http://archive.org/details/archive.org_bot) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: Mozilla/5.0 (compatible; special_archiver/3.2.0 +http://www.loc.gov/webarchiving/notice_to_webmasters.html) bot: name: archive.org bot category: Crawler url: https://archive.org/details/archive.org_bot producer: name: The Internet Archive url: https://archive.org - user_agent: CCBot/2.0 (http://commoncrawl.org/faq/) bot: name: ccBot crawler category: Crawler url: http://commoncrawl.org/faq/ producer: name: reddit inc. url: http://www.reddit.com - user_agent: Mozilla/5.0 eCairn-Grabber/1.0 (+http://ecairn.com/grabber) bot: name: eCairn-Grabber category: Crawler producer: name: eCairn url: https://ecairn.com - user_agent: eZ Publish Link Validator bot: name: eZ Publish Link Validator category: Crawler url: https://ez.no/ producer: name: eZ Systems AS url: https://ez.no/ - user_agent: Mozilla/5.0 (compatible; evc-batch/2.0) bot: name: evc-batch category: Crawler url: "" producer: name: eVenture Capital Partners II, LLC url: http://www.eventures.vc/ - user_agent: Mozilla/5.0 (compatible; inoreader.com; 2 subscribers) bot: name: inoreader category: Feed Reader url: https://www.inoreader.com - user_agent: masscan/1.0 (https://github.com/robertdavidgraham/masscan) bot: name: masscan url: https://github.com/robertdavidgraham/masscan category: Crawler producer: name: Robert Graham url: https://github.com/robertdavidgraham - user_agent: 'Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com' bot: name: nbertaupete95 category: Crawler - user_agent: Mozilla/5.0 (compatible; oBot/2.3.1; http://www.xforce-security.com/crawler/) bot: name: oBot category: Search bot url: https://www.xforce-security.com/crawler/ producer: name: 'IBM Germany Research & Development GmbH' url: https://exchange.xforce.ibmcloud.com/ - user_agent: Mozilla/5.0 (compatible; theoldreader.com; 1 subscribers; feed-id=aaa) bot: name: theoldreader category: Feed Reader url: https://theoldreader.com - user_agent: Sprinklr 2.0 bot: name: Sprinklr category: Crawler url: "" producer: name: Sprinklr, Inc. url: https://www.sprinklr.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64)AppleWebKit/537.36 (KHTML, like Gecko; Google-Ads-Qualify)Chrome/85.0.4183.140 Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 bot: name: BDCbot category: Crawler url: https://bigweb.bigdatacorp.com.br/pages/faq.aspx producer: name: BIG Data Solucoes Em Tecnologia de Informatica LTDA url: https://bigdatacorp.com.br/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 CloudflareDiagnostics/1.0 bot: name: Cloudflare Diagnostics category: Site Monitor url: https://www.cloudflare.com/ producer: name: Cloudflare url: https://www.cloudflare.com/ - user_agent: Cloudflare-diagnostics bot: name: Cloudflare Diagnostics category: Site Monitor url: https://www.cloudflare.com/ producer: name: Cloudflare url: https://www.cloudflare.com/ - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) adbeat.com/policy AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/69.0.3497.105 Mobile/15E148 Safari/605.1 bot: name: Adbeat category: Crawler url: https://www.adbeat.com/operation_policy producer: name: PPC Labs LLC url: https://www.adbeat.com/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.1; bit.ly/2W6Px8S) Chrome/84.0.4147.105 Safari/537.36 bot: name: BuiltWith category: Crawler url: https://builtwith.com/biup producer: name: BuiltWith Pty Ltd url: https://builtwith.com/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com) bot: name: ContentKing category: Site Monitor url: https://whatis.contentkingapp.com/ producer: name: ContentKing BV url: https://www.contentkingapp.com/ - user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36 MicroAdBot/1.1 (https://www.microad.co.jp/contact/) bot: name: MicroAdBot category: Crawler url: https://www.microad.co.jp/ producer: name: MicroAd, Inc. url: https://www.microad.co.jp/ - user_agent: Mozilla/5.0 (compatible; PingAdmin.Ru/1.2; +http://pingadmin.ru/free_test/) bot: name: PingAdmin.Ru category: Site Monitor url: https://ping-admin.ru/ - user_agent: Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html) bot: name: WebDataStats category: Crawler url: https://webdatastats.com/policy.html producer: name: WebTehRazrabotka LLC url: https://webdatastats.com/ - user_agent: Mozilla/5.0 (compatible; parse.ly scraper/0.14; +http://parsely.com) bot: name: parse.ly category: Crawler url: https://www.parse.ly/help/integration/crawler producer: name: Parsely, Inc. url: https://www.parse.ly/ - user_agent: Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com) bot: name: Nimbostratus Bot category: Site Monitor url: http://cloudsystemnetworks.com - user_agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.17) Gecko/20110515 HeartRails_Capture/1.0.4 (+http://capture.heartrails.com/) Namoroka/3.6.17 bot: name: HeartRails Capture category: Service Agent url: http://capture.heartrails.com - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring bot: name: Notify Ninja category: Site Monitor url: http://notifyninja.com - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) Project-Resonance (http://project-resonance.com/) (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 bot: name: Project Resonance category: Crawler url: https://project-resonance.com/ producer: name: RedHunt Labs Limited url: https://redhuntlabs.com/ - user_agent: Mozilla/5.0 (compatible; DataXu/1.0; +http://dataxu.com) bot: name: DataXu category: Service Agent url: https://advertising.roku.com/dataxu producer: name: Roku, Inc. url: https://roku.com - user_agent: Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PingdomTMS/0.8.5 Safari/534.34 bot: name: Pingdom Bot category: Site Monitor url: "" producer: name: Pingdom AB url: https://www.pingdom.com - user_agent: Mozilla/5.0 (compatible; Cocolyzebot/1.0; https://cocolyze.com/bot) bot: name: Cocolyzebot category: Crawler url: https://cocolyze.com/en/cocolyzebot producer: name: VSI INNOVATION SAS url: https://vsi-innovation.com/ - user_agent: veryhip (http://veryhip.com/) bot: name: VeryHip category: Crawler url: https://veryhip.com/ producer: name: VeryHip url: https://veryhip.com/ - user_agent: Mozilla/5.0 (compatible; LinkpadBot/2.3; +http://linkpad.org/robot/) bot: name: LinkpadBot category: Crawler url: https://www.linkpad.org/ producer: name: Solomono LLC url: https://www.linkpad.org/ - user_agent: Mozilla/4.0 (compatible; MuscatFerret/1.7; http://www.webtop.com/) bot: name: MuscatFerret category: Crawler url: http://www.webtop.com/ - user_agent: Quora-Bot/1.0 (http://www.quora.com) bot: name: Quora Bot category: Crawler url: "" producer: name: Quora url: https://www.quora.com/ - user_agent: PageThing.com bot: name: PageThing category: Crawler url: https://www.pagething.com/ producer: name: SPECIALNOISE LTD url: https://www.specialnoise.com/ - user_agent: ArchiveBox/0.5.3 (+https://github.com/ArchiveBox/ArchiveBox/) bot: name: ArchiveBox url: https://archivebox.io/ category: Crawler producer: name: "" url: "" - user_agent: Choosito/1.0 +http://www.choosito.com/ bot: name: Choosito url: https://www.choosito.com/ category: Crawler producer: name: Choosito! Inc. url: https://www.choosito.com/ - user_agent: datagnionbot (+http://www.datagnion.com/bot.html) bot: name: datagnionbot url: https://www.datagnion.com/bot.html category: Crawler producer: name: DATAGNION GMBH url: https://www.datagnion.com/ - user_agent: InterNaetBoten/0.99 (http://www.xn--internt-bxa.nu) bot: name: Generic Bot - user_agent: EasyBib AutoCite (http://autocite-info.citation-api.com/) bot: name: Generic Bot - user_agent: Bidtellect/0.0.643.0 bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; meg/0.2; +https://github.com/tomnomnom/meg) bot: name: Generic Bot - user_agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) WhatCMS/1.0 bot: name: WhatCMS url: https://whatcms.org/ category: Crawler producer: name: Nineteen Ten LLC url: https://whatcms.org/ - user_agent: 'httpx - Open-source project (github.com/projectdiscovery/httpx)' bot: name: httpx url: https://github.com/projectdiscovery/httpx category: Crawler producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: 'Expanse indexes the network perimeters of our customers. If you have any questions or concerns, please reach out to: scaninfo@expanseinc.com' bot: name: Cortex Xpanse category: Security Checker url: https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity producer: name: Palo Alto Networks, Inc. url: https://www.paloaltonetworks.com/ - user_agent: HuaweiWebCatBot/6.0) (To acquire the allowed html pages as reliable information of URL categorization in the automatic process for Huawei Web Categorization.; https://isecurity.huawei.com/; sec at huawei dot com) bot: name: HuaweiWebCatBot category: Crawler url: https://isecurity.huawei.com producer: name: Huawei Technologies Co., Ltd. url: https://huawei.com - user_agent: Mozilla/5.0 (compatible; SputnikFaviconBot/1.2; +http://corp.sputnik.ru/webmaster) bot: name: Sputnik Favicon Bot url: "" category: Crawler - user_agent: Hatena-Favicon2 (http://www.hatena.ne.jp/faq/) bot: name: Hatena Favicon category: Crawler url: https://www.hatena.ne.jp/faq/ producer: name: Hatena Co., Ltd. url: https://www.hatena.ne.jp - user_agent: Mozilla/5.0 (compatible; RyowlEngine/1.0; +https://ryowl.org) bot: name: Ryowl category: Crawler url: https://ryowl.org - user_agent: Mozilla/5.0 (compatible; OdklBot/1.0 like Linux; klass@odnoklassniki.ru) bot: name: Odnoklassniki Bot category: Crawler url: https://odnoklassniki.ru - user_agent: Mediatoolkitbot (complaints@mediatoolkit.com) bot: name: Mediatoolkit Bot category: Crawler url: https://mediatoolkit.com - user_agent: ZoominfoBot (zoominfobot at zoominfo dot com) bot: name: ZoominfoBot category: Crawler url: https://www.zoominfo.com - user_agent: Mozilla/5.0 (compatible; WeViKaBot/1.0; +http://www.wevika.de/) bot: name: WeViKaBot category: Crawler url: http://www.wevika.de - user_agent: Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html) bot: name: SEOkicks category: Crawler url: https://www.seokicks.de/robot.html producer: name: SEOkicks url: https://www.seokicks.de/ - user_agent: Mozilla/5.0 (compatible; Plukkie/1.6; http://www.botje.com/plukkie.htm) bot: name: Plukkie category: Crawler url: http://www.botje.com/plukkie.htm - user_agent: Mozilla/5.0 (compatible; proximic; https://www.comscore.com/Web-Crawler) bot: name: Comscore category: Crawler url: https://www.comscore.com/Web-Crawler - user_agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html) bot: name: SurdotlyBot category: Crawler url: http://sur.ly/bot.html - user_agent: Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com) bot: name: Gowikibot category: Crawler url: http:/www.gowikibot.com - user_agent: Mozilla/5.0 (compatible; SabsimBot/3.0; +https://sabsim.com) bot: name: SabsimBot category: Crawler url: https://sabsim.com - user_agent: Mozilla/5.0 (compatible; LumtelBot/1.0; +http://lumtel.com) bot: name: LumtelBot category: Crawler url: https://lumtel.com - user_agent: Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/) bot: name: PiplBot category: Crawler url: http://www.pipl.com/bot - user_agent: Mozilla/5.0 (compatible; woobot/2.0; +https://www.woorank.com/bot) bot: name: WooRank category: Crawler url: https://www.woorank.com/bot - user_agent: YahooMailProxy; https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html bot: name: Yahoo! Mail Proxy category: Service Agent url: https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cookiebot/1.0; http://cookiebot.com/) Chrome/79.0.3945.117 Safari/537.36' bot: name: Cookiebot category: Crawler url: https://support.cookiebot.com/hc/en-us/articles/360014264140-Scanner-User-Agent producer: name: Cybot A/S url: https://www.cybot.com/ - user_agent: NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com bot: name: NetSystemsResearch category: Security Checker url: https://www.netsystemsresearch.com/ producer: name: NET SYSTEMS RESEARCH LLC url: https://www.netsystemsresearch.com/ - user_agent: 'Mozilla/5.0 (compatible; CensysInspect/1.1; https://about.censys.io/)' bot: name: CensysInspect category: Security Checker url: https://about.censys.io/ producer: name: Censys, Inc. url: https://censys.io/ - user_agent: https://gdnplus.com:Gather Analyze Provide. bot: name: GDNP category: Crawler url: https://gdnplus.com/ producer: name: Global Digital Network Plus, LLC url: https://gdnplus.com/ - user_agent: WellKnownBot/0.1 (https://well-known.dev) bot: name: WellKnownBot category: Crawler url: https://well-known.dev - user_agent: 'Mozilla/5.0 (compatible; Adsbot/3.1; https://seostar.co/robot/)' bot: name: Adsbot category: Crawler url: https://seostar.co/robot/ - user_agent: MTRobot/0.2 (Metrics Tools Analytics Crawler; https://metrics-tools.de/robot.html; crawler@metrics-tools.de) bot: name: MTRobot category: Crawler url: https://metrics-tools.de/robot.html producer: name: Metrics Tools url: https://metrics-tools.de/ - user_agent: My User Agent 1.0 bot: name: Generic Bot - user_agent: serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com) bot: name: serpstatbot category: Crawler url: http://serpstatbot.com/ producer: name: Netpeak Ltd url: https://netpeak.net/ - user_agent: 'colly - https://github.com/gocolly/colly/v2' bot: name: colly category: Crawler url: https://github.com/gocolly/colly/ - user_agent: l9tcpid/v1.0.0 bot: name: l9tcpid category: Security Checker url: https://github.com/LeakIX/l9tcpid - user_agent: Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler) bot: name: MegaIndex category: Crawler url: https://megaindex.com/crawler - user_agent: Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/) bot: name: Seekport category: Crawler url: https://bot.seekport.com/ producer: name: SISTRIX GmbH url: https://www.sistrix.de/ - user_agent: Mozilla/5.0 (compatible; seolyt/1.1; +https://seolyt.com) bot: name: SeolytBot category: Crawler url: https://seolyt.com/ - user_agent: Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com) bot: name: YaK category: Crawler url: https://www.linkfluence.com/ producer: name: Linkfluence SAS url: https://www.linkfluence.com/ - user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 bot: name: KomodiaBot category: Crawler url: http://www.komodia.com/newwiki/index.php/URL_server_crawler producer: name: Komodia Inc. url: https://www.komodia.com/ - user_agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; Catchpoint) like Gecko bot: name: Catchpoint category: Site Monitor url: https://www.catchpoint.com/ producer: name: Catchpoint Systems url: https://www.catchpoint.com/ - user_agent: Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36 bot: name: Better Uptime Bot category: Site Monitor url: https://betteruptime.com/faq producer: name: Better Uptime url: https://betteruptime.com/ - user_agent: gobuster/3.1.0 bot: name: Gobuster url: https://github.com/OJ/gobuster - user_agent: k6/0.31.1 (https://k6.io/) bot: name: K6 url: https://k6.io/ - user_agent: 'Mozilla/5.0 (compatible; Neevabot/1.0; https://neeva.com/neevabot)' bot: name: Neevabot category: Search bot url: https://neeva.com/neevabot producer: name: Neeva Inc. url: https://neeva.com/ - user_agent: LinkPreview/1.5 (http://linkpreview.net) bot: name: LinkPreview category: Service Agent url: https://www.linkpreview.net/ - user_agent: Mozilla/5.0 (compatible; JungleKeyThumbnail/1.1; +http://www.junglekey.fr/) bot: name: JungleKeyThumbnail category: Crawler url: https://junglekey.com/ - user_agent: rocketmonitor 1.0 bot: name: RocketMonitorBot category: Site Monitor url: https://www.radiomast.io/docs/stream-monitoring/technical_details.html producer: name: Radio Mast, Inc. url: https://www.radiomast.io/ - user_agent: RocketMonitorBot/1.0 ( https://www.radiomast.io) bot: name: RocketMonitorBot category: Site Monitor url: https://www.radiomast.io/docs/stream-monitoring/technical_details.html producer: name: Radio Mast, Inc. url: https://www.radiomast.io/ - user_agent: SitemapParser-VIPnytt/1.1 ( https://github.com/VIPnytt/SitemapParser/blob/master/README.md) bot: name: SitemapParser-VIPnytt category: Crawler url: https://github.com/VIPnytt/SitemapParser/ - user_agent: Turnitin (https://bit.ly/2UvnfoQ) bot: name: TurnitinBot category: Crawler url: https://www.turnitin.com/robot/crawlerinfo.html producer: name: iParadigms, LLC. url: https://www.turnitin.com/ - user_agent: Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36 bot: name: Google StoreBot category: Crawler - user_agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.56 Safari/537.36 DMBrowser/2.1 (UV) bot: name: Dotcom Monitor category: Site Monitor url: https://www.dotcom-monitor.com - user_agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:29.0) Gecko/20100101 /29.0 DMBrowser-BV bot: name: Dotcom Monitor category: Site Monitor url: https://www.dotcom-monitor.com - user_agent: Mozilla/5.0 (compatible; ThinkChaos/0.3.0; +In_the_test_phase,_if_the_ThinkChaos_brings_you_trouble,_please_add_disallow_to_the_robots.txt._Thank_you.) bot: name: ThinkChaos category: Crawler - user_agent: Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot) bot: name: DataForSeoBot category: Crawler url: https://dataforseo.com/dataforseo-bot - user_agent: Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com) bot: name: Discord Bot category: Service Agent url: https://discordapp.com - user_agent: facebookcatalog/1.0 bot: name: Facebook Crawler category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: Mozilla/5.0 (compatible; Linespider/1.1; https://lin.ee/4dwXkTH) bot: name: Linespider category: Crawler url: https://lin.ee/4dwXkTH - user_agent: Mozilla/5.0 (compatible; Cincraw/1.0; +http://cincrawdata.net/bot/) bot: name: Cincraw category: Crawler url: http://cincrawdata.net/bot/ - user_agent: CISPA Web Analyzer (https://notify.cispa.de) bot: name: CISPA Web Analyzer category: Crawler url: https://notify.cispa.de/ producer: name: 'CISPA - Helmholtz-Zentrum für Informationssicherheit gGmbH' url: https://cispa.de/en - user_agent: IonCrawl (https://www.ionos.de/terms-gtc/faq-crawler-en/) bot: name: IONOS Crawler category: Crawler url: https://www.ionos.de/terms-gtc/faq-crawler-en/ producer: name: IONOS SE url: https://www.ionos.de/ - user_agent: Crawldad (+https://gist.github.com/jayhardee9/2f2a2c4dba26564ee040ae32e0dd0972) bot: name: Crawldad category: Crawler url: https://gist.github.com/jayhardee9/2f2a2c4dba26564ee040ae32e0dd0972 - user_agent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 ### This is a crawler for security.txt files. For more information visit https://securitytxt-scan.cs.hm.edu/ ###' bot: name: security.txt scanserver category: Security Checker url: https://securitytxt-scan.cs.hm.edu/ producer: name: Hochschule für angewandte Wissenschaften München url: https://www.hm.edu/ - user_agent: Mozilla/5.0 (Linux; CentOS Linux release 7.4.1708) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 (compatible; AdsBot-Naver/1.0; +http://searchad.naver.com) bot: name: Yeti/Naverbot category: Search bot url: http://help.naver.com/robots/ producer: name: Naver url: http://www.naver.com - user_agent: Mozilla/5.0 (compatible; Google-Stale-Content-Probe; +http://www.google.com/bot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: cortex/1.0 bot: name: Generic Bot - user_agent: Mozilla/5.0 AppleWebKit/537.36 Chrome/95.0.4638.69 Safari/537.36 Google-Ads-Conversions bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; TigerBot/8.69; Tiger.ch) bot: name: TigerBot category: Crawler url: https://tiger.ch/ - user_agent: 'Mozilla/5.0 (compatible; TestCrawler/0.2; http://www.comcepta.com)' bot: name: TestCrawler category: Crawler url: https://www.comcepta.com/ - user_agent: CF-UC User Agent v.1d.99613 bot: name: Generic Bot - user_agent: Re-re Studio ( http://vip0.ru/) bot: name: Generic Bot - user_agent: adreview/1.0 bot: name: Generic Bot - user_agent: AHC/2.1 bot: name: Generic Bot - user_agent: NameOfAgent bot: name: Generic Bot - user_agent: Request-Promise bot: name: Generic Bot - user_agent: GoogleAssociationService bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: CrowdTanglebot/1.0 bot: name: CrowdTangle category: Crawler url: https://help.crowdtangle.com/en/articles/3009319-crowdtangle-bot producer: name: CrowdTangle, Inc. url: https://www.crowdtangle.com/ - user_agent: Sellers.Guide Crawler by Primis bot: name: Sellers.Guide category: Crawler url: https://sellers.guide/ producer: name: McCann Disciplines, Ltd. url: https://www.primis.tech/ - user_agent: AhrefsSiteAudit/6.1; http://ahrefs.com/robot/site-audit bot: name: AhrefsSiteAudit category: Site Monitor url: https://ahrefs.com/robot/site-audit producer: name: Ahrefs Pte Ltd url: https://ahrefs.com/ - user_agent: OnalyticaBot bot: name: Onalytica category: Crawler url: https://www.airslate.com/bot/explore/onalytica-bot producer: name: airSlate, Inc. url: https://www.airslate.com/ - user_agent: Snap URL Preview Service; bot; https://developer.snapchat.com/robots bot: name: Snap URL Preview Service category: Service Agent url: https://developers.snap.com/robots producer: name: Snapchat Inc. url: https://www.snapchat.com/ - user_agent: 'deepnoc - https://deepnoc.com/bot' bot: name: deepnoc category: Crawler url: https://deepnoc.com/bot producer: name: deepnoc, GmbH url: https://deepnoc.com/ - user_agent: Newslitbot/0.1; https://www.newslit.co bot: name: Newslitbot category: Crawler url: https://www.newslit.co/ producer: name: Newslit, LLC. url: https://www.newslit.co/ - user_agent: 'Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)' bot: name: uMBot category: Crawler url: https://www.ubermetrics-technologies.com/ producer: name: Ubermetrics Technologies GmbH url: https://www.ubermetrics-technologies.com/ - user_agent: 'Mozilla/5.0 (compatible; Abonti/0.91 - http://www.abonti.com)' bot: name: Abonti category: Crawler url: http://abonti.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36; +collection@infegy.com bot: name: Infegy category: Crawler url: https://infegy.com/ producer: name: Infegy, Inc. url: https://infegy.com/ - user_agent: ALittle Client bot: name: Generic Bot - user_agent: Hello, world bot: name: Generic Bot - user_agent: wp_is_mobile bot: name: Generic Bot - user_agent: HTTP Banner Detection (https://security.ipip.net) bot: name: IPIP category: Security Checker url: https://security.ipip.net/ producer: name: Beijing Tiantexin Tech. Co., Ltd. url: https://en.ipip.net/ - user_agent: Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler) bot: name: Headline category: Crawler url: https://headline.com/legal/crawler producer: name: e.ventures Managementgesellschaft mbH url: https://headline.com/ - user_agent: webprosbot/2.0 (+mailto:abuse-6337@webpros.com) bot: name: WebPros category: Crawler url: https://webpros.com/ producer: name: WebPros Holdco B.V. url: https://webpros.com/ - user_agent: 0xAbyssalDoesntExist bot: name: Generic Bot - user_agent: ELB-HealthChecker/2.0 bot: name: Amazon ELB category: Site Monitor url: https://aws.amazon.com/elasticloadbalancing/ producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Datadog/Synthetics bot: name: Datadog Agent url: https://github.com/DataDog/dd-agent category: Site Monitor producer: name: Datadog url: https://www.datadoghq.com/ - user_agent: DatadogSynthetics bot: name: Datadog Agent url: https://github.com/DataDog/dd-agent category: Site Monitor producer: name: Datadog url: https://www.datadoghq.com/ - user_agent: Wheregoes.com Redirect Checker/1.0 bot: name: WhereGoes category: Crawler url: https://wheregoes.com/ - user_agent: Mozilla/5.0 project_patchwatch bot: name: Project Patchwatch category: Crawler url: http://66.240.192.82/ - user_agent: masscan-ng/1.3 (https://github.com/bi-zone/masscan-ng) bot: name: masscan-ng url: https://github.com/bi-zone/masscan-ng category: Crawler producer: name: BIZON, OOO url: https://bi.zone/ - user_agent: Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/) bot: name: InternetMeasurement category: Crawler url: https://internet-measurement.com/ - user_agent: Mozilla/5.0 (compatible; DomainAppender /1.0; +http://www.profound.net/domainappender) bot: name: DomainAppender category: Crawler url: https://www.profound.net/product/domain_append/ producer: name: Profound Networks, LLC url: https://www.profound.net/ - user_agent: FreeWebMonitoring SiteChecker/0.2 (+https://www.freewebmonitoring.com/bot.html) bot: name: FreeWebMonitoring category: Site Monitor url: https://www.freewebmonitoring.com/bot.html producer: name: GreenWave Online, Inc. url: http://www.greenwaveonline.com/ - user_agent: Page Modified Pinger (+http://www.pagemodified.com/bots) bot: name: Page Modified Pinger category: Site Monitor url: https://www.pagemodified.com/ producer: name: Valley Hosting, LLC url: https://www.pagemodified.com/ - user_agent: adstxtlab.com Crawler bot: name: adstxtlab.com category: Crawler url: https://adstxtlab.com/validator.php producer: name: Jaohawi AB url: https://adstxtlab.com/ - user_agent: Iframely/1.6.1 (+https://metadata.xayn.com) bot: name: Iframely category: Crawler url: https://iframely.com/ producer: name: Itteco Software, Corp. url: https://iframely.com/ - user_agent: DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot) bot: name: DomainStatsBot category: Crawler url: https://domainstats.com/pages/our-bot producer: name: Domainstats Ltd url: https://domainstats.com/ - user_agent: Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about) bot: name: aiHitBot category: Crawler url: https://www.aihitdata.com/about - user_agent: ias-sg/3.1 (+https://www.admantx.com/service-fetcher.html) bot: name: ADmantX Service Fetcher category: Service bot url: https://www.admantx.com/service-fetcher.html - user_agent: DomainCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/wwweb.be) bot: name: DomainCrawler category: Crawler url: https://domaincrawler.com/about-us/ - user_agent: Mozilla/5.0 (compatible; DNSResearchBot/2.1; +http://195.37.190.77) bot: name: DNSResearchBot category: Crawler - user_agent: GitCrawlerBot/0.0.1 (abuse reports to abuse@crissyfield.de) bot: name: GitCrawlerBot category: Crawler - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 AdAuth/1.0; +https://www.adauth.com bot: name: AdAuth category: Crawler url: https://www.adauth.com - user_agent: Mozilla/5.0 (compatible; Faveeo/1.0; +http://www.faveeo.com) bot: name: Faveeo category: Crawler url: http://www.faveeo.com - user_agent: ias-ir/3.1 (+https://www.admantx.com/service-fetcher.html) bot: name: ADmantX Service Fetcher category: Service bot url: https://www.admantx.com/service-fetcher.html - user_agent: ias-jp/3.1 (+https://www.admantx.com/service-fetcher.html) bot: name: ADmantX Service Fetcher category: Service bot url: https://www.admantx.com/service-fetcher.html - user_agent: KOCMOHABT (+https://kozmonavt.tk/) Mozilla/5.0 (compatible; Web Explorer 2) bot: name: Kozmonavt category: Crawler url: https://kozmonavt.ml - user_agent: KOCMOHABT (https://kozmonavt.ml/) Mozilla/5.0 (Web Explorer) bot: name: Kozmonavt category: Crawler url: https://kozmonavt.ml - user_agent: CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/) bot: name: CriteoBot category: Crawler url: https://www.criteo.com/criteo-crawler/ - user_agent: PayPal IPN ( https://www.paypal.com/ipn ) bot: name: PayPal IPN category: Service Agent url: https://developer.paypal.com/api/nvp-soap/ipn/IPNIntro/ producer: name: PayPal, Inc. url: https://www.paypal.com/ - user_agent: l9explore/1.3.0 bot: name: l9explore category: Security Checker url: https://github.com/LeakIX/l9explore - user_agent: 'Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com' bot: name: Cortex Xpanse category: Security Checker url: https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity producer: name: Palo Alto Networks, Inc. url: https://www.paloaltonetworks.com/ - user_agent: Mozilla/5.0 (compatible; MaCoCu; +https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/) bot: name: MaCoCu category: Crawler url: https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/ producer: name: Jožef Stefan Institute url: https://www.ijs.si/ijsw/JSI - user_agent: Mozilla/5.0 (compatible; CLASSLA-web; +https://www.clarin.si/info/classla-web-crawler/) bot: name: CLASSLA-web category: Crawler url: https://www.clarin.si/info/classla-web-crawler/ producer: name: Jožef Stefan Institute url: https://www.ijs.si/ijsw/JSI - user_agent: "Electronic Frontier Foundation's Do Not Track Verifier (for questions or concerns email dnt-policy@eff.org)" bot: name: EFF Do Not Track Verifier category: Crawler url: https://www.eff.org/issues/do-not-track producer: name: Electronic Frontier Foundation url: https://www.eff.org/ - user_agent: Mozilla/5.0 (compatible; InfoTigerBot/1.9; +https://infotiger.com/bot) bot: name: InfoTigerBot category: Crawler url: https://infotiger.com/bot producer: name: Infotiger UG url: https://infotiger.com/ - user_agent: Mozilla/5.0 (compatible; Birdcrawlerbot/0.5; +https://crawla.de) bot: name: Birdcrawlerbot category: Crawler url: https://crawla.de/de/index.php producer: name: Swoppen Systems GmbH url: https://www.swoppen.com/de - user_agent: ScamadviserExternalHit/1.0 bot: name: Scamadviser External Hit category: Crawler url: https://www.scamadviser.com/ producer: name: Ecommerce Operations B.V. url: https://www.scamadviser.com/ - user_agent: ZaldamoSearchBot(www.zaldamo.com/search.html) bot: name: Zaldamo category: Crawler url: https://www.zaldamo.com/search.html producer: name: Zaldamo, LLC. url: https://www.zaldamo.com/ - user_agent: Anarchy99 bot: name: Generic Bot - user_agent: AFB/4.1 (+http://allloadin.com) bot: name: Allloadin Favicon Bot category: Crawler url: https://allloadin.com/ - user_agent: 'Mozilla/5.0 (compatible; SeolytBot/1.0.1; https://seolyt.com)' bot: name: SeolytBot category: Crawler url: https://seolyt.com/ - user_agent: LinkWalker/3.0 (http://www.brandprotect.com) bot: name: LinkWalker category: Crawler url: https://www.phishlabs.com/ producer: name: PhishLabs, Inc. url: https://www.phishlabs.com/ - user_agent: RenovateBot/32.99.9 bot: name: RenovateBot category: Security Checker url: https://github.com/renovatebot/renovate producer: name: White Source Ltd. url: https://www.mend.io/free-developer-tools/renovate/ - user_agent: INETDEX-BOT/1.5 (Mozilla/5.0; https://inetdex.com/; info at inetdex dot com) bot: name: Inetdex Bot category: Crawler url: https://www.inetdex.com/ - user_agent: 'Mozilla/5.0(compatible; NETZZAPPEN is a wysiwyg searchengine, which shows screenshot http://www.netzzappen.com - crawler 033)' bot: name: NETZZAPPEN category: Crawler url: https://www.netzzappen.com/ producer: name: Marc Huemer url: https://www.netzzappen.com/ - user_agent: Mozilla/5.0 (compatible; SerpReputationManagementAgent/1.0) bot: name: Semrush Reputation Management category: Service Agent url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: panscient.com bot: name: Panscient category: Crawler url: https://www.panscient.com/faq.htm producer: name: Panscient, Inc. url: https://www.panscient.com/ - user_agent: daumoa,damoa,daum,daumos,duamoa,duam,duamo bot: name: Daum category: Search bot url: http://tab.search.daum.net/aboutWebSearch_en.html producer: name: Daum Communications Corp. url: http://www.kakaocorp.com/main - user_agent: daumoa,damoa,daum,daumos,duamoa,duam,duamos bot: name: Daum category: Search bot url: http://tab.search.daum.net/aboutWebSearch_en.html producer: name: Daum Communications Corp. url: http://www.kakaocorp.com/main - user_agent: "'Cloud mapping experiment. Contact research@pdrlabs.net'" bot: name: PDR Labs category: Security Checker url: https://web.archive.org/web/20220420054123/http://www.pdrlabs.net/ producer: name: PDR Labs url: https://web.archive.org/web/20220420054123/http://www.pdrlabs.net/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36 bot: name: NiceCrawler category: Crawler url: https://www.nicecrawler.com/ producer: name: Intelium Corp. url: https://www.intelium.com/ - user_agent: Mozilla/5.0 (compatible; t3versionsBot/1.0; +https://www.t3versions.com/bot) bot: name: t3versions category: Crawler url: https://www.t3versions.com/bot producer: name: Torben Hansen url: https://www.t3versions.com/ - user_agent: Mozilla/5.0 (compatible; Crawlson/1.0; +https://www.crawlson.com/domain/example.com) bot: name: Crawlson category: Crawler url: https://www.crawlson.com/about producer: name: Crawlson url: https://www.crawlson.com/ - user_agent: Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io) bot: name: tchelebi category: Crawler url: https://tchelebi.io/ producer: name: NormShield, Inc. url: https://blackkite.com/ - user_agent: Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0 bot: name: JobboerseBot category: Crawler url: https://www.xing.com/jobs producer: name: New Work SE url: https://www.xing.com/ - user_agent: Lkx-TraversalHttpPlugin/0.0.1 (+https://leakix.net/, +https://twitter.com/HaboubiAnis) bot: name: LeakIX category: Security Checker url: https://leakix.net/ producer: name: BaDaaS SRL url: https://leakix.net/ - user_agent: Lkx-Apache2449TraversalPlugin/0.0.1 (+https://leakix.net/, +https://twitter.com/HaboubiAnis) bot: name: LeakIX category: Security Checker url: https://leakix.net/ producer: name: BaDaaS SRL url: https://leakix.net/ - user_agent: RepoLookoutBot/1.0.0 (abuse reports to abuse@repo-lookout.org) bot: name: Repo Lookout category: Security Checker url: https://www.repo-lookout.org/ producer: name: Crissy Field GmbH url: https://www.crissyfield.de/ - user_agent: PATHspider (https://pathspider.net/) bot: name: PATHspider category: Security Checker url: https://pathspider.net/ producer: name: MAMI Project url: https://mami-project.eu/ - user_agent: everyfeed-spider/2.0 (http://www.everyfeed.com) bot: name: Everyfeed url: https://web.archive.org/web/20050930235914/http://www.everyfeed.com/ category: Feed Fetcher producer: name: "" url: "" - user_agent: Mozilla/5.0 (Exchange check) bot: name: Exchange check category: Security Checker url: https://github.com/GossiTheDog/scanning producer: name: Kevin Beaumont url: https://doublepulsar.com/ - user_agent: 'Sublinq Crawler - QR Codes for Interactive Feedback (https://sublinq.com)' bot: name: Sublinq category: Crawler url: https://web.archive.org/web/20220626191617/https://sublinq.com/ producer: name: "" url: "" - user_agent: Gregarius/0.5.2 (+http://devlog.gregarius.net/docs/ua) bot: name: Gregarius category: Feed Fetcher url: https://web.archive.org/web/20100614011837/http://devlog.gregarius.net/docs/ua/ producer: name: "" url: "" - user_agent: revolt bot: name: Generic Bot - user_agent: nvd0rz bot: name: Generic Bot - user_agent: xfa1 bot: name: Generic Bot - user_agent: Hakai/2.0 bot: name: Generic Bot - user_agent: gbrmss/7.29.0 bot: name: Generic Bot - user_agent: fuck-your-hp bot: name: Generic Bot - user_agent: IDBTE4M CODE87 bot: name: Generic Bot - user_agent: Antoine bot: name: Generic Bot - user_agent: Insomania/1.0 bot: name: Generic Bot - user_agent: Hells-Net bot: name: Generic Bot - user_agent: b3astmode bot: name: Generic Bot - user_agent: Hello World bot: name: Generic Bot - user_agent: Linux Gnu (cow) bot: name: Generic Bot - user_agent: '{{.custom_user_agent}}' bot: name: Generic Bot - user_agent: COMODO DCV bot: name: COMODO DCV category: Service Agent url: https://www.comodo.com/ producer: name: Comodo Security Solutions, Inc. url: https://www.comodo.com/ - user_agent: Sectigo DCV bot: name: Sectigo DCV category: Service Agent url: https://sectigo.com/ producer: name: Sectigo Limited url: https://sectigo.com/ - user_agent: KlarnaBot-PriceWatcher/1.0 bot: name: KlarnaBot category: Crawler url: https://docs.klarna.com/klarna-bot/ producer: name: Klarna Bank AB url: https://www.klarna.com/ - user_agent: KlarnaBot-EnrichProducts/1.0 bot: name: KlarnaBot category: Crawler url: https://docs.klarna.com/klarna-bot/ producer: name: Klarna Bank AB url: https://www.klarna.com/ - user_agent: KlarnaBot-DownloadProductImage/1.0 bot: name: KlarnaBot category: Crawler url: https://docs.klarna.com/klarna-bot/ producer: name: Klarna Bank AB url: https://www.klarna.com/ - user_agent: Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com) bot: name: Taboolabot category: Crawler url: https://help.taboola.com/hc/en-us/articles/115002347594-The-Taboola-Crawler producer: name: Taboola, Inc. url: https://www.taboola.com/ - user_agent: 'Sogou head spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)' bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: 'Sogou Orion spider/3.0( http://www.sogou.com/docs/help/webmasters.htm#07)' bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Sogou-Test-Spider/4.0 (compatible; MSIE 5.5; Windows 98) bot: name: Sogou Spider category: Search bot url: http://www.sogou.com/docs/help/webmasters.htm producer: name: Sohu, Inc. url: http://www.sogou.com - user_agent: Asana/1.4.0 WebsiteMetadataRetriever bot: name: Asana category: Crawler url: https://asana.com/ producer: name: Asana, Inc. url: https://asana.com/ - user_agent: Chrome Privacy Preserving Prefetch Proxy bot: name: Chrome Privacy Preserving Prefetch Proxy category: Service Agent url: https://developer.chrome.com/blog/private-prefetch-proxy/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; URLinspectorBot/1.0; +https://www.urlinspector.com/bot) bot: name: URLinspector category: Site Monitor url: https://www.urlinspector.com/bot/ producer: name: LinkResearchTools GmbH url: https://www.linkresearchtools.com/ - user_agent: Test Certificate Info bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; EntferBot/0.1; +https://entfer.com; +a_new_independent_search_engine;) bot: name: Entfer category: Crawler url: https://entfer.com/ producer: name: Entfer Ltd. url: https://entfer.com/ - user_agent: Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; GoogleAdSenseInfeed) Chrome/102.0.5005.61 Mobile Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (iplabel; Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36 bot: name: Generic Bot - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) TagInspector/500.1 Chrome/90.0.4430.72 Safari/537.36 Edg/90.0.818.42 bot: name: Tag Inspector category: Crawler url: https://taginspector.com/ producer: name: InfoTrust, LLC url: https://infotrust.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; pageburst) Chrome/98.0.4758.102 Safari/537.36 bot: name: Pageburst category: Crawler url: https://pageburstls.elsevier.com/ producer: name: Elsevier Ltd url: https://www.elsevier.com/ - user_agent: Snap/1.0 (+https://www.diffbot.com/dev/docs/crawl/) bot: name: Diffbot category: AI Data Scraper url: https://docs.diffbot.com/docs/getting-started-with-crawl producer: name: Diffbot Technologies Corp. url: https://www.diffbot.com/ - user_agent: Doximity-Diffbot bot: name: Doximity-Diffbot category: AI Data Scraper url: https://docs.diffbot.com/docs/getting-started-with-crawl producer: name: Diffbot Technologies Corp. url: https://www.diffbot.com/ - user_agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) bot: name: Diffbot category: AI Data Scraper url: https://docs.diffbot.com/docs/getting-started-with-crawl producer: name: Diffbot Technologies Corp. url: https://www.diffbot.com/ - user_agent: DisqusAdstxtCrawler/0.1 (+https://help.disqus.com/en/articles/1765357-ads-txt-implementation-guide) bot: name: Disqus category: Crawler url: https://help.disqus.com/en/articles/1765357-ads-txt-implementation-guide producer: name: Disqus, Inc. url: https://disqus.com/ - user_agent: Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot) bot: name: start.me category: Crawler url: https://about.start.me/ producer: name: start.me BV url: https://about.start.me/ - user_agent: 2ip bot/1.1 (+http://2ip.io) bot: name: 2ip category: Crawler url: https://2ip.io/ - user_agent: ReqBin Curl Client/1.0 bot: name: ReqBin category: Crawler url: https://reqbin.com/curl - user_agent: Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/) bot: name: XoviBot category: Crawler url: https://www.xovibot.net producer: name: Xovi GmbH url: http://www.xovi.de - user_agent: Overcast/1.0 Podcast Sync (123 subscribers; feed-id=456789; +http://overcast.fm/) bot: name: Overcast Podcast Sync category: Service Agent url: https://overcast.fm/podcasterinfo - user_agent: Verity/1.1 (https://gumgum.com/verity; verity-support@gumgum.com) bot: name: GumGum Verity category: Service Agent url: https://gumgum.com/verity - user_agent: GoogleProber bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: 'hackermention ' bot: name: hackermention category: Feed Reader url: https://github.com/snarfed/hackermention - user_agent: Mozilla/5.0 (compatible; BitSightBot/1.0) bot: name: BitSight category: Security Checker url: https://www.bitsight.com/ producer: name: BitSight Technologies, Inc. url: https://www.bitsight.com/ - user_agent: Mozilla/5.0 (compatible; Ezgif/1.0; +https://ezgif.com/about) bot: name: Ezgif category: Service Agent url: https://ezgif.com/about - user_agent: Mozilla/5.0 (compatible; intelx.io_bot +https://intelx.io) bot: name: Intelligence X category: Crawler url: https://intelx.io/ producer: name: Kleissner Investments s.r.o. url: https://intelx.io/ - user_agent: Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com) bot: name: Femtosearch category: Crawler url: http://femtosearch.com/ producer: name: Grier Forensics, LLC url: https://www.grierforensics.com/ - user_agent: AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler bot: name: AdsTxtCrawler category: Crawler url: https://github.com/InteractiveAdvertisingBureau/adstxtcrawler producer: name: IAB Technology Laboratory, Inc. url: https://iabtechlab.com/ - user_agent: 'Mozilla/5.0 (compatible;Cloudflare-Healthchecks/1.0;+https://www.cloudflare.com/; healthcheck-id: e645d1e65d3b18b4)' bot: name: Cloudflare Health Checks category: Service Agent url: https://developers.cloudflare.com/health-checks/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Mozilla/5.0 (Morningscore/1.0) bot: name: Morningscore Bot category: Crawler url: https://morningscore.io/ producer: name: Morningscore url: https://morningscore.io/ - user_agent: Mozilla/5.0 (Morningscore Bot/1.0) bot: name: Morningscore Bot category: Crawler url: https://morningscore.io/ producer: name: Morningscore url: https://morningscore.io/ - user_agent: Uptime-Kuma/1.21.2 bot: name: Uptime-Kuma category: Site Monitor url: https://github.com/louislam/uptime-kuma - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot bot: name: ChatGPT-User category: AI Assistant url: https://platform.openai.com/docs/bots producer: name: OpenAI OpCo, LLC url: https://openai.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot bot: name: OAI-SearchBot category: AI Search Crawler url: https://platform.openai.com/docs/bots producer: name: OpenAI OpCo, LLC url: https://openai.com/ - user_agent: BrightEdge Crawler/1.0 (crawler@brightedge.com) bot: name: BrightEdge category: Crawler url: https://www.brightedge.com/ producer: name: BrightEdge Technologies, Inc url: https://www.brightedge.com/ - user_agent: sfFeedReader/0.9 bot: name: sfFeedReader category: Feed Fetcher url: https://github.com/diem-project/sfFeed2Plugin - user_agent: HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer) bot: name: Hatena Bookmark category: Crawler url: https://www.hatena.ne.jp/faq/ producer: name: Hatena Co., Ltd. url: https://www.hatena.ne.jp - user_agent: MoodleBot-LinkChecker (+https://docs.moodle.org/en/Usage) bot: name: MoodleBot Linkchecker category: Search bot url: https://docs.moodle.org/en/Usage producer: name: Moodle Pty Ltd url: https://moodle.org/ - user_agent: cyberscan.io bot: name: Cyberscan category: Security Checker url: https://www.cyberscan.io/ producer: name: DGC Verwaltungs GmbH url: https://dgc.org/ - user_agent: Mozilla/5.0 (l9scan/2.0.xxxxxxxxxxxxxxxxxxxxxxxx; +https://leakix.net) bot: name: LeakIX category: Security Checker url: https://leakix.net/ producer: name: BaDaaS SRL url: https://leakix.net/ - user_agent: Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html) bot: name: vuhuvBot category: Search bot url: https://vuhuv.com/bot.html - user_agent: Mozilla/5.0 (compatible; vuhuvRBT/2.0; +https://vuhuv.com/rbt.html) bot: name: vuhuvBot category: Search bot url: https://vuhuv.com/bot.html - user_agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) Image size by Siteimprove.com bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) Probe by Siteimprove.com bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: Mozilla/5.0 (Linux; Android 7.0; SM-G892A Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Mobile Safari/537.36 https://deepcrawl.com/bot bot: name: Lumar category: Crawler url: https://www.lumar.io/spdr/ producer: name: Lumar url: https://www.lumar.io/ - user_agent: RepoLookoutBot/v1.1.0-209-g2b273e8 (abuse reports to abuse@repo-lookout.org) bot: name: Repo Lookout category: Security Checker url: https://www.repo-lookout.org/ producer: name: Crissy Field GmbH url: https://www.crissyfield.de/ - user_agent: Mozilla/5.0 researchscan.comsys.rwth-aachen.de bot: name: Research Scan category: Crawler url: http://researchscan.comsys.rwth-aachen.de/ producer: name: RWTH Aachen University url: https://www.comsys.rwth-aachen.de/ - user_agent: newspaper/0.2.8 bot: name: Scraping Robot category: Crawler url: https://scrapingrobot.com/ producer: name: Sprious LLC url: https://sprious.com/ - user_agent: Google bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) bot: name: GPTBot category: AI Data Scraper url: https://platform.openai.com/docs/bots producer: name: OpenAI OpCo, LLC url: https://openai.com/ - user_agent: Mozilla/5.0 (compatible; Google-InspectionTool/1.0;) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: GoogleOther bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Ant.com beta/1.0 (Crawling antmarks; https://www.ant.com; admin@ant.com) bot: name: Ant category: Crawler url: https://www.ant.com/ producer: name: Ant.com Ltd. url: https://www.ant.com/ - user_agent: Magellan bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; WebwikiBot/2.1; +https://www.webwiki.co.uk) bot: name: Webwiki category: Crawler url: https://www.webwiki.com/ producer: name: webwiki GmbH url: https://www.webwiki.com/ - user_agent: phpMyAdmin bot: name: phpMyAdmin category: Service Agent url: https://www.phpmyadmin.net/ - user_agent: phpMyAdmin/reporting-script bot: name: phpMyAdmin category: Service Agent url: https://www.phpmyadmin.net/ - user_agent: Trackable/0.1 NNjCeA bot: name: Chartable category: Site Monitor url: https://help.chartable.com/article/34-what-is-the-trackable-analytics-prefix producer: name: Chartable url: https://chartable.com - user_agent: Matomo/4.15.1 bot: name: Matomo category: Service Agent url: https://github.com/matomo-org/matomo producer: name: InnoCraft Ltd url: https://matomo.org/ - user_agent: CustomUserAgent/1.0 bot: name: Generic Bot - user_agent: Prometheus/2.40.5 bot: name: Prometheus category: Service Agent url: https://github.com/prometheus/prometheus producer: name: The Linux Foundation url: https://www.cncf.io/ - user_agent: firefox bot: name: Generic Bot - user_agent: Chrome bot: name: Generic Bot - user_agent: ArchiveTeam ArchiveBot/20220523.4a672db (wpull 2.0.3) and not Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36 bot: name: ArchiveBot category: Crawler url: https://wiki.archiveteam.org/index.php?title=ArchiveBot producer: name: ArchiveTeam url: https://wiki.archiveteam.org/ - user_agent: MADBbot/0.1 (Gathering webpages for data analytics; https://madb.zapto.org/bot.html; ma-db-crawl@googlegroups.com) bot: name: MADBbot category: Crawler url: https://madb.zapto.org/bot.html - user_agent: MeltwaterNews www.meltwater.com bot: name: MeltwaterNews category: Crawler producer: name: Meltwater Deutschland GmbH url: https://www.meltwater.com/ - user_agent: Owler@ows.eu/1 bot: name: OWLer category: Crawler url: https://openwebsearch.eu/owler/ producer: name: Open Search Foundation e.V. url: https://openwebsearch.eu/ - user_agent: OWLer/0.1 (built with StormCrawler; https://ows.eu/owler; owl@ow-s.eu bot: name: OWLer category: Crawler url: https://openwebsearch.eu/owler/ producer: name: Open Search Foundation e.V. url: https://openwebsearch.eu/ - user_agent: Owler (ows.eu/owler) bot: name: OWLer category: Crawler url: https://openwebsearch.eu/owler/ producer: name: Open Search Foundation e.V. url: https://openwebsearch.eu/ - user_agent: Page Monitor (https://confluence.dev.bbc.co.uk/display/men/Page+Monitor) bot: name: BBC Page Monitor category: Site Monitor url: https://confluence.dev.bbc.co.uk/display/men/Page+Monitor producer: name: BBC url: https://www.bbc.com/ - user_agent: BBC-Forge-URL-Monitor-Twisted bot: name: BBC Forge URL Monitor category: Site Monitor url: https://www.bbc.com/ producer: name: BBC url: https://www.bbc.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +ClaudeBot@anthropic.com) bot: name: ClaudeBot category: AI Data Scraper url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler producer: name: Anthropic PBC url: https://www.anthropic.com/ - user_agent: TheSafeInternetSearch bot: name: Generic Bot - user_agent: TheSafexInternetSearcher bot: name: Generic Bot - user_agent: TheSafeInternetxSearch bot: name: Generic Bot - user_agent: TheSafexInternetSearch bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com) bot: name: ImageSift category: AI Data Scraper url: https://imagesift.com/ producer: name: Castle Global, Inc. url: https://thehive.ai/ - user_agent: TactiScout/recruit (+http://find-it.world/TempCrawl/Crawltheque.php) bot: name: TactiScout category: Crawler url: https://find-it.world/TempCrawl/Crawltheque.php producer: name: Tactikast - user_agent: Brightbot 1.0 bot: name: BrightBot category: Crawler url: https://www.brightbot.app/ producer: name: Bright Interactive Ltd url: https://www.builtbybright.com/ - user_agent: DaspeedBot/1.0 (+https://daspeed.io/bot-info) bot: name: DaspeedBot category: Crawler url: https://daspeed.io/ producer: name: DAWAP SARL url: https://dawap.fr/ - user_agent: AntBot bot: name: Ant category: Crawler url: https://www.ant.com/ producer: name: Ant.com Ltd. url: https://www.ant.com/ - user_agent: AntBot/1.0 (http://www.ant.com) bot: name: Ant category: Crawler url: https://www.ant.com/ producer: name: Ant.com Ltd. url: https://www.ant.com/ - user_agent: Mozilla/5.0 (compatible; StractBot/0.1; open source search engine; +https://stract.com/webmasters) bot: name: Stract category: Crawler url: https://stract.com/webmasters producer: name: Stract url: https://github.com/StractOrg/stract/ - user_agent: Mozilla/5.0 (compatible; GeedoBot; +http://www.geedo.com/bot.html) bot: name: GeedoBot category: Crawler url: https://geedo.com/bot/ - user_agent: Mozilla/5.0 (compatible; BackupLand/1.0; https://go.backupland.com/; Domain check for viruses;) bot: name: BackupLand category: Crawler url: https://go.backupland.com/ producer: name: ООО «КВАРТА» url: https://go.backupland.com/ - user_agent: Mozilla/5.0 (compatible; Konturbot/1.2; +http://kontur.ru; cargo@kontur.ru) bot: name: Konturbot category: Crawler url: https://kontur.ru/ producer: name: АО «ПФ «СКБ Контур» url: https://kontur.ru/ - user_agent: Mozilla/5.0 (keys-so-bot) bot: name: Keys.so category: Crawler url: https://www.keys.so/ producer: name: ООО «МОДЕСКО» url: https://www.modesco.ru/ - user_agent: Mozilla/5.0 (compatible; LetsearchBot/1.0; +https://letsearch.ru/bots) bot: name: LetSearch category: Crawler url: https://letsearch.ru/bots - user_agent: Mozilla/5.0 (compatible; Example3/1.0; +https://www.example3.com/domain/example.com) bot: name: Example3 category: Crawler url: https://www.example3.com/ - user_agent: P3P Validator bot: name: W3C P3P Validator category: Validator url: https://www.w3.org/P3P/validator.html producer: name: W3C url: https://www.w3.org - user_agent: kirkland-signature bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; StatOnlineRuBot/1.0) bot: name: StatOnline.ru category: Crawler url: https://statonline.ru/ producer: name: ООО «Регистратор доменных имен РЕГ.РУ» url: https://statonline.ru/ - user_agent: Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268 bot: name: YandexRenderResourcesBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots) bot: name: YandexAccessibilityBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexAdNet/1.0; +http://yandex.com/bots) bot: name: YandexAdNet category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots) bot: name: YandexCalendar category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots bot: name: YandexDirectDyn category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexForDomain/1.0; +http://yandex.com/bots) bot: name: YandexForDomain category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots) bot: name: YandexMobileBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexMarket/2.0; +http://yandex.com/bots) bot: name: YandexMarket category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots) bot: name: YandexMedia category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots) bot: name: YandexPagechecker category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots) bot: name: YandexSitelinks category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexSpravBot/1.0; +http://yandex.com/bots) bot: name: YandexSpravBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots) bot: name: YandexVertis category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots) bot: name: YandexVideo category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots) bot: name: YandexVideoParser category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots) bot: name: YandexWebmaster category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots) bot: name: YandexScreenshotBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Google-Safety bot: name: Google-Safety category: Crawler url: https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-Extended bot: name: Google-Extended category: AI Data Scraper url: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Spawning-AI bot: name: Spawning AI category: Crawler url: https://spawning.ai/ producer: name: Spawning, Inc url: https://spawning.ai/ - user_agent: domain research project (+https://trentwil.es/domains.html) bot: name: Domain Research Project category: Crawler url: https://trentwil.es/domains.html producer: name: Trent Wiles url: https://trentwil.es/ - user_agent: Mozilla/5.0 (compatible; VKRobot/1.0) bot: name: VK Robot category: Crawler url: https://dev.vk.com/en/ producer: name: VK url: https://vk.com/ - user_agent: Mozilla/5.0 (compatible; Odin; https://docs.getodin.com/) bot: name: Odin category: Security Checker url: https://docs.getodin.com/ producer: name: Cyble Inc. url: https://cyble.com/ - user_agent: YouBot (+http://www.you.com) bot: name: YouBot category: AI Search Crawler url: https://about.you.com/youbot/ producer: name: SuSea, Inc. url: https://you.com/ - user_agent: Mozilla/5.0 (compatible; YouBot/1.0; +https://about.you.com/youbot/) bot: name: YouBot category: AI Search Crawler url: https://about.you.com/youbot/ producer: name: SuSea, Inc. url: https://you.com/ - user_agent: SiteScoreBot v20210315 - https://sitescore.ai bot: name: SiteScore category: Crawler url: https://sitescore.ai/ - user_agent: Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html) bot: name: Awario category: Search bot url: https://awario.com/bots.html producer: name: TechFusion Ltd. url: https://www.techfusion.com.cy/ - user_agent: MBCrawler/1.0 (https://monitorbacklinks.com/robot) bot: name: Monitor Backlinks category: Crawler url: https://www.seoptimer.com/monitor-backlinks/ producer: name: SEOptimer url: https://www.seoptimer.com/ - user_agent: mariadb-mysql-kbs-bot (+https://github.com/williamdes/mariadb-mysql-kbs; williamdes+mariadb-mysql-kbs@wdes.fr) bot: name: MariaDB/MySQL Knowledge Base category: Crawler url: https://github.com/williamdes/mariadb-mysql-kbs producer: name: WDES SAS url: https://wdes.fr/en/ - user_agent: GitHubCopilotChat/0.8.0 bot: name: GitHubCopilotChat category: AI Assistant url: https://github.com/microsoft/vscode-copilot-chat - user_agent: pdrl.fm Analyzer / 1.0.0 bot: name: Podroll Analyzer category: Crawler url: https://podroll.fm - user_agent: Mozilla/5.0 (compatible; PodUptime/1.0; +https://poduptime.com) bot: name: PodUptime category: Site Monitor url: https://poduptime.com - user_agent: anthropic-ai bot: name: Anthropic AI category: AI Data Scraper url: https://www.anthropic.com/ producer: name: Anthropic, PBC url: https://www.anthropic.com/ - user_agent: bingadsbot bot: name: BingBot category: Search bot url: http://search.msn.com/msnbot.htmn producer: name: Microsoft Corporation url: http://www.microsoft.com - user_agent: default_useragent bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; NetpeakCheckerBot/3.7; +https://netpeaksoftware.com/checker) bot: name: Netpeak Checker category: Crawler url: https://netpeaksoftware.com/checker producer: name: Netpeak LTD url: https://netpeaksoftware.com/ - user_agent: mozilla/5.0 (compatible; SandobaCrawler/1.0; +https://www.sandoba.com/en/crawler/) bot: name: Sandoba//Crawler category: Crawler url: https://www.sandoba.com/en/crawler/ producer: name: SANDOBA//EBUSINESS SOLUTIONS url: https://www.sandoba.com/ - user_agent: SirdataBot (+https://semantic-api.docs.sirdata.net/contextual-api/contextual-api/introduction) bot: name: Sirdata category: Crawler url: https://semantic-api.docs.sirdata.net/contextual-api/contextual-api/introduction producer: name: Sirdata SAS url: https://www.sirdata.com/ - user_agent: CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html) bot: name: CheckMark Network category: Crawler url: https://www.checkmarknetwork.com/spider.html/ producer: name: Exipert, Inc. url: https://www.checkmarknetwork.com/ - user_agent: Mozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/facebookbot/) bot: name: FacebookBot category: Crawler url: https://developers.facebook.com/docs/sharing/bot producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: cohere-ai bot: name: Cohere AI category: AI Data Scraper url: https://cohere.com/ producer: name: Cohere, Inc. url: https://cohere.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) bot: name: PerplexityBot category: AI Search Crawler url: https://docs.perplexity.ai/guides/bots producer: name: Perplexity AI, Inc. url: https://www.perplexity.ai/ - user_agent: Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content) bot: name: The Trade Desk Content category: Crawler url: https://www.thetradedesk.com/us/ttd-content producer: name: The Trade Desk, Inc. url: https://www.thetradedesk.com/ - user_agent: montastic-monitor http://www.montastic.com bot: name: Montastic Monitor category: Site Monitor url: https://www.montastic.com/ producer: name: Metadot, Corp. url: https://www.metadot.com/ - user_agent: Ruby, Twurly v1.1 (https://twurly.org) bot: name: Twurly category: Crawler url: https://twurly.org/ - user_agent: Mozilla/5.0 (Mixnode) AppleWebKit/537.36 (KHTML, like Gecko) bot: name: Mixnode category: Crawler url: https://www.mixnode.com/ producer: name: Mixnode Technologies, Inc. url: https://www.mixnode.com/ - user_agent: MixnodeCache/1.8(+https://cache.mixnode.com/) bot: name: Mixnode category: Crawler url: https://www.mixnode.com/ producer: name: Mixnode Technologies, Inc. url: https://www.mixnode.com/ - user_agent: Mixnode/3.0.1 (+https://www.mixnode.com/; Speed analysis; no data is stored) bot: name: Mixnode category: Crawler url: https://www.mixnode.com/ producer: name: Mixnode Technologies, Inc. url: https://www.mixnode.com/ - user_agent: CSSCheck/1.2.2 bot: name: CSSCheck category: Validator - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.6167.85 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google-PageRenderer Google (+https://developers.google.com/+/web/snippet/) bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MicrosoftPreview/2.0; +https://aka.ms/MicrosoftPreview) Chrome/100.0.4896.127 Safari/537.36 bot: name: Microsoft Preview category: Service Agent url: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 producer: name: Microsoft Corporation url: https://www.microsoft.com/ - user_agent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)' bot: name: VirusTotal Cloud category: Crawler url: https://www.virustotal.com/ producer: name: Chronicle Security Ireland Limited url: https://chronicle.security/ - user_agent: Mozilla/5.0 (Android 10; Mobile; rv:122.0) Gecko/122.0 Firefox/122.0 TinEye/1.0 (via http://www.tineye.com/) bot: name: TinEye category: Crawler url: https://tineye.com/ producer: name: Idée, Inc. url: https://tineye.com/ - user_agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0 AppEngine-Google; (+http://code.google.com/appengine; appid: e~arsnova-filter-system)' bot: name: ARSNova Filter System category: Crawler url: https://particify.de/en/ producer: name: 'Particify Gerhardt & Weingarten OHG' url: https://particify.de/en/ - user_agent: Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com) bot: name: Seekport category: Crawler url: https://bot.seekport.com/ producer: name: SISTRIX GmbH url: https://www.sistrix.de/ - user_agent: Mozilla/5.0 (compatible; botify; http://botify.com) bot: name: Botify category: Crawler url: https://www.botify.com/ producer: name: BOTIFY SAS url: https://www.botify.com/ - user_agent: Mozilla/5.0 (compatible; SnapchatAds/1.0; +https://businesshelp.snapchat.com/s/article/adsbot-crawler) bot: name: Snapchat Ads category: Crawler url: https://businesshelp.snapchat.com/s/article/adsbot-crawler?language=en_US producer: name: Snapchat Inc. url: https://www.snapchat.com/ - user_agent: Mozilla/5.0 (compatible; adscanner/)/1.0 (http://seocompany.store; spider@seocompany.store) bot: name: Adscanner category: Crawler url: https://www.alleyesonscreens.com/ producer: name: AdScanner d.o.o url: https://www.alleyesonscreens.com/ - user_agent: Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com) bot: name: WebCEO category: Crawler url: https://www.webceo.com/ producer: name: WebCEO, LLC url: https://www.webceo.com/ - user_agent: NetTrack Anonymous Web Statistics https://nettrack.info/support.php bot: name: NetTrack category: Crawler url: https://web.archive.org/web/20160607151934/https://nettrack.info/ - user_agent: Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/) bot: name: htmlyse category: Crawler url: https://www.htmlyse.com/ producer: name: Vistex LTD url: https://www.htmlyse.com/ - user_agent: keycdn-tools:https://www.example.com bot: name: KeyCDN Tools category: Service Agent url: https://tools.keycdn.com/geo - user_agent: Google-Transparency-Report bot: name: Google Transparency Report category: Site Monitor url: https://transparencyreport.google.com/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Xenu Link Sleuth/1.3.8 bot: name: Generic Bot - user_agent: Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling) bot: name: Arquivo.pt category: Crawler url: https://sobre.arquivo.pt/en/help/crawling-and-archiving-web-content/ producer: name: FCT|FCCN url: https://www.fct.pt/ - user_agent: Arquivo-web-crawler (compatible; brozzler/1.5 +https://arquivo.pt/faq-crawling) bot: name: Arquivo.pt category: Crawler url: https://sobre.arquivo.pt/en/help/crawling-and-archiving-web-content/ producer: name: FCT|FCCN url: https://www.fct.pt/ - user_agent: Arquivo-web-crawler (compatible; browsertrix/0.8 +https://arquivo.pt/faq-crawling) bot: name: Arquivo.pt category: Crawler url: https://sobre.arquivo.pt/en/help/crawling-and-archiving-web-content/ producer: name: FCT|FCCN url: https://www.fct.pt/ - user_agent: WordPress/6.4.1; https://www.isitwp.com bot: name: IsItWP category: Crawler url: https://www.isitwp.com/ producer: name: WPBeginner, LLC url: https://www.wpbeginner.com/ - user_agent: ZmEu bot: name: Generic Bot - user_agent: Zeus bot: name: Generic Bot - user_agent: Mozilla/5.0 (WhatsMyIP.org GeoIP_Lookups) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org HTTP_Compression_Test) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org HTTP_Headers) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org PageRank_WebStats_Tool) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org MAC_Address_Lookup_Tool) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org URL_Shortener_Preview_Tool) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: Mozilla/5.0 (WhatsMyIP.org Random_Website_Loader) http://whatsmyip.org/ua bot: name: WhatsMyIP.org category: Service Agent url: https://www.whatsmyip.org/ua/ - user_agent: keycdn-tools/perf bot: name: KeyCDN Tools category: Service Agent url: https://tools.keycdn.com/ producer: name: proinity LLC url: https://www.keycdn.com/ - user_agent: keycdn-tools/br bot: name: KeyCDN Tools category: Service Agent url: https://tools.keycdn.com/ producer: name: proinity LLC url: https://www.keycdn.com/ - user_agent: keycdn-tools/h2 bot: name: KeyCDN Tools category: Service Agent url: https://tools.keycdn.com/ producer: name: proinity LLC url: https://www.keycdn.com/ - user_agent: Mozilla/5.0 (compatible; AmazonAdBot/1.0; +https://adbot.amazon.com) bot: name: Amazon AdBot category: Crawler url: https://adbot.amazon.com/ producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/) bot: name: Senuto category: Crawler url: https://www.senuto.com/ producer: name: Senuto Sp. z o.o. url: https://www.senuto.com/ - user_agent: Automattic Analytics Crawler/0.2; http://wordpress.com/crawler/ bot: name: Automattic Analytics category: Crawler url: https://wordpress.com/crawler/ producer: name: Wordpress.org url: https://wordpress.org/ - user_agent: IDG/EU (http://spaziodati.eu/) bot: name: IDG category: Crawler url: https://www.spaziodati.eu/ producer: name: SpazioDati S.r.l. url: https://www.spaziodati.eu/ - user_agent: GozleBot; http://gozle.com.tm bot: name: Gozle category: Crawler url: https://gozle.com.tm/en/blog/post/1 producer: name: Doly Horjun HJ url: https://gozle.com.tm/ - user_agent: Quantcastbot/2.0 (+http://www.quantcast.com/bot) bot: name: Quantcast category: Crawler url: https://www.quantcast.com/bot/ producer: name: Quantcast Corp. url: https://www.quantcast.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/102.0.5005.182 Safari/537.36 FontRadar bot: name: FontRadar category: Crawler url: https://www.fontradar.com/ producer: name: EMDASH SAS url: https://www.fontradar.com/ - user_agent: survey-security-dot-txt/0.1 bot: name: Generic Bot - user_agent: WebAuthn Adoption Study (Contact mb364@hdm-stuttgart.de) bot: name: Generic Bot - user_agent: ViberUrlDownloader bot: name: Viber Url Downloader category: Service Agent url: https://www.viber.com/ producer: name: Viber Media S.à r.l. url: https://www.viber.com/ - user_agent: Zeno bot: name: Zeno category: Crawler url: https://github.com/internetarchive/Zeno producer: name: The Internet Archive url: https://archive.org/ - user_agent: Barracuda Sentinel (EE) bot: name: Barracuda Sentinel category: Service Agent url: https://sentinel.barracudanetworks.com/ producer: name: Barracuda Networks, Inc. url: https://www.barracudanetworks.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.5672.126 Safari/537.36 RuxitSynthetic/1.0 v104032285451 t1437235942278817309 ath5a4d3a2e altpub cvcv=2 smf=0 bot: name: RuxitSynthetic category: Site Monitor url: https://community.dynatrace.com/t5/Troubleshooting/Basic-Commands-for-Synthetic/ta-p/198164 producer: name: Dynatrace LLC url: https://www.dynatrace.com/ - user_agent: DynatraceSynthetic/1.179 bot: name: DynatraceSynthetic category: Site Monitor url: https://community.dynatrace.com/t5/Troubleshooting/Basic-Commands-for-Synthetic/ta-p/198164 producer: name: Dynatrace LLC url: https://www.dynatrace.com/ - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36 +https://sitebulb.com bot: name: Sitebulb category: Crawler url: https://sitebulb.com/ producer: name: Sitebulb Limited url: https://sitebulb.com/ - user_agent: Mozilla/5.0 (compatible; Monsidobot/2.2; +http://monsido.com/bot.html; info@monsido.com) bot: name: Monsidobot category: Crawler url: https://monsido.com/bot-html producer: name: Monsido LLC url: https://monsido.com/ - user_agent: AccompanyBot bot: name: AccompanyBot category: Crawler url: https://www.accompany.com/ producer: name: Accompani, Inc url: https://www.accompany.com/ - user_agent: TheWebInternetSearcher bot: name: Generic Bot - user_agent: Searcherweb bot: name: Generic Bot - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0 Safari/537.36 Ghost Inspector bot: name: Ghost Inspector category: Site Monitor url: 'https://docs.ghostinspector.com/faq/#how-do-i-detect-ghost-inspector-test-runner-traffic-on-my-site' producer: name: Ghost Inspector, Inc. url: https://www.ghostinspector.com/ - user_agent: Google-Area120-PrivacyPolicyFetcher bot: name: Google Area 120 Privacy Policy Fetcher category: Crawler url: https://area120.google.com/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots) bot: name: YandexUserproxy category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: 'Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: )' bot: name: Google Apps Script category: Service Agent url: https://www.google.com/script/start/ - user_agent: Google-Lens bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; SiteOne-Crawler/1.0.7; +https://crawler.siteone.io/bot/?r=xxxxxxxx) bot: name: SiteOne Crawler category: Crawler url: https://crawler.siteone.io/bot/ producer: name: SiteOne s.r.o. url: https://www.siteone.io/ - user_agent: Mozilla/5.0 (compatible; Detectify) +https://detectify.com/bot/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx bot: name: Detectify category: Security Checker url: https://support.detectify.com/support/solutions/articles/48001049001-how-to-allow-detectify-to-access-your-site producer: name: Detectify AB url: https://detectify.com/ - user_agent: Mozilla/5.0 (compatible; Detectify; stateless-http-tests; xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) bot: name: Detectify category: Security Checker url: https://support.detectify.com/support/solutions/articles/48001049001-how-to-allow-detectify-to-access-your-site producer: name: Detectify AB url: https://detectify.com/ - user_agent: Mozilla/5.0 (compatible; Detectify) bot: name: Detectify category: Security Checker url: https://support.detectify.com/support/solutions/articles/48001049001-how-to-allow-detectify-to-access-your-site producer: name: Detectify AB url: https://detectify.com/ - user_agent: DomCopBot (https://www.domcop.com/bot) bot: name: DomCop Bot category: Crawler url: https://www.domcop.com/bot producer: name: Axeman Technology Solutions LLP url: https://axemantech.com/ - user_agent: Mozilla/5.0 (compatible; Paqlebot/2.0; +http://www.paqle.dk/about/paqlebot) bot: name: Paqlebot category: Crawler url: https://www.paqle.dk/about/paqlebot producer: name: Paqle A/S url: https://www.paqle.dk/ - user_agent: Mozilla/4.0 (compatible; Wibybot; https://wiby.me/) bot: name: Wibybot category: Crawler url: https://www.wiby.me/ - user_agent: Synapse (bot; +https://github.com/matrix-org/synapse) bot: name: Synapse category: Crawler url: https://github.com/matrix-org/synapse - user_agent: Mozilla/5.0 (compatible; OSZKbot/3.3 (OSZKBOT,oszkbot) +http://mekosztaly.oszk.hu/mia) on behalf of the National Szechenyi Library bot: name: OSZKbot category: Crawler url: http://mekosztaly.oszk.hu/mia/ producer: name: National Szechenyi Library url: https://webarchivum.oszk.hu/ - user_agent: SplitSignalBot bot: name: SplitSignalBot category: Crawler url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: Mozilla/5.0 (compatible; SiteAuditBot/0.97; http://www.semrush.com/bot.html) bot: name: SiteAuditBot category: Crawler url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: Zoombot (SEOChecker 1.0 http://suite.seozoom.it/bot.html) bot: name: ZoomBot category: Crawler url: https://suite.seozoom.it/bot.html producer: name: SEO Cube S.r.l. url: https://www.seocube.it/ - user_agent: ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html) bot: name: ZoomBot category: Crawler url: https://suite.seozoom.it/bot.html producer: name: SEO Cube S.r.l. url: https://www.seocube.it/ - user_agent: Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/) bot: name: RavenCrawler category: Crawler url: https://raventools.com/site-auditor/ producer: name: TapClicks, Inc. url: https://www.tapclicks.com/ - user_agent: KadoBot (https://www.kadolijst.nl/bot) bot: name: KadoBot category: Crawler url: https://www.kadolijst.nl/bot producer: name: Kadolijst url: https://www.kadolijst.nl/ - user_agent: Investis-Site24x7 bot: name: Site24x7 Website Monitoring category: Site Monitor url: https://www.site24x7.com/site24x7-faq.html producer: name: Site24x7 url: https://www.site24x7.com - user_agent: Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) HLB/1.247 bot: name: Site24x7 Defacement Monitor category: Site Monitor url: https://support.site24x7.com/portal/en/kb/articles/default-user-agent-used-in-website-defacement-monitor producer: name: Site24x7 url: https://www.site24x7.com/ - user_agent: Mozilla/5.0 (compatible; Dubbotbot/0.2; +http://dubbot.com) bot: name: Dubbotbot category: Crawler url: https://help.dubbot.com/en/articles/6746594-example-custom-user-agent producer: name: DubBot url: https://dubbot.com/ - user_agent: Mozilla/5.0 (compatible; Swiftbot/1.0; UID/; +http://swiftype.com/swiftbot) bot: name: Swiftbot category: Crawler url: https://swiftype.com/swiftbot producer: name: Elasticsearch, B.V. url: https://www.elastic.co/ - user_agent: EyeMonIT_bot_version_0.1_(https://www.eyemon.it/) bot: name: EyeMonit category: Site Monitor url: https://eyemonit.com/ producer: name: EyeMonit url: https://eyemonit.com/ - user_agent: ThousandEyes-Dragonfly-x1 bot: name: ThousandEyes category: Site Monitor url: https://www.thousandeyes.com/ producer: name: Cisco Systems, Inc. url: https://www.cisco.com/ - user_agent: OmtrBot/1.0 bot: name: OmtrBot category: Site Monitor - user_agent: WebMon/2.0.0 bot: name: WebMon category: Site Monitor - user_agent: AdsTxtCrawlerTP/1.2 bot: name: AdsTxtCrawlerTP category: Crawler - user_agent: fragFINN.de HTTP-Collector bot: name: fragFINN category: Crawler url: https://www.fragfinn.de/ producer: name: fragFINN e.V. url: https://www.fragfinn.de/ - user_agent: Clickagy Intelligence Bot v2 bot: name: Clickagy category: Crawler url: https://www.clickagy.com/ producer: name: Clickagy, LLC url: https://www.clickagy.com/ - user_agent: kiwitcms-gitops/0.1 bot: name: Kiwi TCMS GitOps category: Service Agent url: https://kiwitcms.org producer: name: Open Technologies Bulgaria, Ltd. url: https://kiwitcms.org - user_agent: kiwitcms-gitops/1 bot: name: Kiwi TCMS GitOps category: Service Agent url: https://kiwitcms.org producer: name: Open Technologies Bulgaria, Ltd. url: https://kiwitcms.org - user_agent: Googlebot-News bot: name: Googlebot News category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.live}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.pro}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.online}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.site}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.fun}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: '${jndi:ldap://${hostName}.useragent.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.oast.me}' bot: name: Interactsh category: Security Checker url: https://github.com/projectdiscovery/interactsh producer: name: ProjectDiscovery, Inc. url: https://projectdiscovery.io/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36 webtru_crawler bot: name: webtru category: Crawler url: https://webtru.io/ producer: name: DataSign Inc. url: https://datasign.jp/ - user_agent: Mozilla / 5.0(Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko; compatible; URLSuMaBot / 1.0; +https://www.urlsuma.de/bot.aspx) Chrome / 70.0.3538.77 Safari / 537.36 bot: name: URLSuMaBot category: Crawler url: https://www.urlsuma.de/ - user_agent: Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 360JK yunjiankong 427691 bot: name: 360JK category: Site Monitor url: http://jk.cloud.360.cn/ producer: name: 360 Security Technology Inc. url: https://www.360.cn/ - user_agent: LinkChain bot: name: Generic Bot - user_agent: Morfeus Fucking Scanner bot: name: Generic Bot - user_agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:80.0) Gecko/20100101 Firefox/80.0 UCSBNetworkMeasurement/2023 (contact; stijn; at; ucsb.edu;) bot: name: UCSB Network Measurement category: Crawler url: https://www.it.ucsb.edu/ producer: name: University of California, Santa Barbara url: https://www.it.ucsb.edu/ - user_agent: Plesk screenshot bot https://support.plesk.com/hc/en-us/articles/10301006946066 bot: name: Plesk Screenshot Service category: Service Agent url: https://support.plesk.com/hc/en-us/articles/13302778306199-What-is-Plesk-Screenshot-Service producer: name: Plesk International GmbH url: https://www.plesk.com/ - user_agent: Y!J-ASR/1.0 crawler (https://support.yahoo-net.jp/PccSearch/s/article/H000007955) bot: name: Yahoo! Japan ASR category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Who.is Bot bot: name: Who.is Bot category: Crawler url: https://who.is/ - user_agent: Mozilla/5.0 (compatible; WireReaderBot/1.0; +https://wirereader.app) bot: name: WireReaderBot category: Feed Fetcher url: https://wirereader.app/ - user_agent: WireReaderBot/1.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) bot: name: WireReaderBot category: Feed Fetcher url: https://wirereader.app/ - user_agent: Mozilla/5.0 (compatible; +https://probely.com/sos) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36 ProbelySPDR/0.2.0 bot: name: Probely category: Security Checker url: https://probely.com/sos/ producer: name: 'Probely - Soluções de Cibersegurança, S.A.' url: https://probely.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Uptime/1.0 (https://uptime.com) bot: name: Uptimebot category: Site Monitor url: https://uptime.com/uptime-bot producer: name: Uptime url: https://uptime.com/ - user_agent: Mozilla/5.0 (Linux; Android 10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.114 Mobile Safari/537.36 Uptime/1.0 (https://uptime.com) bot: name: Uptimebot category: Site Monitor url: https://uptime.com/uptime-bot producer: name: Uptime url: https://uptime.com/ - user_agent: Mozilla/5.0 (compatible; Uptime/1.0; http://uptime.com) bot: name: Uptimebot category: Site Monitor url: https://uptime.com/uptime-bot producer: name: Uptime url: https://uptime.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5002.111 Safari/537.36 Uptimia/1.0 (https://www.uptimia.com) bot: name: Uptimia category: Site Monitor url: https://www.uptimia.com/ producer: name: JJ Online GmbH url: https://www.uptimia.com/ - user_agent: Mozilla/5.0 (compatible; Uptimia; www.uptimia.com) bot: name: Uptimia category: Site Monitor url: https://www.uptimia.com/ producer: name: JJ Online GmbH url: https://www.uptimia.com/ - user_agent: Mozilla/5.0 (compatible; 2GDPR/1.2; https://2gdpr.com) bot: name: 2GDPR category: Service Agent url: https://2gdpr.com/tos producer: name: 2GDPR url: https://2gdpr.com/ - user_agent: abuse.xmco.fr bot: name: Serenety category: Security Checker url: https://abuse.xmco.fr/ producer: name: XMCO, SASU url: https://www.xmco.fr/ - user_agent: CheckHost (https://check-host.net/) bot: name: CheckHost category: Site Monitor url: https://check-host.net/ producer: name: CheckHost url: https://check-host.net/ - user_agent: Mozilla/5.0 (compatible; LAC_IAHarvester/3.3.0; +https://library-archives.canada.ca/eng/services/government-canada/web-social-media-preservation-program/Pages/web-archive.aspx) bot: name: LAC IA Harvester category: Crawler url: https://library-archives.canada.ca/eng/services/government-canada/web-social-media-preservation-program/Pages/web-archive.aspx producer: name: Library and Archives Canada url: https://library-archives.canada.ca/ - user_agent: GoogleSites bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google-Sites-Thumbnails) Chrome/120.0.6099.71 Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-adstxt bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: KvshClient bot: name: Generic Bot - user_agent: Mozilla/5.0 infrawatch/0.1 bot: name: Generic Bot - user_agent: InsytfulBot/1.0; https://www.insytful.com/about-our-bot bot: name: InsytfulBot category: Crawler url: https://www.insytful.com/ producer: name: Zengenti Limited url: https://www.zengenti.com/ - user_agent: statista.com PublicationFinder-Crawler 2.0 bot: name: Statista category: Crawler url: https://www.statista.com/ producer: name: Statista, Inc. url: https://www.statista.com/ - user_agent: SubstackContentFetch/1.0; https://substack.com bot: name: Substack Content Fetch category: Crawler url: https://substack.com/ producer: name: Substack, Inc. url: https://substack.com/ - user_agent: ds9 2.000.ec2(+http://www.deepsearchnine.com/ds9.html) bot: name: Deep SEARCH 9 category: Crawler url: https://www.copyright.com/blog/ccc-expands-corporate-solutions-offering-with-new-technology/ producer: name: Copyright Clearance Center, Inc. url: https://www.copyright.com/ - user_agent: ds9 2.000.ec2 bot: name: Deep SEARCH 9 category: Crawler url: https://www.copyright.com/blog/ccc-expands-corporate-solutions-offering-with-new-technology/ producer: name: Copyright Clearance Center, Inc. url: https://www.copyright.com/ - user_agent: LiveJournal.com (webmaster@livejournal.com; for https://www.livejournal.com/users/example/; 1 readers) bot: name: LiveJournal url: https://www.livejournal.com/ category: Feed Fetcher producer: name: ООО "СИМ" url: https://www.livejournal.com/ - user_agent: bitdiscovery-suggestions bot: name: Tenable.asm category: Security Checker url: https://bitdiscovery.com/ producer: name: Tenable, Inc. url: https://www.tenable.com/ - user_agent: Castopod/1.0 bot: name: Castopod category: Crawler url: https://www.castopod.org/ - user_agent: ivre-masscan/1.3 https://github.com/robertdavidgraham/ bot: name: masscan url: https://github.com/robertdavidgraham/masscan category: Crawler producer: name: Robert Graham url: https://github.com/robertdavidgraham - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36 bot: name: GeedoProductSearch category: Crawler url: https://geedo.com/product-search/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/121.0.6167.57 Safari/537.36 Elastic/Synthetics bot: name: Elastic Synthetics category: Site Monitor url: https://github.com/elastic/synthetics producer: name: Elasticsearch B.V. url: https://www.elastic.co/ - user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0 Time/1710630666672 bot: name: Generic Bot - user_agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN;; rv:1.9.0.9) Gecko/2009042410 Firefox/3.0.9 Wyzo/3.0.3 Time/1715201908281 bot: name: Generic Bot - user_agent: WDG_Validator/1.6.2 bot: name: WDG HTML Validator category: Validator url: http://www.htmlhelp.com/tools/validator/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.0; +http://builtwith.com/biup) Chrome/74.0.3729.131 Safari/537.36 bot: name: BuiltWith category: Crawler url: https://builtwith.com/biup producer: name: BuiltWith Pty Ltd url: https://builtwith.com/ - user_agent: Google Chrome Mozilla/5.0 (Windows NT 10.0; Win64; x64; CONTACT; scan@aegis.network) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 bot: name: Aegis category: Crawler url: https://web.archive.org/web/20180910002802/http://www.aegis.network/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 (compatible; CrawlyProjectCrawler/0.1.3; crawlyproject@digitaldragon.dev +https://crawlyproject.digitaldragon.dev/) bot: name: Crawly Project category: Crawler url: https://web.archive.org/web/20240326141952/https://crawlyproject.digitaldragon.dev/ - user_agent: BDFetch bot: name: BDFetch category: Crawler url: https://web.archive.org/web/20130821043949/http://www.branddimensions.com/ - user_agent: r00ts3c-owned-you bot: name: Generic Bot - user_agent: nvdorz bot: name: Generic Bot - user_agent: Root Slut bot: name: Generic Bot - user_agent: Abcd bot: name: Generic Bot - user_agent: NiggaBalls bot: name: Generic Bot - user_agent: Dark bot: name: Generic Bot - user_agent: BotPoke bot: name: Generic Bot - user_agent: url bot: name: Generic Bot - user_agent: facebookscraper/1.0( http://www.facebook.com/sharescraper_help.php) bot: name: Facebook Crawler category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: PunkMap (https://github.com/openeasm/punkmap) bot: name: Punk Map category: Security Checker url: https://github.com/openeasm/punkmap - user_agent: "'Mozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/networks/ip-networks/deepfield/genome/)'" bot: name: Deepfield Genome category: Crawler url: https://www.nokia.com/networks/ip-networks/deepfield/genome/ producer: name: Nokia Corporation url: https://www.nokia.com/ - user_agent: Gaisbot/3.0 (robot@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php) bot: name: Gaisbot category: Crawler url: https://web.archive.org/web/20090604121511/https://gais.cs.ccu.edu.tw/robot.php - user_agent: FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler) bot: name: AlltheWeb category: Crawler url: https://web.archive.org/web/20041020050801/http://www.alltheweb.com/help/webmaster/crawler - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 (scanner.ducks.party) bot: name: ducks.party category: Security Checker url: https://ducks.party/ - user_agent: Mozilla/5.0 (compatible; Cloudflare-Smart-Transit/1.0; +https://www.cloudflare.com/ bot: name: Cloudflare Smart Transit category: Site Monitor url: https://developers.cloudflare.com/fundamentals/reference/cloudflare-site-crawling/ producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Mozilla/5.0 (Linux; Android 11; Moto G Power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36 CloudflareObservatory/1.0 bot: name: Cloudflare Observatory category: Site Monitor url: https://developers.cloudflare.com/speed/speed-test/run-speed-test producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 CloudflareObservatory/1.0 bot: name: Cloudflare Observatory category: Site Monitor url: https://developers.cloudflare.com/speed/speed-test/run-speed-test producer: name: CloudFlare url: https://www.cloudflare.com/ - user_agent: Mozilla/4.0 (compatible; DepSpid/5.07; +http://about.depspid.net) bot: name: DepSpid category: Crawler url: https://web.archive.org/web/20080321224033/http://about.depspid.net/ - user_agent: GlobalWebSearch bot: name: Generic Bot - user_agent: acme.sectigo.com/v2/DV bot: name: Sectigo DCV category: Service Agent url: https://sectigo.com/ producer: name: Sectigo Limited url: https://sectigo.com/ - user_agent: acme.sectigo.com/v2/OV bot: name: Sectigo DCV category: Service Agent url: https://sectigo.com/ producer: name: Sectigo Limited url: https://sectigo.com/ - user_agent: acme.sectigo.com/v2/EV bot: name: Sectigo DCV category: Service Agent url: https://sectigo.com/ producer: name: Sectigo Limited url: https://sectigo.com/ - user_agent: Mozilla/5.0 (compatible; Website-info.net-Robot; https://website-info.net/robot) bot: name: Website-info category: Crawler url: https://website-info.net/robot producer: name: Meins und Vogel GmbH url: https://muv.com/ - user_agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 (compatible; PagePeeker/3.0; +https://pagepeeker.com/robots/) bot: name: PagePeeker category: Crawler url: https://pagepeeker.com/robots/ producer: name: PAGEPEEKER SRL url: https://pagepeeker.com/ - user_agent: Mozilla/5.0 (compatible; SemrushBot-SWA/0.1; +http://www.semrush.com/bot.html) bot: name: SemrushBot category: Crawler url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: Mozilla/5.0 (compatible; RedekenBot/0.1; +https://www.redeken.com/bot/) bot: name: RedekenBot category: Crawler url: https://www.redeken.com/en/help/bot.html producer: name: Redeken url: https://www.redeken.com/ - user_agent: semaltbot/0.1 (+http://semalt.net) bot: name: semaltbot category: Crawler url: https://semalt.net/ producer: name: Semalt LP url: https://semalt.net/ - user_agent: Mozilla/5.0 (compatible; MakeMerryBot/1.0; +https://makemerry.app/bots) bot: name: MakeMerryBot category: Crawler url: https://makemerry.app/bots - user_agent: Timpibot/0.9 (+http://www.timpi.io) bot: name: Timpibot category: Crawler url: https://timpi.io/ producer: name: Timpi Inc. url: https://timpi.io/ - user_agent: Mozilla/5.0 (compatible; Timpibot/0.8; +http://www.timpi.io) bot: name: Timpibot category: Crawler url: https://timpi.io/ producer: name: Timpi Inc. url: https://timpi.io/ - user_agent: 'Tublm.com/Bot/fubpdfdotcom/Bot/Bot -❤️- +https://tublm.com/game/2048_merge' bot: name: Generic Bot - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15 (compatible; Validbot; +https://www.validbot.com) bot: name: ValidBot category: Crawler url: https://www.validbot.com/ producer: name: Jake Olefsky LLC url: https://www.validbot.com/ - user_agent: NPBot bot: name: NameProtectBot category: Crawler url: https://www.cscglobal.com/cscglobal/home/ producer: name: NameProtect, Inc. url: https://www.cscglobal.com/ - user_agent: Mozilla/5.0 (compatible; CuriousCatgirl Research; +https://curiouscatgirl.cynthia.dev) bot: name: Generic Bot - user_agent: xx032_bo9vs83_2a bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160721-2308 +https://www.domaincodex.com) bot: name: Domain Codex category: Crawler url: https://www.domaincodex.com/ producer: name: Erie Data Systems, LLC url: https://www.eriedatasys.com/ - user_agent: Swisscows Favicons bot: name: Swisscows Favicons category: Crawler url: https://swisscows.com/ producer: name: Swisscows AG url: https://swisscows.com/ - user_agent: Mozilla/4.0 (compatible; fluid/0.0; +http://www.leak.info/bot.html) bot: name: leak.info category: Crawler url: http://www.leak.info/ - user_agent: workona-favicon-service/1.0.0 bot: name: Workona category: Crawler url: https://workona.com/ producer: name: Workona, Inc. url: https://workona.com/ - user_agent: Bloglines/3.1 (http://www.bloglines.com) bot: name: Bloglines category: Crawler url: https://web.archive.org/web/20140309033202/http://www.bloglines.com/ producer: name: Reply!, Inc. url: https://www.reply.com/ - user_agent: 'shadowforce.io - sslshed/0.1' bot: name: Generic Bot - user_agent: search.marginalia.nu bot: name: Marginalia category: Crawler url: https://www.marginalia.nu/marginalia-search/for-webmasters/ producer: name: Marginalia url: https://www.marginalia.nu/ - user_agent: Mozilla/5.0 (compatible;vu-server-health-scanner/1.0;https://130.37.198.75/index.html) bot: name: VU Server Health Scanner category: Security Checker url: https://130.37.198.75/index.html producer: name: VU Amsterdam url: https://vu.nl/en - user_agent: Searcherxweb bot: name: Generic Bot - user_agent: Mozilla/5.0 (platform; rv:geckoversion) Gecko/geckotrail Firefox/firefoxversion bot: name: Generic Bot - user_agent: Report Runner bot: name: Generic Bot - user_agent: Node.js bot: name: Generic Bot - user_agent: Mozilla/5.0 (X11; Windows x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Functionize bot: name: Functionize category: Crawler url: https://www.functionize.com/ producer: name: Functionize, Inc. url: https://www.functionize.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/W.X.Y.Z Safari/537.36 Prerender (+https://github.com/prerender/prerender) bot: name: Prerender category: Crawler url: https://docs.prerender.io/docs/33-overview-of-prerender-crawlers producer: name: saas.group Inc. url: https://saas.group/ - user_agent: Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 Prerender (+https://github.com/prerender/prerender) bot: name: Prerender category: Crawler url: https://docs.prerender.io/docs/33-overview-of-prerender-crawlers producer: name: saas.group Inc. url: https://saas.group/ - user_agent: Prerender (+https://github.com/prerender/prerender) bot: name: Prerender category: Crawler url: https://docs.prerender.io/docs/33-overview-of-prerender-crawlers producer: name: saas.group Inc. url: https://saas.group/ - user_agent: bl.uk_ldfc_bot/3.4.0-20220727 (+https://www.bl.uk/legal-deposit/web-archiving) bot: name: The British Library Legal Deposit Bot category: Crawler url: https://www.bl.uk/ producer: name: The British Library url: https://www.bl.uk/ - user_agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 bot: name: KStandBot category: Crawler url: https://url-classification.io/wiki/index.php?title=URL_server_crawler producer: name: Komodia Inc. url: https://www.komodia.com/ - user_agent: Wordupindexinfo1/0 bot: name: Generic Bot - user_agent: wordupsearchengine-1 bot: name: Generic Bot - user_agent: Wordup-1 bot: name: Generic Bot - user_agent: Mozilla/5.0 (X11; Linux x86_64) Miniature.io/6.0 AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.4 Chrome/69.0.3497.128 Safari/537.36 bot: name: Miniature.io category: Service Agent url: https://miniature.io/ producer: name: LCX Ventures Ltd url: https://www.lcxventures.com/ - user_agent: Mozilla/5.0 (Linux; Android 9; SM-G973F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36 Convertify bot: name: Convertify category: Service Agent url: https://www.convertify.app/ producer: name: Convertify url: https://www.convertify.app/ - user_agent: ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org) bot: name: Zotero Translation Server category: Service Agent url: https://github.com/wikimedia/mediawiki-services-zotero producer: name: The Wikimedia Foundation, Inc. url: https://www.wikimedia.org/ - user_agent: Mozilla/5.0 (compatible; MuckRack/1.0; +https://muckrack.com) bot: name: MuckRack category: Crawler url: https://muckrack.com/ producer: name: Muck Rack, LLC url: https://muckrack.com/ - user_agent: 'Mozilla/5.0 (compatible; um-IC/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' bot: name: uMBot category: Crawler url: https://www.ubermetrics-technologies.com/ producer: name: Ubermetrics Technologies GmbH url: https://www.ubermetrics-technologies.com/ - user_agent: 'Mozilla/5.0 (compatible; um-ANS/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1' bot: name: uMBot category: Crawler url: https://www.ubermetrics-technologies.com/ producer: name: Ubermetrics Technologies GmbH url: https://www.ubermetrics-technologies.com/ - user_agent: 'Mozilla/5.0 (compatible; um-FC/1.0; mailto: techinfo@ubermetrics-technologies.com)' bot: name: uMBot category: Crawler url: https://www.ubermetrics-technologies.com/ producer: name: Ubermetrics Technologies GmbH url: https://www.ubermetrics-technologies.com/ - user_agent: 'Mozilla/5.0 (compatible; um-CC/1.0; mailto: techinfo@ubermetrics-technologies.com)' bot: name: uMBot category: Crawler url: https://www.ubermetrics-technologies.com/ producer: name: Ubermetrics Technologies GmbH url: https://www.ubermetrics-technologies.com/ - user_agent: Mozilla/5.0 (compatible; CyberFind Crawler; +https://cyberfind.net/bot.html)/Nutch-1.20 bot: name: CyberFind Crawler category: Crawler url: https://www.cyberfind.net/bot.html producer: name: Find.tf url: https://find.tf/ - user_agent: Mozilla/5.0 (compatible; CyberFindCrawler; +https://cyberfind.net/bot.html)/1.0 (https://cyberfind.net/bot.html) bot: name: CyberFind Crawler category: Crawler url: https://www.cyberfind.net/bot.html producer: name: Find.tf url: https://find.tf/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/124.0.6367.207 Safari/537.36 WordPress.com mShots bot: name: WordPress.com mShots category: Service Agent url: https://wordpress.org/ producer: name: Wordpress.org url: https://wordpress.org/ - user_agent: wp.com feedbot/1.0 (+https://wp.com) bot: name: wp.com feedbot category: Feed Fetcher url: https://wordpress.com/ producer: name: Automattic, Inc. url: https://automattic.com/ - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/bot bot: name: Lumar category: Crawler url: https://www.lumar.io/spdr/ producer: name: Lumar url: https://www.lumar.io/ - user_agent: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) https://deepcrawl.com/testing bot: name: Lumar category: Crawler url: https://www.lumar.io/spdr/ producer: name: Lumar url: https://www.lumar.io/ - user_agent: Mozilla/7.0 (compatible; Golfe/1.1; +http://www.goo-olfe.ae/bot.html) bot: name: Golfe category: Crawler url: http://www.goo-olfe.ae/bot.html - user_agent: Mozilla/5.0 (compatible; SpiderLing; +https://nlp.fi.muni.cz/projects/biwec/) bot: name: SpiderLing category: Crawler url: https://nlp.fi.muni.cz/projects/biwec/ producer: name: Natural Language Processing Centre url: https://nlp.fi.muni.cz/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36 bot: name: Bravebot category: Search bot url: https://search.brave.com/help/brave-search-crawler producer: name: Brave Software, Inc. url: https://brave.com/ - user_agent: Mozilla/5.0 (compatible; 1001FirmsBot/1.0; +https://www.1001firms.com/1001firmsbot.php) bot: name: 1001FirmsBot category: Crawler url: https://www.1001firms.com/1001firmsbot.php - user_agent: Google-CloudVertexBot bot: name: Google-CloudVertexBot category: AI Data Scraper url: 'https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-cloudvertexbot' producer: name: Google Inc. url: https://www.google.com/ - user_agent: Valve/Steam HTTP Client 1.0 (SteamChatURLLookup) bot: name: Steam Chat URL Lookup category: Service Agent url: https://help.steampowered.com/en/faqs/view/595C-42F4-3B66-E02F producer: name: Valve Corporation url: https://www.valvesoftware.com/ - user_agent: Oh Dear Sitemap check https://ohdear.app bot: name: Oh Dear category: Site Monitor url: https://ohdear.app/docs/faq/what-is-the-oh-dear-crawler-doing-in-my-logs producer: name: Immutable, SNC url: https://ohdear.app/ - user_agent: Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; brokenLinks) bot: name: Oh Dear category: Site Monitor url: https://ohdear.app/docs/faq/what-is-the-oh-dear-crawler-doing-in-my-logs producer: name: Immutable, SNC url: https://ohdear.app/ - user_agent: Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checker; mixedContent) bot: name: Oh Dear category: Site Monitor url: https://ohdear.app/docs/faq/what-is-the-oh-dear-crawler-doing-in-my-logs producer: name: Immutable, SNC url: https://ohdear.app/ - user_agent: Mozilla/5.0 Keydrop bot: name: Generic Bot - user_agent: Mozilla/6.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Inspici (www.inspici.com) bot: name: Inspici category: Crawler url: https://www.inspici.com/ producer: name: Inspici, LLC url: https://www.inspici.com/ - user_agent: meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) bot: name: Meta-ExternalAgent category: AI Assistant url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: meta-externalagent/1.1 bot: name: Meta-ExternalAgent category: AI Assistant url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) bot: name: Meta-ExternalFetcher category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: meta-externalfetcher/1.1 bot: name: Meta-ExternalFetcher category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: peer39_crawler/1.0 bot: name: Peer39 category: Crawler url: https://www.peer39.com/crawler-notice producer: name: Peer39 Tech, LLC url: https://www.peer39.com/ - user_agent: Pandalytics/2.0 (https://domainsbot.com/pandalytics/) bot: name: Pandalytics category: Crawler url: https://www.domainsbot.com/business-intelligence/ producer: name: DomainsBot, Inc. url: https://www.domainsbot.com/ - user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 ( compatible; CloudServerMarketSpider/1.0; +http://cloudservermarket.com/spider.html ) bot: name: CloudServerMarketSpider category: Crawler url: https://web.archive.org/web/20151228225429/https://cloudservermarket.com/spider.html - user_agent: Mozilla/5.0 (compatible; GoogleDocs; jam; +http://docs.google.com) bot: name: Google Docs category: Service Agent url: https://docs.google.com/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; GoogleDocs; +http://docs.google.com; +Google-Document-Conversion) bot: name: Google-Document-Conversion category: Service Agent url: https://support.google.com/drive/answer/176692?hl=en producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com) bot: name: Google Sheets category: Service Agent url: https://workspace.google.com/products/sheets/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; +http://docs.google.com) bot: name: Google Slides category: Service Agent url: https://workspace.google.com/products/slides/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla 5.0 (compatible; Pigafetta/0.5; +http://visual-seo.com/Pigafetta-Bot) bot: name: Pigafetta category: Crawler url: https://visual-seo.com/Pigafetta-Bot producer: name: aStonish Studio Srl url: http://www.astonishstudio.com/ - user_agent: Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/ bot: name: Cotoyogi category: Crawler url: https://ds.rois.ac.jp/center8/crawler/ producer: name: Joint Support-Center for Data Science Research (ROIS-DS) url: https://ds.rois.ac.jp/ - user_agent: SuggestBot/1.0 bot: name: SuggestBot category: Crawler url: https://github.com/nettrom/suggestbot - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 (https://securitee.org/cms-experiment-fall2024/) bot: name: CMS Experiment category: Security Checker url: https://securitee.org/cms-experiment-fall2024/ - user_agent: SiteCheckerBotCrawler/1.0 (+http://sitechecker.pro) bot: name: SiteCheckerBotCrawler category: Crawler url: https://sitechecker.pro/ producer: name: Cyber Circus Limited url: https://sitechecker.pro/ - user_agent: SBIder/0.8-dev (SBIder; http://www.sitesell.com/sbider.html; http://support.sitesell.com/contact-support.html) bot: name: SBIder category: Crawler url: https://www.sitesell.com/sbider.html producer: name: SiteSell Inc. url: https://www.sitesell.com/ - user_agent: Mozilla/5.0 (compatible; BacklinksExtendedBot) bot: name: BacklinksExtendedBot category: Crawler url: https://www.semrush.com/bot/ producer: name: Semrush Inc. url: https://www.semrush.com/ - user_agent: LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) bot: name: LightspeedSystemsCrawler category: Crawler url: https://www.lightspeedsystems.com/ producer: name: Lightspeed Systems, Inc. url: https://www.lightspeedsystems.com/ - user_agent: Research JLU bot: name: Research JLU category: Crawler url: https://www.uni-giessen.de/en/research producer: name: Justus Liebig University Giessen url: https://www.uni-giessen.de/en - user_agent: hgfAlphaXCrawl/1.0 (+https://www.fim.uni-passau.de/data-science/forschung/open-search) bot: name: AlphaXCrawl category: Crawler url: https://www.fim.uni-passau.de/en/data-science/research/open-search producer: name: University of Passau url: https://www.uni-passau.de/en/ - user_agent: IDG/RU (http://spaziodati.eu/) bot: name: IDG category: Crawler url: https://www.spaziodati.eu/ producer: name: SpazioDati S.r.l. url: https://www.spaziodati.eu/ - user_agent: IDG/UK (http://spaziodati.eu/) bot: name: IDG category: Crawler url: https://www.spaziodati.eu/ producer: name: SpazioDati S.r.l. url: https://www.spaziodati.eu/ - user_agent: Chatwork LinkPreview v1 bot: name: Chatwork LinkPreview category: Service Agent url: https://go.chatwork.com/en/ producer: name: kubell Co., Ltd. url: https://www.kubell.com/en/ - user_agent: WPMU DEV Broken Link Checker Local Engine bot: name: WPMU DEV category: Crawler url: 'https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-user-agent' producer: name: Incsub, LLC. url: https://incsub.com/ - user_agent: WPMU DEV Broken Link Checker Spider bot: name: WPMU DEV category: Crawler url: 'https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-user-agent' producer: name: Incsub, LLC. url: https://incsub.com/ - user_agent: Mozilla/5.0 (compatible) bot: name: Generic Bot - user_agent: John Recon/1.0.0 bot: name: Generic Bot - user_agent: 'SPARK COMMIT: 08059e95dacafe0bf6e5782f8e2c8ec9cd8c5a17' bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; SnoopSecInspect/1.1; +https://snoopsec.us.to/) bot: name: SnoopSecInspect category: Security Checker url: https://web.archive.org/web/20241206193253/https://snoopsec.us.to/ - user_agent: Jesus Christ of Nazareth is LORD bot: name: Generic Bot - user_agent: masjesu bot: name: Generic Bot - user_agent: Komaru_The_Cat bot: name: Generic Bot - user_agent: Kowai/1.0 bot: name: Generic Bot - user_agent: Hakai/2.0 bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; ModatScanner/1.0; +https://modat.io/) bot: name: ModatScanner category: Security Checker url: https://www.modat.io/scanning producer: name: Modat B.V. url: https://www.modat.io/ - user_agent: https://researchcyber.net/ bot: name: researchcyber.net category: Security Checker url: https://web.archive.org/web/20241219082407/https://researchcyber.net/ - user_agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; CrystalSemanticsBot http://www.crystalsemantics.com/user-agent/) bot: name: CrystalSemanticsBot category: Crawler url: https://web.archive.org/web/20121230203310/http://www.crystalsemantics.com/user-agent/ producer: name: Crystal Semantics Ltd. url: https://web.archive.org/web/20121029062239/http://www.crystalsemantics.com/ - user_agent: LoliSec/2.0 bot: name: Generic Bot - user_agent: LMAO/2.0 bot: name: Generic Bot - user_agent: Vyhledavac sluzeb | hlavicka smerovani | najdu.s.holubem.eu bot: name: najdu.s.holubem.eu category: Crawler url: https://najdu.s.holubem.eu/ - user_agent: Vyhledavac sluzeb | hlavicka | najdu.s.holubem.eu bot: name: najdu.s.holubem.eu category: Crawler url: https://najdu.s.holubem.eu/ - user_agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.0 Safari/534.34 Siteimprove (Accessibility) bot: name: Siteimprove category: Search bot url: https://siteimprove.com/ producer: name: Siteimprove GmbH url: https://siteimprove.com/ - user_agent: VORTEX/1.2 ( http://marty.anstey.ca/robots/vortex/) bot: name: VORTEX category: Crawler url: https://marty.anstey.ca/robots/vortex - user_agent: xtate/2.12.4 bot: name: xtate category: Crawler url: https://github.com/babycoff/xtate - user_agent: Mozilla/5.0 xtate/2.12.4 bot: name: xtate category: Crawler url: https://github.com/babycoff/xtate - user_agent: FediList Agent/3 (https://fediverse.one/) bot: name: FediList category: Social Media Agent url: https://fedilist.com/ - user_agent: FediList Agent/3 (https://fedilist.com/) bot: name: FediList category: Social Media Agent url: https://fedilist.com/ - user_agent: Grafana/11.1.0 bot: name: Grafana category: Site Monitor url: https://github.com/grafana/grafana producer: name: Grafana Labs url: https://grafana.com/ - user_agent: github-camo (d76dea84) bot: name: Github Camo category: Crawler url: https://github.com/atmos/camo producer: name: Github url: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-anonymized-urls - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bluesky Cardyb/1.1; +mailto:support@bsky.app) Chrome/W.X.Y.Z Safari/537.36 bot: name: Bluesky category: Social Media Agent url: https://bsky.app producer: name: Bluesky Social PBC url: https://bsky.app - user_agent: Mozilla/5.0 (compatible; OpenGraph.io/1.1; +https://opengraph.io/;) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 bot: name: OpenGraph.io category: Crawler url: https://www.opengraph.io producer: name: OpenGraph.io url: https://www.opengraph.io - user_agent: Mozilla/5.0 OnlyScans bot: name: Generic Bot - user_agent: Mozilla/5.0; Keydrop.io/1.0(onlyscans.com/about); bot: name: Generic Bot - user_agent: azure-logic-apps/1.0 (workflow xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx; version xxxxxxxxxxxxxxxxxxxx) microsoft-flow/1.0 bot: name: Microsoft Power Automate category: Service Agent url: https://www.microsoft.com/en-us/power-platform/products/power-automate producer: name: Microsoft Corporation url: https://www.microsoft.com/ - user_agent: Simbiat Software bot: name: Simbiat Software category: Crawler url: https://www.simbiat.eu producer: name: Simbiat Software url: https://www.simbiat.eu - user_agent: Mozilla/5.0 (compatible; Thinkbot/0.5.8; +In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.) bot: name: Thinkbot category: Crawler - user_agent: meta-webindexer bot: name: Meta-WebIndexer category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: meta-externalads bot: name: Meta-ExternalAds category: Social Media Agent url: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers producer: name: Meta Platforms, Inc. url: https://www.meta.com/ - user_agent: Mozilla/5.0 (compatible; YandexAdditional/1.0; +http://yandex.com/bots) bot: name: YandexAdditionalBot category: AI Search Crawler url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexAdditionalBot/1.0; +http://yandex.com/bots) bot: name: YandexAdditionalBot category: AI Search Crawler url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexComBot/3.0; +http://ya.cc/bots) bot: name: YandexComBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (compatible; YandexDialogs/1.0; +http://yandex.com/bots) bot: name: YandexDialogs category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) bot: name: Perplexity-User category: AI Assistant url: https://docs.perplexity.ai/guides/bots producer: name: Perplexity AI, Inc. url: https://www.perplexity.ai/ - user_agent: Mozilla/5.0 (compatible; IbouBot/1.0; +bot@ibou.io; +https://ibou.io/iboubot.html) bot: name: IbouBot category: AI Search Crawler url: https://ibou.io/iboubot.html - user_agent: Mozilla/5.0 (compatible; AddSearchBot/1.0; +http://www.addsearch.com/bot; info@addsearch.com) bot: name: AddSearchBot category: Crawler url: https://www.addsearch.com/docs/indexing/whitelisting-addsearch-bot/ producer: name: AddSearch Oy url: https://www.addsearch.com/ - user_agent: TerraCotta https://github.com/CeramicTeam/CeramicTerracotta bot: name: TerraCotta category: AI Data Scraper url: https://github.com/CeramicTeam/CeramicTerracotta producer: name: Ceramic, Inc. url: https://ceramic.ai/ - user_agent: Mozilla/5.0 (compatible; KeybaseBot; +https://keybase.io) bot: name: KeybaseBot category: Social Media Agent url: http://keybase.io producer: name: 'Zoom Video Communications, Inc' url: 'https://www.zoom.com/' - user_agent: Hello from Palo Alto Networks, find out more about our scans in https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity bot: name: Cortex Xpanse category: Security Checker url: https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity producer: name: Palo Alto Networks, Inc. url: https://www.paloaltonetworks.com/ - user_agent: Mozilla/5.0 (Laravel Reaver .env Presence) bot: name: Generic Bot - user_agent: Mozilla/5.0 (bang2013@atomicmail.io) bot: name: Generic Bot - user_agent: libredtail-http bot: name: Generic Bot - user_agent: Mozilla/5.0 (compatible; Replicate-Bot/1.0; +https://replicate.com/) bot: name: Replicate-Bot category: AI Data Scraper url: https://replicate.com/ producer: name: Replicate, Inc. url: https://replicate.com/ - user_agent: cypex.ai/scanning Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36 bot: name: Cypex category: Security Checker url: https://cypex.ai/scanning/ producer: name: Cypex url: https://cypex.ai/ - user_agent: Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com) bot: name: TikTokSpider category: Search bot url: https://www.tiktok.com/ producer: name: TikTok Inc. url: https://www.tiktok.com/ - user_agent: fhms-its-research-scanner/1.0 (+https://fb02itsscan02.fh-muenster.de) bot: name: FHMS ITS Research Scanner category: Security Checker url: https://fb02itsscan02.fh-muenster.de/ producer: name: University of Applied Sciences Muenster url: https://www.fh-muenster.de/ - user_agent: Mozilla/5.0 (compatible; Together-Bot/1.0; +https://together.ai/) bot: name: Together-Bot category: AI Data Scraper url: https://www.together.ai/ producer: name: Together Computer Inc. url: https://www.together.ai/ - user_agent: Mozilla/5.0 (compatible; xAI-Bot/1.0; +https://x.ai/) bot: name: xAI-Bot category: AI Data Scraper url: https://x.ai/ producer: name: X.AI LLC url: https://x.ai/ - user_agent: Mozilla/5.0 (compatible; Groq-Bot/1.0; +https://groq.com/) bot: name: Groq-Bot category: AI Data Scraper url: https://groq.com/ producer: name: Groq, Inc. url: https://groq.com/ - user_agent: Mozilla/5.0 (compatible; bigsur.ai/1.0) bot: name: Big Sur AI category: AI Assistant url: https://bigsur.ai/ producer: name: Big Sur AI, Inc. url: https://bigsur.ai/ - user_agent: Mozilla/5.0 (compatible; Cohere-Command/1.0; +https://cohere.com/) bot: name: Cohere-Command category: AI Data Scraper url: https://cohere.com/ producer: name: Cohere, Inc. url: https://cohere.com/ - user_agent: cohere-training-data-crawler bot: name: cohere-training-data-crawler category: AI Data Scraper url: https://cohere.com/ producer: name: Cohere, Inc. url: https://cohere.com/ - user_agent: Mozilla/5.0 (compatible; FirecrawlAgent/1.0) bot: name: FirecrawlAgent category: AI Data Scraper url: https://www.firecrawl.dev/ producer: name: SideGuide Technologies, Inc. url: https://www.sideguide.dev/ - user_agent: 'Mozilliqa""' bot: name: Generic Bot headers: Sec-CH-UA: 'Mozilliqa""' Sec-CH-UA-Platform: 'Lindows<?=print(9347655345-4954366);?>' Sec-CH-UA-Mobile: '?0<?=print(9347655345-4954366);?>' - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36 SecurityHeaders bot: name: SecurityHeaders category: Security Checker url: https://securityheaders.com/ producer: name: Snyk Limited url: https://snyk.io/ - user_agent: Mozilla/5.0 (compatible; HaloBot/1.0) bot: name: HaloBot category: Crawler url: https://www.haloscan.com/ producer: name: ARCHI301 url: https://www.haloscan.com/ - user_agent: Mozilla/5.0 (compatible; HaloBot/1.0; +bot@haloscan.com; +https://www.haloscan.com/) bot: name: HaloBot category: Crawler url: https://www.haloscan.com/ producer: name: ARCHI301 url: https://www.haloscan.com/ - user_agent: Mozilla/5.0 (compatible; RSiteAuditor) bot: name: RSiteAuditor category: Crawler url: https://help.agencyanalytics.com/en/articles/3469586-allow-our-site-auditor-via-robots-txt-and-firewall producer: name: My SEO Tool Inc url: https://agencyanalytics.com/ - user_agent: Mozilla/5.0 (compatible; ORT/v1.0.0; visit ORTc.me ) bot: name: OpenRobotsTXT category: Crawler url: https://openrobotstxt.org/ua producer: name: Majestic-12 Limited url: https://majestic.com/ - user_agent: Observer/1.0 (+https://obsrvr.net/about; crawler@obsrvr.net) bot: name: Observer category: Security Checker url: https://obsrvr.net/about - user_agent: AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com) bot: name: AliyunSecBot category: Security Checker url: https://service.alibaba.com/ - user_agent: Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/) bot: name: Mediumbot category: Crawler url: https://medium.com/ producer: name: A Medium Corporation url: https://medium.com/ - user_agent: Mozilla/5.0 (compatible; Heexybot/1.0; +https://heexy.org/bot) bot: name: Heexybot category: Search bot url: https://heexy.org/bot - user_agent: ForwardQR/1.0 (+https://forwardqr.com/bot.html) bot: name: ForwardQR category: Service Agent url: https://forwardqr.com/bot.html - user_agent: CybaaBot/1.0 (+https://cybaa.io) bot: name: CybaaBot category: Security Checker url: https://cybaa.io/bot-policy producer: name: Cybaa Ltd url: https://cybaa.io/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 CybaaBot bot: name: CybaaBot category: Security Checker url: https://cybaa.io/bot-policy producer: name: Cybaa Ltd url: https://cybaa.io/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 CybaaAgent bot: name: CybaaAgent category: Service Agent url: https://cybaa.io/bot-policy producer: name: Cybaa Ltd url: https://cybaa.io/ - user_agent: Secweb-Sectxt/1.0 bot: name: Secweb-Sectxt category: Crawler - user_agent: parser bot: name: Generic Bot - user_agent: Mozilla/5.0 (Windows NT 6.3;compatible; DVbot/1.0; +http://www.doubleverify.com) bot: name: DVbot category: Service Agent url: https://doubleverify.com/ producer: name: DoubleVerify Inc. url: https://doubleverify.com/ - user_agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3122.113 Safari/537.36 Assetnote/1.0.0 bot: name: Assetnote category: Security Checker url: https://www.assetnote.io/ producer: name: Assetnote Pty Ltd url: https://www.assetnote.io/ - user_agent: 'Mozilla/5.0 (compatible; UGAResearchAgent/1.0; Please visit: NISLabUGA.github.io)' bot: name: UGAResearchAgent category: Security Checker url: https://nislabuga-scan.uga.edu/ producer: name: School of Computing at the University of Georgia (UGA) url: https://computing.uga.edu/ - user_agent: Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler) bot: name: SERankingBacklinksBot category: Crawler url: https://help.seranking.com/hc/en-us/articles/23020353159964-SERankingBacklinksBot-Crawler producer: name: SER Acquisition Inc, url: https://seranking.com/ - user_agent: honeygain.org (+https://honeygain.org) bot: name: Generic Bot - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36 bot: name: Amazonbot category: AI Search Crawler url: https://developer.amazon.com/amazonbot producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Mozilla/5.0 (compatible; PrivacyPolicyBot/1.0) bot: name: PrivacyPolicyBot category: Crawler - user_agent: Mozilla/5.0 (compatible; SeoCherryBot/1.0; +https://en.wikipedia.org/wiki/Web_crawler) bot: name: SeoCherryBot category: Crawler - user_agent: HanaleiBot runid=beta-stage-integration-test bot: name: HanaleiBot category: Crawler - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ABEvalBot/0.1) Version/11.1.2 Safari/605.1.15 bot: name: ABEvalBot category: Crawler - user_agent: Google-NotebookLM bot: name: Google-NotebookLM category: AI Assistant url: https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) bot: name: Claude-User category: AI Assistant url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler producer: name: Anthropic PBC url: https://www.anthropic.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +searchbot@anthropic.com) bot: name: Claude-SearchBot category: AI Search Crawler url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler producer: name: Anthropic PBC url: https://www.anthropic.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/119.0.6045.214 Safari/537.36 bot: name: Amzn-User category: AI Assistant url: https://developer.amazon.com/amazonbot producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Safari/600.2.5 (Amazonbot-Video/0.1; +https://developer.amazon.com/support/amazonbot) bot: name: Amazonbot-Video category: Crawler url: https://developer.amazon.com/support/amazonbot producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: webzio-extended (+https://webz.io/bot.html) bot: name: webzio-extended category: AI Data Scraper url: https://docs.webz.io/reference/how-it-works-video producer: name: Webz.io Ltd url: https://webz.io/ - user_agent: webzio (+https://webz.io/bot.html) bot: name: webzio category: AI Search Crawler url: https://docs.webz.io/reference/how-it-works-video producer: name: Webz.io Ltd url: https://webz.io/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 bot: name: Gemini-Deep-Research category: AI Assistant url: https://gemini.google/overview/deep-research/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; Claude-Web/1.0; +https://www.anthropic.com) bot: name: Claude-Web category: AI Search Crawler url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler producer: name: Anthropic PBC url: https://www.anthropic.com/ - user_agent: Mozilla/5.0 (compatible) Ai2Bot-Dolma (+https://www.allenai.org/crawl) bot: name: Ai2Bot-Dolma category: AI Data Scraper url: https://allenai.org/crawler producer: name: The Allen Institute for Artificial Intelligence url: https://allenai.org/ - user_agent: Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler) bot: name: Ai2Bot category: AI Data Scraper url: https://allenai.org/crawler producer: name: The Allen Institute for Artificial Intelligence url: https://allenai.org/ - user_agent: Mozilla/5.0 (compatible; WARDBot/1.0; http://ward.ai/robot) bot: name: WARDBot category: Site Monitor url: https://ward.ai/robot producer: name: W.A.R.D. LLC url: https://ward.ai/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; robots.txt; +https://openai.com/searchbot bot: name: OAI-SearchBot category: AI Search Crawler url: https://platform.openai.com/docs/bots producer: name: OpenAI OpCo, LLC url: https://openai.com/ - user_agent: Mozilla/5.0 (compatible; Webzio-Extended/1.0) bot: name: webzio-extended category: AI Data Scraper url: https://docs.webz.io/reference/how-it-works-video producer: name: Webz.io Ltd url: https://webz.io/ - user_agent: Mozilla/5.0 (compatible; Character-AI/1.0; +https://character.ai/) bot: name: Character-AI category: AI Assistant url: https://character.ai/ producer: name: Character Technologies, Inc. url: https://character.ai/ - user_agent: Mozilla/5.0 (compatible; DeepseekBot/1.0; +https://www.deepseek.com/bot) bot: name: DeepseekBot category: AI Data Scraper url: https://www.deepseek.com/bot producer: name: Hangzhou Deep Search Artificiell Intelligens Basic Technology Research Co., Ltd. url: https://www.deepseek.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots) bot: name: MistralAI-User category: AI Assistant url: https://docs.mistral.ai/robots producer: name: Mistral AI url: https://mistral.ai/ - user_agent: Mozilla/5.0 (compatible; Crawlspace/1.0) bot: name: Crawlspace category: AI Data Scraper url: https://crawlspace.dev/ producer: name: Crawlspace Company url: https://crawlspace.dev/ - user_agent: Mozilla/5.0 (compatible; Devin/1.0) bot: name: Devin category: AI Assistant url: https://cognition.ai/ producer: name: Cognition AI, Inc. url: https://cognition.ai/ - user_agent: Google-Pinpoint bot: name: Google-Pinpoint category: AI Assistant url: 'https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers#google-pinpoint' producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; Andibot/1.0) bot: name: Andibot category: AI Search Crawler url: https://andisearch.com/ producer: name: Andi url: https://andisearch.com/ - user_agent: Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/crawler) bot: name: HypeStat category: Crawler url: https://hypestat.com/crawler - user_agent: Mozilla/5.0 (compatible; StatusNestBacklinkSpider/1.0; +https://statusnest.com/bot) bot: name: StatusNestBacklinkSpider category: Crawler url: https://statusnest.com/bot - user_agent: Mozilla/5.0 (compatible; RunPod-Bot/1.0; +https://runpod.io/) bot: name: RunPod-Bot category: AI Data Scraper url: https://www.runpod.io/ producer: name: RunPod Inc. url: https://www.runpod.io/ - user_agent: Mozilla/5.0 (compatible; HuggingFace-Bot/1.0; +https://huggingface.co/) bot: name: HuggingFace-Bot category: AI Data Scraper url: https://huggingface.co/ producer: name: Hugging Face, Inc url: https://huggingface.co/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleAgent-Mariner; +https://developers.google.com/search/docs/crawling-indexing/google-agent-mariner) Chrome/135.0.0.0 Safari/537.36 bot: name: GoogleAgent-Mariner category: AI Agent url: https://developers.google.com/search/docs/crawling-indexing/google-agent-mariner producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleAgent-Search; +https://developers.google.com/search/docs/crawling-indexing/google-agent-search) Chrome/114.0.0.0 Safari/537.36 bot: name: GoogleAgent-Search category: AI Assistant url: https://developers.google.com/search/docs/crawling-indexing/google-agent-search producer: name: Google Inc. url: https://www.google.com/ - user_agent: Tiberius/0.1 (+https://moosara.com) bot: name: Generic Bot - user_agent: LinkBloom/2.7.11 (Educational Domain Discovery; +https://hamidsoltani.com/linkbloom) bot: name: LinkBloom category: Crawler url: https://hamidsoltani.com/linkbloom - user_agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; test; +https://github.com/rom1504/img2dataset) bot: name: img2dataset category: Crawler url: https://github.com/rom1504/img2dataset - user_agent: Mozilla/5.0 AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; iAskBot/1.0; +https://iask.ai/) Chrome/120.0.6099.119 Safari/605.1.15 bot: name: iAskBot category: AI Data Scraper url: https://iask.ai/ producer: name: Ai Search Inc. url: https://iask.ai/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-Browser/1.0; +https://openai.com/bot) bot: name: ChatGPT-Browser category: AI Assistant url: https://platform.openai.com/docs/bots producer: name: OpenAI OpCo, LLC url: https://openai.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Anthropic-Claude/1.0; +https://www.anthropic.com) bot: name: ClaudeBot category: AI Data Scraper url: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler producer: name: Anthropic PBC url: https://www.anthropic.com/ - user_agent: Mozilla/5.0 (compatible; NovaAct/1.0; +https://www.novaact.com) bot: name: NovaAct category: AI Agent url: https://nova.amazon.com/act producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Mozilla/5.0 (compatible; VertexWP/1.0; +https://vertexwp.com/bot) bot: name: VertexWP category: Security Checker url: https://vertexwp.com/bot - user_agent: Mozilla/5.0 (compatible; AW-WB-Filter/1.0) bot: name: Generic Bot - user_agent: Friendica/2024.12 DatabaseVersion/1576 Request/ContentTypeChecker/1 +https://schmaker.eu bot: name: Friendica category: Social Media Agent - user_agent: Mozilla/5.0 (compatible; Webliobot/0.1; +http://www.weblio.jp/info/crawler.jsp) bot: name: Webliobot category: Crawler url: https://help.weblio.jp/ejje/jpdictionary/j_info/j_i_bcontents/h0021 producer: name: GRAS Group, Inc. url: https://gras-group.co.jp/ - user_agent: sqalix-geral-worker/1.0 bot: name: sqalix category: Crawler url: https://www.sqalix.com/ - user_agent: RecordedFuture Global Inventory Crawler bot: name: RecordedFuture category: Security Checker url: https://www.recordedfuture.com/ producer: name: Recorded Future, Inc. url: https://www.recordedfuture.com/ - user_agent: Valimail/1.0 (domain scanner) bot: name: Valimail category: Service Agent url: https://www.valimail.com/domain-checker/ producer: name: Valimail Inc. url: https://www.valimail.com/ - user_agent: SaferSoftwashLeadGen/2.2 (andrewfegan@local) bot: name: Generic Bot - user_agent: node bot: name: Generic Bot - user_agent: FindFiles.net/1.0 (compatible; +https://findfiles.net/bot) bot: name: FindFiles.net category: Crawler url: https://findfiles.net/bot producer: name: FindFiles.net UG url: https://findfiles.net/ - user_agent: Mozilla/5.0 (X11; Linux x86_64; WanscannerBot/1.2; +https://abuse.pend.re) Gecko/20100101 Firefox/10.0 bot: name: WanscannerBot category: Security Checker url: https://abuse.pend.re/ - user_agent: Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot) bot: name: Kagibot category: Search bot url: https://kagi.com/bot producer: name: Kagi Inc. url: https://kagi.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bard-AI/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) bot: name: Bard-AI category: AI Search Crawler url: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-AI/1.0; +https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) bot: name: Gemini-AI category: AI Search Crawler url: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinerBot/1.0; +https://docs.getliner.com/docs/linerbot) bot: name: LinerBot category: AI Assistant url: https://docs.getliner.com/docs/linerbot producer: name: Liner url: https://liner.com/ - user_agent: Mozilla/5.0 (compatible) AI2Bot-DeepResearchEval (+https://www.allenai.org/crawler) bot: name: Ai2Bot-DeepResearchEval category: AI Assistant url: https://allenai.org/crawler producer: name: The Allen Institute for Artificial Intelligence url: https://allenai.org/ - user_agent: Poggio-Citations 1.0 (+https://docs.poggio.io/api/robots) bot: name: Poggio-Citations category: AI Assistant url: https://docs.poggio.io/dev-resources/robots producer: name: Poggio Labs, Inc. url: https://poggio.io/ - user_agent: Mozilla/5.0 (compatible; QualifiedBot/1.0; +https://www.qualified.com/legal/qualified-crawler-user-agent) bot: name: QualifiedBot category: AI Assistant url: https://www.qualified.com/legal/qualified-crawler-user-agent producer: name: Qualified.com, Inc. url: https://www.qualified.com/ - user_agent: ZanistaBot/1.0 (+https://zanista.ai/crawler-info) bot: name: ZanistaBot category: AI Search Crawler url: https://zanista.ai/crawler-info producer: name: Zanista AI LTD. url: https://zanista.ai/ - user_agent: Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/) bot: name: ChatGLM-Spider category: AI Data Scraper url: https://chatglm.cn/ producer: name: Beijing Zhipu Huazhang Technology Co., Ltd. url: https://zhipuai.cn/ - user_agent: linkReader/1.0 (+https://gochitchat.ai) bot: name: Sider category: AI Data Scraper url: https://sider.ai/ producer: name: Sider AI url: https://sider.ai/ - user_agent: Mozilla/5.0 (compatible; SBIntuitionsBot/0.1; +https://www.sbintuitions.co.jp/bot/) bot: name: SBIntuitionsBot category: AI Data Scraper url: https://www.sbintuitions.co.jp/en/bot/ producer: name: SB Intuitions Corp. url: https://www.sbintuitions.co.jp/ - user_agent: IAS Crawler (ias_crawler; http://integralads.com/site-indexing-policy/) bot: name: IAS Crawler category: AI Data Scraper url: https://integralads.com/ias-privacy-data-management/policies/site-indexing-policy/ producer: name: Integral Ad Science, Inc. url: https://integralads.com/ - user_agent: IAS Wombles (ias_wombles; http://integralads.com/site-indexing-policy/) bot: name: IAS Wombles category: AI Data Scraper url: https://integralads.com/ias-privacy-data-management/policies/site-indexing-policy/ producer: name: Integral Ad Science, Inc. url: https://integralads.com/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0 bot: name: ShapBot category: AI Data Scraper url: https://docs.parallel.ai/resources/crawler producer: name: Parallel Web Systems Inc. url: https://parallel.ai/ - user_agent: Mozilla/5.0 (compatible; LumeWebScan/2.0; +https://lumeweaver.com/) bot: name: LumeWebScan category: Crawler url: https://lumeweaver.com/ producer: name: LumeWeaver Labs url: https://lumeweaver.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36; Devin/1.0; +devin.ai bot: name: Devin category: AI Assistant url: https://cognition.ai/ producer: name: Cognition AI, Inc. url: https://cognition.ai/ - user_agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36 bot: name: Amzn-SearchBot category: AI Search Crawler url: https://developer.amazon.com/amazonbot producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Agent/AmazonBuyForMe bot: name: AmazonBuyForMe category: AI Agent url: https://buyforme.amazon/ producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: 'amazon-kendra-customer-id-' bot: name: Amazon Kendra category: Crawler url: https://docs.aws.amazon.com/kendra/latest/dg/stop-web-crawler.html producer: name: Amazon.com, Inc. url: https://www.amazon.com/ - user_agent: Twingly Recon-Sjostrom/1.0 (+https://app.twingly.com/public-docs/crawler) bot: name: Twingly Recon category: Crawler url: https://app.twingly.com/public-docs/crawler producer: name: Twingly url: https://www.twingly.com - user_agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PanguBot;pangubot@huawei.com) bot: name: PanguBot category: AI Data Scraper url: https://www.huawei.com/ producer: name: Huawei Technologies Co., Ltd. url: https://www.huawei.com/ - user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36; Manus-User/1.0 bot: name: Manus-User category: AI Assistant url: https://manus.im/help/Manus-user producer: name: Butterfly Effect Pte. Ltd. url: https://manus.im/ - user_agent: HubSpotContentSearchBot bot: name: HubSpotContentSearchBot category: Crawler url: https://developers.hubspot.com/docs/cms/start-building/building-blocks/modules/content-search producer: name: HubSpot, Inc. url: https://www.hubspot.com/ - user_agent: ProRataInc (The ProRata Fair Use Crawler; https://prorata.ai; support@prorata.ai) bot: name: ProRata category: AI Search Crawler url: https://platform.gist.ai/docs/gcn-overview producer: name: ProRataAI, Inc. url: https://prorata.ai/ - user_agent: blackboardally bot: name: Blackboard Ally category: Crawler url: https://ally.ac/ producer: name: Anthology Inc. url: https://www.anthology.com/ - user_agent: Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 4.5) bot: name: svEye category: Crawler url: https://sveye.semantic-visions.com/ producer: name: Semantic Visions, s.r.o. url: https://www.semantic-visions.com/ - user_agent: Google-Firebase bot: name: Google-Firebase category: Service Agent url: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests producer: name: Google Inc. url: https://www.google.com/ - user_agent: Google-ilp bot: name: Google-ilp category: Crawler url: https://www.google.com/ producer: name: Google Inc. url: https://www.google.com/ - user_agent: Apache/2.2.2 (Fedora) (internal dummy connection) bot: name: Apache category: Service Agent url: https://www.apache.org/ producer: name: The Apache Software Foundation url: https://www.apache.org/foundation/ - user_agent: Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.0.5) Gecko/2008121621 Ubuntu/8.04 (hardy) Firefox/3.0.5 WebMoney Advisor bot: name: WebMoney Advisor category: Service Agent url: https://advisor.web.money/en/ producer: name: DEED Baltic, UAB url: http://www.deed.lt/ - user_agent: TprAdsTxtCrawler/1.1 bot: name: TprAdsTxtCrawler category: Crawler - user_agent: 'Mozilla/5.0 (compatible; CopyvioDetector/0.4.dev0; +wikipedia.earwig[at]gmail.com)' bot: name: CopyvioDetector category: Crawler url: https://copyvios.toolforge.org/ producer: name: Ben Kurtovic url: https://github.com/earwig - user_agent: Mozilla/5.0 (compatible; Dormouse/1.0) bot: name: Dormouse category: Crawler - user_agent: Mozilla/5.0 (compatible; FormFinder/1.0) bot: name: FormFinder category: Crawler - user_agent: Mozilla/5.0 (compatible; ImportDomains/1.0) bot: name: ImportDomains category: Crawler - user_agent: YourUserAgentHere bot: name: Generic Bot - user_agent: desktop bot: name: Generic Bot - user_agent: StudyBot/1.0 (Research Purposes) bot: name: StudyBot category: Crawler - user_agent: Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots) bot: name: YandexBot-MirrorDetector category: Crawler url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexMedianaBot/1.0; +http://yandex.com/bots) bot: name: YandexMedianaBot category: Search bot url: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html producer: name: Yandex LLC url: https://yandex.com/company/ - user_agent: 'SpeedySpider - http://www.entireweb.com' bot: name: Speedy - user_agent: Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/) bot: name: Speedy - user_agent: nrsbot/6.0(loopip.com/robot.html) bot: name: NetResearchServer - user_agent: NetResearchServer/2.5(loopimprovements.com/robot.html) bot: name: NetResearchServer - user_agent: Google-Cloud-Scheduler bot: name: Google Cloud Scheduler category: Crawler url: https://cloud.google.com/scheduler producer: name: Google Inc. url: https://www.google.com/ - user_agent: GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring) bot: name: Google Stackdriver Monitoring category: Site Monitor url: https://docs.cloud.google.com/monitoring/uptime-checks/using-uptime-checks producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 zgrab/0.x bot: name: zgrab category: Security Checker url: https://github.com/zmap/zgrab - user_agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/) bot: name: Charlotte category: Crawler url: https://web.archive.org/web/20070225121634/http://www.searchme.com/support/pages/spider.php producer: name: Searchme, Inc. url: https://web.archive.org/web/20070217233037/http://www.searchme.com/ - user_agent: Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/) bot: name: Charlotte category: Crawler url: https://web.archive.org/web/20070225121634/http://www.searchme.com/support/pages/spider.php producer: name: Searchme, Inc. url: https://web.archive.org/web/20070217233037/http://www.searchme.com/ - user_agent: CSS Certificate Spider (http://www.css-security.com/certificatespider/) bot: name: CSS Certificate Spider category: Crawler url: http://www.css-security.com/certificatespider/ producer: name: Certified Security Solutions url: https://www.css-security.com/company/about-us/ - user_agent: EmailWolf 1.00 bot: name: EmailWolf - user_agent: Gigabot/3.0 (http://www.gigablast.com/spider.html) bot: name: Gigabot - user_agent: Riddler (http://riddler.io/about) bot: name: Riddler category: Security search bot url: https://riddler.io/about producer: name: F-Secure url: https://www.f-secure.com - user_agent: netEstate NE Crawler (+http://www.website-datenbank.de/) bot: name: netEstate category: Crawler url: http://www.website-datenbank.de/Impressum producer: name: netEstate GmbH url: https://www.netestate.de/en/ - user_agent: Scooter/3.3 bot: name: Scooter - user_agent: Pompos/1.2 http://pompos.iliad.fr bot: name: Pompos category: Crawler url: https://web.archive.org/web/20090124084536/http://dir.com/pompos.html producer: name: ILIAD SA url: https://www.iliad.fr/ - user_agent: 'FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast' bot: name: Lycos - user_agent: Mozilla/5.0 (compatible; WebThumbnail/3.x; Website Thumbnail Generator; +http://webthumbnail.org) bot: name: WebThumbnail - user_agent: Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) bot: name: Inktomi Slurp - user_agent: Willow Internet Crawler by Twotrees V2.1 bot: name: Willow Internet Crawler category: Crawler url: https://twotrees.com/company/indexing-web-crawlers/ producer: name: Twotrees Technologies, LLC url: https://twotrees.com/ - user_agent: YahooCacheSystem; YahooWebServiceClient bot: name: Yahoo! Cache System category: Crawler url: "" producer: name: Yahoo! Inc. url: http://www.yahoo.com - user_agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/99.0 YottaaMonitor bot: name: Yottaa Site Monitor category: Site Monitor url: http://www.yottaa.com/products/site-monitor producer: name: Yottaa url: http://www.yottaa.com/ - user_agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google-BusinessLinkVerification) Chrome/143.0.7499.192 Safari/537.36 bot: name: Googlebot category: Search bot url: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (compatible; Eyeotabot/1.0; +http://www.eyeota.com) bot: name: Eyeotabot category: Crawler url: https://www.eyeota.com/ producer: name: Eyeota Pte Ltd url: https://www.eyeota.com/ - user_agent: Mozilla/5.0 (compatible; YioopBot; +https://www.yioop.com/bot) bot: name: YioopBot category: Search bot url: https://www.yioop.com/bot producer: name: Chris Pollett url: https://www.pollett.org/ - user_agent: Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir) bot: name: yoozBot category: Search bot url: https://www.yooz.ir/ producer: name: yooz url: https://www.yooz.ir/ - user_agent: yoozbot-bear/1.0 (http://yooz.ir/; info@yooz.ir); bot: name: yoozBot category: Search bot url: https://www.yooz.ir/ producer: name: yooz url: https://www.yooz.ir/ - user_agent: Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) bot: name: Yahoo! Japan BRJ category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Y!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) bot: name: Yahoo! Japan PSC category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html) bot: name: Yahoo! Japan BRY category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)) bot: name: Yahoo! Japan category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Y!J-BSC/1.0 (http://help.yahoo.co.jp/help/jp/blog-search/) bot: name: Yahoo! Japan BSC category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Y!J-BRI/0.0.1 crawler ( http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html ) bot: name: Yahoo! Japan BRI category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Y!J-BRO/YFSJ crawler (compatible; Mozilla 4.0; MSIE 5.5; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html; YahooFeedSeekerJp/2.0) bot: name: Yahoo! Japan BRO category: Crawler url: https://support.yahoo-net.jp/PccSearch/s/article/H000007955 producer: name: Yahoo! Japan Corp. url: https://www.yahoo.co.jp/ - user_agent: Mozilla/5.0 (compatible; Google-Site-Verification/1.0) bot: name: Google-Site-Verification category: Validator url: 'https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers#google_site_verifier' producer: name: Google Inc. url: https://www.google.com/ - user_agent: Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943) bot: name: Google-Read-Aloud category: Crawler url: https://developers.google.com/crawling/docs/crawlers-fetchers/read-aloud-user-agent producer: name: Google Inc. url: https://www.google.com/