TollBit - Your complete web stack for the agentic internet
Bot index
Our directory of known bots used across the web. This consists of bots, agents, and crawlers that we reference consistently across analytics, routing, licensing, and documentation.
Filter
90 results
AI2Bot-DeepResearchEval
- Provider: Allen Institute for AI
- Type: AI
- Category: Training
- Description: A research crawler operated by the Allen Institute for AI (AI2) for collecting web data as part of their deep research evaluation projects. This bot appears to be focused on gathering data for AI research and evaluation purposes, likely related to AI2's ongoing research initiatives in natural language processing and machine learning evaluation.
- First seen: Dec 31, 2023
AI2Bot
- Provider: Allen Institute for AI
- Type: AI
- Category: Training
- Description: The Allen Institute for AI's web crawler that collects data for research purposes including AI model training and academic research projects. The bot is used to gather web content for various AI research initiatives conducted by AI2.
- First seen: Dec 31, 2019
Ai2Bot-Dolma
- Provider: Allen Institute for AI
- Type: AI
- Category: Training
- Description: A web crawler operated by the Allen Institute for AI (AI2) to collect training data for the Dolma dataset, which is used for training large language models. The bot systematically crawls web content to build high-quality training corpora for AI research.
- First seen: Dec 31, 2022
Amazonbot
- Provider: Amazon
- Type: AI
- Category: Training
- Description: Amazon's web crawler that indexes web pages for various Amazon services and search features. It crawls publicly available web content to improve Amazon's search and discovery capabilities across the company's platforms.
- First seen: Dec 31, 2021
Amzn-SearchBot
- Provider: Amazon
- Type: AI
- Category: AI Search
- Description: Amazon's search crawler that indexes web content for Amazon's search services. The bot crawls web pages to build search indices and discover content for Amazon's various search and discovery features.
- First seen: Dec 31, 2019
Amzn-User
- Provider: Amazon
- Type: AI
- Category: RAG
- Description: Amazon's internal user agent used for various automated processes and content fetching operations on behalf of Amazon services. The bot appears to perform automated tasks related to Amazon's platform operations but specific details about its functionality are not publicly documented.
- First seen: Apr 20, 2026
ClaudeBot
- Provider: Anthropic
- Type: AI
- Category: Training
- Description: Anthropic's web crawler that collects publicly available data from the web to improve Claude and other AI systems.
- First seen: Dec 31, 2023
Claude-SearchBot
- Provider: Anthropic
- Type: AI
- Category: AI Search
- Description: Anthropic's web crawler that indexes web content to provide up-to-date information in Claude's AI responses. It crawls web pages to enhance Claude's knowledge with current information and enables Claude to answer questions about recent events and topics.
- First seen: Jul 31, 2024
Claude-User
- Provider: Anthropic
- Type: AI
- Category: RAG
- Description: Anthropic's user-agent for Claude AI assistant when fetching web content during conversations. This agent retrieves real-time information from websites to provide up-to-date responses to user queries. It operates on-demand when users request current information or when Claude needs to access specific web resources.
- First seen: Dec 31, 2023
Applebot
- Provider: Apple
- Type: SEO
- Category: Search
- Description: Apple's web crawler that indexes content for Apple's search services and products. It crawls web pages to build search indexes used across Apple's ecosystem, including Siri, Spotlight, and Safari suggestions.
- First seen: Sep 30, 2015
AppleBot-Extended
- Provider: Apple
- Type: AI
- Category: Control
- Description: Apple's web crawler designed for data collection to train Apple's foundational models. This is an extended version of Apple's standard AppleBot crawler that focuses on gathering training data for AI models.
- First seen: Dec 13, 2024
Apple News Bot
- Provider: Apple
- Type: SEO
- Category: Search
- Description: Apple's web crawler that indexes and fetches content for the Apple News platform. It crawls publisher websites to gather articles and news content that appears in the Apple News app and service.
- First seen: Sep 08, 2015
AppleWebKit
- Provider: Apple
- Type: Human
- Category: Human
- Description: AppleWebKit is the rendering engine token that appears in HTTP User-Agent strings sent by real human users of Safari, Chrome, Edge, and other WebKit- or Blink-based browsers across macOS, iOS, Android, and Windows. It is not a bot or crawler; it identifies Apple's open-source WebKit browser engine, which was forked from KDE's KHTML in 2001 and first publicly shipped with Safari in January 2003. The token is present in virtually all modern browser UA strings for compatibility reasons, making it a marker of legitimate human web traffic rather than automated activity.
- First seen: Jan 06, 2003
iPad
- Provider: Apple
- Type: Human
- Category: Human
- Description: The "ipad" user-agent token is a substring present in the standard browser user-agent strings sent by Apple iPads running Safari or other WebKit-based browsers (e.g., Chrome for iOS). It identifies real human users browsing the web on an Apple iPad device. Since iPadOS 13, iPads in "Request Desktop Website" mode may omit the "iPad" token and instead send a macOS-style user-agent, making detection less reliable.
- First seen: Apr 02, 2010
iPhone
- Provider: Apple
- Type: Human
- Category: Human
- Description: The "iphone" token appears in the User-Agent strings of real human users browsing the web on Apple iPhone devices via Safari or other iOS browsers. It is not a bot or crawler of any kind. The iOS User-Agent string identifies the device type, operating system version, and rendering engine to help websites deliver appropriately formatted content. Apple deliberately limits the amount of device-specific information shared in these strings for privacy reasons.
- First seen: Jun 28, 2007
Safari
- Provider: Apple
- Type: Human
- Category: Human
- Description: Safari is Apple's web browser built into macOS, iOS, iPadOS, and visionOS, using the open-source WebKit rendering engine. It is operated by real human users browsing the web and is the second most widely used browser in the world. Safari transmits a user-agent string to web servers identifying the browser version, operating system, and device type so that sites can deliver optimized content. It is not a bot or crawler of any kind.
- First seen: Jan 06, 2003
iboubot
- Provider: Babbar
- Type: SEO
- Category: Search
- Description: Babbar's web crawler that indexes web content for their SEO and web analytics platform. The bot crawls websites to build search indices and gather data for competitive analysis and SEO insights.
- First seen: Dec 31, 2019
Baiduspider
- Provider: Baidu
- Type: SEO
- Category: Search
- Description: Baidu's primary web crawler that indexes web content for the Baidu search engine. It operates from specific IP ranges and identifies itself with various Baiduspider user agent strings.
- First seen: Dec 31, 1999
Baiduspider-image
- Provider: Baidu
- Type: SEO
- Category: Search
- Description: Baidu's specialized web crawler for indexing images across the internet. This bot discovers and analyzes images for inclusion in Baidu's image search results. It operates as part of Baidu's broader web crawling infrastructure to build comprehensive image search capabilities.
- First seen: Dec 31, 2009
BaiduSpider-render
- Provider: Baidu
- Type: SEO
- Category: Search
- Description: BaiduSpider-render is Baidu's specialized web crawler designed to render JavaScript-heavy pages and dynamic content. It works alongside the main BaiduSpider to ensure proper indexing of modern web applications that rely on client-side rendering. The bot processes pages that require JavaScript execution to display their full content for Baidu's search index.
- First seen: Dec 31, 2014
Baiduspider-video
- Provider: Baidu
- Type: SEO
- Category: Search
- Description: Baidu's specialized web crawler for video content discovery and indexing. It crawls web pages to find video files, video metadata, and video-related content for inclusion in Baidu's video search results. The crawler focuses specifically on video content rather than general web pages.
- First seen: Dec 31, 2007
BraveBot
- Provider: Brave
- Type: SEO
- Category: Search
- Description: Brave's web crawler that builds and maintains the search index for Brave Search, an independent search engine. The bot crawls web pages to discover and index content for search results. Brave operates its own web index rather than relying on third-party search providers.
- First seen: Dec 31, 2020
BrightBot
- Provider: Bright Data
- Type: Data Broker
- Category: Data Resale
- Description: An automated web crawler operated by Bright Data for data collection and aggregation purposes. The bot is part of Bright Data's web data platform that collects publicly available web information for commercial data services. It operates as a general-purpose crawler to gather web content for Bright Data's data marketplace and analytics products.
- First seen: Dec 31, 2019
manus-user
- Provider: Butterfly Effect
- Type: AI
- Category: Autonomous
- Description: An AI agent operated by Butterfly Effect that performs autonomous web browsing tasks. Limited official documentation is available about its specific capabilities and intended use cases.
- First seen: Dec 31, 2023
Bytespider
- Provider: ByteDance (TikTok)
- Type: AI
- Category: Training
- Description: ByteDance's web crawler that collects data for AI model training and development.
- First seen: Dec 31, 2020
coccocbot-web
- Provider: Cốc Cốc
- Type: SEO
- Category: Search
- Description: Web crawler operated by Cốc Cốc, a Vietnamese search engine, to index web content for their search services. The bot crawls and indexes web pages to build and maintain Cốc Cốc's search database.
- First seen: Dec 31, 2014
cohere-ai
- Provider: Cohere
- Type: AI
- Category: RAG
- Description: Cohere's web crawler that collects data from publicly available web content to train and improve their large language models.
- First seen: Dec 31, 2020
cohere-training-data-crawler
- Provider: Cohere
- Type: AI
- Category: Training
- Description: Cohere's web crawler designed to collect training data for AI model development. The bot crawls publicly available web content to build datasets used in training Cohere's language models and other AI systems.
- First seen: Dec 31, 2023
CCBot
- Provider: Common Crawl
- Type: AI
- Category: Training
- Description: CCBot is Common Crawl's web crawler that builds an open repository of web crawl data accessible to researchers, companies, and individuals. The bot crawls the web to create datasets that are freely available for analysis, research, and educational purposes. Common Crawl publishes its data monthly and makes it available through Amazon S3.
- First seen: Dec 31, 2007
Diffbot
- Provider: Diffbot
- Type: Data Broker
- Category: Data Resale
- Description: Diffbot is an AI-powered web scraping and data extraction service that crawls websites to extract structured data from web pages. It uses machine learning to automatically identify and extract specific content types like articles, products, and job postings. The service provides APIs for developers and enterprises to integrate web data extraction capabilities into their applications.
- First seen: Dec 31, 2009