7567dd74f3
add : AI2Bot Ai2Bot-Dolma Brightbot CCBot ChatGPT-User cohere-ai cohere-training-data-crawler Crawlspace Diffbot FriendlyCrawler Google-Extended GoogleOther GoogleOther-Image GoogleOther-Video iaskspider ICC-Crawler ImagesiftBot img2dataset ISSCyberRiskCrawler KangarooBot OAI-SearchBot omgili omgilibot PanguBot Perplexity-User PetalBot Scrapy SemrushBot-OCOB SemrushBot-SWA Sidetradeindexerbot Timpibot VelenPublicWebCrawler Webzio-Extended YouBot
76 lines
No EOL
1,000 B
Text
76 lines
No EOL
1,000 B
Text
BLEXBot
|
|
Barkrowler
|
|
Livelapbot
|
|
Franck, the Fediverse Graph Crawler
|
|
YisouSpider
|
|
DotBot
|
|
VelenPublicWebCrawler
|
|
ChatGPT-User
|
|
GoogleOther
|
|
GoogleOther-Image
|
|
GoogleOther-Video
|
|
Applebot
|
|
Applebot-Extended
|
|
AhrefsBot
|
|
Baiduspider
|
|
SemrushBot
|
|
DotBot
|
|
MJ12bot
|
|
PetalBot
|
|
SeznamBot
|
|
Mediapartners-Google
|
|
Bytespider
|
|
Claude-Web
|
|
GPTBot
|
|
PerplexityBot
|
|
Applebot
|
|
Amazonbot
|
|
Meta-ExternalAgent
|
|
Meta-ExternalFetcher
|
|
FacebookBot
|
|
DuckAssistBot
|
|
Anthropic-ai
|
|
Bytedance
|
|
fidget-spinner-bot
|
|
EmailCollector
|
|
WebEMailExtrac
|
|
ClaudeBot
|
|
facebookexternalhit
|
|
facebookcatalog
|
|
YandexBot
|
|
Bingbot
|
|
Googlebot
|
|
AI2Bot
|
|
Ai2Bot-Dolma
|
|
Brightbot
|
|
CCBot
|
|
ChatGPT-User
|
|
cohere-ai
|
|
cohere-training-data-crawler
|
|
Crawlspace
|
|
Diffbot
|
|
FriendlyCrawler
|
|
Google-Extended
|
|
GoogleOther
|
|
GoogleOther-Image
|
|
GoogleOther-Video
|
|
iaskspider
|
|
ICC-Crawler
|
|
ImagesiftBot
|
|
img2dataset
|
|
ISSCyberRiskCrawler
|
|
KangarooBot
|
|
OAI-SearchBot
|
|
omgili
|
|
omgilibot
|
|
PanguBot
|
|
Perplexity-User
|
|
PetalBot
|
|
Scrapy
|
|
SemrushBot-OCOB
|
|
SemrushBot-SWA
|
|
Sidetradeindexerbot
|
|
Timpibot
|
|
VelenPublicWebCrawler
|
|
Webzio-Extended
|
|
YouBot |