diff --git a/Bots and Crawler b/Bots and Crawler index cb14b5d..2cedcf2 100644 --- a/Bots and Crawler +++ b/Bots and Crawler @@ -30,7 +30,6 @@ Meta-ExternalFetcher FacebookBot DuckAssistBot Anthropic-ai -Bytespider Bytedance fidget-spinner-bot EmailCollector @@ -41,3 +40,37 @@ facebookcatalog YandexBot Bingbot Googlebot +AI2Bot +Ai2Bot-Dolma +Brightbot +CCBot +ChatGPT-User +cohere-ai +cohere-training-data-crawler +Crawlspace +Diffbot +FriendlyCrawler +Google-Extended +GoogleOther +GoogleOther-Image +GoogleOther-Video +iaskspider +ICC-Crawler +ImagesiftBot +img2dataset +ISSCyberRiskCrawler +KangarooBot +OAI-SearchBot +omgili +omgilibot +PanguBot +Perplexity-User +PetalBot +Scrapy +SemrushBot-OCOB +SemrushBot-SWA +Sidetradeindexerbot +Timpibot +VelenPublicWebCrawler +Webzio-Extended +YouBot \ No newline at end of file