Sekretaerbaer-Netz/Blocked_Bots_and_Crawler_Sekretaerbaer.de
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Update 20250424

Browse source 
Source hinzugefügt
This commit is contained in:
oldkid 2025-04-24 14:44:23 +02:00
parent 7567dd74f3
commit 2d64cef6e0
3 changed files with 78 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

76
Bots and Crawler.txt Normal file
View file

@ -0,0 +1,76 @@
BLEXBot
Barkrowler
Livelapbot
Franck, the Fediverse Graph Crawler
YisouSpider
DotBot
VelenPublicWebCrawler
ChatGPT-User
GoogleOther
GoogleOther-Image
GoogleOther-Video
Applebot
Applebot-Extended
AhrefsBot
Baiduspider
SemrushBot
DotBot
MJ12bot
PetalBot
SeznamBot
Mediapartners-Google
Bytespider
Claude-Web
GPTBot
PerplexityBot
Applebot
Amazonbot
Meta-ExternalAgent
Meta-ExternalFetcher
FacebookBot
DuckAssistBot
Anthropic-ai
Bytedance
fidget-spinner-bot
EmailCollector
WebEMailExtrac
ClaudeBot
facebookexternalhit
facebookcatalog
YandexBot
Bingbot
Googlebot
AI2Bot
Ai2Bot-Dolma
Brightbot
CCBot
ChatGPT-User
cohere-ai
cohere-training-data-crawler
Crawlspace
Diffbot
FriendlyCrawler
Google-Extended
GoogleOther
GoogleOther-Image
GoogleOther-Video
iaskspider
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
KangarooBot
OAI-SearchBot
omgili
omgilibot
PanguBot
Perplexity-User
PetalBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Sidetradeindexerbot
Timpibot
VelenPublicWebCrawler
Webzio-Extended
YouBot