Sekretaerbaer-Netz/Blocked_Bots_and_Crawler_Sekretaerbaer.de
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge branch 'refs/heads/develop'

Browse source 
add User-Agent: SummalyBot https://datadome.co/bots/summalybot/
This commit is contained in:
oldkid 2025-05-11 08:08:30 +02:00
commit 99e4a6aa2a
4 changed files with 5 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

1
Bots and Crawler.txt
View file

@ -73,6 +73,7 @@ Mediapartners-Google
MJ12bot
SemrushBot
SeznamBot
SummalyBot
VelenPublicWebCrawler
WebEMailExtrac
YandexBot

2
apache-badbots.conf
View file

@ -1 +1 @@
I2Bot|Ai2Bot\-Dolma|aiHitBot|Amazonbot|anthropic\-ai|Applebot|Applebot\-Extended|Brightbot\ 1\.0|Bytespider|CCBot|ChatGPT\-User|Claude\-Web|ClaudeBot|cohere\-ai|cohere\-training\-data\-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google\-Extended|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GPTBot|iaskspider/2\.0|ICC\-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo\ Bot|meta\-externalagent|Meta\-ExternalAgent|meta\-externalfetcher|Meta\-ExternalFetcher|NovaAct|OAI\-SearchBot|omgili|omgilibot|Operator|PanguBot|Perplexity\-User|PerplexityBot|PetalBot|Scrapy|SemrushBot\-OCOB|SemrushBot\-SWA|Sidetrade\ indexer\ bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio\-Extended|YouBot|AhrefsBot|Baiduspider|Barkrowler|Bingbot|BLEXBot|Bytedance|DotBot|EmailCollector|facebookcatalog|facebookexternalhit|fidget-spinner-bot|Franck the Fediverse Graph Crawler|Googlebot|Livelapbot|Mediapartners-Google|MJ12bot|SemrushBot|SeznamBot|VelenPublicWebCrawler|WebEMailExtrac|YandexBot|YisouSpider|
I2Bot|Ai2Bot\-Dolma|aiHitBot|Amazonbot|anthropic\-ai|Applebot|Applebot\-Extended|Brightbot\ 1\.0|Bytespider|CCBot|ChatGPT\-User|Claude\-Web|ClaudeBot|cohere\-ai|cohere\-training\-data\-crawler|Cotoyogi|Crawlspace|Diffbot|DuckAssistBot|FacebookBot|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Google\-Extended|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GPTBot|iaskspider/2\.0|ICC\-Crawler|ImagesiftBot|img2dataset|imgproxy|ISSCyberRiskCrawler|Kangaroo\ Bot|meta\-externalagent|Meta\-ExternalAgent|meta\-externalfetcher|Meta\-ExternalFetcher|NovaAct|OAI\-SearchBot|omgili|omgilibot|Operator|PanguBot|Perplexity\-User|PerplexityBot|PetalBot|Scrapy|SemrushBot\-OCOB|SemrushBot\-SWA|Sidetrade\ indexer\ bot|TikTokSpider|Timpibot|VelenPublicWebCrawler|Webzio\-Extended|YouBot|AhrefsBot|Baiduspider|Barkrowler|Bingbot|BLEXBot|Bytedance|DotBot|EmailCollector|facebookcatalog|facebookexternalhit|fidget-spinner-bot|Franck the Fediverse Graph Crawler|Googlebot|Livelapbot|Mediapartners-Google|MJ12bot|SemrushBot|SeznamBot|SummalyBot|VelenPublicWebCrawler|WebEMailExtrac|YandexBot|YisouSpider|

1
robots.txt
View file

@ -73,6 +73,7 @@ User-agent: Mediapartners-Google
User-agent: MJ12bot
User-agent: SemrushBot
User-agent: SeznamBot
User-Agent: SummalyBot
User-agent: VelenPublicWebCrawler
User-agent: WebEMailExtrac
User-agent: YandexBot

3
source/sekretaerbaer.de/Liste-sekretaerbaer.txt
View file

@ -16,7 +16,8 @@ Mediapartners-Google
MJ12bot
SemrushBot
SeznamBot
SummalyBot
VelenPublicWebCrawler
WebEMailExtrac
YandexBot
YisouSpider
YisouSpider