This commit is contained in:
+4
-1
@@ -4,6 +4,9 @@
|
||||
# Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
|
||||
# See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt
|
||||
|
||||
User-agent: Rivva
|
||||
Allow: /
|
||||
|
||||
# Block all crawlers by default
|
||||
User-agent: *
|
||||
Disallow: /
|
||||
@@ -40,4 +43,4 @@ User-agent: Bytespider
|
||||
User-agent: ImagesiftBot
|
||||
User-agent: PerplexityBot
|
||||
User-agent: cohere-ai
|
||||
Disallow: /
|
||||
Disallow: /
|
||||
|
||||
Reference in New Issue
Block a user