This commit is contained in:
@@ -4,6 +4,9 @@
|
|||||||
# Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
|
# Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
|
||||||
# See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt
|
# See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt
|
||||||
|
|
||||||
|
User-agent: Rivva
|
||||||
|
Allow: /
|
||||||
|
|
||||||
# Block all crawlers by default
|
# Block all crawlers by default
|
||||||
User-agent: *
|
User-agent: *
|
||||||
Disallow: /
|
Disallow: /
|
||||||
|
|||||||
Reference in New Issue
Block a user