svemagie.net/indiekit-blog
1
0
Fork 0
Code Issues Actions Packages Projects Releases Activity

fix: allow Rivva Bot
Build & Deploy / build-and-deploy (push) Successful in 2m49s
Details

Browse Source
This commit is contained in:
svemagie
2026-05-12 15:58:21 +02:00
parent 3afc5554c9
commit fd0b39dc87
1 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
robots.txt
+4 -1
View File
@@ -4,6 +4,9 @@
# Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/ # Reference: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
# See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt # See: https://github.com/MattWilcox/native-base/blob/45f6e7a837104f5ad83a5c7e280fb9a4eb126219/robots.txt
User-agent: Rivva
Allow: /
# Block all crawlers by default # Block all crawlers by default
User-agent: * User-agent: *
Disallow: / Disallow: /
@@ -40,4 +43,4 @@ User-agent: Bytespider
User-agent: ImagesiftBot User-agent: ImagesiftBot
User-agent: PerplexityBot User-agent: PerplexityBot
User-agent: cohere-ai User-agent: cohere-ai
Disallow: / Disallow: /