Updating my robots.txt and other well-known files

Monday, February 19, 2024 - Permalink

Categories: blog -- Tags: #blog #bots

/!\ Warning: This article is older than 555 days, make sure the content is still relevant!

Nota: This post is the first one after updating my deployment process to something way more complex than it should be… I may have made mistake so reach out if anything feels broken!

February 2024 seems to be the “blog update” month here! A quick note as I have updated my robots.txt file to prevent some (AI related) bots from crawling this site. I know it is more a desperate and useless attempt, but that didn’t cost me much to add those (and was inspired by this article :)). I also took the opportunity to add a humans.txt and a security.txt.

The robots.txt is now as follow:

User-agent: *
Disallow:

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Twitterbot
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: YouBot
Disallow: /

Sitemap: https://bacardi55.io/sitemap.xml

The humans.txt:

/* TEAM */
Site Owner: bacardi55
Site: bac + blog (a) rdi55.pl  /* Not like this in txt file */
Fediverse: @bacardi55@social.bacardi55.io

/* SITE */
Created in: 2010
Software: Powered by gohugo.io
Other: No Ads, No Javascript, No tracker

And the security.txt:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Policy: If any vulnerabilities are found related to site code or configuration, please contact me.

Contact: https://social.bacardi55.io/@bacardi55
Contact: mailto : bac + security (a) rdi55 . pl /* Not like this in file */

Encryption: https://bacardi55.io/files/pubkey.txt

Preferred-Languages: en, fr

Canonical: https://bacardi55.io/.well-known/security.txt

Expires: 2024-06-28T21:59:00.000Z

Last-Updated: 2023-02-15
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEC5cOn+pswIkPQKyisnxT0OctJsAFAmXOj1wACgkQsnxT0Oct
JsB58Q//bBdzISQZ1zMeZBHOD/oqX/muiTF2vTSMP3lsSx7Fyp0DAVtOiHYak5ED
/w8qxQqnhUOuI0n1PREOGrJGkkzujmL/AHb5cOKUNUyWtmCaPsy/JkDh8JUShgn4
82o5hkvXde6B7025HLQHZs7ktbsEZDU7wn6ZOQFFm7io1UcpDjeQpomi9AwOlJIA
yDg4gMKnzu5zWugwH2VWLY6ujuUMvPehHso8WOFWwBqr7/nH6LBAvNrPHpVpUhbG
gNQfbf3B5syYJ+3kl5nOy6kuwi+kHyN1nQv7ftAwcowV8L/vehUEwUPtSjVQ3RtD
Xvtk8y6EWtGc3tbVT6/lwJelXI+N1SQcCNaz+sR+OcPFmd8rOrI83B+P/7NFvNfq
5Z3Jktt3UBNjbxR7UMsAakj9ieuZw52CX/TPM0hRNGIvRPN5WnPAe1PigDOQHjft
KxuOlMHB/PunB/D7C7oyGIOo5V1dgIt4XN9Pj3iW2DkcuMSlW2PqifQRmh9JLtn3
YeKgaj9js8xAfFVj12JzMWBirxCZMcgseMNXmQXudNpfZSP2sAV5bvm6SJkaxD9r
xzPK/AsJpdGsaJQwsRHAM5o0zVP+1vK+xwIPRLFlPesYKNuX0PpCkj7pVNwc9NM+
AMuxyaMjxMXhiwlRaatAYnSrcb3VI4EGX0RGL1z6v4j2O1fkEqM=
=CosJ
-----END PGP SIGNATURE-----

Small tips, to sign the security.txt file, just use:

gpg --clearsign security.txt

Contact

If you find any issue or have any question about this article, feel free to reach out to me via webmentions, email, mastodon, matrix or even IRC, see the About page for details.

Updating my robots.txt and other well-known files

Contact

Releated Posts