Robots
A quick note because I have to get out the door and go to work… gah.
I recently removed the restrictions on my robots.txt file.
This site has been denying access to most bots for the past year:
# Welcome to Tregeagle
######################
User-agent: archive.org_bot
Allow: /
User-agent: ia_archiver
Allow: /
User-agent: search.marginalia.nu
Allow: /
User-agent: WilbyBot
Allow: /
# Allow me show you the door
############################
# Anti-SEO
User-agent: *
Allow: /pages/about.html
Disallow: /
Sitemap: https://tregeagle.com/sitemap.xml
Allowing bots to crawl through my pages I immediately noticed that Google indexed it quick as light. Search engines like Google, Bing and DuckDuckGo are not relevant to me these days. I do not want this site to be found, I just like tinkering with it. I opened my robots file mainly to allow access to Archive.org and Marginalia.nu which are both admirable projects.
None of this affects my human readers, all three of you should be ok.
Love Ruben x
Have a comment?