Robots
A quick note because I have to get out the door and go to work… gah.
I recently removed the restrictions on my robots.txt file.
This site has been denying access to most bots for the past year:
# Welcome to Tregeagle ###################### User-agent: archive.org_bot Allow: / User-agent: ia_archiver Allow: / User-agent: search.marginalia.nu Allow: / User-agent: WilbyBot Allow: / # Allow me show you the door ############################ # Anti-SEO User-agent: * Allow: /pages/about.html Disallow: / Sitemap: https://tregeagle.com/sitemap.xmlAllowing bots to crawl through my pages I immediately noticed that Google indexed it quick as light. Search engines like Google, Bing and DuckDuckGo are not relevant to me these days. I do not want this site to be found, I just like tinkering with it. I opened my robots file mainly to allow access to Archive.org and Marginalia.nu which are both admirable projects.
None of this affects my human readers, all three of you should be ok.
Love Ruben x
Comments
Add a comment?There are no comments yet.