About LondonRefurbsBot
If this address appeared in your server logs, our crawler visited your website. This page explains who we are, what we did, and how to stop us if you'd rather we didn't.
Who we are
LondonRefurbs is a directory of refurbishment and trade companies across London. Your company is listed here — or is being considered for listing — and we fetched a few pages of your website so your listing can describe your work accurately, in your own words, rather than from second-hand data.
What the crawler does
- It fetches your homepage, then up to eight more pages that your homepage's own navigation links to — at most three from any one section. It follows the pages your site presents as its main ones, not everything it can find, and it does not go a second hop deeper.
- It skips checkout, account, login and legal pages. Those say nothing about your work.
- It reads and obeys your
robots.txt, includingCrawl-delayand any group naming it directly. - It spaces requests to one host several seconds apart, honours
Retry-After, and stands down entirely if your server answers itsrobots.txtrequest with a server error. - It fetches page markup only — it does not download your images, media or fonts.
- It keeps the fetched pages, with the URL and time of each fetch, as the source of record for what your listing says.
How it identifies itself
Every request carries this User-Agent:
Mozilla/5.0 (compatible; LondonRefurbsBot/1.0; +https://londonrefurbs.com/bot)
How to block it
Add this to the robots.txt file at the root of your website and the crawler will not fetch anything from it:
User-agent: LondonRefurbsBot Disallow: /
The rule takes effect the next time the crawler checks your robots.txt, before it requests any page.
Who to contact
Email bot@londonrefurbs.com. Tell us your domain and we will confirm what we hold, and remove it on request.