Our crawler, and how to turn it away.

• It says who it is. Every request carries the user agent StoreBayBot/1.0 (+https://storebay.co.uk/bot).

• It reads your robots.txt before anything else, and never relies on a copy more than a day old.

• It asks for one page at a time, at least 2 seconds apart, or slower if your robots.txt sets a Crawl-delay (up to 30 seconds).

• It reads at most 12 pages of a site, choosing the ones most likely to give contact details, locations and prices. For a business with many sites, it may read up to 60 location pages.

• It never logs in, fills in a form, makes a booking or gets past a bot check.

• The facts a business publishes about itself: its name, its sites and their addresses, its phone numbers, a general email address such as info@, its company number, and whether it takes bookings and shows prices online. We note the page each fact came from, and when.

• Not your pages. Apart from a short quote showing why we think a site offers self storage, we delete the text we read within 30 days.

• StoreBay makes software for storage operators, and this list is how our sales team knows who to contact.

To ask what we hold about your business, or to be taken off the list, email privacy@storebay.co.uk.

User-agent: StoreBayBot
Disallow: /

To keep it off part of your site, disallow that path instead. It follows the rules written for StoreBayBot if there are any, and otherwise the rules for every robot (User-agent: *).

Turning the crawler away stops it reading your site. It doesn’t remove what we already hold: for that, email privacy@storebay.co.uk.