ABOUT OUR CRAWLER
If you found this page in your server logs, ProfileBot read a few pages of your website. This page explains exactly what it read, why, and how to stop it.
You, or a colleague, have a listing on muniSERV, Canada's municipal professional network. Canadian municipalities post RFPs and requests for proposals here, and there are usually more than a hundred open at any time. Rather than expect you to read all of them, we try to surface the handful that actually fit your practice.
To do that well we need to know what you actually do, and a directory profile is often a line or two. Your own website says it far better. So ProfileBot reads a few of your public pages and drafts a short summary of your practice, which we then show you on your muniSERV dashboard to confirm or correct before it is used for anything.
We only read the sites of providers who are listed with us on a paid plan. We do not crawl the web at large, and we have no interest in anyone else’s site.
ProfileBot requests your robots.txt before anything else and honours the User-agent: * rules. If a page is disallowed, it is never requested: not retried, not fetched by another route. If your site blocks us, we record that we were blocked and move on.
At most six pages per site: your homepage, plus up to five pages it can tell are about your services or practice. It never leaves your domain, never follows links deeper than one level, and re-reads a site at most once every 90 days.
At least two seconds between requests, more if your robots.txt asks for more. Your stated crawl delay always wins over ours.
The user agent above is sent on every request, including the robots.txt request. We never disguise the bot as a browser, and we do not attempt to work around blocks, rate limits, or bot protection. If your site refuses an identified bot, that is an answer, and we stop.
We keep the short practice summary and the list of page addresses read, both of which we show the member. The page text itself is used to write that summary and then discarded. We do not copy your content, republish it, or store it.
Any one of these is enough, and none of them requires talking to us:
Turning it off does not affect your muniSERV listing or your matches. We simply fall back to the service categories on your profile, and tell you that is what we are using.
Questions about any of this go to info@muniserv.ca, and a person will answer. More about muniSERV →