What it is
SiteRefactorBot is the automated client SiteRefactor uses to analyze a website when a visitor submits its address. The analysis looks at public signals only: which technologies the site uses, how its pages are structured, SEO basics, HTTPS and security headers. The results feed a report with migration options and indicative costs.
The crawler does not run continuously and does not build a search index. It visits a site only when someone requests an analysis of it, and the same site can be analyzed only a limited number of times within a short period (by default, three analyses per site per ten minutes).
User agent
Requests from the crawler identify themselves with this user-agent string:
Mozilla/5.0 (compatible; SiteRefactorBot/1.0; +https://siterefactor.com/bot)The robots.txt token is SiteRefactorBot.
What it fetches
- robots.txt, first, to check whether it may proceed and which paths it should avoid.
- The homepage, including its response headers and the HTTPS/TLS connection details. A single extra request checks whether plain HTTP redirects to HTTPS.
- The XML sitemap, to estimate the number of pages and discover URLs.
- Up to 20 internal pages per analysis, discovered through homepage and navigation links and the sitemap.
- One rendering of the homepage in a sandboxed headless browser (Chromium), to detect JavaScript frameworks and take a screenshot. The browser loads the page's scripts, styles and images much as a visitor's browser would, within a fixed request budget and with audio and video downloads blocked. It starts with no cookies or stored data.
Security headers and HTTPS settings are checked passively, by reading the responses to these ordinary requests.
What it never does
- It only sends GET requests and never submits forms. During the single homepage rendering, your page's own scripts run as they would for any visitor, but every request they attempt other than GET, HEAD or OPTIONS (for example POST beacons or form submissions) is blocked inside the sandbox before it leaves our infrastructure.
- It never logs in, creates accounts or accesses content behind authentication.
- It never tries to bypass bot protection, CAPTCHAs or paywalls. If a site blocks it, the report says so.
- It does not scan ports or probe for vulnerabilities.
- It does not analyze private or reserved network addresses.
Request rate
- At most 2 concurrent requests to your site.
- At least about 400 ms between requests.
- A
Crawl-delaydirective is honored up to 5 seconds. If you set a crawl delay above 3 seconds, the analysis also inspects fewer pages (at most 10). - Each analysis has an overall time limit, after which crawling stops.
Blocking or limiting it
To opt out of analyses completely, add this to your robots.txt:
User-agent: SiteRefactorBot
Disallow: /If the homepage is disallowed, page inspection is skipped entirely and the report explains why. The same happens if your robots.txt returns a server error (5xx). If there is no group for SiteRefactorBot, the rules in your User-agent: * group apply.
To keep it out of specific sections only, or to slow it down:
User-agent: SiteRefactorBot
Disallow: /internal/
Crawl-delay: 2Disallowed URLs are skipped and the report notes how many were left out.
PageSpeed Insights requests
Performance results come from Google PageSpeed Insights. Google fetches the page separately, using its own infrastructure and its own user agent. Those requests do not come from SiteRefactorBot, so robots.txt rules for SiteRefactorBot do not apply to them; they are governed by Google's policies for that service.
Contact
If you believe the crawler is misbehaving, or you want to report abuse, email biuro@gacode.pl with the affected domain and approximate times of the requests. Our crawler and its browser connect from the IP address 54.36.162.208. PageSpeed tests are run by Google from Google's own infrastructure. More about how we handle data is in our Privacy Policy.