The Records Labs crawler

What it does

Each Records Labs customer is an organization that chooses which websites to add, for example its own website, help center or product documentation. The crawler visits only those sites, the pages they redirect to and the files their pages link to, and only the parts the organization asked for: a whole site, a sitemap or a list of pages. It does not browse the web at large.

Reading a page can also load the images and other files that page uses, which may come from another host such as a content delivery network. To keep the knowledge current, the crawler checks the site again on a schedule, which the organization can change or turn off.

Some links to videos and video channels that an organization adds, on their own or inside a list of pages, are read with other tools. Those tools do not always send our user agent or come from the addresses below, and they may try another way when a request is refused. The rest of this page does not describe them.

How to recognize it

The crawler identifies itself with this user agent:

Mozilla/5.0 (compatible; RecordsLabsIngestion/1.0; +https://recordslabs.ai/ingestion)

It starts with Mozilla/5.0, like the user agents of many crawlers, so that sites that expect a browser-style user agent still answer it. It names the crawler and links to this page, and it does not claim to be a browser.

Every HTTPS request the crawler sends to a site an organization added is also signed with Web Bot Auth. The Signature-Agent, Signature-Input and Signature headers let a verifier check that the request came from Records Labs. Our public key directory publishes the keys used to verify those signatures. Signing identifies the crawler; it does not override your site's access rules.

The crawler, and the other Records Labs requests described on this page, come from these IP addresses:

• 152.55.176.240
• 152.55.177.181
• 152.55.177.192
• 162.220.232.251
• 162.220.232.252

These addresses come from our hosting provider and may also carry other companies' traffic, so please do not rely on the address alone. Check the user agent as well.

How to control it with robots.txt

When an organization adds your whole site, your sitemap or a list of your pages, the crawler reads your robots.txt first and follows it for every request it then makes to your site: each page, each redirect, each sitemap file, each linked file and each image. Some requests outside a crawl do not read robots.txt; see Other requests from Records Labs. Some video links are read with other tools. Files that a page on another site loads from your site can also be an exception, described under How fast it goes.

Write a group for RecordsLabsIngestion. If you have none, the crawler follows your rules for all crawlers (User-agent: *). The most specific rule wins, and * and $ work as in the robots.txt standard. Changes reach the crawler within a day.

If you write a RecordsLabsIngestion group, the crawler follows only that group and no longer reads your User-agent: * rules, as the robots.txt standard says. Copy into it any of those rules you still want the crawler to follow.

Keep the crawler out of one folder

User-agent: RecordsLabsIngestion
Disallow: /private/

Keep the crawler out of your whole site

User-agent: RecordsLabsIngestion
Disallow: /

Slow it down

User-agent: RecordsLabsIngestion
Disallow: /admin/
Crawl-delay: 10

Copy into this group any Disallow lines you have for all crawlers, like /admin/ here. With this group, the crawler waits at least 10 seconds between pages. Each image download, each page render and each second request for the same page also waits its turn, or is skipped when that turn is too far away. Request-rate: 1/20 works too: one page every 20 seconds. If you set both, the crawler uses the slower one. A delay longer than one day counts as one day.

Turn the crawler away with Content-Signal

User-agent: *
Content-Signal: ai-input=no

When the group the crawler follows, or a Content-Signal line at the top of the file before any User-agent line, says ai-input=no, the crawler treats it like Disallow: / and reads none of your pages. This stops future reads; it does not remove pages the crawler has already read. If you have a RecordsLabsIngestion group, put the line in that group too, or at the top of the file. The requests under Other requests from Records Labs do not read robots.txt, so this line does not stop them.

When robots.txt cannot be read

If your robots.txt cannot be reached, answers with a server error or asks the crawler to slow down (429), the crawler does not guess. It reads none of your pages, tries again a few times, and then stops for that round. If a firewall or bot check that we recognize blocks your robots.txt, the crawler reads none of your pages. If your robots.txt answers 404 or another plain 4xx code, the crawler treats your site as having no rules, as the robots.txt standard says.

The crawler also respects Content-Signal: search=no and Content-Signal: use=immediate, which does not permit keeping pages in a search index. use=reference permits indexing with source links, and ai-train=no does not stop this crawler because Records Labs does not train AI models on crawled content.

How fast it goes

The crawler requests one page at a time from each host name of your site, normally waiting about one second after each page before it requests the next one, or longer when your robots.txt sets a Crawl-delay or Request-rate. Files your pages link to, such as PDFs, wait their turn the same way. This holds even when several organizations add your site. Sitemap files are read one after another.

A page can bring a few more requests with it, right after it: a redirect on your site, the images it shows, sometimes a render in a browser engine (below), and now and then a second request for the same page. That second request happens when the crawler checks a page again and your server answers 304 Not Modified: about 1 time in 50, the crawler then asks for the full page once more, to make sure it really did not change. Without a Crawl-delay or Request-rate these requests do not wait a second of their own, and an image download can overlap with the next page's request. With one, each image download, each render and each second request for the same page waits its own turn, or is skipped when that turn is too far away.

Sometimes the crawler has to run a page's scripts, for example when the page's text or images only appear after its scripts run. It then loads the page in a browser engine that sends the same user agent and does not hide that it is automated. The browser engine loads the page and the files and data the page asks for, several at a time as a browser does, except the ones your robots.txt disallows. It reads another site's data only when that site allows it, as a browser does. If the page's scripts try to open another page, the browser engine does not go there; a page on your site found that way may be read later as its own page, in its own turn.

Files and data such a page uses from another host, such as a content delivery network's scripts, fonts and images, load as part of that page, the way a browser loads them: at that page's pace, with our user agent, not signed, and following that host's robots.txt only if the crawler has already read it.

If your server answers 429 Too Many Requests or 503 Service Unavailable, the crawler tries that page again a few times, at the pace above, and then leaves it for that round.

When your site refuses it

When your site refuses the crawler, it never sends that request again under another name. It does not switch to a browser user agent, to addresses outside the list above or to proxy services, and it does not solve CAPTCHAs.

If your site refuses almost every page we ask for, the crawler stops for now. It won't ask again until the next scheduled check, or until the organization that added your site tries again. That organization sees that your site turned us away, so they can ask you for access.

When a person opens your site

Some sites ask for a sign-in or a cookie choice before they show their pages. A person in the organization can open a site in a secure browser that Records Labs runs, for example to sign in or make that choice themselves. While that person browses, the requests are theirs: they use a standard browser user agent and come from the addresses above. Before the browser opens, a quick request with our user agent checks where the address leads. When the person saves that access, the crawler uses what was saved, such as the sign-in cookies, to read the site, still as RecordsLabsIngestion and still following your robots.txt and pace.

Other requests from Records Labs

Some requests from Records Labs are not part of a crawl, so they do not read robots.txt. These use our crawler's user agent:

• a file or data file that a person in the organization adds by its web address;
• the captions, media, images and other files of an online course or lesson that a person in the organization imports or captures;
• a quick check of where an address leads, before a person opens it in the secure browser described above;
• a test a person runs of the access they saved for a site;
• downloading a file again from its original address when we have no saved copy of it, including a file the crawler found during an earlier crawl, either to read it again (this can happen automatically, when an earlier reading of it was incomplete) or so that a person can open it in Records Labs.

When an organization imports an online course, Records Labs opens the course's pages in a browser engine with a standard browser user agent, and with a sign-in the organization saved if it provides one. Those page loads do not read robots.txt or follow the pace above. Neither do the caption, media and other files it then downloads with our user agent.

When a person in an organization asks a Records Labs assistant a question, the assistant may read web pages the question is about while it answers, such as a page the person named or linked. Those requests use the user agent RecordsLabs-AnswerFetch/1, come from the addresses above, and do not read robots.txt or follow the pace above.

How the content is used

Pages the crawler reads go into the knowledge base of the organization that added your site. That organization's people and assistants use them to answer questions, and can link back to your pages as the source.

Records Labs does not train AI models on crawled content. Each organization chooses whether the model providers that write its answers may train on data; by default they may not. Providers used to read documents and build the search index operate under their own terms.

Letting the crawler through a firewall

If your firewall or bot protection blocks the crawler and you want to let it in, allow requests that use our user agent and come from the addresses above. Matching both is safer than either one alone; verifying the signature also proves the request came from Records Labs. We are glad to help: write to us with your domain and the time of a blocked request.

Contact us

Questions about the crawler, or want it to slow down or stop on your site? Email support@recordslabs.ai with your domain and, if you can, the time of a request you saw.