Four Dartmouth-Area Contractors Let AI Crawlers In. Their Servers Threw Them Out Anyway.

A general contractor in Dartmouth does not get work from a website. He gets it from the guy whose kitchen he did in 2019, from the lumberyard counter, from the sign on the truck outside a job on Slocum Road. That is a real and durable way to run a business, and it has outlasted every marketing fashion of the last thirty years. Nothing in this article suggests otherwise.

But the referral now has a new middleman. When a homeowner in Padanaram types "who can reframe a porch in South Dartmouth" into ChatGPT instead of Google, something has to hand that assistant a set of facts about you. That something is your website — not as it looks in Chrome, but as it arrives at a crawler that does not run JavaScript, does not scroll, and does not wait.

So we asked exactly one question of 18 general contractor websites in and around Dartmouth. Not "is it pretty." Not "does it rank." Just: when an AI crawler knocks, what does it get?

What we measured, and how#

On 26 August 2026 we requested the homepage of each site six times from the same machine, changing only the user-agent string: ClaudeBot, GPTBot, PerplexityBot, Googlebot, an iPhone Safari browser, and an invented bot name that appears in no blocklist anywhere. That last one is the control, and it is the reason we can say what we say below. We then fetched /robots.txt for each, and parsed the raw HTML — the bytes the server actually sent, before any JavaScript runs — looking for four facts a person would need in order to hire someone: a phone number, a street address, hours of operation, and structured business data.

The 18 businesses came from the BBB's Dartmouth general contractor category, the One SouthCoast Chamber member directory, and organic search, cross-checked against each company's own site for a real street address. Firms that only rank for "Dartmouth" through out-of-area landing pages were excluded. Seventeen of the eighteen answered a browser on the first pass; the eighteenth timed out once and answered normally on retest. None of this is a story about dead websites.

The finding: robots.txt said yes, the server said no#

robots.txt is a note on the front door. It is polite, public, and entirely advisory. The server is the door.

On four of these five sites, the note and the door disagree.

Site N is the only honest row in that table. Its owner, or someone acting for its owner, made a decision: keep the generative-AI crawlers out, keep Googlebot in. The robots.txt says so and the server enforces it, right down to serving Googlebot a clean 200. Agree or disagree, it is coherent.

The other four made no decision at all. We know this because of the control column. We sent each of them a made-up bot name — an agent string that appears in no blocklist anywhere, belonging to nothing. All four served it a normal 200. This is not a security posture that catches unknown robots. It is a filter that recognises one specific published crawler name and slams the door on it, while waving an anonymous stranger straight through.

None of those four contractors typed that rule. It arrived with the hosting. We read the Server header on each: three of the five sit behind Cloudflare, one behind LiteSpeed, one behind OpenResty. That is what we measured, and it is where the measuring stops — we cannot see inside anyone's control panel, so we cannot prove which toggle did this. What we can say is that all three of those platforms ship AI-crawler blocking as an option that can be switched on in one click, often by someone who is not the business owner. A contractor who has never logged into a DNS panel in his life is now, silently, invisible to one of the assistants his customers are using.

The quieter finding: there was not much to read anyway#

Blocking the crawler is dramatic. What the other thirteen sites served up is more mundane and probably costs more work.

Of the 18 sites, 15 do not state their hours anywhere in the raw HTML. Nine have no street address in the raw HTML. Three have no phone number in it. Eight publish no structured data whatsoever — not a single machine-readable tag. The median site returned about 505 words of readable text; the thinnest returned 98.

Ninety-eight words. That site — Site I — also has no phone number, no address, no hours, no structured data, no working HTTPS, and a 403 for ClaudeBot. If an assistant is asked to name a contractor in Dartmouth, there is functionally nothing there to name.

One correction we owe our own tool. Our checker scores a site as having business schema only if it finds a type from a fixed list, and GeneralContractor was not on that list. Two sites we initially marked as missing structured data are in fact publishing proper GeneralContractor markup, complete with address, geo-coordinates and opening hours — better markup than several sites the checker passed. That is our error, not theirs, and the list has been widened. It is also a good illustration of why "the tool said so" is not the same as "it is so."

This is almost never a content problem. These businesses have the information; it is on the page, in a slider, in a footer widget, in a "Contact" tab that loads after the fact. It is painted on by JavaScript a fraction of a second after the crawler has already left with an empty page.

That research, published with Vercel in December 2024, found that none of the major AI crawlers render JavaScript at all — OpenAI's, Anthropic's, Perplexity's, ByteDance's. Twenty months on, our own results are consistent with it: the sites with the least raw text are the ones built to assemble themselves in the browser.

What a Dartmouth contractor does on Monday#

Cheapest first. The first one is free and takes four minutes.

1. Check your own front door. Free, 4 minutes. Go to yoursite.com/robots.txt in any browser. If you see a Disallow: / under GPTBot or ClaudeBot, that is a deliberate block someone set — decide whether you want it. If you see nothing, or the file 404s, that is fine; absence means allowed. Then, separately, call your web person or your host and ask one question: "is AI crawler blocking turned on for my site?" On Cloudflare it is a single toggle in the Security settings, and three of the five blocked sites we found are on Cloudflare. We cannot prove that toggle is what did it — but ask the question anyway, because on four of those five nobody involved appears to have known there was a block at all.

2. Put your hours and address in plain text in your footer. Free, 20 minutes. Not in a map widget. Not in an image of a business card. Typed text, in the HTML, on every page: company name, street address, town, ZIP, phone, and the hours you actually answer the phone. Fifteen of eighteen sites here are missing the hours. This is the single highest-value twenty minutes available to any contractor reading this.

3. Add structured business data. Free if you are on WordPress, an hour otherwise. Eight of eighteen publish none at all. It is a small block of code — GeneralContractor is the right type for most of you — that tells any machine, unambiguously, your name, service area, phone and hours. Google's own guidance is worth reading before you buy anything: it says plainly that you do not need special AI files or Markdown to appear in its generative results. The basics are the work.

4. Get the facts out of JavaScript. This one costs money. If your homepage serves 98 or 141 words of raw text, the fix is structural and a developer has to do it. But do steps 1 through 3 first — on most of these sites they close the majority of the gap for the price of an afternoon.

What we could not measure#

We tested from one machine, in one place, on one day. A 403 to ClaudeBot's user-agent from our IP is strong evidence of a user-agent rule — the same server returned 200 to a browser, to Googlebot, and to an invented bot name from that identical IP, and the result reproduced across three separate runs. It is not proof that Anthropic's production crawler IPs are treated the same way, and we do not claim it is.

We did not test every page, only the homepage. We read each server's Server header but we cannot see anyone's hosting control panel, so the specific setting responsible for each block is inferred, not measured. We could not verify Facebook or Instagram presence for any of these firms server-side, so we did not look. And three of the twenty-one contractors we identified have no website of their own at all — only a chamber listing or a BBB profile. They are excluded from every number above, because a business with no site is a different article.

We also do not know how many of these blocks were switched on deliberately by a web developer acting for the owner. Site N's was. We would guess the other four were not, because the control test shows the rule is doing something no one would choose on purpose. A guess is what that is.

Anonymised throughout. These firms did not ask to be graded, and several of them do excellent work; the problem here is a default setting on a hosting account, not a lack of craft.

If you want to know which row in that table is yours, I will tell you for nothing — it takes me about ten minutes and I do not need access to anything. Avalon Partner builds and fixes websites for South Coast trades.

774.559.8992 · Joshua.Amado@AvalonPartner.com