We Tested 20 Charlotte Marketing Agency Websites for AI Search Readiness (2026)

Charlotte marketing agency websites are mostly open to AI search crawlers, but not entirely: as of October 15, 2026, 17 of the 20 agency homepages we measured let every major AI crawler in, 3 of 20 still return an error to OAI-SearchBot, the crawler that decides whether a site appears in ChatGPT search, and none block anything in robots.txt.

October 2026 update

We re-measured the same 20 homepages on October 15, 2026, with the same scripts and user agents. Two agencies that had returned HTTP 400 to GPTBot now serve it, so server-side refusals fell from 5 of 20 to 3 of 20. The three homepages refusing OAI-SearchBot in September still refuse it.

Structure barely moved: of the 16 homepages scored both times, none changed by more than 4 points. The fifth first-100-words pass is CGR Creative, scored for the first time now that its server answers GPTBot.

The headline findings

Twenty Charlotte-area agency homepages, measured on October 15, 2026, produced these numbers: 0 of 20 restrict an AI crawler in robots.txt, 3 of 20 refuse at least one AI crawler at the server, 18 of 18 fetchable homepages deliver real text in raw HTML, 19 of 20 carry JSON-LD, and 5 of 18 pass the first-100-words test.

Who blocks which crawler

Nobody in the sample blocks an AI crawler in robots.txt: all 20 files let the six crawlers fetch the homepage. Access is lost one layer down. Three of 20 homepages return HTTP 403 to at least one crawler's documented user agent while serving 200 to a browser, down from five in September.

Two of the three refuse all six user agents; one refuses OAI-SearchBot, ClaudeBot and Claude-SearchBot but serves GPTBot. The two that dropped off the list had refused only GPTBot, the training crawler, while serving every search crawler. That is the separation OpenAI's crawler documentation describes: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

Three agencies wrote deliberate rules: Overtop Media, Knowmad Digital Marketing and Idea Forge Studios name the AI crawlers in robots.txt and allow them. The other 17 files allow everything by default.

What an AI crawler actually sees

Every homepage we could fetch as GPTBot delivers its text in raw HTML. Across the 18 fetchable homepages the median is 1,150 words, the range is 538 to 5,697, and none falls under 500 words, the region where a client-rendered page starts to look empty to a crawler that does not run JavaScript.

That matters because, per Lantern's analysis updated in June 2026, no major AI crawler renders JavaScript. Med Rank Interactive and Overtop Media exceed 5,000 words of raw text on both dates.

Structure and the first 100 words

Structure is where the sample is weakest. On the extractability script, which grades a page the way AI retrieval reads it, the 18 scored homepages have a median of 51.5 out of 100 and a range of 29 to 88. Only 5 of 18 pass the first-100-words check; 8 are partial and 5 fail.

The check asks whether the first 100 words of body text contain a definition or a sentence on the page's own topic, plus a number, without a pronoun opener. Surfer's June 2026 analysis of 100,000 citation placements across 10,000 AI Overviews found 38% of citations come from those words. Med Rank Interactive scored 88; Kashmer Interactive, The Social Rook, Organic Clicks and CGR Creative also pass.

Schema, titles and canonicals

Structured data is nearly universal and mostly generic: 19 of 20 homepages carry JSON-LD, 16 declare an Organization or Corporation, 7 a LocalBusiness or ProfessionalService, 2 a FAQPage, and 5 a Person node anywhere in the graph. Google's guide is explicit: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add."

Titles run long: the median tag is 64 characters, 13 of 20 exceed 60, and 11 of 20 contain the word Charlotte. Canonicals are clean on 19 of 20; one points at a different hostname, as it did in September.

What this means if you are choosing a Charlotte agency

Ask any agency to fetch its own homepage with the OAI-SearchBot user agent before you sign. An agency whose server turns that crawler away cannot be expected to know the rule for your site, and as of October 2026 the same 3 of 20 as six weeks earlier are in that position.

Read the numbers with care: a 403 to a user-agent string is often a hosting firewall's default, fixed in minutes once noticed, as two agencies here did between runs. Whether the agency will notice on your site is a question for the sales call.

Mirastart's own numbers

Mirastart's homepage, measured the same way on October 15: HTTP 200 to all six crawlers, a robots.txt that names and allows all six, 2,446 words of raw HTML as GPTBot, an extractability score of 94 with a first-100-words pass, JSON-LD including a Person node for the founder, and a 57-character title containing Charlotte.

In September the same homepage scored 65 and carried no Person node; it has been rebuilt since, and the founder's Person schema now sits on the homepage and the About page. Read the 94 with that disclosure: Mirastart wrote the scoring script, so the score measures how closely a page follows our own checklist.

Method

The study measured 21 homepages on September 3 and October 15, 2026, with stdlib Python scripts at one request per second, raw HTML only, no JavaScript rendering. The sample is 20 agencies with a Charlotte-area office, drawn from our two Charlotte agency lists and Expertise.com's Charlotte pages, plus Mirastart as a separately reported 21st row.

• Word count: homepage fetched as GPTBot; script, style, noscript, template and SVG blocks removed; visible words counted.
• Crawler access: robots.txt parsed with Python's urllib.robotparser for six tokens (no file counts as allowed), then the homepage fetched once under each vendor's documented user-agent string; non-200 responses re-checked once with 5-second gaps, and two October HTTP 429 rate limits that cleared on re-check count as served. Only the string was tested, not the crawlers' published IP ranges.
• Structure: scripts/extractability.py scored the raw HTML and returned the first-100-words verdict; JSON-LD, title and canonical were read from the same HTML. The script was revised on September 10, so October's pages were also scored with the September 3 version: the two versions agree within 4 points on every homepage.
• People: a Person node in JSON-LD or a visibly named founder on the homepage or the page at /about (then /about-us); a /about that redirected to the homepage or a blog post was recorded as unclear. This column was checked by hand in September and not re-verified in October, except for Mirastart's own row.
• Limits: homepages only, two measurements per site six weeks apart. Clutch's Charlotte lists returned HTTP 403 and were not used; Expertise.com's pages were last updated September 1, 2026. Raw data: the October CSV (21 rows) and the September CSV, both linked under Sources.