# SEO and AI-search (AEO) data without DataForSEO

DataForSEO is switched off (gateway setting `disabled_providers`). Every dataset we used to buy from it has a cheaper
FetchAPI endpoint, listed below with:
- what DataForSEO cost us (measured from our own billing log, Oct 3-26 2026, $250 total);
- what the replacement costs (measured on live runs, 2026-11-13);
- how close its numbers are.

Call the `/api/v1/<job>` endpoints with header `x-api-key`. Each job picks the cheapest working route and falls back on
its own, and each answer's `meta` shows `provider`, `tried` and `cost_usd`. Apify runs settle a moment later
(`cost_settles_later`). Endpoints marked "owner keys" use scraped Semrush, Ahrefs, Moz or SpyFu data: internal use only,
and don't republish it.

## Which endpoint for which data

| You need | Call | Replacement cost | DataForSEO cost (measured) |
|---|---|---|---|
| Search volume, CPC, 12-month trend | `GET /api/v1/keywords/volume?keywords=a,b,c` | **$0.0005-0.0007 a keyword** up to 250 keywords (Ubersuggest); $0.19 a batch of up to 1,000 above that (Google Keyword Planner) | $0.09 a batch (`google_ads/search_volume`) |
| Keywords a domain ranks for (keyword, position, volume, CPC, traffic, URL) | `GET /api/v1/domain/keywords?domain=x.com&limit=10` | **$0.0045 for the top 10** (Semrush, owner keys); $0.002 a keyword for 11+ (Semrush); `limit=5` = Similarweb's top 5, no positions, $0.0003 | $0.015 a domain (`ranked_keywords`) |
| Google top 10 + **AI Overview** text and cited URLs | `GET /api/v1/google/search?q=...&ai_overview=1` | **$0.00025 a keyword** (litescrape), then $0.002 (scrape.badger), then $0.004 (Apify + add-on) | $0.002-0.005 a page |
| Google top 10 only | `GET /api/v1/google/search?q=...` | free daily allowance (TinyFish, owner keys), then $0.0005 | $0.002-0.005 |
| ChatGPT answer + cited sources | `GET /api/v1/ai/answers?engine=chatgpt&prompt=...` | **$0.004** (Apify, one run a prompt in parallel, 70-330 s; a prompt not answered in 150 s goes to scrape.badger), $0.0072 (scrape.badger, ~20 s, more citations) | $0.039 a prompt |
| Perplexity answer + sources | `...engine=perplexity` | $0.0072 (scrape.badger), $0.01 (Apify, Sonar API, 10 citations) | $0.006 |
| Google AI Mode answer + sources | `...engine=ai_mode` | **$0.00025** (litescrape), $0.003 (Apify) | $0.004 |
| Gemini answer + sources | `...engine=gemini` | $0.004 (Apify, one run a prompt in parallel, 1-2 min) | $0.038 |
| Claude answer (no sources) | `...engine=claude` | $0.01 (Claude Haiku 4.5 through Anthropic's API via cheapapi: **no web search, no citations**, what the model remembers; ~10 s) | – |
| Microsoft Copilot answer + sources | `...engine=copilot` | $0.0025 a prompt + $0.009 a call (clearpath, Bing AI mode, GPT-5 mini; all prompts in one run, ~25 s) | – |
| **AI search volume** (how often people ask AI about a term) | `GET /api/v1/keywords/ai-volume?keywords=a,b` (up to 50) | $0.015 a call + $0.0005 a term (cheapapi; short head terms, long prompts return 0) | $0.01+ (`ai_keyword_data`) |
| **Whole AEO block for a brand** | `GET /api/v1/aeo/brand?brand=X&domain=x.com&category=...` | lean: ~$0.14 first brand in a category, ~$0.05 each further brand, ~$0.002 a re-run within 7 days; deep: ~$0.22 (see below) | – |
| Keyword difficulty | `GET /api/v1/keywords/difficulty?keywords=...` | $0.008 a keyword (Ahrefs KD + backlinks needed), $0.0045 + $0.025 a batch (wrenfield) | $0.012 a batch |
| Referring domains, backlinks, spam score | `GET /api/v1/web/backlinks?domains=a.com,b.com` | ~$0.0012 a domain + ~$0.035 a run (up to 1,000 domains; DataForSEO-backed reseller on Apify) | $0.024 a domain (`backlinks/summary`) |
| **Backlink history (monthly since 2019) and dated links** | `GET /api/v1/web/backlinks-history?domains=...&months=24`; `web/backlinks?top=100&list=referring_domains` | $0.071 a domain for 24 months; dated rows $0.0008 each (guide: https://fetchapi.co/docs/backlinks.md) | $0.025 a domain (`backlinks/history`) |
| Top linking domains (with Moz DA, spam score) | `...web/backlinks?domain=x.com&top=100` | ~$0.009 for 100 (Moz via radeance, owner keys) | $0.033 (`referring_domains`) |
| US Google organic traffic + monthly history since 2019 | `GET /api/v1/web/organic?domains=a.com,...` (up to 100) | **$0.03 a run** for 1-100 domains (SpyFu, owner keys) | $0.012 a batch (`bulk_traffic_estimation`) |
| SEO competitors (shared Google keywords) | `GET /api/v1/company/competitors?domain=x.com` | $0.015 (SpyFu), after Crunchbase (~$0.008) | $0.014 |
| Does a brand run Google ads, how many creatives | `GET /api/v1/ads/all?brand=Linear&domain=linear.app` | **free**: `google.runs_ads`, `google.creative_count` from Google's Ads Transparency Center; big advertisers get a range (`creative_count_upper`) | $0.002 a query |
| Domain authority | `GET /api/v1/web/authority?domains=...` | ~$0.0005 a domain (Ahrefs DR, else Moz DA) | $0.024 |
| AI Overview / chatbot mention counts | count from stored `ai_overview.cited` and `ai/answers` citations, in a view | $0 extra | $0.10-0.28 a query (`llm_mentions`) |
| Keyword ranking history | store `google/search` positions daily (history starts the day you start) | free allowance / $0.0005 a keyword | $0.13 a domain |

**Still not replaced:**
- **SERP screenshots:** the Apify Google route keeps an HTML snapshot (`htmlSnapshotUrl`); litescrape sells screenshots at $0.15 per 1,000 (untested).
- **Past ranking history** (positions before we started tracking).

## Data quality: benchmark against DataForSEO-equivalent data (2026-11-13)

Raw numbers and scripts: `.context/seo-benchmark/` (CSVs per test, one JSON per run). Cost of the benchmark: $1.79.
- **Baseline:** DataForSEO-backed resellers on Apify (santhej for keywords, cheapapi for backlinks and traffic). Our own
  DataForSEO account was not called.
- **Positions:** checked against live Google.

### Search volume (30 keywords: head, mid, long-tail, brand, and 10 of resellbot.com's)

| Source | vs Google Keyword Planner: median ratio | within ±30% | rank correlation | CPC within ±30% |
|---|---|---|---|---|
| **Ubersuggest** (`keywords/volume`, ≤250 keywords) | **1.00** | 86% | 0.99 | 60% |
| DataForSEO-equivalent | 1.00 | 96% | 1.00 | 83% |
| Semrush | 0.66 | 22% | 0.96 | 24% |

- **Ubersuggest** uses Google's volume buckets. Its median ratio is 1.00 for every kind of keyword, so it can replace
  DataForSEO for volume. Its CPC is looser.
- **Semrush** orders keywords the same way, but its volumes run about a third lower (about 0.45x on head and mid terms,
  1.0x on brand terms). Never mix Semrush volumes with Google or Ubersuggest volumes in one list.
- **True long-tail** ("best crm for small real estate team") came back null from every source.
- **Google Ads vs Semrush can differ by 25x on a fast-rising term.** "sold comps": Planner 12,100, Ubersuggest 8,100,
  Semrush 480. Report the source with the number.

### Ranked keywords (resellbot.com, linear.app, sold-comps.com, posthog.com, calendly.com)

| Source | Positions vs live Google: exact | off by 1-2 | wrong | not ranking | cost per domain |
|---|---|---|---|---|---|
| **Semrush top 10** (`limit<=10`) | **19/26** | 4 | 2 | 1 | $0.0045 |
| Semrush 30 rows (`limit>10`) | 17/29 | 6 | 6 | 0 | $0.060 |
| DataForSEO-equivalent (30 rows) | 16/25 | 5 | 3 | 1 | $0.070 |

- **Big sites (linear.app, posthog.com):** every source was exact on every check.
- **Small sites (resellbot.com, sold-comps.com):** every source was only 20-45% exact. Treat small-site positions as
  approximate.
- **Semrush and DataForSEO pick different top-10 keywords** (overlap 0.27), so a site's "top keywords" depend on the index.
- **Similarweb's free top 5** has no positions and overlaps little with either; use it only as a free "top terms" glance.

### Backlinks (referring domains)

| Domain | DataForSEO-equiv. | Ahrefs | Moz |
|---|---|---|---|
| linear.app | 11,242 | 17,501 | 15,601 |
| posthog.com | 17,598 | 35,262 | 27,773 |
| calendly.com | 665,853 | 883,954 | 721,724 |
| resellbot.com | 208 | 1,635 | 214 |
| sold-comps.com | 76 | 1,299 | 261 |

- **Large sites:** Moz is within 1.1-1.6x of DataForSEO, Ahrefs within 1.3-2x.
- **Small sites:** Ahrefs counts 8-17x more, so the index decides the number. Compare domains only within one source.
- **Top-linker lists barely overlap** between indexes (0-3 of 20 shared), partly because each sorts differently.
- **Ahrefs DR** from `web/authority` matched the Ahrefs backlink tool on all 5 domains.

### Organic traffic (US Google clicks, Oct 2026)

| Domain | SpyFu (`web/organic`) | DataForSEO-equivalent | ratio |
|---|---|---|---|
| resellbot.com | 2,133 | 1,925 | 1.11 |
| linear.app | 26,250 | 31,277 | 0.84 |
| posthog.com | 6,230 | 10,636 | 0.59 |
| calendly.com | 94,960 | 166,856 | 0.57 |
| sold-comps.com | 17 | 643 | 0.03 |

- **SpyFu** is within 0.57-1.11x on 4 of 5 sites, at about the same cost when batched (100 domains a run). It fails on very
  small sites.
- **Similarweb organic** (global visits x organic share) runs 17-45x higher. It counts branded and worldwide search, so
  don't substitute one for the other.

## AEO block in one call: aeo/brand

`GET /api/v1/aeo/brand` asks every engine the same buyer prompts in parallel and scores the answers for one brand. It
runs `ai/answers` per engine, `google/search ai_overview=1` per prompt, `keywords/ai-volume` and `web/traffic-bulk ai=1`
for you, and adds up their cost.

**Answers are shared through a cache** (table `ai_answer_cache`, migration 0056).
- Every answer is stored per (engine, normalised prompt, country) and reused for 7 days. AI volume is stored per term
  for 30 days.
- Prompts come in two kinds:
  - **Category prompts** don't name the brand ("best note taking app"). They are bought once a week per category, and
    every brand in that category is scored from the same answers.
  - **Brand prompts** name the brand ("Notion alternatives"). They are bought for each new brand.
- `fresh=1` asks again, and the new answers are still written to the cache.

| mode | engines | prompts | cache | first brand in a category | each further brand (category warm) | same brand again within 7 days |
|---|---|---|---|---|---|---|
| `lean` (default) | chatgpt, perplexity, ai_mode, ai_overview, copilot | 5 category + 3 brand | reads and writes | **$0.14** (live: Notion $0.142, Resellbot $0.149) | **$0.054** (live: Obsidian after Notion) | **$0.002** (live: Notion, Evernote) |
| `deep` | all 7: the five above + gemini and claude, 5 prompts each | 5 category + 5 brand | writes only (fresh answers) | **$0.22** (live: Notion $0.223) | $0.22 | $0.22 |

- A lean report for a further brand costs about $0.054. Its 3 brand prompts break down as Perplexity $0.022, Copilot
  $0.0165 (its $0.009 run fee included), ChatGPT $0.012, and AI Mode + AI Overview $0.0015, plus AI traffic about $0.003.
- Perplexity (scrape.badger, $0.0072 an answer) is the biggest line in both modes. The $0.003 "Perplexity" actor tested
  as an alternative (dev00/perplexity-ai-search-api) returns a canned template answer with Google News links, not
  Perplexity's answer, so it is not used.
- `engines=`, `prompts=` and `fresh=` override either mode.

```bash
curl -s -G "https://fetchapi.co/api/v1/aeo/brand" -H "x-api-key: $FETCHAPI_KEY" \
  --data-urlencode "brand=Resellbot" --data-urlencode "domain=resellbot.com" --data-urlencode "category=ebay sold comps tool"
```

**Params**

| Param | Meaning |
|---|---|
| `brand` (required) | the brand name, matched case-insensitively as a whole word (or its domain) |
| `domain` (required) | the brand's site; cited URLs on it (or its subdomains) are the brand's own citations |
| `category` | what it sells ("note taking app"); builds the prompts and the AI-volume head terms |
| `prompts` | up to 10, newline separated. Default 5 category prompts (`best {category}`, `best {category} for small teams`, `best free {category}`, `which {category} do you recommend for a beginner`, `most popular {category} right now`) and 3 brand prompts (`{brand} alternatives`, `is {brand} worth it`, `{brand} vs {first competitor}` or `{brand} review`). Deep mode adds `{brand} vs {second competitor}` or `{brand} pros and cons`, and `{brand} pricing and plans` |
| `competitors` | comma list, `Name` or `Name\|domain.com`. Default: the 5 brands the answers name most (words of the category, like "eBay", don't count) |
| `mode` | `lean` (default) or `deep` (see the table above) |
| `engines` | default (lean) `chatgpt,perplexity,ai_mode,ai_overview,copilot`. `gemini` and `claude` are asked at most 5 prompts each (3 category + 2 brand) |
| `fresh` | `1`: ask every engine again instead of reusing answers from the last 7 days (default on in deep mode) |
| `ai_traffic` | `1` (default): Similarweb AI-chatbot referrals for the brand's domain and competitor domains (given, or a cited domain named like the competitor); `0` off |
| `ai_volume` | `1` (default): AI search volume for `{category}` and `best {category}` (cached 30 days, so once a month per category); `0` off |
| `country` | default US |

**Output**

- `mode`, `fresh`: as run.
- `engines.<engine>` has one block per engine:
  - `prompts`: prompts sent to it. `from_cache` and `asked_live` say how many answers came from the cache and how many
    were bought. `cost_usd`: this report's spend on the engine.
  - `answers`: answers received. For `ai_overview` this counts only prompts where Google showed an overview (`overview_shown` of `prompts_asked`).
  - `mentioned`: answers naming the brand. `visibility` = mentioned / answers.
  - `avg_position`: the brand's mean rank, by first appearance, among the brands each answer names (1 = named first).
  - `share_of_voice`: answers naming the brand divided by the sum, over brand + competitors, of answers naming each.
  - `citations`: cited URLs. `own_citations`: those on the brand's domain. `own_pages_cited`: [{url, prompt}].
  - `top_cited_domains`: [{domain, n}], top 10.
  - `competitors_ahead`: [{name, n}], the brands named before the brand. When the brand is absent, the first 3 named count.
  - `rows`: one per answer, [{prompt, branded_prompt, mentioned, position, brands_named, citations [{url, domain}]}].
- `overall` pools every engine's answers:
  - `visibility`, plus `visibility_unbranded`, which leaves out prompts that name the brand (those always mention it).
  - `avg_position` and `share_of_voice`, plus `share_of_voice_by_brand`.
  - `citation_share`: own citations / all citations.
  - `own_pages_cited`: unique URLs, with the engines and prompts that cited each.
  - `competitors_ahead`: top 5. `engines_mentioning`: engines with at least one mention.
- `ai_traffic` is the Similarweb part:
  - The brand's `share_of_traffic` from AI chatbots, AI `visits` and `chatbots` [{chatbot, share_pct, rank}].
  - The same for `competitors`.
  - `status: pending` means Similarweb had not answered within the wait; call `web/traffic-bulk` again later.
  - Licence: internal use.
- `ai_demand`: [{term, volume (last month), avg_monthly, change_pct_year, trend (12 months, oldest first)}].
- Cost and housekeeping:
  - `cost_usd` adds the known costs to the routes' estimates for Apify runs, which settle a few minutes later. `cost_known_usd` is the part already known.
  - `calls[]` lists every sub-call with its route, usd and seconds.
  - `notes` and `seconds` describe the run.

**Brand names in answers:**
- The dictionary has three parts: the brand, the competitors, and product names pulled from bold text, list items,
  headings and table cells that recur in 2 or more answers. Generic headers like "Pricing" or "Best for" are dropped.
- A longer name ending in a shorter one counts as the shorter one ("Microsoft OneNote" = "OneNote").
- The dictionary is good enough to rank and count brands, but not exact. Check `rows[].brands_named` before quoting a position.

**Time (live 2026-11-13):**
- A cold report takes 4-5.5 min. ChatGPT on Apify sets the pace: 70-330 s a prompt. A prompt with no answer after
  150 s is re-asked on scrape.badger, and that re-ask is not in the estimate.
- A fully cached report takes about 2 min, all of it the AI-traffic wait.

**Limits:**
- `claude` is the API model with no web search, so it never cites. It measures what Claude remembers, not what
  claude.ai would find.
- `copilot` is Copilot's Bing AI mode. Its chat modes returned only an "Update Copilot" notice on 2026-11-13.
- The AI Overview engine counts only prompts Google showed an overview for.
- Each prompt is asked once per engine, so answers vary from run to run. Compare runs over time, not single numbers.
  Answers from the cache are up to 7 days old.
- `ai_traffic` came back `pending` for every domain on 2026-11-13 (notion.com included). In that window the chg lane of
  `web/traffic-bulk ai=1` answered 503, and that job is unchanged here.
- ChatGPT costs $0.004 an answer on Apify; a prompt that falls back to scrape.badger costs $0.0072, which
  `cost_usd` does not yet include.

## Recipes

**Daily SEO/AEO scan (1,000 money keywords + 100 ChatGPT + 20 Perplexity prompts): about $1 a day** (was ~$10+ on DataForSEO)

```bash
# Google top 10 + AI Overview citations, one keyword per call (~$0.00025 each, ~20 s; run ~10 in parallel)
curl -s "https://fetchapi.co/api/v1/google/search?q=best%20crm%20for%20small%20business&ai_overview=1" -H "x-api-key: $FETCHAPI_KEY"
# -> results[] {position, url, domain, title}, ai_overview {text, cited [{url, domain, title}]} or null

# Chatbot answers, up to 20 prompts a call (newline separated)
curl -s -G "https://fetchapi.co/api/v1/ai/answers" -H "x-api-key: $FETCHAPI_KEY" \
  --data-urlencode "engine=chatgpt" --data-urlencode $'prompts=best crm for a small business\nbest ebay sold comps tool'
# -> per prompt {engine, prompt, answer, citations [{url, domain, title}]}
```

Store every answer. Count AI Overview and chatbot mentions per domain in a view, not as stored numbers.

**Keyword list sizing:** `keywords/volume` with up to 250 keywords per call ($0.0005-0.0007 each). Above about 300, send up
to 1,000 per call: the job switches to the $0.19 Keyword Planner batch.

**Top keywords block on a report:** `domain/keywords?domain=x.com&limit=10` ($0.0045) gives keyword, position, volume, CPC,
traffic and URL.

**Backlink block:** use `web/backlinks?domains=...` for counts and spam score on many domains at once. Add `top=20` for the
top linkers of one domain.

**Organic size and growth for many sites:** `web/organic?domains=` takes 100 per call for $0.03 and includes monthly
history. For a category, take its top 1,000 sites from Similarweb: 10 calls, $0.30.

## Notes

- **Owner keys only** (scraped data, scope `seo-scraped`): Semrush top 10, Moz top linkers, the Ahrefs backlink
  fallback and SpyFu. Other keys get the next route.
- **DataForSEO-backed resellers** are still used: santhej (ranked keywords fallback) and cheapapi (backlink counts). They
  resell DataForSEO data through Apify at lower prices, without our DataForSEO account.
- **Bright Data's** Google route needs a new token in /admin (it returns 401 Token expired).
- **Research files** with every provider found (about 130 new routes) are in `staging/` (aeo-*, seo-keywords-*,
  seo-backlinks-*), not yet ingested.