# Bulk company enrichment pass-throughs

Use these paths for large company lists. Do not loop over the ordinary single-domain endpoints until you
have checked the native batch route, remaining quota, pagination cost and cache. Both workflows below are
resumable and intended for owner/internal use; Similarweb and Crunchbase-derived data have redistribution
restrictions.

## Similarweb traffic: native batches of 1,000 domains

Provider: `rapidapi-similarweb-scraper1`. Submit no more than 1,000 normalized DNS domains per request:

```bash
curl -X POST \
  "https://fetchapi.co/api/gw/rapidapi-similarweb-scraper1/api/similarweb/batches" \
  -H "x-api-key: $FETCHAPI_KEY" \
  -H "content-type: application/json" \
  -d '{"domains":["stripe.com","supabase.com"]}'
```

Persist the returned `id` before polling. A batch submission is one RapidAPI request, regardless of whether
it contains one or 1,000 domains. Poll no more often than every 30 seconds. Results page at 100 records, so
follow `pagination.next_after` until it is null:

```bash
curl \
  "https://fetchapi.co/api/gw/rapidapi-similarweb-scraper1/api/similarweb/batches/$BATCH_ID?limit=100&after=0" \
  -H "x-api-key: $FETCHAPI_KEY"
```

Each status/result GET is also one request. The reusable resumable collector is:

```bash
node scripts/load_inc5000_traffic.js
```

It filters malformed domains, submits 1,000-domain batches, stores batch IDs before polling, follows
100-record result pages and upserts one cached traffic row per domain. Output includes monthly visits,
three-month history, ranks, engagement, traffic sources, countries and keywords.

## Company and funding enrichment: true bulk first, Crunchbase fallback

For owner jobs, use Recent Funding's private bulk endpoint once it is live. It accepts up to 1,000
normalized websites and builds the company dumps with batched database queries. Verify an authenticated
test returns `200` before starting a large job; if the route is not deployed yet, pause the job instead of
falling back to tens of thousands of single-domain requests:

```bash
curl -X POST \
  "https://recentfunding.com/api/admin/companies-by-websites" \
  -H "Authorization: Bearer $RECENTFUNDING_ADMIN_TOKEN" \
  -H "content-type: application/json" \
  -d '{"websites":["stripe.com","supabase.com"]}'
```

Never put the owner token in code, logs or chat. It belongs in local/runtime secrets. Public Recent Funding
API keys cannot call this route. Preserve the input order and cache both found and not-found results. Each
matched company dump can include the raw company row, accounts, assets, discoveries and signals, people,
profiles/photos, investors, rounds and round-investor links, evidence/provenance, landscape assignments,
acquisitions, rejected rounds, merge history and photo rejects.

The single-domain endpoint remains useful for spot checks:

```bash
curl \
  "https://recentfunding.com/api/admin/company-by-website?website=supabase.com" \
  -H "Authorization: Bearer $RECENTFUNDING_ADMIN_TOKEN"
```

The resumable bulk collector is:

```bash
node scripts/load_inc5000_funding.js
```

It sends no more than 1,000 domains per request, appends every complete result to
`.context/inc5000-funding/bulk-responses.jsonl`, and resumes from that checkpoint. The production bulk
route was verified at about 3.2 seconds cold and 0.8 seconds warm for 1,000 inputs. The former
single-domain sweep measured only about 3.4 domains/second and was stopped; do not recreate it.

For a website that Recent Funding does not know, use the existing Crunchbase pass-through. Crunchbase has
no verified multi-company batch route in this provider; cache the resolved slug so it costs at most two calls
the first time and one thereafter:

```bash
# 1. Resolve name or domain to a Crunchbase organization slug.
curl \
  "https://fetchapi.co/api/gw/rapidapi-crunchbase-extractor-full-api3/autocomplete?q=supabase.com&type=org&limit=10" \
  -H "x-api-key: $FETCHAPI_KEY"

# 2. Fetch the rich company record. Keep fresh=false: fresh=true costs five requests.
curl \
  "https://fetchapi.co/api/gw/rapidapi-crunchbase-extractor-full-api3/company/details?id=supabase&fresh=false" \
  -H "x-api-key: $FETCHAPI_KEY"
```

`company/details` already contains the profile, website, headquarters, categories, employee range, founders,
key employees, investors, round slugs/types/leads, acquisition/exit/IPO status, products, tech stack, press,
social links and a Semrush traffic estimate. Do not fan out to `company/people`, `company/financials` or
`company/tech` unless the caller specifically needs fields absent from `company/details`.

## Required preflight for another agent

1. Normalize and deduplicate domains before submission.
2. Run a 2-10 record test and compare provider quota before/after.
3. Persist batch IDs or append each completed domain before starting another request.
4. Cache successful and not-found answers; never pay repeatedly for a permanent miss.
5. Use bounded concurrency and exponential backoff for 429/5xx responses.
6. Estimate submission, polling and pagination requests—not only input records.
7. Keep provider-restricted data internal and store distributable artifacts separately.

## Final portable dataset

Do not hand another project a database connection or a raw checkpoint. Build the private download bundle
once both enrichments finish, or publish the current usable snapshot with `--allow-incomplete`:

```bash
python3 scripts/export_inc5000_dataset.py \
  --funding-jsonl .context/inc5000-funding/bulk-responses.jsonl \
  --upload

# Publish immediately when partial coverage is acceptable:
python3 scripts/export_inc5000_dataset.py \
  --funding-jsonl .context/inc5000-funding/bulk-responses.jsonl \
  --allow-incomplete --upload
```

The exporter refuses to create a final bundle until every domain has a complete Similarweb raw response
and a Recent Funding found/not-found result. It produces `companies.csv.gz` for ordinary analysis,
`companies.full.jsonl.gz` with the complete raw Similarweb response and Recent Funding company dump,
`manifest.json` with coverage and SHA-256 hashes, a data dictionary, and one `inc5000-enriched.zip` file.
`--upload` stores that ZIP in the private `raw` bucket and prints a seven-day signed download URL. Use
`--check-only` to report readiness. Incomplete bundles are marked `final: false`, and include their exact
coverage counts in the manifest and owner-only download endpoint response.

After publication, another owner agent should request one fresh download URL instead of paging through the
database or copying credentials:

```bash
curl "https://fetchapi.co/api/v1/inc5000/export?hours=24" \
  -H "x-api-key: $FETCHAPI_KEY"
```

The response contains `ready`, `partial`, full coverage counts, artifact names/checksums, and
`download_url`. `ready: false, partial: true` means the ZIP is usable but not all enrichment lookups were
completed. Download that private ZIP directly from storage. Only owner keys can call this endpoint; the
URL expires after the requested 1-168 hours, while a later endpoint call issues a new one.