crawlgraphtry free

where backlink data comes from.

every backlink number you have ever seen is a count over somebody's crawl, and no crawler sees the whole web. that one fact explains why two tools report different totals for the same site, why a new link takes weeks to show up, and why the free sources show you so little.

the posts here go source by source: the commercial indexes and how often they recrawl, the first-party data google and bing will give you about your own site for free, and the open common crawl hyperlink graph this site is built on.

8 posts · about 64 min of reading

commercial indexes

how the paid tools crawl, and why they lag behind the live web.

  1. where ahrefs and moz get their backlink data

    Ahrefs, Moz, Semrush, and Majestic find backlinks by running their own web crawlers, storing separate indexes, and applying proprietary deduplication and scoring. Their counts disagree because each measures a different slice of the web. Common Crawl is the open alternative: anyone can inspect its archive and hyperlink graph, although its snapshots are less current than commercial live indexes.

    11 min read

  2. how often do backlink tools crawl? why your new link is not showing up yet

    no tool sees the web live. a link must be crawled, indexed, then published before it appears. how to read your tool's date and confirm a link yourself.

    6 min read

  3. moz's crawler explained: dotbot, rogerbot, and the crawled flag

    moz runs two crawlers: dotbot builds the link index behind DA, rogerbot runs site crawl in a moz pro campaign. what crawled means and why DA moves.

    14 min read

free first-party sources

what google and bing will show you about links to your own site.

  1. google search console backlinks: what it shows

    gsc is the only free backlink data source straight from google - but it caps the report at ~1,000 referring domains, only shows data for sites you've verified, and surfaces a filtered subset of what google actually indexed. here's the full walkthrough plus how to get around the limits.

    9 min read

  2. bing webmaster tools backlinks (free, no signup)

    bing webmaster tools shows backlink data for free - and unlike google search console, it surfaces backlinks for sites you don't own through the similar sites report. here's the full walkthrough, the gsc comparison, and the export workflow.

    9 min read

the open graph: common crawl

choose between archived pages, the URL index API and the hyperlink graph, then check the release dates.

  1. common crawl explained: URL index API and web graph

    Common Crawl is a public web archive. Its URL Index API finds captured pages, while its separate web graph maps links between hosts or domains. Neither dataset is a complete or live map of the web. Choose the index for capture research and the graph for backlink analysis, then pin the exact collection or release used.

    6 min read

  2. common crawl release schedule (next crawl)

    common crawl publishes new web crawls every 1-2 months. here's the recent release cadence, crawlgraph's current composite, and why the lag matters for backlink freshness.

    5 min read

  3. Common Crawl index API 2026: the current CC-MAIN index IDs

    The current index is cc-main-2026-apr-may-jun, built from CC-MAIN-2026-17, CC-MAIN-2026-21 and CC-MAIN-2026-25. Copy-paste API URLs for both the Common Crawl CDX index and crawlgraph.

    4 min read