How much of the web does a crawler see?
Every web crawl is a sample. Common Crawl collects billions of pages a month; a focused academic crawl collects a hundred million URLs. Both are samples of something larger: everything the crawler’s configuration could have reached, had it continued sampling under its policy. The crawl size is known exactly. The size of that reachable web is not, and neither is the share of it a crawl holds.
The classical answer, from ecology, is to catch, mark, release and catch again: the overlap between two samples reveals the population. Applied to a crawler, the trick has a twist. The ecologist assumes that nothing enters or leaves the population between the two catches. The web does both: it keeps changing between the two crawls, so pages the first crawl missed and pages that no longer exist both show up as “not seen again”, and a naive overlap mixes them.
Our paper, Measuring What the Crawler Sees (TPDL 2026), separates the two, missing and gone, with an urn model with turnover. Two numbers read off the overlaps of a crawl series, a coverage and a retention, tell you what share of the reachable web each crawl holds, how fast that web is replaced, and how large it is, without ever counting it. On Common Crawl’s domains the simple model breaks in an instructive way, and the repair reveals a persistent core of roughly 40 % of all domains.
The model is at work below. Entities drift from right to left, and their speed is how fast they are replaced. The circle is the reach of the crawler, and every pulse of its rim is a crawl that draws from whatever is inside. The best ranks, the core, sit in the centre; the ranks fall off toward the top and bottom, the frontier. In the two regimes with a known answer, the homogeneous web and the core + shell web, the containment and discovery curves under the circle recover the numbers the web was given. The data behind the charts is from Common Crawl’s crawl statistics.
- The model: the web as an urn with turnover, played live, and its two parameters, coverage and retention.
- The metrics: containment and discovery, what they read on a homogeneous web, and the regimes in which they disagree.
- The data: Common Crawl’s domains and URLs, read with the two metrics.
1. The model
The web is an urn. A crawl draws a share of it, and between two crawls a share of it is replaced. Two parameters, the coverage \(c\) and the retention \(\alpha\), set the model, and the rest of this page concerns their estimation.
The simulation
The animation plays the model; the two charts show the two quantities a crawler can measure on it. The numbered guide under the figure says how to read it, and the regimes it offers are the subject of section 2.
Containment overlap vs. time apart
Discovery seen so far vs. run length
- The rim is the policy boundary. Inside, the policy space: the urn of \(N\) elements the crawler can reach. Beyond it, in grey, the rest of the web, never sampled.
- Each rim pulse is a crawl of size \(M\): a random draw of a share \(c\) of the dots, wherever they stand. The drawn dots light up, or the unseen ones, with the switch under the circle.
- Dots move through the policy space as time runs with the slider. An element enters at the right when it appears, crosses at the pace of its own lifetime, and leaves at the left when it disappears. One crossing is one lifetime.
- The charts read the last thirteen crawls. Above, containment: how much of a crawl a later crawl still holds. Below, discovery: how much a run of crawls has seen. The tags under each title show \(c\), \(\alpha\), \(N\) as set in the model → as the fit reads them back.
- Regimes. Static web and Turnover are homogeneous. Core + shell and Rank profile split the circle into rank strips, the shade is the share drawn. Fine-tune edits them.
Fine-tune: the web, the reading, the playback
The web
One row per strip, middle first. Each row is one slider with three handles. The box sets the strip's retention α, shown as the expected lifetime T = 1/(1 − α) in crawls, on a log scale; at the far right α = 1 and the dots never leave. The grip beyond the whisker spreads the lifetimes within the strip. The dot sets the strip's coverage c, the fraction of the strip each crawl draws. Handles take the arrow keys.
N, the number of elements the crawler could reach. The budget M follows from the coverages.
Things to try
- Static web. Nothing is born or dies. The containment means sit flat at c, and discovery climbs toward N/M and stops: keep crawling and you have seen everything.
- Turnover. Follow one dot: it enters on the right, is lit each time a crawl happens to draw it, and is gone at the left edge. The containment line now tilts, and its slope is α; its intercept is still c. Discovery straightens out, because the web keeps making new things to find.
- Core + shell. The middle strips stand still and are drawn every time. Containment settles on a floor; the reading takes it out and reads the shell on its own. Both figures recover κ, the core's share of the web.
- Rank profile. The middle strips drift slowly and are bright, the frontier streams past and is dim. Compare the tags: the two figures disagree on α, and the regimes of section 2 say why. The unseen side of the web sits at the right edge, where elements enter, and in the frontier, never in the core; switch to the drawn for the other side.
- Fine-tune. Pull a strip's grip to the right: the slow dots crowd the strip, the long-lag means lift, and α̂ climbs above the strip's median. Scrub the crawl bar: every window of thirteen crawls gives its own fit, and the scatter is the sampling noise of a run this short.
The urn with turnover
A crawl is large, a hundred million URLs in a focused academic crawl, billions of pages in Common Crawl, but it is a sample of a larger set: everything the crawler’s configuration could have reached had it continued sampling under its policy. We denote the size of that set by \(N\). The size of a crawl, \(M\), is known to the operator. \(N\) is not, and neither is the coverage, \(c = M/N\), the share of the reachable web that a crawl holds. The simulation plays the simplest model in which \(N\) can be estimated from repeated crawls.
| Each round | What happens | Parameter |
|---|---|---|
| Sampling | A share \(c\) of the \(N\) elements is drawn at random. The draw is the crawl, of size \(M = cN\). | c, coverage |
| Churning | A share \(\alpha\) of the \(N\) survives to the next round. The rest, \(\bar\alpha = 1-\alpha\), is replaced by elements that nobody has seen. | α, retention |
This is the urn with turnover. An element lives \(1/\bar\alpha\) crawls on average, and spends a share \(c\) of them in a crawl and the rest unseen. The model is stationary: the population keeps its size, and nothing in it depends on the round. A fixed crawl configuration is what justifies this, since it holds the sampling procedure constant while the population changes. Everything the model says about a crawler follows from these two parameters, and the next section is about reading them off the crawls.
2. The metrics
Containment reads a pair of crawls, discovery reads a run. On a homogeneous web the two agree. Where they disagree, the web has more than one kind of element, and the disagreement says which kind.
Containment
The problem is a classical one. In ecology, animals are caught, marked, released and caught again; the marked share of the second catch estimates the coverage of the first, the share of the population it took (Petersen 1896, Lincoln 1930). If the two catches are independent draws from a closed population, \(N = M_1 M_2 / |u_1 \cap u_2|\).
Lawrence and Giles (1998) brought the estimator to the web with the roles rearranged. The query is what they hold fixed: it defines a slice of the web. The search engines are what varies: they are the sampling devices. The same queries went to several engines, each returned its set of URLs, and the pairwise intersections of those sets took the place of the recapture. The fraction of one engine’s results that a second engine also returned estimates the coverage of the second, and the size of the indexable web follows from that coverage and the reported size of the second engine’s index. The samples are simultaneous and the population is treated as fixed. The estimate is a lower bound, for a reason they state: engines do not sample independently, since authors register pages with several engines and every engine favours well-linked pages, so the overlap overstates the coverage.
A crawler rearranges the roles once more. The policy is what stays fixed, and it defines the reachable web as the query defined its slice. What varies is the time. There is only one sampling device, so the two samples are taken in sequence rather than side by side, and the overlap depends on the time between them. And the population is not closed: between two crawls, pages disappear and new pages appear, so the overlap confounds what the first crawl missed with what no longer exists. In full generality, with every page following its own trajectory and the crawler its own selection rules, the overlap cannot separate the two. The urn can.
| Design | Held fixed | Varied | Population | The overlap gives |
|---|---|---|---|---|
| Ecology | The trap and the ground. | The visit: two catches, the marks carried from the first to the second. | Closed between the catches. | N, the population size |
| Search engines | The query, a slice of the web. | The engine: several result sets for the same query, at the same time. | The indexable web, taken as fixed. | a lower bound on N; engines are correlated samplers |
| A crawler | The policy, the reachable web. | The time: the crawls of one crawler, in sequence, \(\Delta t\) apart. | Open: elements leave and enter between crawls. | g(Δt): coverage and turnover, together |
The containment of a crawl in a later one is the share of the first that the second still holds. Let \(u_t\) be the crawl of round \(t\) and take an element of \(u_i\). Between round \(i\) and round \(j = i + \Delta t\) it must survive \(\Delta t\) churns, each with probability \(\alpha\) and independent of the others, and crawl \(j\) must then draw it, with probability \(c\) and independent of its history, since every crawl draws uniformly from whatever is present. So the element is in \(u_j\) with probability \(c\,\alpha^{\Delta t}\). The containment is the mean of that indicator over the \(M\) elements of \(u_i\), and for a crawl of millions of elements the mean is its expectation:
\[ g_{ij} = \frac{|u_i \cap u_j|}{|u_i|}, \qquad \mathbb{E}\,g_{ij} = c\,\alpha^{\Delta t}, \qquad \log g = \log c + \Delta t\,\log\alpha . \]The parameters are the same in every round, so the containment of a pair depends on how far apart the two crawls are and not on when they were taken: a pair from the start of the window and a pair from its end, at the same lag, are two readings of the same value, and all the pairs of a window lie on the one line. On a logarithmic scale the slope is the retention and the intercept the coverage. The intercept is what two crawls taken at the same instant would share, which cannot be observed, so the line is fitted through the pairs, as means per lag, and extended back to \(\Delta t = 0\); \(\hat N = M/\hat c\) follows. The upper chart of the simulation does this for the 78 pairs of its window of thirteen crawls; the black tick marks the \(c\) the model was set to.
Discovery
A run of \(T\) crawls \(u_0, \dots, u_{T-1}\) sees \(U(T) = |\bigcup_{t < T} u_t|\) distinct elements. Each crawl adds what no earlier crawl saw, so \(U(T)\) is a sum over the run of the new part of each crawl, and the new share of a crawl is one minus the share of revisits. Let \(\rho_t\) be the share of the population at round \(t\) that some earlier crawl has seen. Since crawl \(t\) draws uniformly, the share of revisits in \(u_t\) is \(\rho_t\) as well, and
\[ \frac{U(T)}{M} = \sum_{t < T} \overbrace{\bigl(1 - \rho_t\bigr)}^{\text{new}}, \qquad \rho_0 = 0 . \]One round moves \(\rho\) in two steps. The crawl draws a share \(c\) of the unseen part, so after it the seen share is \(\rho + c\,(1 - \rho)\): the revisits plus the newly seen. The churn then keeps a share \(\alpha\) of everything and replaces the rest by elements nobody has seen, so the seen share is multiplied by \(\alpha\):
\[ \rho_{t+1} = \alpha\,\bigl(\rho_t + c\,(1 - \rho_t)\bigr) = \alpha c + \alpha\bar c\,\rho_t . \]This is a linear recursion with the fixed point \(\rho^\ast = \alpha c/(1 - \alpha\bar c)\), approached geometrically with ratio \(\alpha\bar c\), the probability that an element survives a round without being drawn:
\[ \rho_t = \rho^\ast\bigl(1 - (\alpha\bar c)^t\bigr), \qquad 1 - \rho^\ast = \frac{\bar\alpha}{\bar\alpha + \alpha c}, \]where the last form uses \(1 - \alpha\bar c = \bar\alpha + \alpha c\). Summing the new shares over the run gives the discovery curve, a geometric sum:
\[ \frac{U(T)}{M} = \sum_{t < T} \bigl(1 - \rho^\ast + \rho^\ast (\alpha\bar c)^t\bigr) = \frac{T\,\bar\alpha}{\bar\alpha + \alpha c} + \frac{\alpha c\,\bigl(1 - (\alpha\bar c)^{T}\bigr)}{(\bar\alpha + \alpha c)^2} =: B(T;\,\alpha, c). \]The curve is a straight part and a transient. The slope, \(1 - \rho^\ast\), is the share of a crawl that is new once the run has settled; the transient is the first crawls, which find more because nothing has been seen yet, and it dies off with ratio \(\alpha\bar c\). Three checks: at \(T = 2\) the curve is \(2 - \alpha c = 2 - g(1)\), the union of two sets by inclusion and exclusion, so the two metrics coincide at the first lag; without churn, \(\alpha = 1\), the slope is zero and the curve saturates at \(1/c = N/M\), the dotted line in the lower chart; with full coverage, \(c = 1\), every crawl is the whole population and the curve is the line \(1 + (T-1)\bar\alpha\). The lower chart fits \(B\) to the union of the crawls in the window, over \(\alpha\) and \(c\).
Regimes
Homogeneous. Every element has the same \(c\) and the same \(\alpha\). The two metrics then read the same pair: they coincide at the first lag, and on the urn the agreement carries to every lag, since both are functions of the same two numbers. Static web and Turnover in the simulation are this regime, and their tags agree.
Heterogeneous. Real populations are not homogeneous, and the strips of the simulation allow heterogeneity along the axis a crawler itself uses, its rank. Each departure has its own signature in the two charts.
A core that never leaves. A share \(\kappa\) of the \(N\) elements is in every crawl. The rest, the shell, is an urn of its own with coverage \(c_\partial\) and retention \(\alpha_\partial\). A crawl holds the whole core and a share \(c_\partial\) of the shell, so its size is \(M = \kappa N + c_\partial (1-\kappa) N\), and the core’s share of a crawl exceeds its share of the population, because the shell is only partly drawn:
\[ \mu = \frac{\kappa}{\kappa + (1-\kappa)\,c_\partial} > \kappa . \]An element of \(u_i\) is core with probability \(\mu\) and then in every later crawl, or shell with probability \(1 - \mu\) and then follows the shell’s urn. Over a run, the core is all in the first crawl and adds nothing afterwards, while the shell discovers as an urn of its own with \(M_\partial = (1-\mu) M\):
\[ g(\Delta t) = \mu + (1-\mu)\,c_\partial\,\alpha_\partial^{\Delta t}, \qquad \frac{U(T)}{M} = \mu + (1-\mu)\,B(T;\,\alpha_\partial, c_\partial). \]The containment now decays to the floor \(\mu\) instead of zero. Discovery starts higher and rises more slowly. In the simulation, the core + shell reading fits three numbers at once: the floor, and the intercept and slope of the shell. The upper chart then plots \(g - \mu\), so the shell is a straight line again, and both charts report \(\hat\kappa\). When turnover is slow, a window of thirteen crawls is almost straight, and the floor is hard to tell from the slope. The fitted \(\hat\mu\) then varies between windows: a short window of slow turnover holds little information about the floor.
A spread of lifetimes. Open Fine-tune and pull the grip of a strip to the right. At any moment the strip consists mostly of long-lived elements, because a long-lived element occupies its place for longer, and these are the ones still present at the next crawl. The means at long lags lie above the line the short lags predict, \(\hat\alpha\) exceeds the strip’s median, and the discovery curve lies below the homogeneous expectation, because long-lived elements are seen again rather than replaced.
A gradient over rank. Select Rank profile. Coverage and retention both fall from the middle to the frontier, as they do for a crawler that prioritises by rank: the pages it reaches first are the pages that persist, the pages it reaches last are the pages that are replaced. Unequal coverage lifts the intercept above \(M/N\) even without turnover, so \(N\) is underestimated; unequal retention bends the line. The two metrics now disagree in a specific direction: containment is weighted by recurrence and dominated by the middle, discovery by new arrivals and dominated by the frontier. Section 3 finds this gradient in the data.
| Departure | Containment intercept \(\hat c\) | Containment slope \(\hat\alpha\) | Discovery curve |
|---|---|---|---|
| Core that never leaves | overshoots; floor at \(\mu\) | too close to 1 | starts higher, rises more slowly |
| Spread of lifetimes | near \(c\) | climbs above the median | below the one-speed expectation |
| Gradient over rank | above \(M/N\) | bends, long lags to the slow strips | disagrees with containment, toward the frontier |
Operational quantities, and the assumption behind both metrics
Operational quantities. The two parameters translate into quantities a crawl operator can use. The lifetime is a property of the web; the crawl policy decides which part of it an element spends seen and which part unseen. The last column gives the values for the domain-level reading of section 3, with one crawl about every six weeks.
| Quantity | Formula | Meaning | Turnover reading |
|---|---|---|---|
| Life expectancy | \(\ell = \alpha/\bar\alpha\) | number of crawls an element remains reachable | 42 crawls, about 5 years |
| Time visible | \(\eta = c\,\ell\) | number of those crawls in which it is drawn | 35 crawls |
| Time hidden | \(\bar\eta = \bar c\,\ell\) | number of those crawls in which it is not drawn | 7 crawls |
| Half-life | \(\ln 2 / (-\ln\alpha)\) | lag at which the containment has halved | 30 crawls |
| Fresh share | \(\bar\alpha/(\bar\alpha + \alpha c)\) | share of a crawl that is new, in the long run | 2.7 % |
| Seen after 13 crawls | \(B(13;\,\alpha, c)\) | distinct elements, in units of the crawl size | 1.5 |
Independence. Both derivations use that each crawl is a uniform random draw from what is present, independent of earlier draws. A crawler that repeatedly collects the same fixed region of the web violates this: consecutive crawls contain the same elements, the intercept is 1 regardless of the coverage, and the overlap measures the sampling procedure rather than the population. Simultaneous connections to the same Twitter sample stream agree to more than 96 percent for this reason (Joseph, Landwehr and Carley 2014). A crawler with a fixed policy lies between the fixed region and the random draw; the containment residual \(R_{ij}\) of the paper quantifies the difference.
3. The data
The crawl policy can now be observed in terms of coverage. At domain level, about half of the population is persistent and the remainder is replaced within about a year. At URL level, consecutive crawls have overlapped by only a few percent since 2017.
Domains, 2020 to 2025
The blue marks in the charts of the simulation are measurements on Common Crawl: the registered domains of 51 monthly crawls, with the intersection of every pair computed exactly and the union of every run estimated with cardinality sketches. The presets of the model are set to the fits of the containment chart, so that simulation and measurement coincide.
| Reading | Coverage | Retention per crawl | Lifetime | Core share κ |
|---|---|---|---|---|
| One exponential | c = 0.84 | α = 0.977 | 42 crawls, about 5 years | – |
| Core + shell | c∂ = 0.70 | α∂ = 0.877 | 7 crawls, about 10 months | κ = 0.50 |
Over the full six years, the homogeneous fits disagree on the retention: \(\alpha \approx 0.90\) per year with \(c \approx 0.78\) from containment, against \(\alpha \approx 0.71\) per year from the discovery curve, and no single pair fits both. The two-component fits agree on the core and on the shell’s retention, and disagree on the shell’s coverage:
| Two-component fit, six years | Core share κ | Shell retention per year | Shell coverage |
|---|---|---|---|
| From containment | κ = 0.42 | α∂ = 0.68 | c∂ = 0.70 |
| From discovery | κ = 0.46 | α∂ = 0.64 | c∂ = 0.20 |
Drawn across, each fit misses the other curve: the containment fit climbs too slowly as a discovery curve, the discovery fit starts far below the containment points. This is the simulation’s core + shell trade-off, μ against \(c_\partial\), at a scale the trade-off cannot explain, and it is the gradient of section 2 in the data. Containment is weighted by recurrence and reads the shell where it recurs, in the well-covered middle ranks; discovery is weighted by new arrivals and reads it at the frontier, where coverage is low. The shell is not homogeneous, and rank is the axis along which it varies.
URLs, 2008 to 2026
Common Crawl also publishes, as part of its crawl statistics, the overlap of every pair of monthly crawls at URL level, as a heatmap of the Jaccard similarity of unique URLs. The values behind the figure, recomputed from the same HyperLogLog sketches for all 129 crawls from 2008 to 2026, are provided in this repository as data/cc-url-overlap.csv. Each row holds the sizes of the two crawls, their intersection and their union, so the containment is taken directly from the sketches, not from the Jaccard similarity under an equal-size assumption. The chart below shows it for a window of thirteen consecutive crawls, as the simulation does above; the slider moves the window across the series, and the play button runs it through the history.
Containment, URL level
Blue: every pair of the thirteen crawls in the window as a small square and the mean per lag as a large one, from the cardinality sketches of Common Crawl’s crawl statistics. Red: one exponential in time through the means more than 2.5 months apart, since the first two lags measure the crawler’s URL selection rather than the population; ĉ where it meets Δt = 0, retention per year. Marks left of the dashed line are not in the fit. Values below 0.02 are within the error of the sketches. Hover a mark for its crawls.
Two regimes are visible. Up to 2016, consecutive crawls shared about three quarters of their URLs. Since 2017 they have shared a few percent, and the containment increases from the first lag to the third: the crawler defers URLs fetched recently in favour of URLs not yet fetched, so the first two lags measure the selection policy rather than the population. From the third lag onward the window reads as an urn with a retention of about 0.93 per crawl and an intercept of about 0.2. Taken together with the domain level, a containment of 0.84 over one near 0.02, this is the gradient of section 2 in the data: the crawler retains the domains and rotates the pages within them.
References
- Paris, M., Çelikkanat, H., Foppiano, L. (2026). Measuring What the Crawler Sees: Discovery Curves, Core Persistence, and Shell Dynamics in Longitudinal Web Crawls. TPDL 2026, Faro. arXiv:2607.13636.
- Paris, M., Paris, G., Baumann, M. (2026). Estimating the Absolute Coverage of a Web Crawl. arXiv:2603.15416.
- Lawrence, S., Giles, C. L. (1998). Searching the World Wide Web. Science 280, 98–100.
- Bharat, K., Broder, A. (1998). A technique for measuring the relative size and overlap of public Web search engines. WWW7.
- Broder, A. (1997). On the resemblance and containment of documents. SEQUENCES 1997.
- Joseph, K., Landwehr, P. M., Carley, K. M. (2014). Two 1%s don't make a whole: Comparing simultaneous samples from Twitter's Streaming API. SBP 2014.
- Common Crawl Foundation. Statistics of Common Crawl monthly archives: overlaps between monthly crawls. commoncrawl.github.io/cc-crawl-statistics/plots/crawloverlap, code in cc-crawl-statistics.
- Petersen, C. G. J. (1896). The yearly immigration of young plaice into the Limfjord from the German Sea. Lincoln, F. C. (1930). Calculating waterfowl abundance on the basis of banding returns. USDA Circular 118.