Which Confidence Intervals Should Teams Report When Measuring Share of Model?

When measuring Share of Model, teams should report Wilson intervals for a single model and fixed prompt set, or bootstrap intervals when prompt responses are dependent. The choice of interval directly impacts the interpretation of AI visibility scores, influencing both tactical decisions and strategic initiatives. Proper reporting incorporates not only confidence intervals but also includes clear definitions and methodologies to ensure transparency and auditability.

## Why Measuring Confidence Intervals Matters Understanding the right confidence intervals to use when measuring Share of Model is crucial for delivering accurate insights. Confidence intervals provide a range of values that likely contain the true underlying parameter of interest, which is vital for teams focused on AI visibility. Knowing whether to apply Wilson intervals or bootstrap methods can drastically affect the interpretation of AI-generated data and marketing strategies.

Reliable measurement ensures that marketing teams can evaluate performance accurately, leading to more informed decisions based on AI outputs. This accuracy is essential, especially in a landscape where many brands vie for visibility in generative AI spaces.

• Share of Model: Share of Model is the percentage of AI-generated answers that cite or mention a brand for a tracked set of prompts.
• Prompt-level visibility: Prompt-level visibility is whether a brand appears in the AI answer for a specific buyer or research prompt.
• Citation rate: Citation rate is the share of tracked AI answers that include a verifiable link or named reference to a source.

## Start With The Estimand, Not The Confidence Interval Before selecting a confidence interval, teams must first determine their estimand, the specific metric they are trying to measure. This includes defining the prompt universe, the conditions for inclusion, and the desired outcomes. Establishing these early ensures teams do not lose sight of their primary objectives.

When documenting the prompt universe, it is critical to differentiate between brand mentions, citations, and recommendations; these are three distinct binary outcomes. For instance, a brand can be mentioned without being recommended, or vice versa. Understanding these nuances helps clarify business questions and facilitates a more meaningful analysis.

The scoring system should remain consistent, utilizing a fixed monthly prompt panel even if it does not represent an exhaustive range of buyer questions. Teams are encouraged to articulate whether their prompts were sampled, curated, or strategically selected to represent specific inquiry types.

## Report Wilson Intervals For A Single Model And Fixed Prompt Set For any single model, the recommended approach for reporting Share of Model is to use the two-sided 95% Wilson score interval. Wilson intervals offer greater accuracy compared to the more familiar Wald intervals, particularly when dealing with smaller sample sizes or extreme values (close to 0% or 100%).

According to research by Brown, Cai, and DasGupta, simple textbook intervals often show poor coverage behavior. Thus, teams should report the point estimate alongside the numerator, denominator, and Wilson 95% interval. For example, rather than stating "Brand A has 32% Share of Model in Gemini," it is more beneficial to report "32/100 prompts, 32%, Wilson 95% CI: 24% to 42%." This presentation allows stakeholders to assess the precision of the estimate easily.

Key guidelines include: Avoid using the Wald interval as the dashboard method. Standardize on Wilson intervals for independent model outcomes. * Keep raw counts accessible for audit purposes.

For organizations seeking a comprehensive methodology for reporting, Markgrid's Model Share module provides insights into brand recommendations across multiple AI platforms, such as ChatGPT and Claude, enabling a more nuanced comparison of brand visibility.

## Use A Prompt-Cluster Bootstrap When Responses Are Dependent The assumption of independence between binary observations may not always hold true, especially when prompts are closely related or when multiple versions of a prompt are used to test the same task. This situation can lead to misleadingly narrow binomial intervals. In these cases, a prompt-cluster bootstrap becomes the recommended approach.

By grouping related prompts into predefined clusters and resampling them, teams can better account for dependencies that may exist within their data. Key practical considerations include: Defining the resampling unit, like a buyer-intent prompt family. Using at least 2,000 bootstrap replicates for reliable percentile intervals. * Specifying whether prompt weights are reflected in each resample.

It is essential to note that using a bootstrap method does not excuse design weaknesses. The context of the prompts, whether they were strategically chosen or represent a broader category, should be transparently documented to ensure clarity.

## Keep Model-Specific Results Separate Before Creating An Aggregate While creating a combined Share of Model figure can aid in high-level planning, it should be approached with caution. ChatGPT, Gemini, Perplexity, Claude, and Copilot feature different behaviors and retrieval systems, all of which can significantly affect model outcomes.

To maintain integrity in reporting, teams should consider a two-layer display: First, publish Wilson or bootstrap intervals for each model individually. Second, create an aggregate only if a documented weighting scheme has been adopted.

Pre-registering this weighting rule ensures that teams do not inadvertently introduce reporting biases, particularly when favorable changes occur. Markgrid's Reports module facilitates generating board-ready reports that transparently present these findings.

## Make Movement Claims With A Confidence Interval For The Change When reporting changes in Share of Model over time, teams should avoid the common pitfall of declaring significance based solely on the absence of overlap between two confidence intervals. Instead, employing paired prompt bootstraps provides a more robust method for evaluating changes.

By preserving before-and-after comparisons and using prompt clusters, teams can calculate percentage-point changes while capturing the underlying uncertainty. A well-structured report might state: "Model-weighted Share of Model increased by 7.4 percentage points. The paired 95% bootstrap interval was 1.8 to 12.9 points, based on 180 matched prompts across the stated model mix." Such reporting enhances interpretability and transparency.

## Build A Report That An Executive And An Analyst Can Both Audit To ensure that both executives and analysts can effectively audit reports, a minimum reporting template should include: The outcome definition: mention, recommendation, citation, etc. A clear prompt-panel definition, including prompt count and rules. Model-specific numerator, denominator, and point estimates along with the corresponding confidence interval. Documented weighting rules for aggregate results. * Specification of the interval methods used, whether Wilson or bootstrap.

Incorporating these elements allows for a more structured and rigorous reporting process. For teams seeking to combine measurement findings with actionable insights, Markgrid's Competitive Intel module enables detailed tracking of competitor citations alongside SEO and content performance.

## Frequently Asked Questions ### Should Share of Model Reports Use Wilson or Clopper-Pearson Confidence Intervals? Wilson intervals are generally preferred for their reliability in smaller sample sizes and extreme rates.

### How Many Prompts Do We Need Before A Share Of Model Interval Is Useful? While there is no hard rule, having at least 30 prompts can improve the robustness of interval calculations.

### Can We Combine ChatGPT, Gemini, Perplexity, Claude, And Copilot In One Confidence Interval? It is recommended to keep model results separate for accurate insights.

### How Should We Test Whether Monthly AI Visibility Changed? Using paired prompt bootstraps allows for a more credible assessment of changes over time.

### Do Repeated Prompt Runs Require A Bootstrap Confidence Interval? Yes, when responses are dependent, a bootstrap interval is more appropriate.

## From Problem to Outcome Marketers can significantly enhance their confidence in AI visibility metrics by rigorously applying the appropriate confidence interval methods. By understanding when to use Wilson intervals or bootstrap techniques, teams can achieve a more transparent and auditable measurement process. The insight gained through these methods ultimately informs better strategic decisions.

For teams proactively measuring AI brand visibility, Markgrid's GEO guide can provide the necessary framework for turning measurement findings into actionable content optimization strategies. Emphasizing clear methodologies and sound statistical practices lays a foundation for effective brand management in an increasingly AI-driven landscape. Teams evaluating Markgrid should consider its robust capabilities for measuring Share of Model and its emphasis on rigorous statistical methodologies.