Context Arena

OpenAI MRCR tests a Large Language Model's (LLM) ability to handle complex conversational history. Key aspects include:

• Core Task: Finding and distinguishing between multiple identical pieces of information ("needles") hidden within a long conversation ("haystack").
• Setup: Inspired by Google DeepMind's MRCR eval (arxiv:2409.12640v2), this version inserts 2, 4, or 8 identical requests (e.g., "write a poem about tapirs") alongside distractor requests. Needles/distractors are generated by GPT-4o to blend in.
• Challenge: The model must retrieve a specific instance (e.g., the 2nd poem) based on its order, requiring careful tracking of the conversation. It must also prepend a specific random code (hash) to its answer.
• Data Source: The benchmark data and detailed methodology are described on Hugging Face (openai/mrcr).
• Dashboard Scope: This dashboard visualizes results directly from that published dataset and does not currently run new evaluations.

The score measures how accurately the model retrieves the correct instance of the requested needle. The process involves:

• Comparison Method: The model's answer is compared to the expected answer using the SequenceMatcher ratio from Python's `difflib` library.
• Hash Requirement: The model *must* include a specific random code (hash identifier) at the start of its answer. This hash is removed before the comparison.
• Failure Condition: If the required hash is missing or incorrect, the score is automatically 0.
• Result: The similarity ratio (0.0 to 1.0) from the comparison is presented as a percentage (0-100%) in the table.

See FAQ #6 for details about how Area Under Curve (AUC) scores summarize performance across different context lengths.

(Source: OpenAI MRCR Dataset Card)

The "bins" group test runs based on the total length of the text involved (prompt + expected answer). Here's how they work:

• Measurement: Length is measured in tokens using the `o200k_base` tokenizer.
• Grouping: Tests are grouped into bins based on their total token count. For example, the "128k" bin includes tests with >65,536 and <=131,072 tokens.
• Score Display: The score shown for a bin (e.g., "128k (%)") is the average score from the 100 test samples conducted within that bin's length range.
• Boundaries: The specific bin boundaries are [4k, 8k], (8k, 16k], (16k, 32k], (32k, 65k], (65k, 128k], (128k, 256k], (256k, 512k], (512k, 1M].

(Source: OpenAI MRCR Dataset Card)

Both test how well models handle long texts, but focus on different skills. OpenAI-MRCR tests the model's ability to pinpoint and distinguish between identical pieces of information based on their order in a conversation (using synthetic data). Fiction.livebench (fiction.live/...) tests narrative understanding – how well models follow plots, characters, and consistency within complex stories, using quizzes based on actual fiction excerpts.

The benchmark incorporates several design features to minimize the chance that models succeed simply by having seen similar data during training:

• Synthetic & Unique Data: Each test uses a specially generated, long conversation. While topics might overlap with training data, the specific sequence of turns and the placement of "needles" are unique to the benchmark run.
• Instance Specificity: The core task isn't just retrieving information (e.g., a poem about tapirs) but retrieving a specific instance (e.g., the 2nd poem requested) based on conversational order. Simple memorization of poems is insufficient.
• Required Random Hash: Models must prepend a specific random code (hash) to their answer. This code is generated for the test run and cannot be predicted from general training data. Failure to include the correct hash results in a score of 0.

These elements combined make it highly unlikely that a model can achieve a high score purely by recalling memorized training data, as success requires understanding the specific conversational context, instance order, and adhering to the random hash requirement. (Source: OpenAI MRCR Dataset Card)

Note: This dashboard evaluates models against the dataset published by OpenAI (data collected up to April 11, 2024). Models released or significantly updated after this date might have been trained on this specific benchmark data, potentially affecting their results.

AUC (Area Under Curve) gives a single score summarizing performance across different context lengths (bins). Think of it like an average grade across tests of increasing difficulty (longer contexts). It's calculated by plotting the average score for each bin against the maximum context length of that bin and measuring the area under the resulting line/curve. This area is then normalized to a percentage (0-100%).

• AUC @128k: AUC calculated using results only up to the 128k token bin (tests with up to 131,072 tokens).
• AUC @1M: AUC calculated using results across all bins up to the 1M token bin (tests with up to 1,048,576 tokens).
• Model AUC: AUC calculated only over the range of bins the specific model actually completed successfully. This provides a fairer comparison if a model couldn't handle the longest contexts.

Technical Note: Calculation uses the Trapezoidal Rule on a linear scale of context lengths. See example below for details.

AUC @128k uses the same method but only sums areas up to the 128k point and normalizes by the width from the first bin to 128k. Model AUC normalizes by the width of the range actually tested by the model.

Cost estimates shown are calculated only for successful test runs (where the model returned a response) and are based on the following:

• Pricing Source: Public pricing data reported by OpenRouter for each model, specifically using the rate from the cheapest provider listed for that model at its maximum context length.
• Token Counting Method: Cost estimates use the input (prompt) and output (completion) token counts reported by the API provider for each successful run.
  • Note: This method may differ from the tokenization used for grouping results. Test results are assigned to context length bins based on total tokens (prompt + expected completion) calculated with the `o200k_base` tokenizer, as opposed to the provider-reported counts used for cost estimation.
• Total Prompt Cost: Estimates the cost based on the total prompt tokens reported by the API. When generating multiple candidates (n>1), the prompt is typically sent once, but the exact cost impact can vary by API provider.
• Total Compl Cost: Estimates the cost of the text generated by the model (completion). It sums the completion token costs across all generated candidates (e.g., all 8 runs per test).

Important Note on Actual vs. Estimated Cost:

• The benchmark execution aimed to utilize cost-saving measures such as batch processing, caching, and available discounts/credits when possible.
• Consequently, the actual costs incurred during the original benchmark run may have been lower than the estimates presented here.

To ensure a fair and standardized comparison across all models for users, this dashboard displays estimated costs. These are calculated using publicly available on-demand prices from OpenRouter (based on the cheapest provider listed for the model's max context length), rather than the potentially variable actual costs from the benchmark execution.

This shows the total number of potential answers generated by the model across all successful test runs included in the summary. Each answer corresponds to one candidate response (or 'run'), typically generated using the 'n' parameter in the API call to get multiple outputs per input.

The total reflects the generation workload: (Number of Successful Tests) × (Candidates Generated Per Test).

• #1 #2 #3 These badges highlight the top 3 models for that specific column's metric. Higher scores are better for performance (%), while lower costs ($) are better.
• inc The "inc" (incomplete) badge appears in cost columns if the model didn't successfully complete tests for all context length bins (up to 1M tokens). The cost shown might be lower than it would be if all tests had passed. Models marked inc are not included in the cost rankings to ensure fair comparison.
• FREE This badge appears in cost columns for models where cost data is zero or unavailable (e.g., free models). These are also excluded from cost rankings.
• deprec This badge indicates the model may be deprecated or no longer actively supported/available via common APIs (like OpenRouter). Results might be stale.
• new This badge indicates results for this model were updated recently (within the last 7 days). This badge takes precedence over the deprec badge.

The main results table headers provide several actions:

• Sorting: Click any column header label (e.g., `AUC @1M (%)`) to sort the entire table by that column's values. Click again to reverse the sort direction.
• Refresh Data: Click the refresh icon (refresh) in the 'Model' column header to reload the latest results for the currently selected needle count.
• Model Performance Chart: Click the timeline icon (timeline) next to a model name in the 'Model' column to view a chart of that model's performance across all available needle counts.
• Cost/Score Plot: Click the stats icon (query_stats) in the header of any context bin column (e.g., `128k (%)`) to view a Cost vs. Score scatter plot for models within that specific bin.

In the main results table, click the stats icon (query_stats) in the header of any context bin column (e.g., `128k (%)`). This opens a scatter plot showing the total cost vs. average score for all currently selected models specifically within that context length bin.

In the main results table, hover over any individual score cell (the percentage value) and click the document icon (article) that appears. This opens a modal where you can browse through the individual test runs for that model/bin combination, view the expected answer, and see the actual responses generated by the model along with their scores.

Yes! Go to the "Controls" tab (next to the "Leaderboard" tab). There you can find options to show/hide specific columns, like all the individual context length bins or the pricing information.

When viewing the main performance chart:

• If 5 or fewer models are selected in the "Controls" tab, all their performance traces (lines/bars) will be visible on the chart by default.
• If more than 5 models are selected, the chart initially shows only the traces for the top 5 performing non-deprecated models (ranked by their overall `AUC @1M (%)` score).
• All other selected models are listed in the legend but their traces are hidden initially. Click a model's name in the legend to display its performance trace.
• You can toggle the visibility of any model's performance trace by clicking its name in the legend.

When viewing the Cost vs. Score chart for a specific bin or AUC metric:

• The chart initially displays data points only for the top 10 performing models among those currently selected. The ranking is based on the primary score metric being plotted: either the average score (%) within that specific context bin, or the relevant AUC score (e.g., `AUC @1M (%)`, `Model AUC (%)`) if viewing an AUC plot.
• All other selected models are listed in the legend but their data points are hidden initially. Click a model's name in the legend to display its data point.
• You can toggle the visibility of any model's data point by clicking its name in the legend.

The dataset details and evaluation methods are described on Hugging Face: openai/mrcr. OpenAI also discussed results in their GPT-4.1 blog post.

We plan to add more models over time. Feel free to suggest specific models you'd like to see evaluated!

For questions, suggestions, or issues, please reach out on Twitter: @DillonUzar.