Context Arena

OpenAI MRCR tests a Large Language Model's (LLM) ability to handle complex conversational history. Key aspects include:

• Core Task: Finding and distinguishing between multiple identical pieces of information ("needles") hidden within a long conversation ("haystack").
• Setup: Inspired by Google DeepMind's MRCR eval (arxiv:2409.12640v2), this version inserts 2, 4, or 8 identical requests (e.g., "write a poem about tapirs") alongside distractor requests. Needles/distractors are generated by GPT-4o to blend in.
• Challenge: The model must retrieve a specific instance (e.g., the 2nd poem) based on its order, requiring careful tracking of the conversation. It must also prepend a specific random code (hash) to its answer.
• Data Source: The benchmark data and detailed methodology are described on Hugging Face (openai/mrcr).
• Dashboard Scope: This dashboard visualizes results directly from that published dataset and does not currently run new evaluations.

The score measures how accurately the model retrieves the correct instance of the requested needle. The process involves:

• Comparison Method: The model's answer is compared to the expected answer using the SequenceMatcher ratio from Python's `difflib` library.
• Hash Requirement: The model *must* include a specific random code (hash identifier) at the start of its answer. This hash is removed before the comparison.
• Failure Condition: If the required hash is missing or incorrect, the score is automatically 0.
• Result: The similarity ratio (0.0 to 1.0) from the comparison is presented as a percentage (0-100%) in the table.

(Source: OpenAI MRCR Dataset Card)

The "bins" group test runs based on the total length of the text involved (prompt + expected answer). Here's how they work:

• Measurement: Length is measured in tokens using the `o200k_base` tokenizer.
• Grouping: Tests are grouped into bins based on their total token count. For example, the "128k" bin includes tests with >65,536 and <=131,072 tokens.
• Score Display: The score shown for a bin (e.g., "128k (%)") is the average score from the 100 test samples conducted within that bin's length range.
• Boundaries: The specific bin boundaries are [4k, 8k], (8k, 16k], (16k, 32k], (32k, 65k], (65k, 128k], (128k, 256k], (256k, 512k], (512k, 1M].

(Source: OpenAI MRCR Dataset Card)

Both test how well models handle long texts, but focus on different skills. OpenAI-MRCR tests the model's ability to pinpoint and distinguish between identical pieces of information based on their order in a conversation (using synthetic data). Fiction.livebench (fiction.live/...) tests narrative understanding – how well models follow plots, characters, and consistency within complex stories, using quizzes based on actual fiction excerpts.

The benchmark incorporates several design features to minimize the chance that models succeed simply by having seen similar data during training:

• Synthetic & Unique Data: Each test uses a specially generated, long conversation. While topics might overlap with training data, the specific sequence of turns and the placement of "needles" are unique to the benchmark run.
• Instance Specificity: The core task isn't just retrieving information (e.g., a poem about tapirs) but retrieving a specific instance (e.g., the 2nd poem requested) based on conversational order. Simple memorization of poems is insufficient.
• Required Random Hash: Models must prepend a specific random code (hash) to their answer. This code is generated for the test run and cannot be predicted from general training data. Failure to include the correct hash results in a score of 0.

These elements combined make it highly unlikely that a model can achieve a high score purely by recalling memorized training data, as success requires understanding the specific conversational context, instance order, and adhering to the random hash requirement. (Source: OpenAI MRCR Dataset Card)

Note: This dashboard evaluates models against the dataset published by OpenAI (data collected up to April 11, 2024). Models released or significantly updated after this date might have been trained on this specific benchmark data, potentially affecting their results.

AUC (Area Under Curve) gives a single score summarizing performance across different context lengths (bins). Think of it like an average grade across tests of increasing difficulty (longer contexts). It's calculated by plotting the average score for each bin against the maximum context length of that bin and measuring the area under the resulting line/curve. This area is then normalized to a percentage (0-100%).

• AUC @128k: AUC calculated using results only up to the 128k token bin (tests with up to 131,072 tokens).
• AUC @1M: AUC calculated using results across all bins up to the 1M token bin (tests with up to 1,048,576 tokens).
• Model AUC: AUC calculated only over the range of bins the specific model actually completed successfully. This provides a fairer comparison if a model couldn't handle the longest contexts.

All AUC scores are normalized to a maximum of 100%.

Costs are estimates based on the public API pricing for each model and the number of tokens reported by the API for each test run (typically using the `o200k_base` tokenizer).

• Total Prompt Cost: Estimates the cost of sending the input text (prompt) to the model. It assumes a separate API call for each candidate response generated, multiplying the base prompt cost by the number of candidates.
• Total Compl Cost: Estimates the cost of the text generated by the model (completion). It sums the completion token costs across all candidates generated for all successful runs.

Note: Actual costs might differ due to factors like batching requests, different pricing tiers, or API provider adjustments not captured here.

This shows the total number of candidate responses (potential answers) generated by the model across all the successful test runs included in the summary. It reflects the total generation workload: (Number of Successful Tests) × (Candidates Generated Per Test).

• #1 #2 #3 These badges highlight the top 3 models for that specific column's metric. Higher scores are better for performance (%), while lower costs ($) are better.
• inc The "inc" (incomplete) badge appears in cost columns if the model didn't successfully complete tests for all context length bins (up to 1M tokens). The cost shown might be lower than it would be if all tests had passed. Models marked inc are not included in the cost rankings to ensure fair comparison.
• FREE This badge appears in cost columns for models where cost data is zero or unavailable (e.g., free models). These are also excluded from cost rankings.
• deprec This badge indicates the model may be deprecated or no longer actively supported/available via common APIs (like OpenRouter). Results might be stale.
• new This badge indicates results for this model were updated recently (within the last 7 days). This badge takes precedence over the deprec badge.

In the main results table, click the stats icon (query_stats) in the header of any context bin column (e.g., `128k (%)`). This opens a scatter plot showing the total cost vs. average score for all currently selected models specifically within that context length bin.

In the main results table, hover over any individual score cell (the percentage value) and click the document icon (article) that appears. This opens a modal where you can browse through the individual test runs for that model/bin combination, view the expected answer, and see the actual responses generated by the model along with their scores.

Yes! Go to the "Controls" tab (next to the "Leaderboard" tab). There you can find options to show/hide specific columns, like all the individual context length bins or the pricing information.

The dataset details and evaluation methods are described on Hugging Face: openai/mrcr. OpenAI also discussed results in their GPT-4.1 blog post.

We plan to add more models over time. Feel free to suggest specific models you'd like to see evaluated!

For questions, suggestions, or issues, please reach out on Twitter: @DillonUzar.