
Choose what must be new at evaluation time: another row, an unseen source family, or a later case. Preserve that boundary in a versioned manifest before tuning the prompt. A reproducible split can still be the wrong split.
A New Case ID Can Hide an Old Problem
Imagine a support assistant evaluated on questions derived from resolved tickets. Ticket A produces three cases: the original question, a paraphrase, and a follow-up asking for the same remedy. You tune the prompt on the original and put the paraphrase in the final test. Their IDs differ, but the remedy and the surrounding story have already shaped development.
This is a constructed example, not a measured benchmark. If the release claim is that the assistant handles unseen ticket families, the assignment does not test that claim cleanly. It might still be a useful regression test for known material. The problem is calling it evidence about a different population.
scikit-learn’s cross-validation documentation explains how tuning against test results can leak information into model selection. Here we apply that principle to prompt and retrieval-setting selection; no model weight update is required for a developer to learn from test failures.
Pick the Generalization Question First
Write the intended claim before picking a percentage. “Works on more cases from our current support library” and “works on new incident families next month” need different evidence. A single split need not answer both questions. Keep separate named evaluations when the product needs both.
| Assignment | Question it can support | What to inspect |
|---|---|---|
| Random rows | Performance on withheld rows from the sampled population, subject to the sampling assumptions. | Shared tickets, templates and derived variants can undermine an unseen-family claim. |
| Whole source families | Performance on the reserved families represented by that sampling design. | The family definition may miss shared templates or merge unrelated problems. |
| Availability-time cutoff | Performance on later cases using information available before the cutoff. | Earlier and later revisions may still belong to the same family. |
The scikit-learn documentation describes group-aware splitting and explains why shuffled evaluation can misrepresent time-dependent data. Those are methodological building blocks, not a universal prescription for LLM tests. A group holdout changes the task mix when large families cluster in one partition. A time holdout can expose a changed product or policy rather than just a weaker prompt. Report those differences instead of trying seeds until the preferred score appears.
Make Source Lineage Part of the Dataset
Give each case a stable case ID and a source-family ID that follows its origin. Three questions generated from one incident should carry the same incident family even when their wording differs. If a translated document and its original express the same source material, decide whether that relationship belongs in the same family for your evaluation claim.
Do not equate a filename with provenance. Copies can have different names, and a single file can contain unrelated cases. Record the original source, revision, derivation method, and the rule used to form families. For a temporal evaluation, use the time information became available to the system, not just the export date. A November export can contain answers written after the date your historical experiment claims to stop.
Our suggested manifest has case_id, source_family_id, partition, source_revision, available_at, and dataset_version. Keep sensitive source content outside a broadly accessible manifest; an opaque ID still needs appropriate access controls when it identifies a private record. Freeze the manifest with the prompt, retrieval snapshot, and scoring rubric used for the release comparison. The production eval guide covers the surrounding harness.
Run a Small Audit Before Running the Model
Save the following program as split-audit.cjs and run node split-audit.cjs. It constructs twelve rows: three dated revisions for each of four families. The row assignment intentionally puts revision one in development and later revisions in test. It is a clear counterexample, not a random-split simulation. The group assignment reserves A/B for development and C/D for test.
// Synthetic source-family audit; Node.js built-ins only, no model calls.
const rows = ['A', 'B', 'C', 'D'].flatMap(source_family_id =>
[1, 2, 3].map(revision => ({
case_id: `${source_family_id}-${revision}`,
source_family_id,
available_at: `2026-11-0${revision}`
})));
function audit(manifest) {
const ids = new Set();
const families = { development: new Set(), test: new Set() };
for (const row of manifest) {
if (!row.case_id || ids.has(row.case_id) || !row.source_family_id ||
!Object.hasOwn(families, row.partition)) {
throw new Error('Invalid or duplicate case record');
}
ids.add(row.case_id);
families[row.partition].add(row.source_family_id);
}
if (families.development.size === 0 || families.test.size === 0) {
throw new Error('Both partitions must be nonempty');
}
return [...families.test].filter(id => families.development.has(id)).sort();
}
const rowSplit = rows.map(row => ({ ...row,
partition: row.case_id.endsWith('-1') ? 'development' : 'test' }));
const groupSplit = rows.map(row => ({ ...row,
partition: ['A', 'B'].includes(row.source_family_id) ? 'development' : 'test' }));
const timeSplit = rows.map(row => ({ ...row,
partition: row.available_at <= '2026-11-01' ? 'development' : 'test' }));
const output = { runtime: process.version, dependencies: 'Node.js built-ins only',
row_shared_families: audit(rowSplit), group_shared_families: audit(groupSplit),
time_shared_families: audit(timeSplit), group_manifest: groupSplit };
if (output.group_shared_families.length !== 0 || output.row_shared_families.length !== 4 ||
output.time_shared_families.length !== 4) throw new Error('Unexpected fixture result');
for (const bad of [[...groupSplit, groupSplit[0]],
groupSplit.map((row, i) => i === 0 ? { ...row, source_family_id: '' } : row)]) {
let rejected = false;
try { audit(bad); } catch { rejected = true; }
if (!rejected) throw new Error('Invalid manifest accepted');
}
console.log(JSON.stringify(output, null, 2));
Executed locally on 2026-11-14 using Node.js v24.13.0 with built-ins only. The row and time assignments each shared families A, B, C and D across partitions. The group assignment shared none and produced six development rows and six test rows. The program also rejected a duplicate case ID and a missing family ID. No model calls or quality measurements were made.
Read the output as an audit of the labels you supplied. An empty shared-family list does not prove semantic independence: someone may have assigned two versions of the same incident to different families. Exact-text hashes can help find copies, but paraphrases require a broader review. A similarity search can supply candidate pairs for inspection; a threshold is a chosen review aid, not proof of common origin.
For an unseen-family release claim, fail the audit when a family crosses the boundary. For a time evaluation on known families, that overlap can be intentional; record it and name the test accordingly. If you require both future cases and new families, reserve later families explicitly and report what was excluded. A simple chronological cutoff does not enforce both constraints.
Keep the Holdout Out of Development Decisions
Partitioning once is insufficient if test failures become tomorrow’s prompt examples. Decide who may inspect the final test outputs and what happens after that inspection changes the system. A practical choice is to move consumed holdout cases into a regression suite and reserve a new version for the next untouched comparison. Preserve the old results rather than silently replacing difficult cases.
scikit-learn’s leakage guidance also warns against learning transformations from the test subset. In this LLM workflow, use the analogous separation for choosing examples, retrieval settings, and task-specific decision thresholds. Labeling source families to reconcile duplicates is a data correction and may use the test cases. Adjusting prompts or settings after seeing test scores is tuning. Document the first, and never let test scores decide which split you keep.
When a duplicate is discovered after evaluation, record the affected cases and the original assignment. Rebuild the split under the declared rule and rerun the affected evaluation before making a new release claim. Do not delete only the failing duplicate: that turns a data correction into score selection. There is no universal split fraction here; reserve enough families and task coverage for the decision, while reporting the resulting imbalance.
State What the Audit Leaves Unresolved
This procedure checks your dataset construction. It does not detect whether a foundation model encountered the source during pretraining, establish statistical independence, or certify that the scorer measures the right behavior. The answer-key isolation article addresses a separate path: an agent reading grader-only information during a run. Both failures can exist in the same evaluation.
Your decision record should identify the target population, grouping rule, time rule, dataset version, case and family counts, overlap result, exclusions, and whether test results have influenced development. Use confidence intervals only after defining the units and comparison they describe. A precise interval around a contaminated comparison does not repair its design.
FAQ
Does a fixed random seed prevent data leakage?
No. A seed can reproduce an assignment, but related cases can still land on both sides of that assignment.
Can a chronological split also contain the same source families?
Yes. A later revision can enter the test set while an earlier revision stays in development. That split evaluates later cases, not necessarily unseen source families.
Does this audit detect contamination in a model’s pretraining data?
No. It checks the declared case IDs, source-family labels, and partitions in your own evaluation manifest. It cannot establish what a foundation model saw during pretraining.
Can I keep using a holdout after tuning on its failures?
You can retain it as a regression set, but it has influenced development. Use a fresh reserved set for the next evaluation that is meant to be untouched.
Primary sources checked 2026-11-14
- scikit-learn: cross-validation — held-out evaluation, group-aware splitting, tuning and time dependence.
- scikit-learn: common pitfalls — leakage from development and learned preprocessing.
The source-family manifest, workflow recommendations and synthetic fixture are this article’s applications of those principles. They do not constitute a vendor standard or a production benchmark.