ML Experiment

Turn an empirical question into evidence that supports a decision: a comparison that can answer it, training data fit for its intended use, and a serving configuration whose results you can attribute. A trained checkpoint, a working runner, or a higher number is not by itself that evidence; a negative finding or a bounded unresolved question can finish the work.

Use only the parts the task needs. A training comparison, a serving comparison, and a reproduction each stand alone, and one question often needs two. What an evaluation measures, and whether a built evaluation can be trusted, belong to the evaluation skills below.

Start from the question

Recover the question, the decision the answer could change, existing results, the model, data, and serving setup in use, and the constraints before asking. Name the unit being compared (a run, record, response, or task completion) and keep counts in that unit; records, source groups, and independent observations are not interchangeable.

A design or review request returns findings. Building, training, adapting, or executing needs the corresponding request. Fast checks within the same question and always-on resources may proceed. Present the observation, the proposed change, and one sentence predicting the result before changing direction, before any run that spends beyond the always-on tier or the project's stated budget, and before long new work, unless that activity is already authorized. Spend is the gate, not duration: a short run on paid accelerators needs the same presentation as a multi-day one. When the project has a praxis profile, its compute ladder and approval threshold define the tiers. Data access, shared hardware, external annotation, and provider limits still apply to short runs, and public availability does not establish permission or consent.

Use the project's runner, records, identifiers, schemas, and plan. Do not create a new tracking framework, registry, or harness.

Experiment

State which alternatives the next observation could distinguish and how the answer would change the work. For performance improvement, identify the intended benefit and a credible current baseline; for reproduction, the original claim and its conditions. Choose the smallest comparison that can answer this round: the changed factor, the baseline, comparison conditions, meaningful outputs, and why it is informative. Tune nuisance settings fairly when they interact with the factor under study; identical settings are not always a fair comparison. Separate exploratory selection from a confirmatory comparison.

With little compute, iterate at the smallest model and data scale at which the phenomenon still appears, and say when a result may not transfer to full scale. With little data, spend it on evaluation before training: a few hundred labelled examples usually buy a credible measurement and rarely a credible fine-tune.