Executive summary
Eight generated answers do not always describe the same process. A new study fixes the candidate count at eight and changes only execution: all answers in one batch or spread across as many as eight serial calls. On A100 graphics processors, serial calls use 4.64 to 4.86 times as much gross device energy and reach 5.77 to 6.12 times the P95 latency. For FQI, this is a general quality problem: an outcome metric without a released process recipe can hide changes that materially affect cost, throughput and operational suitability.
FQI takeaway
- Release the process recipe: Candidate count, batch size, call count, hardware and parallelism belong in the controlled configuration.
- State measurement boundaries: Energy and timing values are comparable only when warm-up, model loading, idle power, device scope and statistics use the same definitions.
- Assess quality and system impact together: Accuracy gains support release only when latency, energy, throughput and failure risk remain acceptable in context.
The principle at a glance
Test-time scaling generates several candidate solutions and selects the most frequent answer. The number N commonly describes this budget. The study shows that N captures quantity only. For N = 8, it compares four schedules: one call with eight candidates, two calls with four each, four calls with two each and eight calls with one each.
Terms explained
- Batch: A group of candidates processed in one model call.
- P95 latency: The time threshold within which 95% of measured queries complete.
- Throughput: The number of queries processed per unit of time.
- Gross GPU energy: Graphics-processor energy during the measured interval without subtracting idle consumption.
- Measurement boundary: The definition of which components, process phases and energy uses a metric includes or excludes.
- AI: Artificial intelligence; systems that derive patterns from data or support tasks automatically.
- Prompt: A task instruction or input for a language-processing system.
- NIST: The US National Institute of Standards and Technology.
- Token: A technical text-processing unit, usually a word or word part.
Deep dive
The paper first confirms the expected value of additional candidates. Across 500 GSM8K mathematics task instructions, or prompts, accuracy rises from N = 1 to N = 8 by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. These accuracy runs have no per-N energy or latency measurements.
The main experiment then fixes N at eight. All four schedules run on the same 100 GSM8K prompts over three repetitions. Moving from one batched call to eight single-candidate calls raises gross GPU energy to 4.64 times the baseline for Phi-3 and 4.86 times for Qwen. P95 latency rises to 5.77 and 6.12 times the baseline, while throughput falls to 16.7% and 17.9% of the batched reference.
Two additional A100 jobs per model confirm the direction with narrow ratio ranges. A separate experiment using 500 short SciQ answers on V100 hardware also finds higher energy and latency with more calls, although the factors differ. The source therefore does not claim one universal ratio for all systems.
The study measures graphics-processor energy reported by NVML from the start of a query until after voting and synchronisation. Prompt processing, generation and answer selection are included. Model loading, warm-up, final grading and reporting-time counting of technical text-processing units, or tokens, are excluded. Idle energy is not subtracted; host, cooling and power-supply energy are outside the boundary.
The FQI transfer is process-recipe release. A controlled artificial intelligence (AI) configuration records more than model, prompt and N. It includes call structure, batch size, parallelism, hardware class, precision, stopping rules and measurement boundary. A change request evaluates domain quality, P95 or P99 latency, throughput and resource impact. A schedule change may require revalidation.
The National Institute of Standards and Technology (NIST), the US measurement agency, similarly calls for evaluation under deployment-like conditions, documented generalisation limits and assessment of environmental impact. This guidance does not define universal energy or latency limits.
Limitations and risks
The preprint has not been peer reviewed and covers two models, two workloads, and A100 and V100 hardware using batch-scheduled generation. Larger models, dynamic or continuous batching, shared hardware, other inference systems and multi-GPU execution may behave differently. The study does not isolate repeated prompt processing, synchronisation and batching effects individually. Gross GPU energy is not a complete data-centre or environmental account. Its percentages and ratios must not be adopted as universal planning values.
FQI conclusion
Candidate count is an outcome metric, not a complete process description. A team releasing AI inspection or decision support must treat execution as a controlled recipe. Only the combination of domain result, schedule, measurement boundary and system impact makes comparisons reproducible. This prevents two processes from looking identical on paper while one requires several times as much time and energy in operation.
Sources & transparency
- Kashaniyan and Jannesari: Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling, arXiv:2609.19499v1, especially Sections IV–VII.
- National Institute of Standards and Technology: AI Risk Management Framework Core, especially MEASURE 2.3, 2.5, 2.12 and 2.13.
Editorial note: models, workloads, schedules and measurements are reported source findings. Process-recipe release and its transfer to change control and process validation are FQI interpretation. AI-assisted and checked against the cited sources.