FP8
Compare performance across different hardware configurations. See each full report for detailed metrics.
Per-user generation speed across context lengths. The top edge of each band shows single-user performance; the bottom edge shows performance at maximum tested concurrency.
Time until the first token is generated. The bottom edge of each band shows single-user TTFT; the top edge shows TTFT at maximum tested concurrency. Lower is better for responsive user experience.
Maximum concurrent requests while maintaining acceptable user experience. Thresholds vary by use case. See individual reports for details.
"+" indicates capacity exceeded tested concurrency range. "~" indicates estimated from available data.
Benchmark methodology →Get a custom recommendation based on your specific workload, budget, and performance requirements.
1x H100 SXM
February 05, 2026
Infrastructure decisions require real performance data. This report measures user-facing performance, showing how many concurrent users a configuration can support at a given context length before performance degrades.
This benchmark evaluates Devstral-Small-2-24B-Instruct-2512 (Mistral AI, 24B parameters, Dense) running in FP8 precision on 1x H100 SXM (80GB VRAM).
Test parameters: Context lengths from 1K - 256K tokens. Concurrency from 1 - 3 requests. 1024 output tokens per request. No prompt caching. No speculative decoding. Full-precision KV cache.
| Metric | Result |
|---|---|
| Peak System Throughput | 273.6 tok/s @ 3 concurrent requests, 1K context |
| TTFT Single Request | 45ms (1K context) → 49.2s (256K context) |
| Generation Speed Single Request | 103.0 tok/s (1K context) → 56.3 tok/s (256K context) |
| Chatbot Capacity | 8 concurrent requests @ 32K context |
| Throughput Scaling | 2.9× from 1 to 3 concurrent requests |
| Success Rate | 100.0% across 3.8K requests |
The table below maps this configuration's performance to common deployment scenarios. Capacity limits are where TTFT or generation speed falls below accepted thresholds for a comfortable user experience.
| Use Case | TTFT Threshold | Speed Threshold | Notes | Details |
|---|---|---|---|---|
| Code Completion | 2s e2e | N/A | Supports ~21 concurrent requests within accepted thresholds. | View details |
| Short-form Chatbot | 10s | 10 tok/s | Supports ~42 concurrent requests within accepted thresholds. | View details |
| General Chatbot | 8s | 15 tok/s | Supports 8 concurrent requests within accepted thresholds. | View details |
| Long Document Processing | 12s | 15 tok/s | Supports 3 concurrent requests within accepted thresholds. | View details |
| Automated Coding Assistant | 12s | 20 tok/s | Best suited for single-user agentic workflows. For team environments, enable prompt caching or consider a smaller model. | View details |
The limits shown are conservative. Beyond these points, the system continues functioning with slower response times that may still be acceptable for your specific use case.
Want to validate your specific configuration?
Aggregate token generation across all concurrent requests. Measures output tokens only. Prompt tokens processed during prefill are excluded.
Average system throughput across 1K - 256K tokens context at 1 - 3 concurrent requests.
| Condition | Throughput |
|---|---|
| Peak (1K context, 3 requests) | 273.6 tok/s |
| 32K context, 3 requests | 114.7 tok/s |
| 128K context, 3 requests | 31.4 tok/s |
At peak throughput, this configuration produces approximately 985K tokens per hour. This is relevant for batch workloads like document processing, synthetic data generation, or offline analysis. Higher concurrency or shorter contexts can increase this further.
Token generation rate experienced by each individual user. This is the speed at which text streams into their response, also referred to as "decode speed" or "decode throughput." As concurrency increases, per-user speed decreases since GPU resources are shared across requests.
Average per-user generation speed across 1K - 256K tokens context at 1 - 3 concurrent requests.
| Speed | User Experience |
|---|---|
| < 15 tok/s | Slow; may be slower than reading speed |
| 15–25 tok/s | Acceptable; keeps pace with reading |
| 25–50 tok/s | Fast; exceeds reading speed |
| > 50 tok/s | Very fast; text appears nearly instantly |
At 28.4 tok/s (the lowest measured point: 128K context, 3 concurrent requests), this configuration remains fast across all tested scenarios. Single-user performance at 1K context reaches 103.0 tok/s.
Time from request submission to first response token. The primary metric for perceived responsiveness. TTFT has two components:
At low concurrency, prefill dominates. Under load, queue wait becomes the larger factor. See Queue Wait Times and Prefill Speed in Technical Analysis.
Average TTFT across 1K - 256K tokens context at 1 - 3 concurrent requests.
| TTFT | User Experience |
|---|---|
| < 500ms | Feels instant |
| 500ms–2s | Feels responsive |
| 2–5s | Noticeable but still acceptable |
| 5–10s | Feels slow; generally acceptable at higher context lengths |
| > 10s | Can be frustrating; users may retry or abandon |
How many concurrent requests can this configuration handle for different workloads? Each chart below shows performance metrics as concurrency increases at a specific context length. Dashed lines indicate quality thresholds, the point where user experience degrades below acceptable levels. The "capacity limit" is the tested or estimated point where the first threshold is reached.
Inline code suggestions in IDEs, like autocomplete. Responsiveness is critical. This test generates 128 output tokens per request (vs. 1024 elsewhere) to match typical autocomplete length. The key metric is end-to-end latency, not TTFT.
Threshold: End-to-end latency < 2,000ms
Average end-to-end latency and throughput at 1K context. Dashed line indicates quality threshold.
| Metric | @ 1 request | @ 21 requests | @ 100 requests |
|---|---|---|---|
| End-to-end latency | 1,288ms | ~2,010ms (threshold exceeded) | 6110ms (threshold exceeded) |
| Throughput | 103 tok/s | ~74 tok/s | 30 tok/s |
Capacity limit: ~21 concurrent requests
At 21 concurrent requests, end-to-end latency reaches ~2010ms, just above the 2,000ms threshold.
Quick conversational exchanges: customer support queries, simple Q&A, single-turn requests. 8K context accommodates a few back-and-forth messages plus system prompt. User expectations are more forgiving for these scenarios. 10+ tok/s is acceptable for reading streamed responses from a support chatbot.
Thresholds: TTFT < 10s, generation speed > 10 tok/s
Average per-user generation speed and TTFT at 8K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 42 requests | @ 100 requests |
|---|---|---|---|
| TTFT | 0.4s | ~9.1s | 69.8s (threshold exceeded) |
| Generation speed | 105 tok/s | ~27 tok/s | 25 tok/s |
Capacity limit: ~42 concurrent requests
At 42 concurrent requests, TTFT reaches ~9.1 seconds, just under the 10-second threshold. Generation speed at this concurrency is ~27 tok/s, above the 10 tok/s minimum.
ChatGPT-style chatbot. If you're deploying a multi-turn conversational chatbot, this benchmark shows how many concurrent requests you can support while matching acceptable responsiveness. 32K context matches ChatGPT's limit.
Thresholds: TTFT < 8s, generation speed > 15 tok/s
Average per-user generation speed and TTFT at 32K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 8 requests | @ 11 requests |
|---|---|---|---|
| TTFT | 2.1s | 8.1s (threshold exceeded) | 12.8s (threshold exceeded) |
| Generation speed | 92 tok/s | 32 tok/s | 25 tok/s |
Capacity limit: 8 concurrent requests
At 8 concurrent requests, TTFT reaches 8.1 seconds, just above the 8-second threshold. Generation speed at this concurrency is 32 tok/s, above the 15 tok/s minimum.
Summarizing reports, extracting data from contracts, analyzing lengthy documents. 64K tokens handles documents up to roughly 125-160 pages depending on formatting and density.
Users typically tolerate higher latency for document processing since they understand large inputs require more processing time. However, generation speed still needs to stay at or above reading speed.
Thresholds: TTFT < 12s, generation speed > 15 tok/s
Average per-user generation speed and TTFT at 64K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 3 requests | @ 5 requests |
|---|---|---|---|
| TTFT | 5.3s | 10.3s | 15.1s (threshold exceeded) |
| Generation speed | 82 tok/s | 41 tok/s | 29 tok/s |
Capacity limit: 3 concurrent requests
At 3 concurrent requests, TTFT reaches 10.3 seconds, just under the 12-second threshold. Generation speed at this concurrency is 41 tok/s, above the 15 tok/s minimum.
Agentic coding workloads: AI assistants that read large portions of a codebase to answer questions, refactor code, or implement features. 96K tokens handles roughly 8,000-9,000 lines of code, enough for significant repository context.
Agentic workflows chain multiple LLM calls (tool use, retrieval, iterative refinement). With caching properly configured, context persists between requests and only new tokens require processing, dramatically reducing TTFT for each step. These results represent worst-case TTFT where all context is processed at once.
Thresholds: TTFT < 12s, generation speed > 20 tok/s
Average per-user generation speed and TTFT at 96K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 2 requests | @ 3 requests |
|---|---|---|---|
| TTFT | 9.7s | 14.4s (threshold exceeded) | 19.3s (threshold exceeded) |
| Generation speed | 81 tok/s | 40 tok/s | 37 tok/s |
Capacity limit: 1 request
This configuration handles single-user agentic coding workloads with 9.7s TTFT and 81 tok/s generation speed, acceptable for individual use.
Infrastructure-level metrics that explain user-facing performance. Queue depth, prefill throughput, token generation latency, and scaling efficiency across load conditions. These help diagnose bottlenecks and validate infrastructure decisions.
Time a request waits for GPU availability before processing begins. At low concurrency, queue wait is near zero. As load increases, requests queue and wait times grow.
Queue wait is included in TTFT. Breaking it out separately helps identify whether latency is caused by GPU saturation (high queue wait) or context processing (high prefill time).
Average queue wait time across 1K - 256K tokens context at 1 - 3 concurrent requests.
At single concurrency, queue wait is effectively zero regardless of context length. At 3 concurrent requests with 128K context, queue wait reaches 11.4 seconds. Rising queue times signal GPU saturation, meaning requests are waiting for resources rather than being processed immediately.
Rate at which the model processes input context before generating output. Prefill speed determines the non-queue portion of TTFT. Higher prefill speeds mean faster time-to-first-token at a given context length.
Average per-user prefill speed across 1K - 256K tokens context at 1 - 3 concurrent requests.
| Concurrent Requests | Peaks At | Peak Speed |
|---|---|---|
| 1 | 1K context | 23,103 tok/s |
| 2 | 8K context | 20,586 tok/s |
| 3 | 1K context | 19,080 tok/s |
Prefill speed peaks at a certain context length and then declines as additional context increases computational overhead. This peak can reflect GPU saturation (compute or memory bandwidth fully utilized) or engine configuration such as chunked prefill limits, which cap tokens processed per forward pass to maintain responsiveness under load. On the chart, this appears as lines that peak before reaching the longest context.
Time between consecutive tokens during generation. Determines the smoothness of streaming responses. Lower latency produces more fluid text output. ITL helps diagnose the underlying token-level behavior.
Average inter-token latency across 1K - 256K tokens context at 1 - 3 concurrent requests.
At single-user short context, inter-token latency is imperceptible (10ms). The highest latency observed was 39ms at 128K context with 3 concurrent requests, still imperceptible to users.
Percentage of ideal linear scaling achieved as concurrency increases. 100% efficiency means doubling concurrent requests doubles total throughput with no per-user degradation. Real-world efficiency is always lower due to shared GPU resources.
Scaling efficiency across 1K - 128K tokens context at 1 - 3 concurrent requests.
Efficiency remains high at low concurrency where there are sufficient GPU resources to serve requests without contention. At higher concurrency, efficiency drops as requests compete for shared resources. High efficiency at your target concurrency indicates headroom for more users. Sharply dropping efficiency signals diminishing returns.
This page shows averages. Full percentile breakdowns (P50–P99) and GPU metrics (utilization, VRAM, temperature) available on request
GPU power draw under varying load conditions. Relevant for operational cost estimation, cooling requirements, and data center power budgeting.
Average GPU power draw across 1K - 256K tokens context at 1 - 3 concurrent requests.
Power consumption scales with both context length and concurrency. The highest power draw observed was 605W at 64K context with 2 concurrent requests, costing approximately $0.06/hour at $0.10/kWh. Higher concurrency or sustained load beyond tested conditions may increase power consumption further. For infrastructure planning, budget for peak power draw.
Get a custom benchmark for your configuration, or talk through your requirements with our team.