FP8
Compare performance across different hardware configurations. See each full report for detailed metrics.
Per-user generation speed across context lengths. The top edge of each band shows single-user performance; the bottom edge shows performance at maximum tested concurrency.
Time until the first token is generated. The bottom edge of each band shows single-user TTFT; the top edge shows TTFT at maximum tested concurrency. Lower is better for responsive user experience.
Maximum concurrent requests while maintaining acceptable user experience. Thresholds vary by use case. See individual reports for details.
"+" indicates capacity exceeded tested concurrency range. "~" indicates estimated from available data.
Benchmark methodology →Get a custom recommendation based on your specific workload, budget, and performance requirements.
1x H200 SXM
February 03, 2026
Infrastructure decisions require real performance data. This report measures user-facing performance, showing how many concurrent users a configuration can support at a given context length before performance degrades.
This benchmark evaluates Qwen3-Coder-30B-A3B-Instruct (Qwen, 30B parameters, Mixture-of-Experts) running in FP8 precision on 1x H200 SXM (141GB VRAM).
Test parameters: Context lengths from 1K - 256K tokens. Concurrency from 1 - 6 requests. 1024 output tokens per request. No prompt caching. No speculative decoding. Full-precision KV cache.
| Metric | Result |
|---|---|
| Peak System Throughput | 599.6 tok/s @ 6 concurrent requests, 1K context |
| TTFT Single Request | 76ms (1K context) → 47.6s (256K context) |
| Generation Speed Single Request | 164.9 tok/s (1K context) → 79.9 tok/s (256K context) |
| Chatbot Capacity | ~15 concurrent requests @ 32K context |
| Throughput Scaling | 4.6× from 1 to 6 concurrent requests |
| Success Rate | 100.0% across 5.4K requests |
The table below maps this configuration's performance to common deployment scenarios. Capacity limits are where TTFT or generation speed falls below accepted thresholds for a comfortable user experience.
| Use Case | TTFT Threshold | Speed Threshold | Notes | Details |
|---|---|---|---|---|
| Code Completion | 2s e2e | N/A | Supports ~42 concurrent requests within accepted thresholds. | View details |
| Short-form Chatbot | 10s | 10 tok/s | Supports 125+ concurrent requests with fast responses. Additional capacity likely available. | View details |
| General Chatbot | 8s | 15 tok/s | Supports ~15 concurrent requests within accepted thresholds. | View details |
| Long Document Processing | 12s | 15 tok/s | Supports 5 concurrent requests within accepted thresholds. | View details |
| Automated Coding Assistant | 12s | 20 tok/s | Supports 2 concurrent requests within accepted thresholds. | View details |
The limits shown are conservative. Beyond these points, the system continues functioning with slower response times that may still be acceptable for your specific use case.
Want to validate your specific configuration?
Aggregate token generation across all concurrent requests. Measures output tokens only. Prompt tokens processed during prefill are excluded.
Average system throughput across 1K - 256K tokens context at 1 - 6 concurrent requests.
| Condition | Throughput |
|---|---|
| Peak (1K context, 6 requests) | 599.6 tok/s |
| 32K context, 6 requests | 238.4 tok/s |
| 128K context, 6 requests | 42.3 tok/s |
At peak throughput, this configuration produces approximately 2.2 million tokens per hour. This is relevant for batch workloads like document processing, synthetic data generation, or offline analysis. Higher concurrency or shorter contexts can increase this further.
Token generation rate experienced by each individual user. This is the speed at which text streams into their response, also referred to as "decode speed" or "decode throughput." As concurrency increases, per-user speed decreases since GPU resources are shared across requests.
Average per-user generation speed across 1K - 256K tokens context at 1 - 6 concurrent requests.
| Speed | User Experience |
|---|---|
| < 15 tok/s | Slow; may be slower than reading speed |
| 15–25 tok/s | Acceptable; keeps pace with reading |
| 25–50 tok/s | Fast; exceeds reading speed |
| > 50 tok/s | Very fast; text appears nearly instantly |
At 10.5 tok/s (the lowest measured point: 256K context, 4 concurrent requests), this configuration slows below fast reading speed in the most demanding scenarios. Single-user performance at 1K context reaches 164.9 tok/s.
Time from request submission to first response token. The primary metric for perceived responsiveness. TTFT has two components:
At low concurrency, prefill dominates. Under load, queue wait becomes the larger factor. See Queue Wait Times and Prefill Speed in Technical Analysis.
Average TTFT across 1K - 256K tokens context at 1 - 6 concurrent requests.
| TTFT | User Experience |
|---|---|
| < 500ms | Feels instant |
| 500ms–2s | Feels responsive |
| 2–5s | Noticeable but still acceptable |
| 5–10s | Feels slow; generally acceptable at higher context lengths |
| > 10s | Can be frustrating; users may retry or abandon |
How many concurrent requests can this configuration handle for different workloads? Each chart below shows performance metrics as concurrency increases at a specific context length. Dashed lines indicate quality thresholds, the point where user experience degrades below acceptable levels. The "capacity limit" is the tested or estimated point where the first threshold is reached.
Inline code suggestions in IDEs, like autocomplete. Responsiveness is critical. This test generates 128 output tokens per request (vs. 1024 elsewhere) to match typical autocomplete length. The key metric is end-to-end latency, not TTFT.
Threshold: End-to-end latency < 2,000ms
Average end-to-end latency and throughput at 1K context. Dashed line indicates quality threshold.
| Metric | @ 1 request | @ 42 requests | @ 100 requests |
|---|---|---|---|
| End-to-end latency | 852ms | ~2,009ms (threshold exceeded) | 3491ms (threshold exceeded) |
| Throughput | 165 tok/s | ~70 tok/s | 39 tok/s |
Capacity limit: ~42 concurrent requests
At 42 concurrent requests, end-to-end latency reaches ~2009ms, just above the 2,000ms threshold.
Quick conversational exchanges: customer support queries, simple Q&A, single-turn requests. 8K context accommodates a few back-and-forth messages plus system prompt. User expectations are more forgiving for these scenarios. 10+ tok/s is acceptable for reading streamed responses from a support chatbot.
Thresholds: TTFT < 10s, generation speed > 10 tok/s
Average per-user generation speed and TTFT at 8K context.
| Metric | @ 1 request | @ 75 requests | @ 125 requests |
|---|---|---|---|
| TTFT | 0.2s | 3.0s | 4.0s |
| Generation speed | 149 tok/s | 25 tok/s | 21 tok/s |
Capacity limit: 125+ concurrent requests
At 125 concurrent requests, TTFT is 4.0 seconds and generation speed is 21 tok/s, both well within acceptable bounds. Capacity likely extends higher.
ChatGPT-style chatbot. If you're deploying a multi-turn conversational chatbot, this benchmark shows how many concurrent requests you can support while matching acceptable responsiveness. 32K context matches ChatGPT's limit.
Thresholds: TTFT < 8s, generation speed > 15 tok/s
Average per-user generation speed and TTFT at 32K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 15 requests | @ 21 requests |
|---|---|---|---|
| TTFT | 1.2s | ~8.1s (threshold exceeded) | 9.5s (threshold exceeded) |
| Generation speed | 142 tok/s | ~33 tok/s | 25 tok/s |
Capacity limit: ~15 concurrent requests
At 15 concurrent requests, TTFT reaches ~8.1 seconds, just above the 8-second threshold. Generation speed at this concurrency is ~33 tok/s, above the 15 tok/s minimum.
Summarizing reports, extracting data from contracts, analyzing lengthy documents. 64K tokens handles documents up to roughly 125-160 pages depending on formatting and density.
Users typically tolerate higher latency for document processing since they understand large inputs require more processing time. However, generation speed still needs to stay at or above reading speed.
Thresholds: TTFT < 12s, generation speed > 15 tok/s
Average per-user generation speed and TTFT at 64K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 5 requests | @ 7 requests |
|---|---|---|---|
| TTFT | 3.8s | 10.2s | 14.2s (threshold exceeded) |
| Generation speed | 123 tok/s | 44 tok/s | 30 tok/s |
Capacity limit: 5 concurrent requests
At 5 concurrent requests, TTFT reaches 10.2 seconds, just under the 12-second threshold. Generation speed at this concurrency is 44 tok/s, above the 15 tok/s minimum.
Agentic coding workloads: AI assistants that read large portions of a codebase to answer questions, refactor code, or implement features. 96K tokens handles roughly 8,000-9,000 lines of code, enough for significant repository context.
Agentic workflows chain multiple LLM calls (tool use, retrieval, iterative refinement). With caching properly configured, context persists between requests and only new tokens require processing, dramatically reducing TTFT for each step. These results represent worst-case TTFT where all context is processed at once.
Thresholds: TTFT < 12s, generation speed > 20 tok/s
Average per-user generation speed and TTFT at 96K context. Dashed lines indicate quality thresholds.
| Metric | @ 1 request | @ 2 requests | @ 6 requests |
|---|---|---|---|
| TTFT | 7.8s | 11.4s | 26.6s (threshold exceeded) |
| Generation speed | 113 tok/s | 70 tok/s | 25 tok/s |
Capacity limit: 2 concurrent requests
At 2 concurrent requests, TTFT reaches 11.4 seconds, just under the 12-second threshold. Generation speed at this concurrency is 70 tok/s, above the 20 tok/s minimum.
Infrastructure-level metrics that explain user-facing performance. Queue depth, prefill throughput, token generation latency, and scaling efficiency across load conditions. These help diagnose bottlenecks and validate infrastructure decisions.
Time a request waits for GPU availability before processing begins. At low concurrency, queue wait is near zero. As load increases, requests queue and wait times grow.
Queue wait is included in TTFT. Breaking it out separately helps identify whether latency is caused by GPU saturation (high queue wait) or context processing (high prefill time).
Average queue wait time across 1K - 256K tokens context at 1 - 6 concurrent requests.
At single concurrency, queue wait is effectively zero regardless of context length. At 4 concurrent requests with 256K context, queue wait reaches 67.4 seconds. Rising queue times signal GPU saturation, meaning requests are waiting for resources rather than being processed immediately.
Rate at which the model processes input context before generating output. Prefill speed determines the non-queue portion of TTFT. Higher prefill speeds mean faster time-to-first-token at a given context length.
Average per-user prefill speed across 1K - 256K tokens context at 1 - 6 concurrent requests.
| Concurrent Requests | Peaks At | Peak Speed |
|---|---|---|
| 1 | 8K context | 45,004 tok/s |
| 2 | 8K context | 42,899 tok/s |
| 4 | 8K context | 41,429 tok/s |
| 6 | 8K context | 43,384 tok/s |
Prefill speed peaks at a certain context length and then declines as additional context increases computational overhead. This peak can reflect GPU saturation (compute or memory bandwidth fully utilized) or engine configuration such as chunked prefill limits, which cap tokens processed per forward pass to maintain responsiveness under load. On the chart, this appears as lines that peak before reaching the longest context.
Time between consecutive tokens during generation. Determines the smoothness of streaming responses. Lower latency produces more fluid text output. ITL helps diagnose the underlying token-level behavior.
Average inter-token latency across 1K - 256K tokens context at 1 - 6 concurrent requests.
At single-user short context, inter-token latency is imperceptible (6ms). The highest latency observed was 149ms at 256K context with 4 concurrent requests, where individual tokens become visible as they stream.
Percentage of ideal linear scaling achieved as concurrency increases. 100% efficiency means doubling concurrent requests doubles total throughput with no per-user degradation. Real-world efficiency is always lower due to shared GPU resources.
Scaling efficiency across 1K - 256K tokens context at 1 - 6 concurrent requests.
Efficiency remains high at low concurrency where there are sufficient GPU resources to serve requests without contention. At higher concurrency, efficiency drops as requests compete for shared resources. High efficiency at your target concurrency indicates headroom for more users. Sharply dropping efficiency signals diminishing returns.
This page shows averages. Full percentile breakdowns (P50–P99) and GPU metrics (utilization, VRAM, temperature) available on request
GPU power draw under varying load conditions. Relevant for operational cost estimation, cooling requirements, and data center power budgeting.
Average GPU power draw across 1K - 256K tokens context at 1 - 6 concurrent requests.
Power consumption scales with both context length and concurrency. The highest power draw observed was 610W at 256K context with 4 concurrent requests, costing approximately $0.06/hour at $0.10/kWh. Higher concurrency or sustained load beyond tested conditions may increase power consumption further. For infrastructure planning, budget for peak power draw.
Get a custom benchmark for your configuration, or talk through your requirements with our team.