327 lines
7.5 KiB
Markdown
327 lines
7.5 KiB
Markdown
# LLM Benchmarking Tool
|
||
|
||
The following benchmarks were used to 1) measure throughput of configured models, on the available hardware (NVIDIA RTX 6000 PRO GPUs), aswell as to debug connection issues that arised during the configuration of the pipelines.
|
||
|
||
Benchmarks were created for qwen 3.5 and gpt oss. Mainly GPT OSS was used during the project (because of throughput and
|
||
output quality)
|
||
|
||
|
||
|
||
------------------
|
||
# How-to benchmark:
|
||
|
||
Benchmark LLM deployments using **batch request patterns** - sends N requests simultaneously to measure concurrent throughput.
|
||
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
## Dataset Generation (Optional)
|
||
|
||
You have **3 input options**:
|
||
|
||
### 1. Generated Prompts (Default)
|
||
Automatically generates synthetic text to match token counts.
|
||
|
||
### 2. Real Conversations
|
||
Use conversations from HuggingFace datasets:
|
||
|
||
```bash
|
||
# Generate conversation dataset (takes ~5 minutes)
|
||
python create_test_dataset.py
|
||
|
||
# Custom buckets
|
||
python create_test_dataset.py --buckets 1000 5000 10000 --chains_per_bucket 64
|
||
|
||
# Output to custom location
|
||
python create_test_dataset.py --output data/conversations.json
|
||
```
|
||
|
||
This creates a JSON file with real conversations bucketed by token count. The benchmark will cycle through these conversations instead of repeating the same synthetic prompt.
|
||
|
||
### 3. Custom Text
|
||
Provide your own text directly:
|
||
|
||
```bash
|
||
# Via CLI
|
||
python benchmark_llm.py --text "Your custom text here..."
|
||
|
||
# Or in config file
|
||
text: "Analyze this large document about..."
|
||
```
|
||
|
||
## Quick Start
|
||
|
||
### 1. Create Configuration File
|
||
|
||
```yaml
|
||
endpoint:
|
||
url: https://b5cee612-b599-4524-a893-7698c9e75948.services.ubiops.development.vlam.ai
|
||
api_key: your-api-key
|
||
model_name: your-model
|
||
|
||
benchmark:
|
||
input_tokens: [1000, 5000, 10000]
|
||
batch_sizes: [16, 32, 64, 128]
|
||
num_batches: 10
|
||
output_tokens: 256
|
||
dataset: test_conversations.json # Optional: real conversations
|
||
text: null # Optional: custom text input
|
||
|
||
runtime:
|
||
request_timeout: 300
|
||
delay_between_runs: 5
|
||
log_io: false
|
||
wait_for_ready: true
|
||
```
|
||
|
||
### 2. Run Benchmark
|
||
|
||
```bash
|
||
python benchmark_llm.py --config benchmark_config.yaml
|
||
```
|
||
|
||
### 3. Generate Visualizations
|
||
|
||
```bash
|
||
python visualize_results.py --input results/results_your-model/benchmark_results.json
|
||
```
|
||
|
||
## Usage
|
||
|
||
### Configuration File
|
||
|
||
```bash
|
||
python benchmark_llm.py --config benchmark_config.yaml
|
||
```
|
||
|
||
### CLI Arguments
|
||
|
||
```bash
|
||
# With dataset
|
||
python benchmark_llm.py \
|
||
--endpoint_url https://api.example.com/v1 \
|
||
--api_key YOUR_KEY \
|
||
--model_name gpt-4 \
|
||
--input_tokens 1000 5000 10000 \
|
||
--batch_sizes 16 32 64 128 \
|
||
--num_batches 10 \
|
||
--output_tokens 256 \
|
||
--dataset test_conversations.json
|
||
|
||
# With custom text
|
||
python benchmark_llm.py \
|
||
--endpoint_url https://api.example.com/v1 \
|
||
--api_key YOUR_KEY \
|
||
--model_name gpt-4 \
|
||
--batch_sizes 32 \
|
||
--num_batches 10 \
|
||
--text "Analyze the following document about cloud architecture..."
|
||
```
|
||
|
||
## How It Works
|
||
|
||
### Batch Execution
|
||
|
||
The tool sends batches of N requests **simultaneously**:
|
||
|
||
```
|
||
Batch 0: [Req 1, Req 2, ..., Req 32] ← All start at exact same time
|
||
[Wait for all to complete]
|
||
|
||
Batch 1: [Req 33, Req 34, ..., Req 64] ← All start at exact same time
|
||
[Wait for all to complete]
|
||
```
|
||
|
||
This ensures:
|
||
- All requests in a batch have **identical** `time_created` timestamps
|
||
- Concurrent load testing
|
||
- Accurate burst performance measurement
|
||
|
||
### Request Calculation
|
||
|
||
```
|
||
total_requests = num_batches × batch_size
|
||
```
|
||
|
||
**Example:**
|
||
```yaml
|
||
batch_sizes: [32]
|
||
num_batches: 10
|
||
|
||
# Result: 10 batches × 32 requests = 320 total requests
|
||
# Each batch sends 32 requests simultaneously
|
||
```
|
||
|
||
## Key Metrics
|
||
|
||
### Throughput
|
||
- **Tokens/second** across all requests in a batch
|
||
- Measures system's ability to handle concurrent load
|
||
- Higher is better
|
||
|
||
### Time to First Token (TTFT)
|
||
- Latency until first content token appears
|
||
- Critical for user experience
|
||
- Lower is better
|
||
|
||
### Latency Percentiles
|
||
- **P50 (median)**: Typical request latency
|
||
- **P95**: 95% of requests complete faster
|
||
- **P99**: 99% of requests complete faster
|
||
|
||
### Batch Metrics
|
||
```json
|
||
{
|
||
"batch_metrics": {
|
||
"num_batches": 10,
|
||
"avg_batch_throughput": 2456.78,
|
||
"min_batch_throughput": 2301.45,
|
||
"max_batch_throughput": 2589.12
|
||
}
|
||
}
|
||
```
|
||
|
||
## Output Structure
|
||
|
||
```
|
||
results/
|
||
└── results_your-model/
|
||
├── benchmark_results.json # Raw benchmark data
|
||
├── benchmark_io.log # I/O logs (if enabled)
|
||
├── config_used.yaml # Config copy (API key redacted)
|
||
├── throughput.png # Throughput vs batch size
|
||
├── ttft.png # TTFT vs batch size
|
||
└── latency_percentiles.png # Latency distribution
|
||
```
|
||
|
||
## Configuration Reference
|
||
|
||
### Endpoint Configuration
|
||
|
||
```yaml
|
||
endpoint:
|
||
url: string # OpenAI-compatible endpoint URL
|
||
api_key: string # API authentication key
|
||
model_name: string # Model identifier
|
||
```
|
||
|
||
### Benchmark Configuration
|
||
|
||
```yaml
|
||
benchmark:
|
||
input_tokens: list[int] # Token counts to test [1000, 5000, 10000]
|
||
batch_sizes: list[int] # Batch sizes to test [16, 32, 64, 128]
|
||
num_batches: int # Number of batches per config (default: 10)
|
||
output_tokens: int # Max output tokens (default: 256)
|
||
```
|
||
|
||
**Understanding batch_sizes:**
|
||
- `batch_sizes: [16]` → Sends 16 requests simultaneously
|
||
- `batch_sizes: [32]` → Sends 32 requests simultaneously
|
||
- `batch_sizes: [16, 32, 64]` → Tests 3 different batch sizes
|
||
|
||
### Runtime Configuration
|
||
|
||
```yaml
|
||
runtime:
|
||
request_timeout: int # Timeout per request in seconds (default: 300)
|
||
delay_between_runs: int # Delay between configs in seconds (default: 5)
|
||
log_io: bool # Enable I/O logging (default: false)
|
||
wait_for_ready: bool # Wait for model init (default: true)
|
||
max_init_retries: int # Max init attempts (default: 10)
|
||
init_retry_delay: int # Delay between init attempts (default: 30)
|
||
```
|
||
|
||
## Example Output
|
||
|
||
```
|
||
Starting benchmark: 10 batches × 32 requests/batch = 320 total
|
||
Input: 5000 tokens, Output: 256 tokens
|
||
============================================================
|
||
|
||
Batch 0: 32/32 successful, 12.34s, 2456.78 tok/s
|
||
Batch 1: 32/32 successful, 12.45s, 2401.23 tok/s
|
||
Batch 2: 32/32 successful, 12.56s, 2389.45 tok/s
|
||
...
|
||
|
||
✓ Benchmark complete in 125.67s
|
||
Success: 100% (320/320)
|
||
P95 Latency: 13.45s
|
||
Throughput: 2428.56 tokens/s
|
||
Avg Batch Throughput: 2429.01 tokens/s
|
||
```
|
||
|
||
## Use Cases
|
||
|
||
### 1. Finding Optimal Batch Size
|
||
|
||
Test multiple batch sizes to find the sweet spot:
|
||
|
||
```yaml
|
||
batch_sizes: [16, 32, 64, 128, 256]
|
||
num_batches: 10
|
||
```
|
||
|
||
Compare the `throughput.png` to see where throughput peaks.
|
||
|
||
### 2. Stress Testing
|
||
|
||
Test maximum burst capacity:
|
||
|
||
```yaml
|
||
batch_sizes: [256]
|
||
num_batches: 5
|
||
```
|
||
|
||
Sends 256 simultaneous requests per batch.
|
||
|
||
### 3. Performance Profiling
|
||
|
||
Test different input sizes at various batch sizes:
|
||
|
||
```yaml
|
||
input_tokens: [1000, 2500, 5000, 10000]
|
||
batch_sizes: [16, 32, 64, 128]
|
||
```
|
||
|
||
Comprehensive performance matrix across configurations.
|
||
|
||
## Advanced Usage
|
||
|
||
### Enable I/O Logging
|
||
|
||
Log all input prompts and outputs for debugging:
|
||
|
||
```bash
|
||
python benchmark_llm.py --config benchmark_config.yaml
|
||
# Set log_io: true in config
|
||
```
|
||
|
||
Or:
|
||
|
||
```bash
|
||
python benchmark_llm.py --log_io ...
|
||
```
|
||
|
||
Results saved to `benchmark_io.log`.
|
||
|
||
### Skip Model Initialization
|
||
|
||
If model is already warm:
|
||
|
||
```bash
|
||
python benchmark_llm.py --config benchmark_config.yaml --skip_init_wait
|
||
```
|
||
|
||
### Custom Timeout
|
||
|
||
For large batches or slow responses:
|
||
|
||
```bash
|
||
python benchmark_llm.py --request_timeout 600 ...
|
||
```
|