From 27111b7f5309d2592590612ab6216663964daa78 Mon Sep 17 00:00:00 2001 From: figmar Date: Sun, 9 Aug 2026 08:19:24 +0800 Subject: [PATCH] Publish hermes-skills-mlops via gitea-publish skill --- audiocraft/SKILL.md | 568 ++++++++++++ audiocraft/references/advanced-usage.md | 666 ++++++++++++++ audiocraft/references/troubleshooting.md | 504 +++++++++++ dspy/SKILL.md | 594 ++++++++++++ dspy/references/examples.md | 663 ++++++++++++++ dspy/references/modules.md | 475 ++++++++++ dspy/references/optimizers.md | 566 ++++++++++++ evaluating-llms-harness/SKILL.md | 498 ++++++++++ .../references/api-evaluation.md | 490 ++++++++++ .../references/benchmark-guide.md | 488 ++++++++++ .../references/custom-tasks.md | 602 +++++++++++++ .../references/distributed-eval.md | 519 +++++++++++ huggingface-hub/SKILL.md | 81 ++ llama-cpp/SKILL.md | 249 +++++ llama-cpp/references/advanced-usage.md | 504 +++++++++++ llama-cpp/references/hub-discovery.md | 168 ++++ llama-cpp/references/optimization.md | 89 ++ llama-cpp/references/quantization.md | 243 +++++ llama-cpp/references/server.md | 150 ++++ llama-cpp/references/troubleshooting.md | 442 +++++++++ obliteratus/SKILL.md | 342 +++++++ obliteratus/references/analysis-modules.md | 166 ++++ obliteratus/references/methods-guide.md | 141 +++ .../templates/abliteration-config.yaml | 33 + obliteratus/templates/analysis-study.yaml | 40 + obliteratus/templates/batch-abliteration.yaml | 41 + segment-anything/SKILL.md | 506 +++++++++++ segment-anything/references/advanced-usage.md | 589 ++++++++++++ .../references/troubleshooting.md | 484 ++++++++++ serving-llms-vllm/SKILL.md | 373 ++++++++ serving-llms-vllm/references/optimization.md | 226 +++++ serving-llms-vllm/references/quantization.md | 284 ++++++ .../references/server-deployment.md | 255 ++++++ .../references/troubleshooting.md | 447 +++++++++ weights-and-biases/SKILL.md | 598 +++++++++++++ weights-and-biases/references/artifacts.md | 584 ++++++++++++ weights-and-biases/references/integrations.md | 700 +++++++++++++++ weights-and-biases/references/sweeps.md | 847 ++++++++++++++++++ 38 files changed, 15215 insertions(+) create mode 100644 audiocraft/SKILL.md create mode 100644 audiocraft/references/advanced-usage.md create mode 100644 audiocraft/references/troubleshooting.md create mode 100644 dspy/SKILL.md create mode 100644 dspy/references/examples.md create mode 100644 dspy/references/modules.md create mode 100644 dspy/references/optimizers.md create mode 100644 evaluating-llms-harness/SKILL.md create mode 100644 evaluating-llms-harness/references/api-evaluation.md create mode 100644 evaluating-llms-harness/references/benchmark-guide.md create mode 100644 evaluating-llms-harness/references/custom-tasks.md create mode 100644 evaluating-llms-harness/references/distributed-eval.md create mode 100644 huggingface-hub/SKILL.md create mode 100644 llama-cpp/SKILL.md create mode 100644 llama-cpp/references/advanced-usage.md create mode 100644 llama-cpp/references/hub-discovery.md create mode 100644 llama-cpp/references/optimization.md create mode 100644 llama-cpp/references/quantization.md create mode 100644 llama-cpp/references/server.md create mode 100644 llama-cpp/references/troubleshooting.md create mode 100644 obliteratus/SKILL.md create mode 100644 obliteratus/references/analysis-modules.md create mode 100644 obliteratus/references/methods-guide.md create mode 100644 obliteratus/templates/abliteration-config.yaml create mode 100644 obliteratus/templates/analysis-study.yaml create mode 100644 obliteratus/templates/batch-abliteration.yaml create mode 100644 segment-anything/SKILL.md create mode 100644 segment-anything/references/advanced-usage.md create mode 100644 segment-anything/references/troubleshooting.md create mode 100644 serving-llms-vllm/SKILL.md create mode 100644 serving-llms-vllm/references/optimization.md create mode 100644 serving-llms-vllm/references/quantization.md create mode 100644 serving-llms-vllm/references/server-deployment.md create mode 100644 serving-llms-vllm/references/troubleshooting.md create mode 100644 weights-and-biases/SKILL.md create mode 100644 weights-and-biases/references/artifacts.md create mode 100644 weights-and-biases/references/integrations.md create mode 100644 weights-and-biases/references/sweeps.md diff --git a/audiocraft/SKILL.md b/audiocraft/SKILL.md new file mode 100644 index 0000000..824147f --- /dev/null +++ b/audiocraft/SKILL.md @@ -0,0 +1,568 @@ +--- +name: audiocraft-audio-generation +description: "AudioCraft: MusicGen text-to-music, AudioGen text-to-sound." +version: 1.0.0 +author: Orchestra Research +license: MIT +dependencies: [audiocraft, torch>=2.0.0, transformers>=4.30.0] +platforms: [linux, macos] +metadata: + hermes: + tags: [Multimodal, Audio Generation, Text-to-Music, Text-to-Audio, MusicGen] + +--- + +# AudioCraft: Audio Generation + +Comprehensive guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec. + +## When to use AudioCraft + +**Use AudioCraft when:** +- Need to generate music from text descriptions +- Creating sound effects and environmental audio +- Building music generation applications +- Need melody-conditioned music generation +- Want stereo audio output +- Require controllable music generation with style transfer + +**Key features:** +- **MusicGen**: Text-to-music generation with melody conditioning +- **AudioGen**: Text-to-sound effects generation +- **EnCodec**: High-fidelity neural audio codec +- **Multiple model sizes**: Small (300M) to Large (3.3B) +- **Stereo support**: Full stereo audio generation +- **Style conditioning**: MusicGen-Style for reference-based generation + +**Use alternatives instead:** +- **Stable Audio**: For longer commercial music generation +- **Bark**: For text-to-speech with music/sound effects +- **Riffusion**: For spectogram-based music generation +- **OpenAI Jukebox**: For raw audio generation with lyrics + +## Quick start + +### Installation + +```bash +# From PyPI +pip install audiocraft + +# From GitHub (latest) +pip install git+https://github.com/facebookresearch/audiocraft.git + +# Or use HuggingFace Transformers +pip install transformers torch torchaudio +``` + +### Basic text-to-music (AudioCraft) + +```python +import torchaudio +from audiocraft.models import MusicGen + +# Load model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Set generation parameters +model.set_generation_params( + duration=8, # seconds + top_k=250, + temperature=1.0 +) + +# Generate from text +descriptions = ["happy upbeat electronic dance music with synths"] +wav = model.generate(descriptions) + +# Save audio +torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000) +``` + +### Using HuggingFace Transformers + +```python +from transformers import AutoProcessor, MusicgenForConditionalGeneration +import scipy + +# Load model and processor +processor = AutoProcessor.from_pretrained("facebook/musicgen-small") +model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small") +model.to("cuda") + +# Generate music +inputs = processor( + text=["80s pop track with bassy drums and synth"], + padding=True, + return_tensors="pt" +).to("cuda") + +audio_values = model.generate( + **inputs, + do_sample=True, + guidance_scale=3, + max_new_tokens=256 +) + +# Save +sampling_rate = model.config.audio_encoder.sampling_rate +scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy()) +``` + +### Text-to-sound with AudioGen + +```python +from audiocraft.models import AudioGen + +# Load AudioGen +model = AudioGen.get_pretrained('facebook/audiogen-medium') + +model.set_generation_params(duration=5) + +# Generate sound effects +descriptions = ["dog barking in a park with birds chirping"] +wav = model.generate(descriptions) + +torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000) +``` + +## Core concepts + +### Architecture overview + +``` +AudioCraft Architecture: +┌──────────────────────────────────────────────────────────────┐ +│ Text Encoder (T5) │ +│ │ │ +│ Text Embeddings │ +└────────────────────────┬─────────────────────────────────────┘ + │ +┌────────────────────────▼─────────────────────────────────────┐ +│ Transformer Decoder (LM) │ +│ Auto-regressively generates audio tokens │ +│ Using efficient token interleaving patterns │ +└────────────────────────┬─────────────────────────────────────┘ + │ +┌────────────────────────▼─────────────────────────────────────┐ +│ EnCodec Audio Decoder │ +│ Converts tokens back to audio waveform │ +└──────────────────────────────────────────────────────────────┘ +``` + +### Model variants + +| Model | Size | Description | Use Case | +|-------|------|-------------|----------| +| `musicgen-small` | 300M | Text-to-music | Quick generation | +| `musicgen-medium` | 1.5B | Text-to-music | Balanced | +| `musicgen-large` | 3.3B | Text-to-music | Best quality | +| `musicgen-melody` | 1.5B | Text + melody | Melody conditioning | +| `musicgen-melody-large` | 3.3B | Text + melody | Best melody | +| `musicgen-stereo-*` | Varies | Stereo output | Stereo generation | +| `musicgen-style` | 1.5B | Style transfer | Reference-based | +| `audiogen-medium` | 1.5B | Text-to-sound | Sound effects | + +### Generation parameters + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `duration` | 8.0 | Length in seconds (1-120) | +| `top_k` | 250 | Top-k sampling | +| `top_p` | 0.0 | Nucleus sampling (0 = disabled) | +| `temperature` | 1.0 | Sampling temperature | +| `cfg_coef` | 3.0 | Classifier-free guidance | + +## MusicGen usage + +### Text-to-music generation + +```python +from audiocraft.models import MusicGen +import torchaudio + +model = MusicGen.get_pretrained('facebook/musicgen-medium') + +# Configure generation +model.set_generation_params( + duration=30, # Up to 30 seconds + top_k=250, # Sampling diversity + top_p=0.0, # 0 = use top_k only + temperature=1.0, # Creativity (higher = more varied) + cfg_coef=3.0 # Text adherence (higher = stricter) +) + +# Generate multiple samples +descriptions = [ + "epic orchestral soundtrack with strings and brass", + "chill lo-fi hip hop beat with jazzy piano", + "energetic rock song with electric guitar" +] + +# Generate (returns [batch, channels, samples]) +wav = model.generate(descriptions) + +# Save each +for i, audio in enumerate(wav): + torchaudio.save(f"music_{i}.wav", audio.cpu(), sample_rate=32000) +``` + +### Melody-conditioned generation + +```python +from audiocraft.models import MusicGen +import torchaudio + +# Load melody model +model = MusicGen.get_pretrained('facebook/musicgen-melody') +model.set_generation_params(duration=30) + +# Load melody audio +melody, sr = torchaudio.load("melody.wav") + +# Generate with melody conditioning +descriptions = ["acoustic guitar folk song"] +wav = model.generate_with_chroma(descriptions, melody, sr) + +torchaudio.save("melody_conditioned.wav", wav[0].cpu(), sample_rate=32000) +``` + +### Stereo generation + +```python +from audiocraft.models import MusicGen + +# Load stereo model +model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium') +model.set_generation_params(duration=15) + +descriptions = ["ambient electronic music with wide stereo panning"] +wav = model.generate(descriptions) + +# wav shape: [batch, 2, samples] for stereo +print(f"Stereo shape: {wav.shape}") # [1, 2, 480000] +torchaudio.save("stereo.wav", wav[0].cpu(), sample_rate=32000) +``` + +### Audio continuation + +```python +from transformers import AutoProcessor, MusicgenForConditionalGeneration + +processor = AutoProcessor.from_pretrained("facebook/musicgen-medium") +model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium") + +# Load audio to continue +import torchaudio +audio, sr = torchaudio.load("intro.wav") + +# Process with text and audio +inputs = processor( + audio=audio.squeeze().numpy(), + sampling_rate=sr, + text=["continue with a epic chorus"], + padding=True, + return_tensors="pt" +) + +# Generate continuation +audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512) +``` + +## MusicGen-Style usage + +### Style-conditioned generation + +```python +from audiocraft.models import MusicGen + +# Load style model +model = MusicGen.get_pretrained('facebook/musicgen-style') + +# Configure generation with style +model.set_generation_params( + duration=30, + cfg_coef=3.0, + cfg_coef_beta=5.0 # Style influence +) + +# Configure style conditioner +model.set_style_conditioner_params( + eval_q=3, # RVQ quantizers (1-6) + excerpt_length=3.0 # Style excerpt length +) + +# Load style reference +style_audio, sr = torchaudio.load("reference_style.wav") + +# Generate with text + style +descriptions = ["upbeat dance track"] +wav = model.generate_with_style(descriptions, style_audio, sr) +``` + +### Style-only generation (no text) + +```python +# Generate matching style without text prompt +model.set_generation_params( + duration=30, + cfg_coef=3.0, + cfg_coef_beta=None # Disable double CFG for style-only +) + +wav = model.generate_with_style([None], style_audio, sr) +``` + +## AudioGen usage + +### Sound effect generation + +```python +from audiocraft.models import AudioGen +import torchaudio + +model = AudioGen.get_pretrained('facebook/audiogen-medium') +model.set_generation_params(duration=10) + +# Generate various sounds +descriptions = [ + "thunderstorm with heavy rain and lightning", + "busy city traffic with car horns", + "ocean waves crashing on rocks", + "crackling campfire in forest" +] + +wav = model.generate(descriptions) + +for i, audio in enumerate(wav): + torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000) +``` + +## EnCodec usage + +### Audio compression + +```python +from audiocraft.models import CompressionModel +import torch +import torchaudio + +# Load EnCodec +model = CompressionModel.get_pretrained('facebook/encodec_32khz') + +# Load audio +wav, sr = torchaudio.load("audio.wav") + +# Ensure correct sample rate +if sr != 32000: + resampler = torchaudio.transforms.Resample(sr, 32000) + wav = resampler(wav) + +# Encode to tokens +with torch.no_grad(): + encoded = model.encode(wav.unsqueeze(0)) + codes = encoded[0] # Audio codes + +# Decode back to audio +with torch.no_grad(): + decoded = model.decode(codes) + +torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000) +``` + +## Common workflows + +### Workflow 1: Music generation pipeline + +```python +import torch +import torchaudio +from audiocraft.models import MusicGen + +class MusicGenerator: + def __init__(self, model_name="facebook/musicgen-medium"): + self.model = MusicGen.get_pretrained(model_name) + self.sample_rate = 32000 + + def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0): + self.model.set_generation_params( + duration=duration, + top_k=250, + temperature=temperature, + cfg_coef=cfg + ) + + with torch.no_grad(): + wav = self.model.generate([prompt]) + + return wav[0].cpu() + + def generate_batch(self, prompts, duration=30): + self.model.set_generation_params(duration=duration) + + with torch.no_grad(): + wav = self.model.generate(prompts) + + return wav.cpu() + + def save(self, audio, path): + torchaudio.save(path, audio, sample_rate=self.sample_rate) + +# Usage +generator = MusicGenerator() +audio = generator.generate( + "epic cinematic orchestral music", + duration=30, + temperature=1.0 +) +generator.save(audio, "epic_music.wav") +``` + +### Workflow 2: Sound design batch processing + +```python +import json +from pathlib import Path +from audiocraft.models import AudioGen +import torchaudio + +def batch_generate_sounds(sound_specs, output_dir): + """ + Generate multiple sounds from specifications. + + Args: + sound_specs: list of {"name": str, "description": str, "duration": float} + output_dir: output directory path + """ + model = AudioGen.get_pretrained('facebook/audiogen-medium') + output_dir = Path(output_dir) + output_dir.mkdir(exist_ok=True) + + results = [] + + for spec in sound_specs: + model.set_generation_params(duration=spec.get("duration", 5)) + + wav = model.generate([spec["description"]]) + + output_path = output_dir / f"{spec['name']}.wav" + torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000) + + results.append({ + "name": spec["name"], + "path": str(output_path), + "description": spec["description"] + }) + + return results + +# Usage +sounds = [ + {"name": "explosion", "description": "massive explosion with debris", "duration": 3}, + {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5}, + {"name": "door", "description": "wooden door creaking and closing", "duration": 2} +] + +results = batch_generate_sounds(sounds, "sound_effects/") +``` + +### Workflow 3: Gradio demo + +```python +import gradio as gr +import torch +import torchaudio +from audiocraft.models import MusicGen + +model = MusicGen.get_pretrained('facebook/musicgen-small') + +def generate_music(prompt, duration, temperature, cfg_coef): + model.set_generation_params( + duration=duration, + temperature=temperature, + cfg_coef=cfg_coef + ) + + with torch.no_grad(): + wav = model.generate([prompt]) + + # Save to temp file + path = "temp_output.wav" + torchaudio.save(path, wav[0].cpu(), sample_rate=32000) + return path + +demo = gr.Interface( + fn=generate_music, + inputs=[ + gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"), + gr.Slider(1, 30, value=8, label="Duration (seconds)"), + gr.Slider(0.5, 2.0, value=1.0, label="Temperature"), + gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient") + ], + outputs=gr.Audio(label="Generated Music"), + title="MusicGen Demo" +) + +demo.launch() +``` + +## Performance optimization + +### Memory optimization + +```python +# Use smaller model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Clear cache between generations +torch.cuda.empty_cache() + +# Generate shorter durations +model.set_generation_params(duration=10) # Instead of 30 + +# Use half precision +model = model.half() +``` + +### Batch processing efficiency + +```python +# Process multiple prompts at once (more efficient) +descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"] +wav = model.generate(descriptions) # Single batch + +# Instead of +for desc in descriptions: + wav = model.generate([desc]) # Multiple batches (slower) +``` + +### GPU memory requirements + +| Model | FP32 VRAM | FP16 VRAM | +|-------|-----------|-----------| +| musicgen-small | ~4GB | ~2GB | +| musicgen-medium | ~8GB | ~4GB | +| musicgen-large | ~16GB | ~8GB | + +## Common issues + +| Issue | Solution | +|-------|----------| +| CUDA OOM | Use smaller model, reduce duration | +| Poor quality | Increase cfg_coef, better prompts | +| Generation too short | Check max duration setting | +| Audio artifacts | Try different temperature | +| Stereo not working | Use stereo model variant | + +## References + +- **[Advanced Usage](references/advanced-usage.md)** - Training, fine-tuning, deployment +- **[Troubleshooting](references/troubleshooting.md)** - Common issues and solutions + +## Resources + +- **GitHub**: https://github.com/facebookresearch/audiocraft +- **Paper (MusicGen)**: https://arxiv.org/abs/2306.05284 +- **Paper (AudioGen)**: https://arxiv.org/abs/2209.15352 +- **HuggingFace**: https://huggingface.co/facebook/musicgen-small +- **Demo**: https://huggingface.co/spaces/facebook/MusicGen diff --git a/audiocraft/references/advanced-usage.md b/audiocraft/references/advanced-usage.md new file mode 100644 index 0000000..953be2b --- /dev/null +++ b/audiocraft/references/advanced-usage.md @@ -0,0 +1,666 @@ +# AudioCraft Advanced Usage Guide + +## Fine-tuning MusicGen + +### Custom dataset preparation + +```python +import os +import json +from pathlib import Path +import torchaudio + +def prepare_dataset(audio_dir, output_dir, metadata_file): + """ + Prepare dataset for MusicGen fine-tuning. + + Directory structure: + output_dir/ + ├── audio/ + │ ├── 0001.wav + │ ├── 0002.wav + │ └── ... + └── metadata.json + """ + output_dir = Path(output_dir) + audio_output = output_dir / "audio" + audio_output.mkdir(parents=True, exist_ok=True) + + # Load metadata (format: {"path": "...", "description": "..."}) + with open(metadata_file) as f: + metadata = json.load(f) + + processed = [] + + for idx, item in enumerate(metadata): + audio_path = Path(audio_dir) / item["path"] + + # Load and resample to 32kHz + wav, sr = torchaudio.load(str(audio_path)) + if sr != 32000: + resampler = torchaudio.transforms.Resample(sr, 32000) + wav = resampler(wav) + + # Convert to mono if stereo + if wav.shape[0] > 1: + wav = wav.mean(dim=0, keepdim=True) + + # Save processed audio + output_path = audio_output / f"{idx:04d}.wav" + torchaudio.save(str(output_path), wav, sample_rate=32000) + + processed.append({ + "path": str(output_path.relative_to(output_dir)), + "description": item["description"], + "duration": wav.shape[1] / 32000 + }) + + # Save processed metadata + with open(output_dir / "metadata.json", "w") as f: + json.dump(processed, f, indent=2) + + print(f"Processed {len(processed)} samples") + return processed +``` + +### Fine-tuning with dora + +```bash +# AudioCraft uses dora for experiment management +# Install dora +pip install dora-search + +# Clone AudioCraft +git clone https://github.com/facebookresearch/audiocraft.git +cd audiocraft + +# Create config for fine-tuning +cat > config/solver/musicgen/finetune.yaml << 'EOF' +defaults: + - musicgen/musicgen_base + - /model: lm/musicgen_lm + - /conditioner: cond_base + +solver: musicgen +autocast: true +autocast_dtype: float16 + +optim: + epochs: 100 + batch_size: 4 + lr: 1e-4 + ema: 0.999 + optimizer: adamw + +dataset: + batch_size: 4 + num_workers: 4 + train: + - dset: your_dataset + root: /path/to/dataset + valid: + - dset: your_dataset + root: /path/to/dataset + +checkpoint: + save_every: 10 + keep_every_states: null +EOF + +# Run fine-tuning +dora run solver=musicgen/finetune +``` + +### LoRA fine-tuning + +```python +from peft import LoraConfig, get_peft_model +from audiocraft.models import MusicGen +import torch + +# Load base model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Get the language model component +lm = model.lm + +# Configure LoRA +lora_config = LoraConfig( + r=8, + lora_alpha=16, + target_modules=["q_proj", "v_proj", "k_proj", "out_proj"], + lora_dropout=0.05, + bias="none" +) + +# Apply LoRA +lm = get_peft_model(lm, lora_config) +lm.print_trainable_parameters() +``` + +## Multi-GPU Training + +### DataParallel + +```python +import torch +import torch.nn as nn +from audiocraft.models import MusicGen + +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Wrap LM with DataParallel +if torch.cuda.device_count() > 1: + model.lm = nn.DataParallel(model.lm) + +model.to("cuda") +``` + +### DistributedDataParallel + +```python +import torch.distributed as dist +from torch.nn.parallel import DistributedDataParallel as DDP + +def setup(rank, world_size): + dist.init_process_group("nccl", rank=rank, world_size=world_size) + torch.cuda.set_device(rank) + +def train(rank, world_size): + setup(rank, world_size) + + model = MusicGen.get_pretrained('facebook/musicgen-small') + model.lm = model.lm.to(rank) + model.lm = DDP(model.lm, device_ids=[rank]) + + # Training loop + # ... + + dist.destroy_process_group() +``` + +## Custom Conditioning + +### Adding new conditioners + +```python +from audiocraft.modules.conditioners import BaseConditioner +import torch + +class CustomConditioner(BaseConditioner): + """Custom conditioner for additional control signals.""" + + def __init__(self, dim, output_dim): + super().__init__(dim, output_dim) + self.embed = torch.nn.Linear(dim, output_dim) + + def forward(self, x): + return self.embed(x) + + def tokenize(self, x): + # Tokenize input for conditioning + return x + +# Use with MusicGen +from audiocraft.models.builders import get_lm_model + +# Modify model config to include custom conditioner +# This requires editing the model configuration +``` + +### Melody conditioning internals + +```python +from audiocraft.models import MusicGen +from audiocraft.modules.codebooks_patterns import DelayedPatternProvider +import torch + +model = MusicGen.get_pretrained('facebook/musicgen-melody') + +# Access chroma extractor +chroma_extractor = model.lm.condition_provider.conditioners.get('chroma') + +# Manual chroma extraction +def extract_chroma(audio, sr): + """Extract chroma features from audio.""" + import librosa + + # Compute chroma + chroma = librosa.feature.chroma_cqt(y=audio.numpy(), sr=sr) + + return torch.from_numpy(chroma).float() + +# Use extracted chroma for conditioning +chroma = extract_chroma(melody_audio, sample_rate) +``` + +## EnCodec Deep Dive + +### Custom compression settings + +```python +from audiocraft.models import CompressionModel +import torch + +# Load EnCodec +encodec = CompressionModel.get_pretrained('facebook/encodec_32khz') + +# Access codec parameters +print(f"Sample rate: {encodec.sample_rate}") +print(f"Channels: {encodec.channels}") +print(f"Cardinality: {encodec.cardinality}") # Codebook size +print(f"Num codebooks: {encodec.num_codebooks}") +print(f"Frame rate: {encodec.frame_rate}") + +# Encode with specific bandwidth +# Lower bandwidth = more compression, lower quality +encodec.set_target_bandwidth(6.0) # 6 kbps + +audio = torch.randn(1, 1, 32000) # 1 second +encoded = encodec.encode(audio) +decoded = encodec.decode(encoded[0]) +``` + +### Streaming encoding + +```python +import torch +from audiocraft.models import CompressionModel + +encodec = CompressionModel.get_pretrained('facebook/encodec_32khz') + +def encode_streaming(audio_stream, chunk_size=32000): + """Encode audio in streaming fashion.""" + all_codes = [] + + for chunk in audio_stream: + # Ensure chunk is right shape + if chunk.dim() == 1: + chunk = chunk.unsqueeze(0).unsqueeze(0) + + with torch.no_grad(): + codes = encodec.encode(chunk)[0] + all_codes.append(codes) + + return torch.cat(all_codes, dim=-1) + +def decode_streaming(codes_stream, output_stream): + """Decode codes in streaming fashion.""" + for codes in codes_stream: + with torch.no_grad(): + audio = encodec.decode(codes) + output_stream.write(audio.cpu().numpy()) +``` + +## MultiBand Diffusion + +### Using MBD for enhanced quality + +```python +from audiocraft.models import MusicGen, MultiBandDiffusion + +# Load MusicGen +model = MusicGen.get_pretrained('facebook/musicgen-medium') + +# Load MultiBand Diffusion +mbd = MultiBandDiffusion.get_mbd_musicgen() + +model.set_generation_params(duration=10) + +# Generate with standard decoder +descriptions = ["epic orchestral music"] +wav_standard = model.generate(descriptions) + +# Generate tokens and use MBD decoder +with torch.no_grad(): + # Get tokens + gen_tokens = model.generate_tokens(descriptions) + + # Decode with MBD + wav_mbd = mbd.tokens_to_wav(gen_tokens) + +# Compare quality +print(f"Standard shape: {wav_standard.shape}") +print(f"MBD shape: {wav_mbd.shape}") +``` + +## API Server Deployment + +### FastAPI server + +```python +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel +import torch +import torchaudio +from audiocraft.models import MusicGen +import io +import base64 + +app = FastAPI() + +# Load model at startup +model = None + +@app.on_event("startup") +async def load_model(): + global model + model = MusicGen.get_pretrained('facebook/musicgen-small') + model.set_generation_params(duration=10) + +class GenerateRequest(BaseModel): + prompt: str + duration: float = 10.0 + temperature: float = 1.0 + cfg_coef: float = 3.0 + +class GenerateResponse(BaseModel): + audio_base64: str + sample_rate: int + duration: float + +@app.post("/generate", response_model=GenerateResponse) +async def generate(request: GenerateRequest): + if model is None: + raise HTTPException(status_code=500, detail="Model not loaded") + + try: + model.set_generation_params( + duration=min(request.duration, 30), + temperature=request.temperature, + cfg_coef=request.cfg_coef + ) + + with torch.no_grad(): + wav = model.generate([request.prompt]) + + # Convert to bytes + buffer = io.BytesIO() + torchaudio.save(buffer, wav[0].cpu(), sample_rate=32000, format="wav") + buffer.seek(0) + + audio_base64 = base64.b64encode(buffer.read()).decode() + + return GenerateResponse( + audio_base64=audio_base64, + sample_rate=32000, + duration=wav.shape[-1] / 32000 + ) + + except Exception as e: + raise HTTPException(status_code=500, detail=str(e)) + +@app.get("/health") +async def health(): + return {"status": "ok", "model_loaded": model is not None} + +# Run: uvicorn server:app --host 0.0.0.0 --port 8000 +``` + +### Batch processing service + +```python +import asyncio +from concurrent.futures import ThreadPoolExecutor +import torch +from audiocraft.models import MusicGen + +class MusicGenService: + def __init__(self, model_name='facebook/musicgen-small', max_workers=2): + self.model = MusicGen.get_pretrained(model_name) + self.executor = ThreadPoolExecutor(max_workers=max_workers) + self.lock = asyncio.Lock() + + async def generate_async(self, prompt, duration=10): + """Async generation with thread pool.""" + loop = asyncio.get_event_loop() + + def _generate(): + with torch.no_grad(): + self.model.set_generation_params(duration=duration) + return self.model.generate([prompt]) + + # Run in thread pool + wav = await loop.run_in_executor(self.executor, _generate) + return wav[0].cpu() + + async def generate_batch_async(self, prompts, duration=10): + """Process multiple prompts concurrently.""" + tasks = [self.generate_async(p, duration) for p in prompts] + return await asyncio.gather(*tasks) + +# Usage +service = MusicGenService() + +async def main(): + prompts = ["jazz piano", "rock guitar", "electronic beats"] + results = await service.generate_batch_async(prompts) + return results +``` + +## Integration Patterns + +### LangChain tool + +```python +from langchain.tools import BaseTool +import torch +import torchaudio +from audiocraft.models import MusicGen +import tempfile + +class MusicGeneratorTool(BaseTool): + name = "music_generator" + description = "Generate music from a text description. Input should be a detailed description of the music style, mood, and instruments." + + def __init__(self): + super().__init__() + self.model = MusicGen.get_pretrained('facebook/musicgen-small') + self.model.set_generation_params(duration=15) + + def _run(self, description: str) -> str: + with torch.no_grad(): + wav = self.model.generate([description]) + + # Save to temp file + with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f: + torchaudio.save(f.name, wav[0].cpu(), sample_rate=32000) + return f"Generated music saved to: {f.name}" + + async def _arun(self, description: str) -> str: + return self._run(description) +``` + +### Gradio with advanced controls + +```python +import gradio as gr +import torch +import torchaudio +from audiocraft.models import MusicGen + +models = {} + +def load_model(model_size): + if model_size not in models: + model_name = f"facebook/musicgen-{model_size}" + models[model_size] = MusicGen.get_pretrained(model_name) + return models[model_size] + +def generate(prompt, duration, temperature, cfg_coef, top_k, model_size): + model = load_model(model_size) + + model.set_generation_params( + duration=duration, + temperature=temperature, + cfg_coef=cfg_coef, + top_k=top_k + ) + + with torch.no_grad(): + wav = model.generate([prompt]) + + # Save + path = "output.wav" + torchaudio.save(path, wav[0].cpu(), sample_rate=32000) + return path + +demo = gr.Interface( + fn=generate, + inputs=[ + gr.Textbox(label="Prompt", lines=3), + gr.Slider(1, 30, value=10, label="Duration (s)"), + gr.Slider(0.1, 2.0, value=1.0, label="Temperature"), + gr.Slider(0.5, 10.0, value=3.0, label="CFG Coefficient"), + gr.Slider(50, 500, value=250, step=50, label="Top-K"), + gr.Dropdown(["small", "medium", "large"], value="small", label="Model Size") + ], + outputs=gr.Audio(label="Generated Music"), + title="MusicGen Advanced", + allow_flagging="never" +) + +demo.launch(share=True) +``` + +## Audio Processing Pipeline + +### Post-processing chain + +```python +import torch +import torchaudio +import torchaudio.transforms as T +import numpy as np + +class AudioPostProcessor: + def __init__(self, sample_rate=32000): + self.sample_rate = sample_rate + + def normalize(self, audio, target_db=-14.0): + """Normalize audio to target loudness.""" + rms = torch.sqrt(torch.mean(audio ** 2)) + target_rms = 10 ** (target_db / 20) + gain = target_rms / (rms + 1e-8) + return audio * gain + + def fade_in_out(self, audio, fade_duration=0.1): + """Apply fade in/out.""" + fade_samples = int(fade_duration * self.sample_rate) + + # Create fade curves + fade_in = torch.linspace(0, 1, fade_samples) + fade_out = torch.linspace(1, 0, fade_samples) + + # Apply fades + audio[..., :fade_samples] *= fade_in + audio[..., -fade_samples:] *= fade_out + + return audio + + def apply_reverb(self, audio, decay=0.5): + """Apply simple reverb effect.""" + impulse = torch.zeros(int(self.sample_rate * 0.5)) + impulse[0] = 1.0 + impulse[int(self.sample_rate * 0.1)] = decay * 0.5 + impulse[int(self.sample_rate * 0.2)] = decay * 0.25 + + # Convolve + audio = torch.nn.functional.conv1d( + audio.unsqueeze(0), + impulse.unsqueeze(0).unsqueeze(0), + padding=len(impulse) // 2 + ).squeeze(0) + + return audio + + def process(self, audio): + """Full processing pipeline.""" + audio = self.normalize(audio) + audio = self.fade_in_out(audio) + return audio + +# Usage with MusicGen +from audiocraft.models import MusicGen + +model = MusicGen.get_pretrained('facebook/musicgen-small') +model.set_generation_params(duration=10) + +wav = model.generate(["chill ambient music"]) +processor = AudioPostProcessor() +wav_processed = processor.process(wav[0].cpu()) + +torchaudio.save("processed.wav", wav_processed, sample_rate=32000) +``` + +## Evaluation + +### Audio quality metrics + +```python +import torch +from audiocraft.metrics import CLAPTextConsistencyMetric +from audiocraft.data.audio import audio_read + +def evaluate_generation(audio_path, text_prompt): + """Evaluate generated audio quality.""" + # Load audio + wav, sr = audio_read(audio_path) + + # CLAP consistency (text-audio alignment) + clap_metric = CLAPTextConsistencyMetric() + clap_score = clap_metric.compute(wav, [text_prompt]) + + return { + "clap_score": clap_score, + "duration": wav.shape[-1] / sr + } + +# Batch evaluation +def evaluate_batch(generations): + """Evaluate multiple generations.""" + results = [] + for gen in generations: + result = evaluate_generation(gen["path"], gen["prompt"]) + result["prompt"] = gen["prompt"] + results.append(result) + + # Aggregate + avg_clap = sum(r["clap_score"] for r in results) / len(results) + return { + "individual": results, + "average_clap": avg_clap + } +``` + +## Model Comparison + +### MusicGen variants benchmark + +| Model | CLAP Score | Generation Time (10s) | VRAM | +|-------|------------|----------------------|------| +| musicgen-small | 0.35 | ~5s | 2GB | +| musicgen-medium | 0.42 | ~15s | 4GB | +| musicgen-large | 0.48 | ~30s | 8GB | +| musicgen-melody | 0.45 | ~15s | 4GB | +| musicgen-stereo-medium | 0.41 | ~18s | 5GB | + +### Prompt engineering tips + +```python +# Good prompts - specific and descriptive +good_prompts = [ + "upbeat electronic dance music with synthesizer leads and punchy drums at 128 bpm", + "melancholic piano ballad with strings, slow tempo, emotional and cinematic", + "funky disco groove with slap bass, brass section, and rhythmic guitar" +] + +# Bad prompts - too vague +bad_prompts = [ + "nice music", + "song", + "good beat" +] + +# Structure: [mood] [genre] with [instruments] at [tempo/style] +``` diff --git a/audiocraft/references/troubleshooting.md b/audiocraft/references/troubleshooting.md new file mode 100644 index 0000000..7b83e86 --- /dev/null +++ b/audiocraft/references/troubleshooting.md @@ -0,0 +1,504 @@ +# AudioCraft Troubleshooting Guide + +## Installation Issues + +### Import errors + +**Error**: `ModuleNotFoundError: No module named 'audiocraft'` + +**Solutions**: +```bash +# Install from PyPI +pip install audiocraft + +# Or from GitHub +pip install git+https://github.com/facebookresearch/audiocraft.git + +# Verify installation +python -c "from audiocraft.models import MusicGen; print('OK')" +``` + +### FFmpeg not found + +**Error**: `RuntimeError: ffmpeg not found` + +**Solutions**: +```bash +# Ubuntu/Debian +sudo apt-get install ffmpeg + +# macOS +brew install ffmpeg + +# Windows (using conda) +conda install -c conda-forge ffmpeg + +# Verify +ffmpeg -version +``` + +### PyTorch CUDA mismatch + +**Error**: `RuntimeError: CUDA error: no kernel image is available` + +**Solutions**: +```bash +# Check CUDA version +nvcc --version +python -c "import torch; print(torch.version.cuda)" + +# Install matching PyTorch +pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121 + +# For CUDA 11.8 +pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu118 +``` + +### xformers issues + +**Error**: `ImportError: xformers` related errors + +**Solutions**: +```bash +# Install xformers for memory efficiency +pip install xformers + +# Or disable xformers +export AUDIOCRAFT_USE_XFORMERS=0 + +# In Python +import os +os.environ["AUDIOCRAFT_USE_XFORMERS"] = "0" +from audiocraft.models import MusicGen +``` + +## Model Loading Issues + +### Out of memory during load + +**Error**: `torch.cuda.OutOfMemoryError` during model loading + +**Solutions**: +```python +# Use smaller model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Force CPU loading first +import torch +device = "cpu" +model = MusicGen.get_pretrained('facebook/musicgen-small', device=device) +model = model.to("cuda") + +# Use HuggingFace with device_map +from transformers import MusicgenForConditionalGeneration +model = MusicgenForConditionalGeneration.from_pretrained( + "facebook/musicgen-small", + device_map="auto" +) +``` + +### Download failures + +**Error**: Connection errors or incomplete downloads + +**Solutions**: +```python +# Set cache directory +import os +os.environ["AUDIOCRAFT_CACHE_DIR"] = "/path/to/cache" + +# Or for HuggingFace +os.environ["HF_HOME"] = "/path/to/hf_cache" + +# Resume download +from huggingface_hub import snapshot_download +snapshot_download("facebook/musicgen-small", resume_download=True) + +# Use local files +model = MusicGen.get_pretrained('/local/path/to/model') +``` + +### Wrong model type + +**Error**: Loading wrong model for task + +**Solutions**: +```python +# For text-to-music: use MusicGen +from audiocraft.models import MusicGen +model = MusicGen.get_pretrained('facebook/musicgen-medium') + +# For text-to-sound: use AudioGen +from audiocraft.models import AudioGen +model = AudioGen.get_pretrained('facebook/audiogen-medium') + +# For melody conditioning: use melody variant +model = MusicGen.get_pretrained('facebook/musicgen-melody') + +# For stereo: use stereo variant +model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium') +``` + +## Generation Issues + +### Empty or silent output + +**Problem**: Generated audio is silent or very quiet + +**Solutions**: +```python +import torch + +# Check output +wav = model.generate(["upbeat music"]) +print(f"Shape: {wav.shape}") +print(f"Max amplitude: {wav.abs().max().item()}") +print(f"Mean amplitude: {wav.abs().mean().item()}") + +# If too quiet, normalize +def normalize_audio(audio, target_db=-14.0): + rms = torch.sqrt(torch.mean(audio ** 2)) + target_rms = 10 ** (target_db / 20) + gain = target_rms / (rms + 1e-8) + return audio * gain + +wav_normalized = normalize_audio(wav) +``` + +### Poor quality output + +**Problem**: Generated music sounds bad or noisy + +**Solutions**: +```python +# Use larger model +model = MusicGen.get_pretrained('facebook/musicgen-large') + +# Adjust generation parameters +model.set_generation_params( + duration=15, + top_k=250, # Increase for more diversity + temperature=0.8, # Lower for more focused output + cfg_coef=4.0 # Increase for better text adherence +) + +# Use better prompts +# Bad: "music" +# Good: "upbeat electronic dance music with synthesizers and punchy drums" + +# Try MultiBand Diffusion +from audiocraft.models import MultiBandDiffusion +mbd = MultiBandDiffusion.get_mbd_musicgen() +tokens = model.generate_tokens(["prompt"]) +wav = mbd.tokens_to_wav(tokens) +``` + +### Generation too short + +**Problem**: Audio shorter than expected + +**Solutions**: +```python +# Check duration setting +model.set_generation_params(duration=30) # Set before generate + +# Verify in generation +print(f"Duration setting: {model.generation_params}") + +# Check output shape +wav = model.generate(["prompt"]) +actual_duration = wav.shape[-1] / 32000 +print(f"Actual duration: {actual_duration}s") + +# Note: max duration is typically 30s +``` + +### Melody conditioning fails + +**Error**: Issues with melody-conditioned generation + +**Solutions**: +```python +import torchaudio +from audiocraft.models import MusicGen + +# Load melody model (not base model) +model = MusicGen.get_pretrained('facebook/musicgen-melody') + +# Load and prepare melody +melody, sr = torchaudio.load("melody.wav") + +# Resample to model sample rate if needed +if sr != 32000: + resampler = torchaudio.transforms.Resample(sr, 32000) + melody = resampler(melody) + +# Ensure correct shape [batch, channels, samples] +if melody.dim() == 1: + melody = melody.unsqueeze(0).unsqueeze(0) +elif melody.dim() == 2: + melody = melody.unsqueeze(0) + +# Convert stereo to mono +if melody.shape[1] > 1: + melody = melody.mean(dim=1, keepdim=True) + +# Generate with melody +model.set_generation_params(duration=min(melody.shape[-1] / 32000, 30)) +wav = model.generate_with_chroma(["piano cover"], melody, 32000) +``` + +## Memory Issues + +### CUDA out of memory + +**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory` + +**Solutions**: +```python +import torch + +# Clear cache before generation +torch.cuda.empty_cache() + +# Use smaller model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Reduce duration +model.set_generation_params(duration=10) # Instead of 30 + +# Generate one at a time +for prompt in prompts: + wav = model.generate([prompt]) + save_audio(wav) + torch.cuda.empty_cache() + +# Use CPU for very large generations +model = MusicGen.get_pretrained('facebook/musicgen-small', device="cpu") +``` + +### Memory leak during batch processing + +**Problem**: Memory grows over time + +**Solutions**: +```python +import gc +import torch + +def generate_with_cleanup(model, prompts): + results = [] + + for prompt in prompts: + with torch.no_grad(): + wav = model.generate([prompt]) + results.append(wav.cpu()) + + # Cleanup + del wav + gc.collect() + torch.cuda.empty_cache() + + return results + +# Use context manager +with torch.inference_mode(): + wav = model.generate(["prompt"]) +``` + +## Audio Format Issues + +### Wrong sample rate + +**Problem**: Audio plays at wrong speed + +**Solutions**: +```python +import torchaudio + +# MusicGen outputs at 32kHz +sample_rate = 32000 + +# AudioGen outputs at 16kHz +sample_rate = 16000 + +# Always use correct rate when saving +torchaudio.save("output.wav", wav[0].cpu(), sample_rate=sample_rate) + +# Resample if needed +resampler = torchaudio.transforms.Resample(32000, 44100) +wav_resampled = resampler(wav) +``` + +### Stereo/mono mismatch + +**Problem**: Wrong number of channels + +**Solutions**: +```python +# Check model type +print(f"Audio channels: {wav.shape}") +# Mono: [batch, 1, samples] +# Stereo: [batch, 2, samples] + +# Convert mono to stereo +if wav.shape[1] == 1: + wav_stereo = wav.repeat(1, 2, 1) + +# Convert stereo to mono +if wav.shape[1] == 2: + wav_mono = wav.mean(dim=1, keepdim=True) + +# Use stereo model for stereo output +model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium') +``` + +### Clipping and distortion + +**Problem**: Audio has clipping or distortion + +**Solutions**: +```python +import torch + +# Check for clipping +max_val = wav.abs().max().item() +print(f"Max amplitude: {max_val}") + +# Normalize to prevent clipping +if max_val > 1.0: + wav = wav / max_val + +# Apply soft clipping +def soft_clip(x, threshold=0.9): + return torch.tanh(x / threshold) * threshold + +wav_clipped = soft_clip(wav) + +# Lower temperature during generation +model.set_generation_params(temperature=0.7) # More controlled +``` + +## HuggingFace Transformers Issues + +### Processor errors + +**Error**: Issues with MusicgenProcessor + +**Solutions**: +```python +from transformers import AutoProcessor, MusicgenForConditionalGeneration + +# Load matching processor and model +processor = AutoProcessor.from_pretrained("facebook/musicgen-small") +model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small") + +# Ensure inputs are on same device +inputs = processor( + text=["prompt"], + padding=True, + return_tensors="pt" +).to("cuda") + +# Check processor configuration +print(processor.tokenizer) +print(processor.feature_extractor) +``` + +### Generation parameter errors + +**Error**: Invalid generation parameters + +**Solutions**: +```python +# HuggingFace uses different parameter names +audio_values = model.generate( + **inputs, + do_sample=True, # Enable sampling + guidance_scale=3.0, # CFG (not cfg_coef) + max_new_tokens=256, # Token limit (not duration) + temperature=1.0 +) + +# Calculate tokens from duration +# ~50 tokens per second +duration_seconds = 10 +max_tokens = duration_seconds * 50 +audio_values = model.generate(**inputs, max_new_tokens=max_tokens) +``` + +## Performance Issues + +### Slow generation + +**Problem**: Generation takes too long + +**Solutions**: +```python +# Use smaller model +model = MusicGen.get_pretrained('facebook/musicgen-small') + +# Reduce duration +model.set_generation_params(duration=10) + +# Use GPU +model.to("cuda") + +# Enable flash attention if available +# (requires compatible hardware) + +# Batch multiple prompts +prompts = ["prompt1", "prompt2", "prompt3"] +wav = model.generate(prompts) # Single batch is faster than loop + +# Use compile (PyTorch 2.0+) +model.lm = torch.compile(model.lm) +``` + +### CPU fallback + +**Problem**: Generation running on CPU instead of GPU + +**Solutions**: +```python +import torch + +# Check CUDA availability +print(f"CUDA available: {torch.cuda.is_available()}") +print(f"CUDA device: {torch.cuda.get_device_name(0)}") + +# Explicitly move to GPU +model = MusicGen.get_pretrained('facebook/musicgen-small') +model.to("cuda") + +# Verify model device +print(f"Model device: {next(model.lm.parameters()).device}") +``` + +## Common Error Messages + +| Error | Cause | Solution | +|-------|-------|----------| +| `CUDA out of memory` | Model too large | Use smaller model, reduce duration | +| `ffmpeg not found` | FFmpeg not installed | Install FFmpeg | +| `No module named 'audiocraft'` | Not installed | `pip install audiocraft` | +| `RuntimeError: Expected 3D tensor` | Wrong input shape | Check tensor dimensions | +| `KeyError: 'melody'` | Wrong model for melody | Use musicgen-melody | +| `Sample rate mismatch` | Wrong audio format | Resample to model rate | + +## Getting Help + +1. **GitHub Issues**: https://github.com/facebookresearch/audiocraft/issues +2. **HuggingFace Forums**: https://discuss.huggingface.co +3. **Paper**: https://arxiv.org/abs/2306.05284 + +### Reporting Issues + +Include: +- Python version +- PyTorch version +- CUDA version +- AudioCraft version: `pip show audiocraft` +- Full error traceback +- Minimal reproducible code +- Hardware (GPU model, VRAM) diff --git a/dspy/SKILL.md b/dspy/SKILL.md new file mode 100644 index 0000000..674cb7e --- /dev/null +++ b/dspy/SKILL.md @@ -0,0 +1,594 @@ +--- +name: dspy +description: "DSPy: declarative LM programs, auto-optimize prompts, RAG." +version: 1.0.0 +author: Orchestra Research +license: MIT +dependencies: [dspy, openai, anthropic] +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [Prompt Engineering, DSPy, Declarative Programming, RAG, Agents, Prompt Optimization, LM Programming, Stanford NLP, Automatic Optimization, Modular AI] + +--- + +# DSPy: Declarative Language Model Programming + +## When to Use This Skill + +Use DSPy when you need to: +- **Build complex AI systems** with multiple components and workflows +- **Program LMs declaratively** instead of manual prompt engineering +- **Optimize prompts automatically** using data-driven methods +- **Create modular AI pipelines** that are maintainable and portable +- **Improve model outputs systematically** with optimizers +- **Build RAG systems, agents, or classifiers** with better reliability + +**GitHub Stars**: 22,000+ | **Created By**: Stanford NLP + +## Installation + +```bash +# Stable release +pip install dspy + +# Latest development version +pip install git+https://github.com/stanfordnlp/dspy.git + +# With specific LM providers +pip install dspy[openai] # OpenAI +pip install dspy[anthropic] # Anthropic Claude +pip install dspy[all] # All providers +``` + +## Quick Start + +### Basic Example: Question Answering + +```python +import dspy + +# Configure your language model +lm = dspy.Claude(model="claude-sonnet-4-5-20250929") +dspy.settings.configure(lm=lm) + +# Define a signature (input → output) +class QA(dspy.Signature): + """Answer questions with short factual answers.""" + question = dspy.InputField() + answer = dspy.OutputField(desc="often between 1 and 5 words") + +# Create a module +qa = dspy.Predict(QA) + +# Use it +response = qa(question="What is the capital of France?") +print(response.answer) # "Paris" +``` + +### Chain of Thought Reasoning + +```python +import dspy + +lm = dspy.Claude(model="claude-sonnet-4-5-20250929") +dspy.settings.configure(lm=lm) + +# Use ChainOfThought for better reasoning +class MathProblem(dspy.Signature): + """Solve math word problems.""" + problem = dspy.InputField() + answer = dspy.OutputField(desc="numerical answer") + +# ChainOfThought generates reasoning steps automatically +cot = dspy.ChainOfThought(MathProblem) + +response = cot(problem="If John has 5 apples and gives 2 to Mary, how many does he have?") +print(response.rationale) # Shows reasoning steps +print(response.answer) # "3" +``` + +## Core Concepts + +### 1. Signatures + +Signatures define the structure of your AI task (inputs → outputs): + +```python +# Inline signature (simple) +qa = dspy.Predict("question -> answer") + +# Class signature (detailed) +class Summarize(dspy.Signature): + """Summarize text into key points.""" + text = dspy.InputField() + summary = dspy.OutputField(desc="bullet points, 3-5 items") + +summarizer = dspy.ChainOfThought(Summarize) +``` + +**When to use each:** +- **Inline**: Quick prototyping, simple tasks +- **Class**: Complex tasks, type hints, better documentation + +### 2. Modules + +Modules are reusable components that transform inputs to outputs: + +#### dspy.Predict +Basic prediction module: + +```python +predictor = dspy.Predict("context, question -> answer") +result = predictor(context="Paris is the capital of France", + question="What is the capital?") +``` + +#### dspy.ChainOfThought +Generates reasoning steps before answering: + +```python +cot = dspy.ChainOfThought("question -> answer") +result = cot(question="Why is the sky blue?") +print(result.rationale) # Reasoning steps +print(result.answer) # Final answer +``` + +#### dspy.ReAct +Agent-like reasoning with tools: + +```python +from dspy.predict import ReAct + +class SearchQA(dspy.Signature): + """Answer questions using search.""" + question = dspy.InputField() + answer = dspy.OutputField() + +def search_tool(query: str) -> str: + """Search Wikipedia.""" + # Your search implementation + return results + +react = ReAct(SearchQA, tools=[search_tool]) +result = react(question="When was Python created?") +``` + +#### dspy.ProgramOfThought +Generates and executes code for reasoning: + +```python +pot = dspy.ProgramOfThought("question -> answer") +result = pot(question="What is 15% of 240?") +# Generates: answer = 240 * 0.15 +``` + +### 3. Optimizers + +Optimizers improve your modules automatically using training data: + +#### BootstrapFewShot +Learns from examples: + +```python +from dspy.teleprompt import BootstrapFewShot + +# Training data +trainset = [ + dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"), + dspy.Example(question="What is 3+5?", answer="8").with_inputs("question"), +] + +# Define metric +def validate_answer(example, pred, trace=None): + return example.answer == pred.answer + +# Optimize +optimizer = BootstrapFewShot(metric=validate_answer, max_bootstrapped_demos=3) +optimized_qa = optimizer.compile(qa, trainset=trainset) + +# Now optimized_qa performs better! +``` + +#### MIPRO (Most Important Prompt Optimization) +Iteratively improves prompts: + +```python +from dspy.teleprompt import MIPRO + +optimizer = MIPRO( + metric=validate_answer, + num_candidates=10, + init_temperature=1.0 +) + +optimized_cot = optimizer.compile( + cot, + trainset=trainset, + num_trials=100 +) +``` + +#### BootstrapFinetune +Creates datasets for model fine-tuning: + +```python +from dspy.teleprompt import BootstrapFinetune + +optimizer = BootstrapFinetune(metric=validate_answer) +optimized_module = optimizer.compile(qa, trainset=trainset) + +# Exports training data for fine-tuning +``` + +### 4. Building Complex Systems + +#### Multi-Stage Pipeline + +```python +import dspy + +class MultiHopQA(dspy.Module): + def __init__(self): + super().__init__() + self.retrieve = dspy.Retrieve(k=3) + self.generate_query = dspy.ChainOfThought("question -> search_query") + self.generate_answer = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + # Stage 1: Generate search query + search_query = self.generate_query(question=question).search_query + + # Stage 2: Retrieve context + passages = self.retrieve(search_query).passages + context = "\n".join(passages) + + # Stage 3: Generate answer + answer = self.generate_answer(context=context, question=question).answer + return dspy.Prediction(answer=answer, context=context) + +# Use the pipeline +qa_system = MultiHopQA() +result = qa_system(question="Who wrote the book that inspired the movie Blade Runner?") +``` + +#### RAG System with Optimization + +```python +import dspy +from dspy.retrieve.chromadb_rm import ChromadbRM + +# Configure retriever +retriever = ChromadbRM( + collection_name="documents", + persist_directory="./chroma_db" +) + +class RAG(dspy.Module): + def __init__(self, num_passages=3): + super().__init__() + self.retrieve = dspy.Retrieve(k=num_passages) + self.generate = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + context = self.retrieve(question).passages + return self.generate(context=context, question=question) + +# Create and optimize +rag = RAG() + +# Optimize with training data +from dspy.teleprompt import BootstrapFewShot + +optimizer = BootstrapFewShot(metric=validate_answer) +optimized_rag = optimizer.compile(rag, trainset=trainset) +``` + +## LM Provider Configuration + +### Anthropic Claude + +```python +import dspy + +lm = dspy.Claude( + model="claude-sonnet-4-5-20250929", + api_key="your-api-key", # Or set ANTHROPIC_API_KEY env var + max_tokens=1000, + temperature=0.7 +) +dspy.settings.configure(lm=lm) +``` + +### OpenAI + +```python +lm = dspy.OpenAI( + model="gpt-4", + api_key="your-api-key", + max_tokens=1000 +) +dspy.settings.configure(lm=lm) +``` + +### Local Models (Ollama) + +```python +lm = dspy.OllamaLocal( + model="llama3.1", + base_url="http://localhost:11434" +) +dspy.settings.configure(lm=lm) +``` + +### Multiple Models + +```python +# Different models for different tasks +cheap_lm = dspy.OpenAI(model="gpt-3.5-turbo") +strong_lm = dspy.Claude(model="claude-sonnet-4-5-20250929") + +# Use cheap model for retrieval, strong model for reasoning +with dspy.settings.context(lm=cheap_lm): + context = retriever(question) + +with dspy.settings.context(lm=strong_lm): + answer = generator(context=context, question=question) +``` + +## Common Patterns + +### Pattern 1: Structured Output + +```python +from pydantic import BaseModel, Field + +class PersonInfo(BaseModel): + name: str = Field(description="Full name") + age: int = Field(description="Age in years") + occupation: str = Field(description="Current job") + +class ExtractPerson(dspy.Signature): + """Extract person information from text.""" + text = dspy.InputField() + person: PersonInfo = dspy.OutputField() + +extractor = dspy.TypedPredictor(ExtractPerson) +result = extractor(text="John Doe is a 35-year-old software engineer.") +print(result.person.name) # "John Doe" +print(result.person.age) # 35 +``` + +### Pattern 2: Assertion-Driven Optimization + +```python +import dspy +from dspy.primitives.assertions import assert_transform_module, backtrack_handler + +class MathQA(dspy.Module): + def __init__(self): + super().__init__() + self.solve = dspy.ChainOfThought("problem -> solution: float") + + def forward(self, problem): + solution = self.solve(problem=problem).solution + + # Assert solution is numeric + dspy.Assert( + isinstance(float(solution), float), + "Solution must be a number", + backtrack=backtrack_handler + ) + + return dspy.Prediction(solution=solution) +``` + +### Pattern 3: Self-Consistency + +```python +import dspy +from collections import Counter + +class ConsistentQA(dspy.Module): + def __init__(self, num_samples=5): + super().__init__() + self.qa = dspy.ChainOfThought("question -> answer") + self.num_samples = num_samples + + def forward(self, question): + # Generate multiple answers + answers = [] + for _ in range(self.num_samples): + result = self.qa(question=question) + answers.append(result.answer) + + # Return most common answer + most_common = Counter(answers).most_common(1)[0][0] + return dspy.Prediction(answer=most_common) +``` + +### Pattern 4: Retrieval with Reranking + +```python +class RerankedRAG(dspy.Module): + def __init__(self): + super().__init__() + self.retrieve = dspy.Retrieve(k=10) + self.rerank = dspy.Predict("question, passage -> relevance_score: float") + self.answer = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + # Retrieve candidates + passages = self.retrieve(question).passages + + # Rerank passages + scored = [] + for passage in passages: + score = float(self.rerank(question=question, passage=passage).relevance_score) + scored.append((score, passage)) + + # Take top 3 + top_passages = [p for _, p in sorted(scored, reverse=True)[:3]] + context = "\n\n".join(top_passages) + + # Generate answer + return self.answer(context=context, question=question) +``` + +## Evaluation and Metrics + +### Custom Metrics + +```python +def exact_match(example, pred, trace=None): + """Exact match metric.""" + return example.answer.lower() == pred.answer.lower() + +def f1_score(example, pred, trace=None): + """F1 score for text overlap.""" + pred_tokens = set(pred.answer.lower().split()) + gold_tokens = set(example.answer.lower().split()) + + if not pred_tokens: + return 0.0 + + precision = len(pred_tokens & gold_tokens) / len(pred_tokens) + recall = len(pred_tokens & gold_tokens) / len(gold_tokens) + + if precision + recall == 0: + return 0.0 + + return 2 * (precision * recall) / (precision + recall) +``` + +### Evaluation + +```python +from dspy.evaluate import Evaluate + +# Create evaluator +evaluator = Evaluate( + devset=testset, + metric=exact_match, + num_threads=4, + display_progress=True +) + +# Evaluate model +score = evaluator(qa_system) +print(f"Accuracy: {score}") + +# Compare optimized vs unoptimized +score_before = evaluator(qa) +score_after = evaluator(optimized_qa) +print(f"Improvement: {score_after - score_before:.2%}") +``` + +## Best Practices + +### 1. Start Simple, Iterate + +```python +# Start with Predict +qa = dspy.Predict("question -> answer") + +# Add reasoning if needed +qa = dspy.ChainOfThought("question -> answer") + +# Add optimization when you have data +optimized_qa = optimizer.compile(qa, trainset=data) +``` + +### 2. Use Descriptive Signatures + +```python +# ❌ Bad: Vague +class Task(dspy.Signature): + input = dspy.InputField() + output = dspy.OutputField() + +# ✅ Good: Descriptive +class SummarizeArticle(dspy.Signature): + """Summarize news articles into 3-5 key points.""" + article = dspy.InputField(desc="full article text") + summary = dspy.OutputField(desc="bullet points, 3-5 items") +``` + +### 3. Optimize with Representative Data + +```python +# Create diverse training examples +trainset = [ + dspy.Example(question="factual", answer="...).with_inputs("question"), + dspy.Example(question="reasoning", answer="...").with_inputs("question"), + dspy.Example(question="calculation", answer="...").with_inputs("question"), +] + +# Use validation set for metric +def metric(example, pred, trace=None): + return example.answer in pred.answer +``` + +### 4. Save and Load Optimized Models + +```python +# Save +optimized_qa.save("models/qa_v1.json") + +# Load +loaded_qa = dspy.ChainOfThought("question -> answer") +loaded_qa.load("models/qa_v1.json") +``` + +### 5. Monitor and Debug + +```python +# Enable tracing +dspy.settings.configure(lm=lm, trace=[]) + +# Run prediction +result = qa(question="...") + +# Inspect trace +for call in dspy.settings.trace: + print(f"Prompt: {call['prompt']}") + print(f"Response: {call['response']}") +``` + +## Comparison to Other Approaches + +| Feature | Manual Prompting | LangChain | DSPy | +|---------|-----------------|-----------|------| +| Prompt Engineering | Manual | Manual | Automatic | +| Optimization | Trial & error | None | Data-driven | +| Modularity | Low | Medium | High | +| Type Safety | No | Limited | Yes (Signatures) | +| Portability | Low | Medium | High | +| Learning Curve | Low | Medium | Medium-High | + +**When to choose DSPy:** +- You have training data or can generate it +- You need systematic prompt improvement +- You're building complex multi-stage systems +- You want to optimize across different LMs + +**When to choose alternatives:** +- Quick prototypes (manual prompting) +- Simple chains with existing tools (LangChain) +- Custom optimization logic needed + +## Resources + +- **Documentation**: https://dspy.ai +- **GitHub**: https://github.com/stanfordnlp/dspy (22k+ stars) +- **Discord**: https://discord.gg/XCGy2WDCQB +- **Twitter**: @DSPyOSS +- **Paper**: "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines" + +## See Also + +- `references/modules.md` - Detailed module guide (Predict, ChainOfThought, ReAct, ProgramOfThought) +- `references/optimizers.md` - Optimization algorithms (BootstrapFewShot, MIPRO, BootstrapFinetune) +- `references/examples.md` - Real-world examples (RAG, agents, classifiers) + + diff --git a/dspy/references/examples.md b/dspy/references/examples.md new file mode 100644 index 0000000..2f568c7 --- /dev/null +++ b/dspy/references/examples.md @@ -0,0 +1,663 @@ +# DSPy Real-World Examples + +Practical examples of building production systems with DSPy. + +## Table of Contents +- RAG Systems +- Agent Systems +- Classification +- Data Processing +- Multi-Stage Pipelines + +## RAG Systems + +### Basic RAG + +```python +import dspy + +class BasicRAG(dspy.Module): + def __init__(self, num_passages=3): + super().__init__() + self.retrieve = dspy.Retrieve(k=num_passages) + self.generate = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + passages = self.retrieve(question).passages + context = "\n\n".join(passages) + return self.generate(context=context, question=question) + +# Configure retriever (example with Chroma) +from dspy.retrieve.chromadb_rm import ChromadbRM + +retriever = ChromadbRM( + collection_name="my_docs", + persist_directory="./chroma_db", + k=3 +) +dspy.settings.configure(rm=retriever) + +# Use RAG +rag = BasicRAG() +result = rag(question="What is DSPy?") +print(result.answer) +``` + +### Optimized RAG + +```python +from dspy.teleprompt import BootstrapFewShot + +# Training data with question-answer pairs +trainset = [ + dspy.Example( + question="What is retrieval augmented generation?", + answer="RAG combines retrieval of relevant documents with generation..." + ).with_inputs("question"), + # ... more examples +] + +# Define metric +def answer_correctness(example, pred, trace=None): + # Check if answer contains key information + return example.answer.lower() in pred.answer.lower() + +# Optimize RAG +optimizer = BootstrapFewShot(metric=answer_correctness) +optimized_rag = optimizer.compile(rag, trainset=trainset) + +# Optimized RAG performs better on similar questions +result = optimized_rag(question="Explain RAG systems") +``` + +### Multi-Hop RAG + +```python +class MultiHopRAG(dspy.Module): + """RAG that follows chains of reasoning across documents.""" + + def __init__(self): + super().__init__() + self.retrieve = dspy.Retrieve(k=3) + self.generate_query = dspy.ChainOfThought("question -> search_query") + self.generate_answer = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + # First retrieval + query1 = self.generate_query(question=question).search_query + passages1 = self.retrieve(query1).passages + + # Generate follow-up query based on first results + context1 = "\n".join(passages1) + query2 = self.generate_query( + question=f"Based on: {context1}\nFollow-up: {question}" + ).search_query + + # Second retrieval + passages2 = self.retrieve(query2).passages + + # Combine all context + all_context = "\n\n".join(passages1 + passages2) + + # Generate final answer + return self.generate_answer(context=all_context, question=question) + +# Use multi-hop RAG +multi_rag = MultiHopRAG() +result = multi_rag(question="Who wrote the book that inspired Blade Runner?") +# Hop 1: Find "Blade Runner was based on..." +# Hop 2: Find author of that book +``` + +### RAG with Reranking + +```python +class RerankedRAG(dspy.Module): + """RAG with learned reranking of retrieved passages.""" + + def __init__(self): + super().__init__() + self.retrieve = dspy.Retrieve(k=10) # Get more candidates + self.rerank = dspy.Predict("question, passage -> relevance_score: float") + self.answer = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + # Retrieve candidates + passages = self.retrieve(question).passages + + # Rerank passages + scored_passages = [] + for passage in passages: + score = float(self.rerank( + question=question, + passage=passage + ).relevance_score) + scored_passages.append((score, passage)) + + # Take top 3 after reranking + top_passages = [p for _, p in sorted(scored_passages, reverse=True)[:3]] + context = "\n\n".join(top_passages) + + # Generate answer from reranked context + return self.answer(context=context, question=question) +``` + +## Agent Systems + +### ReAct Agent + +```python +from dspy.predict import ReAct + +# Define tools +def search_wikipedia(query: str) -> str: + """Search Wikipedia for information.""" + import wikipedia + try: + return wikipedia.summary(query, sentences=3) + except: + return "No results found" + +def calculate(expression: str) -> str: + """Evaluate mathematical expression safely.""" + try: + # Use safe eval + result = eval(expression, {"__builtins__": {}}, {}) + return str(result) + except: + return "Invalid expression" + +def search_web(query: str) -> str: + """Search the web.""" + # Your web search implementation + return results + +# Create agent signature +class ResearchAgent(dspy.Signature): + """Answer questions using available tools.""" + question = dspy.InputField() + answer = dspy.OutputField() + +# Create ReAct agent +agent = ReAct(ResearchAgent, tools=[search_wikipedia, calculate, search_web]) + +# Agent decides which tools to use +result = agent(question="What is the population of France divided by 10?") +# Agent: +# 1. Thinks: "Need population of France" +# 2. Acts: search_wikipedia("France population") +# 3. Thinks: "Got 67 million, need to divide" +# 4. Acts: calculate("67000000 / 10") +# 5. Returns: "6,700,000" +``` + +### Multi-Agent System + +```python +class MultiAgentSystem(dspy.Module): + """System with specialized agents for different tasks.""" + + def __init__(self): + super().__init__() + + # Router agent + self.router = dspy.Predict("question -> agent_type: str") + + # Specialized agents + self.research_agent = ReAct( + ResearchAgent, + tools=[search_wikipedia, search_web] + ) + self.math_agent = dspy.ProgramOfThought("problem -> answer") + self.reasoning_agent = dspy.ChainOfThought("question -> answer") + + def forward(self, question): + # Route to appropriate agent + agent_type = self.router(question=question).agent_type + + if agent_type == "research": + return self.research_agent(question=question) + elif agent_type == "math": + return self.math_agent(problem=question) + else: + return self.reasoning_agent(question=question) + +# Use multi-agent system +mas = MultiAgentSystem() +result = mas(question="What is 15% of the GDP of France?") +# Routes to research_agent for GDP, then to math_agent for calculation +``` + +## Classification + +### Binary Classifier + +```python +class SentimentClassifier(dspy.Module): + def __init__(self): + super().__init__() + self.classify = dspy.Predict("text -> sentiment: str") + + def forward(self, text): + return self.classify(text=text) + +# Training data +trainset = [ + dspy.Example(text="I love this!", sentiment="positive").with_inputs("text"), + dspy.Example(text="Terrible experience", sentiment="negative").with_inputs("text"), + # ... more examples +] + +# Optimize +def accuracy(example, pred, trace=None): + return example.sentiment == pred.sentiment + +optimizer = BootstrapFewShot(metric=accuracy, max_bootstrapped_demos=5) +classifier = SentimentClassifier() +optimized_classifier = optimizer.compile(classifier, trainset=trainset) + +# Use classifier +result = optimized_classifier(text="This product is amazing!") +print(result.sentiment) # "positive" +``` + +### Multi-Class Classifier + +```python +class TopicClassifier(dspy.Module): + def __init__(self): + super().__init__() + self.classify = dspy.ChainOfThought( + "text -> category: str, confidence: float" + ) + + def forward(self, text): + result = self.classify(text=text) + return dspy.Prediction( + category=result.category, + confidence=float(result.confidence) + ) + +# Define categories in signature +class TopicSignature(dspy.Signature): + """Classify text into one of: technology, sports, politics, entertainment.""" + text = dspy.InputField() + category = dspy.OutputField(desc="one of: technology, sports, politics, entertainment") + confidence = dspy.OutputField(desc="0.0 to 1.0") + +classifier = dspy.ChainOfThought(TopicSignature) +result = classifier(text="The Lakers won the championship") +print(result.category) # "sports" +print(result.confidence) # 0.95 +``` + +### Hierarchical Classifier + +```python +class HierarchicalClassifier(dspy.Module): + """Two-stage classification: coarse then fine-grained.""" + + def __init__(self): + super().__init__() + self.coarse = dspy.Predict("text -> broad_category: str") + self.fine_tech = dspy.Predict("text -> tech_subcategory: str") + self.fine_sports = dspy.Predict("text -> sports_subcategory: str") + + def forward(self, text): + # Stage 1: Broad category + broad = self.coarse(text=text).broad_category + + # Stage 2: Fine-grained based on broad + if broad == "technology": + fine = self.fine_tech(text=text).tech_subcategory + elif broad == "sports": + fine = self.fine_sports(text=text).sports_subcategory + else: + fine = "other" + + return dspy.Prediction(broad_category=broad, fine_category=fine) +``` + +## Data Processing + +### Text Summarization + +```python +class AdaptiveSummarizer(dspy.Module): + """Summarizes text to target length.""" + + def __init__(self): + super().__init__() + self.summarize = dspy.ChainOfThought("text, target_length -> summary") + + def forward(self, text, target_length="3 sentences"): + return self.summarize(text=text, target_length=target_length) + +# Use summarizer +summarizer = AdaptiveSummarizer() +long_text = "..." # Long article + +short_summary = summarizer(long_text, target_length="1 sentence") +medium_summary = summarizer(long_text, target_length="3 sentences") +detailed_summary = summarizer(long_text, target_length="1 paragraph") +``` + +### Information Extraction + +```python +from pydantic import BaseModel, Field + +class PersonInfo(BaseModel): + name: str = Field(description="Full name") + age: int = Field(description="Age in years") + occupation: str = Field(description="Job title") + location: str = Field(description="City and country") + +class ExtractPerson(dspy.Signature): + """Extract person information from text.""" + text = dspy.InputField() + person: PersonInfo = dspy.OutputField() + +extractor = dspy.TypedPredictor(ExtractPerson) + +text = "Dr. Jane Smith, 42, is a neuroscientist at Stanford University in Palo Alto, California." +result = extractor(text=text) + +print(result.person.name) # "Dr. Jane Smith" +print(result.person.age) # 42 +print(result.person.occupation) # "neuroscientist" +print(result.person.location) # "Palo Alto, California" +``` + +### Batch Processing + +```python +class BatchProcessor(dspy.Module): + """Process large datasets efficiently.""" + + def __init__(self): + super().__init__() + self.process = dspy.Predict("text -> processed_text") + + def forward(self, texts): + # Batch processing for efficiency + return self.process.batch([{"text": t} for t in texts]) + +# Process 1000 documents +processor = BatchProcessor() +results = processor(texts=large_dataset) + +# Results are returned in order +for original, result in zip(large_dataset, results): + print(f"{original} -> {result.processed_text}") +``` + +## Multi-Stage Pipelines + +### Document Processing Pipeline + +```python +class DocumentPipeline(dspy.Module): + """Multi-stage document processing.""" + + def __init__(self): + super().__init__() + self.extract = dspy.Predict("document -> key_points") + self.classify = dspy.Predict("key_points -> category") + self.summarize = dspy.ChainOfThought("key_points, category -> summary") + self.tag = dspy.Predict("summary -> tags") + + def forward(self, document): + # Stage 1: Extract key points + key_points = self.extract(document=document).key_points + + # Stage 2: Classify + category = self.classify(key_points=key_points).category + + # Stage 3: Summarize + summary = self.summarize( + key_points=key_points, + category=category + ).summary + + # Stage 4: Generate tags + tags = self.tag(summary=summary).tags + + return dspy.Prediction( + key_points=key_points, + category=category, + summary=summary, + tags=tags + ) +``` + +### Quality Control Pipeline + +```python +class QualityControlPipeline(dspy.Module): + """Generate output and verify quality.""" + + def __init__(self): + super().__init__() + self.generate = dspy.ChainOfThought("prompt -> output") + self.verify = dspy.Predict("output -> is_valid: bool, issues: str") + self.improve = dspy.ChainOfThought("output, issues -> improved_output") + + def forward(self, prompt, max_iterations=3): + output = self.generate(prompt=prompt).output + + for _ in range(max_iterations): + # Verify output + verification = self.verify(output=output) + + if verification.is_valid: + return dspy.Prediction(output=output, iterations=_ + 1) + + # Improve based on issues + output = self.improve( + output=output, + issues=verification.issues + ).improved_output + + return dspy.Prediction(output=output, iterations=max_iterations) +``` + +## Production Tips + +### 1. Caching for Performance + +```python +from functools import lru_cache + +class CachedRAG(dspy.Module): + def __init__(self): + super().__init__() + self.retrieve = dspy.Retrieve(k=3) + self.generate = dspy.ChainOfThought("context, question -> answer") + + @lru_cache(maxsize=1000) + def forward(self, question): + passages = self.retrieve(question).passages + context = "\n".join(passages) + return self.generate(context=context, question=question).answer +``` + +### 2. Error Handling + +```python +class RobustModule(dspy.Module): + def __init__(self): + super().__init__() + self.process = dspy.ChainOfThought("input -> output") + + def forward(self, input): + try: + result = self.process(input=input) + return result + except Exception as e: + # Log error + print(f"Error processing {input}: {e}") + # Return fallback + return dspy.Prediction(output="Error: could not process input") +``` + +### 3. Monitoring + +```python +class MonitoredModule(dspy.Module): + def __init__(self): + super().__init__() + self.process = dspy.ChainOfThought("input -> output") + self.call_count = 0 + self.errors = 0 + + def forward(self, input): + self.call_count += 1 + + try: + result = self.process(input=input) + return result + except Exception as e: + self.errors += 1 + raise + + def get_stats(self): + return { + "calls": self.call_count, + "errors": self.errors, + "error_rate": self.errors / max(self.call_count, 1) + } +``` + +### 4. A/B Testing + +```python +class ABTestModule(dspy.Module): + """Run two variants and compare.""" + + def __init__(self, variant_a, variant_b): + super().__init__() + self.variant_a = variant_a + self.variant_b = variant_b + self.a_calls = 0 + self.b_calls = 0 + + def forward(self, input, variant="a"): + if variant == "a": + self.a_calls += 1 + return self.variant_a(input=input) + else: + self.b_calls += 1 + return self.variant_b(input=input) + +# Compare two optimizers +baseline = dspy.ChainOfThought("question -> answer") +optimized = BootstrapFewShot(...).compile(baseline, trainset=trainset) + +ab_test = ABTestModule(variant_a=baseline, variant_b=optimized) + +# Route 50% to each +import random +variant = "a" if random.random() < 0.5 else "b" +result = ab_test(input=question, variant=variant) +``` + +## Complete Example: Customer Support Bot + +```python +import dspy +from dspy.teleprompt import BootstrapFewShot + +class CustomerSupportBot(dspy.Module): + """Complete customer support system.""" + + def __init__(self): + super().__init__() + + # Classify intent + self.classify_intent = dspy.Predict("message -> intent: str") + + # Specialized handlers + self.technical_handler = dspy.ChainOfThought("message, history -> response") + self.billing_handler = dspy.ChainOfThought("message, history -> response") + self.general_handler = dspy.Predict("message, history -> response") + + # Retrieve relevant docs + self.retrieve = dspy.Retrieve(k=3) + + # Conversation history + self.history = [] + + def forward(self, message): + # Classify intent + intent = self.classify_intent(message=message).intent + + # Retrieve relevant documentation + docs = self.retrieve(message).passages + context = "\n".join(docs) + + # Add context to history + history_str = "\n".join(self.history) + full_message = f"Context: {context}\n\nMessage: {message}" + + # Route to appropriate handler + if intent == "technical": + response = self.technical_handler( + message=full_message, + history=history_str + ).response + elif intent == "billing": + response = self.billing_handler( + message=full_message, + history=history_str + ).response + else: + response = self.general_handler( + message=full_message, + history=history_str + ).response + + # Update history + self.history.append(f"User: {message}") + self.history.append(f"Bot: {response}") + + return dspy.Prediction(response=response, intent=intent) + +# Training data +trainset = [ + dspy.Example( + message="My account isn't working", + intent="technical", + response="I'd be happy to help. What error are you seeing?" + ).with_inputs("message"), + # ... more examples +] + +# Define metric +def response_quality(example, pred, trace=None): + # Check if response is helpful + if len(pred.response) < 20: + return 0.0 + if example.intent != pred.intent: + return 0.3 + return 1.0 + +# Optimize +optimizer = BootstrapFewShot(metric=response_quality) +bot = CustomerSupportBot() +optimized_bot = optimizer.compile(bot, trainset=trainset) + +# Use in production +optimized_bot.save("models/support_bot_v1.json") + +# Later, load and use +loaded_bot = CustomerSupportBot() +loaded_bot.load("models/support_bot_v1.json") +response = loaded_bot(message="I can't log in") +``` + +## Resources + +- **Documentation**: https://dspy.ai +- **Examples Repo**: https://github.com/stanfordnlp/dspy/tree/main/examples +- **Discord**: https://discord.gg/XCGy2WDCQB diff --git a/dspy/references/modules.md b/dspy/references/modules.md new file mode 100644 index 0000000..aa373d0 --- /dev/null +++ b/dspy/references/modules.md @@ -0,0 +1,475 @@ +# DSPy Modules + +Complete guide to DSPy's built-in modules for language model programming. + +## Module Basics + +DSPy modules are composable building blocks inspired by PyTorch's NN modules: +- Have learnable parameters (prompts, few-shot examples) +- Can be composed using Python control flow +- Generalized to handle any signature +- Optimizable with DSPy optimizers + +### Base Module Pattern + +```python +import dspy + +class CustomModule(dspy.Module): + def __init__(self): + super().__init__() + # Initialize sub-modules + self.predictor = dspy.Predict("input -> output") + + def forward(self, input): + # Module logic + result = self.predictor(input=input) + return result +``` + +## Core Modules + +### dspy.Predict + +**Basic prediction module** - Makes LM calls without reasoning steps. + +```python +# Inline signature +qa = dspy.Predict("question -> answer") +result = qa(question="What is 2+2?") + +# Class signature +class QA(dspy.Signature): + """Answer questions concisely.""" + question = dspy.InputField() + answer = dspy.OutputField(desc="short, factual answer") + +qa = dspy.Predict(QA) +result = qa(question="What is the capital of France?") +print(result.answer) # "Paris" +``` + +**When to use:** +- Simple, direct predictions +- No reasoning steps needed +- Fast responses required + +### dspy.ChainOfThought + +**Step-by-step reasoning** - Generates rationale before answer. + +**Parameters:** +- `signature`: Task signature +- `rationale_field`: Custom reasoning field (optional) +- `rationale_field_type`: Type for rationale (default: `str`) + +```python +# Basic usage +cot = dspy.ChainOfThought("question -> answer") +result = cot(question="If I have 5 apples and give away 2, how many remain?") +print(result.rationale) # "Let's think step by step..." +print(result.answer) # "3" + +# Custom rationale field +cot = dspy.ChainOfThought( + signature="problem -> solution", + rationale_field=dspy.OutputField( + prefix="Reasoning: Let's break this down step by step to" + ) +) +``` + +**When to use:** +- Complex reasoning tasks +- Math word problems +- Logical deduction +- Quality > speed + +**Performance:** +- ~2x slower than Predict +- Significantly better accuracy on reasoning tasks + +### dspy.ProgramOfThought + +**Code-based reasoning** - Generates and executes Python code. + +```python +pot = dspy.ProgramOfThought("question -> answer") + +result = pot(question="What is 15% of 240?") +# Internally generates: answer = 240 * 0.15 +# Executes code and returns result +print(result.answer) # 36.0 + +result = pot(question="If a train travels 60 mph for 2.5 hours, how far does it go?") +# Generates: distance = 60 * 2.5 +print(result.answer) # 150.0 +``` + +**When to use:** +- Arithmetic calculations +- Symbolic math +- Data transformations +- Deterministic computations + +**Benefits:** +- More reliable than text-based math +- Handles complex calculations +- Transparent (shows generated code) + +### dspy.ReAct + +**Reasoning + Acting** - Agent that uses tools iteratively. + +```python +from dspy.predict import ReAct + +# Define tools +def search_wikipedia(query: str) -> str: + """Search Wikipedia for information.""" + # Your search implementation + return search_results + +def calculate(expression: str) -> float: + """Evaluate a mathematical expression.""" + return eval(expression) + +# Create ReAct agent +class ResearchQA(dspy.Signature): + """Answer questions using available tools.""" + question = dspy.InputField() + answer = dspy.OutputField() + +react = ReAct(ResearchQA, tools=[search_wikipedia, calculate]) + +# Agent decides which tools to use +result = react(question="How old was Einstein when he published special relativity?") +# Internally: +# 1. Thinks: "Need birth year and publication year" +# 2. Acts: search_wikipedia("Albert Einstein") +# 3. Acts: search_wikipedia("Special relativity 1905") +# 4. Acts: calculate("1905 - 1879") +# 5. Returns: "26 years old" +``` + +**When to use:** +- Multi-step research tasks +- Tool-using agents +- Complex information retrieval +- Tasks requiring multiple API calls + +**Best practices:** +- Keep tool descriptions clear and specific +- Limit to 5-7 tools (too many = confusion) +- Provide tool usage examples in docstrings + +### dspy.MultiChainComparison + +**Generate multiple outputs and compare** - Self-consistency pattern. + +```python +mcc = dspy.MultiChainComparison("question -> answer", M=5) + +result = mcc(question="What is the capital of France?") +# Generates 5 candidate answers +# Compares and selects most consistent +print(result.answer) # "Paris" +print(result.candidates) # All 5 generated answers +``` + +**Parameters:** +- `M`: Number of candidates to generate (default: 5) +- `temperature`: Sampling temperature for diversity + +**When to use:** +- High-stakes decisions +- Ambiguous questions +- When single answer may be unreliable + +**Tradeoff:** +- M times slower (M parallel calls) +- Higher accuracy on ambiguous tasks + +### dspy.majority + +**Majority voting over multiple predictions.** + +```python +from dspy.primitives import majority + +# Generate multiple predictions +predictor = dspy.Predict("question -> answer") +predictions = [predictor(question="What is 2+2?") for _ in range(5)] + +# Take majority vote +answer = majority([p.answer for p in predictions]) +print(answer) # "4" +``` + +**When to use:** +- Combining multiple model outputs +- Reducing variance in predictions +- Ensemble approaches + +## Advanced Modules + +### dspy.TypedPredictor + +**Structured output with Pydantic models.** + +```python +from pydantic import BaseModel, Field + +class PersonInfo(BaseModel): + name: str = Field(description="Full name") + age: int = Field(description="Age in years") + occupation: str = Field(description="Current job") + +class ExtractPerson(dspy.Signature): + """Extract person information from text.""" + text = dspy.InputField() + person: PersonInfo = dspy.OutputField() + +extractor = dspy.TypedPredictor(ExtractPerson) +result = extractor(text="John Doe is a 35-year-old software engineer.") + +print(result.person.name) # "John Doe" +print(result.person.age) # 35 +print(result.person.occupation) # "software engineer" +``` + +**Benefits:** +- Type safety +- Automatic validation +- JSON schema generation +- IDE autocomplete + +### dspy.Retry + +**Automatic retry with validation.** + +```python +from dspy.primitives import Retry + +def validate_number(example, pred, trace=None): + """Validate output is a number.""" + try: + float(pred.answer) + return True + except ValueError: + return False + +# Retry up to 3 times if validation fails +qa = Retry( + dspy.ChainOfThought("question -> answer"), + validate=validate_number, + max_retries=3 +) + +result = qa(question="What is 15% of 80?") +# If first attempt returns non-numeric, retries automatically +``` + +### dspy.Assert + +**Assertion-driven optimization.** + +```python +import dspy +from dspy.primitives.assertions import assert_transform_module, backtrack_handler + +class ValidatedQA(dspy.Module): + def __init__(self): + super().__init__() + self.qa = dspy.ChainOfThought("question -> answer: float") + + def forward(self, question): + answer = self.qa(question=question).answer + + # Assert answer is numeric + dspy.Assert( + isinstance(float(answer), float), + "Answer must be a number", + backtrack=backtrack_handler + ) + + return dspy.Prediction(answer=answer) +``` + +**Benefits:** +- Catches errors during optimization +- Guides LM toward valid outputs +- Better than post-hoc filtering + +## Module Composition + +### Sequential Pipeline + +```python +class Pipeline(dspy.Module): + def __init__(self): + super().__init__() + self.stage1 = dspy.Predict("input -> intermediate") + self.stage2 = dspy.ChainOfThought("intermediate -> output") + + def forward(self, input): + intermediate = self.stage1(input=input).intermediate + output = self.stage2(intermediate=intermediate).output + return dspy.Prediction(output=output) +``` + +### Conditional Logic + +```python +class ConditionalModule(dspy.Module): + def __init__(self): + super().__init__() + self.router = dspy.Predict("question -> category: str") + self.simple_qa = dspy.Predict("question -> answer") + self.complex_qa = dspy.ChainOfThought("question -> answer") + + def forward(self, question): + category = self.router(question=question).category + + if category == "simple": + return self.simple_qa(question=question) + else: + return self.complex_qa(question=question) +``` + +### Parallel Execution + +```python +class ParallelModule(dspy.Module): + def __init__(self): + super().__init__() + self.approach1 = dspy.ChainOfThought("question -> answer") + self.approach2 = dspy.ProgramOfThought("question -> answer") + + def forward(self, question): + # Run both approaches + answer1 = self.approach1(question=question).answer + answer2 = self.approach2(question=question).answer + + # Compare or combine results + if answer1 == answer2: + return dspy.Prediction(answer=answer1, confidence="high") + else: + return dspy.Prediction(answer=answer1, confidence="low") +``` + +## Batch Processing + +All modules support batch processing for efficiency: + +```python +cot = dspy.ChainOfThought("question -> answer") + +questions = [ + "What is 2+2?", + "What is 3+3?", + "What is 4+4?" +] + +# Process all at once +results = cot.batch([{"question": q} for q in questions]) + +for result in results: + print(result.answer) +``` + +## Saving and Loading + +```python +# Save module +qa = dspy.ChainOfThought("question -> answer") +qa.save("models/qa_v1.json") + +# Load module +loaded_qa = dspy.ChainOfThought("question -> answer") +loaded_qa.load("models/qa_v1.json") +``` + +**What gets saved:** +- Few-shot examples +- Prompt instructions +- Module configuration + +**What doesn't get saved:** +- Model weights (DSPy doesn't fine-tune by default) +- LM provider configuration + +## Module Selection Guide + +| Task | Module | Reason | +|------|--------|--------| +| Simple classification | Predict | Fast, direct | +| Math word problems | ProgramOfThought | Reliable calculations | +| Logical reasoning | ChainOfThought | Better with steps | +| Multi-step research | ReAct | Tool usage | +| High-stakes decisions | MultiChainComparison | Self-consistency | +| Structured extraction | TypedPredictor | Type safety | +| Ambiguous questions | MultiChainComparison | Multiple perspectives | + +## Performance Tips + +1. **Start with Predict**, add reasoning only if needed +2. **Use batch processing** for multiple inputs +3. **Cache predictions** for repeated queries +4. **Profile token usage** with `track_usage=True` +5. **Optimize after prototyping** with teleprompters + +## Common Patterns + +### Pattern: Retrieval + Generation + +```python +class RAG(dspy.Module): + def __init__(self, k=3): + super().__init__() + self.retrieve = dspy.Retrieve(k=k) + self.generate = dspy.ChainOfThought("context, question -> answer") + + def forward(self, question): + context = self.retrieve(question).passages + return self.generate(context=context, question=question) +``` + +### Pattern: Verification Loop + +```python +class VerifiedQA(dspy.Module): + def __init__(self): + super().__init__() + self.answer = dspy.ChainOfThought("question -> answer") + self.verify = dspy.Predict("question, answer -> is_correct: bool") + + def forward(self, question, max_attempts=3): + for _ in range(max_attempts): + answer = self.answer(question=question).answer + is_correct = self.verify(question=question, answer=answer).is_correct + + if is_correct: + return dspy.Prediction(answer=answer) + + return dspy.Prediction(answer="Unable to verify answer") +``` + +### Pattern: Multi-Turn Dialog + +```python +class DialogAgent(dspy.Module): + def __init__(self): + super().__init__() + self.respond = dspy.Predict("history, user_message -> assistant_message") + self.history = [] + + def forward(self, user_message): + history_str = "\n".join(self.history) + response = self.respond(history=history_str, user_message=user_message) + + self.history.append(f"User: {user_message}") + self.history.append(f"Assistant: {response.assistant_message}") + + return response +``` diff --git a/dspy/references/optimizers.md b/dspy/references/optimizers.md new file mode 100644 index 0000000..62bba96 --- /dev/null +++ b/dspy/references/optimizers.md @@ -0,0 +1,566 @@ +# DSPy Optimizers (Teleprompters) + +Complete guide to DSPy's optimization algorithms for improving prompts and model weights. + +## What are Optimizers? + +DSPy optimizers (called "teleprompters") automatically improve your modules by: +- **Synthesizing few-shot examples** from training data +- **Proposing better instructions** through search +- **Fine-tuning model weights** (optional) + +**Key idea**: Instead of manually tuning prompts, define a metric and let DSPy optimize. + +## Optimizer Selection Guide + +| Optimizer | Best For | Speed | Quality | Data Needed | +|-----------|----------|-------|---------|-------------| +| BootstrapFewShot | General purpose | Fast | Good | 10-50 examples | +| MIPRO | Instruction tuning | Medium | Excellent | 50-200 examples | +| BootstrapFinetune | Fine-tuning | Slow | Excellent | 100+ examples | +| COPRO | Prompt optimization | Medium | Good | 20-100 examples | +| KNNFewShot | Quick baseline | Very fast | Fair | 10+ examples | + +## Core Optimizers + +### BootstrapFewShot + +**Most popular optimizer** - Generates few-shot demonstrations from training data. + +**How it works:** +1. Takes your training examples +2. Uses your module to generate predictions +3. Selects high-quality predictions (based on metric) +4. Uses these as few-shot examples in future prompts + +**Parameters:** +- `metric`: Function that scores predictions (required) +- `max_bootstrapped_demos`: Max demonstrations to generate (default: 4) +- `max_labeled_demos`: Max labeled examples to use (default: 16) +- `max_rounds`: Optimization iterations (default: 1) +- `metric_threshold`: Minimum score to accept (optional) + +```python +import dspy +from dspy.teleprompt import BootstrapFewShot + +# Define metric +def validate_answer(example, pred, trace=None): + """Return True if prediction matches gold answer.""" + return example.answer.lower() == pred.answer.lower() + +# Training data +trainset = [ + dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"), + dspy.Example(question="What is 3+5?", answer="8").with_inputs("question"), + dspy.Example(question="What is 10-3?", answer="7").with_inputs("question"), +] + +# Create module +qa = dspy.ChainOfThought("question -> answer") + +# Optimize +optimizer = BootstrapFewShot( + metric=validate_answer, + max_bootstrapped_demos=3, + max_rounds=2 +) + +optimized_qa = optimizer.compile(qa, trainset=trainset) + +# Now optimized_qa has learned few-shot examples! +result = optimized_qa(question="What is 5+7?") +``` + +**Best practices:** +- Start with 10-50 training examples +- Use diverse examples covering edge cases +- Set `max_bootstrapped_demos=3-5` for most tasks +- Increase `max_rounds=2-3` for better quality + +**When to use:** +- First optimizer to try +- You have 10+ labeled examples +- Want quick improvements +- General-purpose tasks + +### MIPRO (Most Important Prompt Optimization) + +**State-of-the-art optimizer** - Iteratively searches for better instructions. + +**How it works:** +1. Generates candidate instructions +2. Tests each on validation set +3. Selects best-performing instructions +4. Iterates to refine further + +**Parameters:** +- `metric`: Evaluation metric (required) +- `num_candidates`: Instructions to try per iteration (default: 10) +- `init_temperature`: Sampling temperature (default: 1.0) +- `verbose`: Show progress (default: False) + +```python +from dspy.teleprompt import MIPRO + +# Define metric with more nuance +def answer_quality(example, pred, trace=None): + """Score answer quality 0-1.""" + if example.answer.lower() in pred.answer.lower(): + return 1.0 + # Partial credit for similar answers + return 0.5 if len(set(example.answer.split()) & set(pred.answer.split())) > 0 else 0.0 + +# Larger training set (MIPRO benefits from more data) +trainset = [...] # 50-200 examples +valset = [...] # 20-50 examples + +# Create module +qa = dspy.ChainOfThought("question -> answer") + +# Optimize with MIPRO +optimizer = MIPRO( + metric=answer_quality, + num_candidates=10, + init_temperature=1.0, + verbose=True +) + +optimized_qa = optimizer.compile( + student=qa, + trainset=trainset, + valset=valset, # MIPRO uses separate validation set + num_trials=100 # More trials = better quality +) +``` + +**Best practices:** +- Use 50-200 training examples +- Separate validation set (20-50 examples) +- Run 100-200 trials for best results +- Takes 10-30 minutes typically + +**When to use:** +- You have 50+ labeled examples +- Want state-of-the-art performance +- Willing to wait for optimization +- Complex reasoning tasks + +### BootstrapFinetune + +**Fine-tune model weights** - Creates training dataset for fine-tuning. + +**How it works:** +1. Generates synthetic training data +2. Exports data in fine-tuning format +3. You fine-tune model separately +4. Load fine-tuned model back + +**Parameters:** +- `metric`: Evaluation metric (required) +- `max_bootstrapped_demos`: Demonstrations to generate (default: 4) +- `max_rounds`: Data generation rounds (default: 1) + +```python +from dspy.teleprompt import BootstrapFinetune + +# Training data +trainset = [...] # 100+ examples recommended + +# Define metric +def validate(example, pred, trace=None): + return example.answer == pred.answer + +# Create module +qa = dspy.ChainOfThought("question -> answer") + +# Generate fine-tuning data +optimizer = BootstrapFinetune(metric=validate) +optimized_qa = optimizer.compile(qa, trainset=trainset) + +# Exports training data to file +# You then fine-tune using your LM provider's API + +# After fine-tuning, load your model: +finetuned_lm = dspy.OpenAI(model="ft:gpt-3.5-turbo:your-model-id") +dspy.settings.configure(lm=finetuned_lm) +``` + +**Best practices:** +- Use 100+ training examples +- Validate on held-out test set +- Monitor for overfitting +- Compare with prompt-based methods first + +**When to use:** +- You have 100+ examples +- Latency is critical (fine-tuned models faster) +- Task is narrow and well-defined +- Prompt optimization isn't enough + +### COPRO (Coordinate Prompt Optimization) + +**Optimize prompts via gradient-free search.** + +**How it works:** +1. Generates prompt variants +2. Evaluates each variant +3. Selects best prompts +4. Iterates to refine + +```python +from dspy.teleprompt import COPRO + +# Training data +trainset = [...] + +# Define metric +def metric(example, pred, trace=None): + return example.answer == pred.answer + +# Create module +qa = dspy.ChainOfThought("question -> answer") + +# Optimize with COPRO +optimizer = COPRO( + metric=metric, + breadth=10, # Candidates per iteration + depth=3 # Optimization rounds +) + +optimized_qa = optimizer.compile(qa, trainset=trainset) +``` + +**When to use:** +- Want prompt optimization +- Have 20-100 examples +- MIPRO too slow + +### KNNFewShot + +**Simple k-nearest neighbors** - Selects similar examples for each query. + +**How it works:** +1. Embeds all training examples +2. For each query, finds k most similar examples +3. Uses these as few-shot demonstrations + +```python +from dspy.teleprompt import KNNFewShot + +trainset = [...] + +# No metric needed - just selects similar examples +optimizer = KNNFewShot(k=3) +optimized_qa = optimizer.compile(qa, trainset=trainset) + +# For each query, uses 3 most similar examples from trainset +``` + +**When to use:** +- Quick baseline +- Have diverse training examples +- Similarity is good proxy for helpfulness + +## Writing Metrics + +Metrics are functions that score predictions. They're critical for optimization. + +### Binary Metrics + +```python +def exact_match(example, pred, trace=None): + """Return True if prediction exactly matches gold.""" + return example.answer == pred.answer + +def contains_answer(example, pred, trace=None): + """Return True if prediction contains gold answer.""" + return example.answer.lower() in pred.answer.lower() +``` + +### Continuous Metrics + +```python +def f1_score(example, pred, trace=None): + """F1 score between prediction and gold.""" + pred_tokens = set(pred.answer.lower().split()) + gold_tokens = set(example.answer.lower().split()) + + if not pred_tokens: + return 0.0 + + precision = len(pred_tokens & gold_tokens) / len(pred_tokens) + recall = len(pred_tokens & gold_tokens) / len(gold_tokens) + + if precision + recall == 0: + return 0.0 + + return 2 * (precision * recall) / (precision + recall) + +def semantic_similarity(example, pred, trace=None): + """Embedding similarity between prediction and gold.""" + from sentence_transformers import SentenceTransformer + model = SentenceTransformer('all-MiniLM-L6-v2') + + emb1 = model.encode(example.answer) + emb2 = model.encode(pred.answer) + + similarity = cosine_similarity(emb1, emb2) + return similarity +``` + +### Multi-Factor Metrics + +```python +def comprehensive_metric(example, pred, trace=None): + """Combine multiple factors.""" + score = 0.0 + + # Correctness (50%) + if example.answer.lower() in pred.answer.lower(): + score += 0.5 + + # Conciseness (25%) + if len(pred.answer.split()) <= 20: + score += 0.25 + + # Citation (25%) + if "source:" in pred.answer.lower(): + score += 0.25 + + return score +``` + +### Using Trace for Debugging + +```python +def metric_with_trace(example, pred, trace=None): + """Metric that uses trace for debugging.""" + is_correct = example.answer == pred.answer + + if trace is not None and not is_correct: + # Log failures for analysis + print(f"Failed on: {example.question}") + print(f"Expected: {example.answer}") + print(f"Got: {pred.answer}") + + return is_correct +``` + +## Evaluation Best Practices + +### Train/Val/Test Split + +```python +# Split data +trainset = data[:100] # 70% +valset = data[100:120] # 15% +testset = data[120:] # 15% + +# Optimize on train +optimized = optimizer.compile(module, trainset=trainset) + +# Validate during optimization (for MIPRO) +optimized = optimizer.compile(module, trainset=trainset, valset=valset) + +# Evaluate on test +from dspy.evaluate import Evaluate +evaluator = Evaluate(devset=testset, metric=metric) +score = evaluator(optimized) +``` + +### Cross-Validation + +```python +from sklearn.model_selection import KFold + +kfold = KFold(n_splits=5) +scores = [] + +for train_idx, val_idx in kfold.split(data): + trainset = [data[i] for i in train_idx] + valset = [data[i] for i in val_idx] + + optimized = optimizer.compile(module, trainset=trainset) + score = evaluator(optimized, devset=valset) + scores.append(score) + +print(f"Average score: {sum(scores) / len(scores):.2f}") +``` + +### Comparing Optimizers + +```python +results = {} + +for opt_name, optimizer in [ + ("baseline", None), + ("fewshot", BootstrapFewShot(metric=metric)), + ("mipro", MIPRO(metric=metric)), +]: + if optimizer is None: + module_opt = module + else: + module_opt = optimizer.compile(module, trainset=trainset) + + score = evaluator(module_opt, devset=testset) + results[opt_name] = score + +print(results) +# {'baseline': 0.65, 'fewshot': 0.78, 'mipro': 0.85} +``` + +## Advanced Patterns + +### Custom Optimizer + +```python +from dspy.teleprompt import Teleprompter + +class CustomOptimizer(Teleprompter): + def __init__(self, metric): + self.metric = metric + + def compile(self, student, trainset, **kwargs): + # Your optimization logic here + # Return optimized student module + return student +``` + +### Multi-Stage Optimization + +```python +# Stage 1: Bootstrap few-shot +stage1 = BootstrapFewShot(metric=metric, max_bootstrapped_demos=3) +optimized1 = stage1.compile(module, trainset=trainset) + +# Stage 2: Instruction tuning +stage2 = MIPRO(metric=metric, num_candidates=10) +optimized2 = stage2.compile(optimized1, trainset=trainset, valset=valset) + +# Final optimized module +final_module = optimized2 +``` + +### Ensemble Optimization + +```python +class EnsembleModule(dspy.Module): + def __init__(self, modules): + super().__init__() + self.modules = modules + + def forward(self, question): + predictions = [m(question=question).answer for m in self.modules] + # Vote or average + return dspy.Prediction(answer=max(set(predictions), key=predictions.count)) + +# Optimize multiple modules +opt1 = BootstrapFewShot(metric=metric).compile(module, trainset=trainset) +opt2 = MIPRO(metric=metric).compile(module, trainset=trainset) +opt3 = COPRO(metric=metric).compile(module, trainset=trainset) + +# Ensemble +ensemble = EnsembleModule([opt1, opt2, opt3]) +``` + +## Optimization Workflow + +### 1. Start with Baseline + +```python +# No optimization +baseline = dspy.ChainOfThought("question -> answer") +baseline_score = evaluator(baseline, devset=testset) +print(f"Baseline: {baseline_score}") +``` + +### 2. Try BootstrapFewShot + +```python +# Quick optimization +fewshot = BootstrapFewShot(metric=metric, max_bootstrapped_demos=3) +optimized = fewshot.compile(baseline, trainset=trainset) +fewshot_score = evaluator(optimized, devset=testset) +print(f"Few-shot: {fewshot_score} (+{fewshot_score - baseline_score:.2f})") +``` + +### 3. If More Data Available, Try MIPRO + +```python +# State-of-the-art optimization +mipro = MIPRO(metric=metric, num_candidates=10) +optimized_mipro = mipro.compile(baseline, trainset=trainset, valset=valset) +mipro_score = evaluator(optimized_mipro, devset=testset) +print(f"MIPRO: {mipro_score} (+{mipro_score - baseline_score:.2f})") +``` + +### 4. Save Best Model + +```python +if mipro_score > fewshot_score: + optimized_mipro.save("models/best_model.json") +else: + optimized.save("models/best_model.json") +``` + +## Common Pitfalls + +### 1. Overfitting to Training Data + +```python +# ❌ Bad: Too many demos +optimizer = BootstrapFewShot(max_bootstrapped_demos=20) # Overfits! + +# ✅ Good: Moderate demos +optimizer = BootstrapFewShot(max_bootstrapped_demos=3-5) +``` + +### 2. Metric Doesn't Match Task + +```python +# ❌ Bad: Binary metric for nuanced task +def bad_metric(example, pred, trace=None): + return example.answer == pred.answer # Too strict! + +# ✅ Good: Graded metric +def good_metric(example, pred, trace=None): + return f1_score(example.answer, pred.answer) # Allows partial credit +``` + +### 3. Insufficient Training Data + +```python +# ❌ Bad: Too little data +trainset = data[:5] # Not enough! + +# ✅ Good: Sufficient data +trainset = data[:50] # Better +``` + +### 4. No Validation Set + +```python +# ❌ Bad: Optimizing on test set +optimizer.compile(module, trainset=testset) # Cheating! + +# ✅ Good: Proper splits +optimizer.compile(module, trainset=trainset, valset=valset) +evaluator(optimized, devset=testset) +``` + +## Performance Tips + +1. **Start simple**: BootstrapFewShot first +2. **Use representative data**: Cover edge cases +3. **Monitor overfitting**: Validate on held-out set +4. **Iterate metrics**: Refine based on failures +5. **Save checkpoints**: Don't lose progress +6. **Compare to baseline**: Measure improvement +7. **Test multiple optimizers**: Find best fit + +## Resources + +- **Paper**: "DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines" +- **GitHub**: https://github.com/stanfordnlp/dspy +- **Discord**: https://discord.gg/XCGy2WDCQB diff --git a/evaluating-llms-harness/SKILL.md b/evaluating-llms-harness/SKILL.md new file mode 100644 index 0000000..306b4f6 --- /dev/null +++ b/evaluating-llms-harness/SKILL.md @@ -0,0 +1,498 @@ +--- +name: evaluating-llms-harness +description: "lm-eval-harness: benchmark LLMs (MMLU, GSM8K, etc.)." +version: 1.0.1 +author: Orchestra Research +license: MIT +dependencies: [lm-eval, transformers, vllm] +platforms: [linux, macos] +metadata: + hermes: + tags: [Evaluation, LM Evaluation Harness, Benchmarking, MMLU, HumanEval, GSM8K, EleutherAI, Model Quality, Academic Benchmarks, Industry Standard] + +--- + +# lm-evaluation-harness - LLM Benchmarking + +## What's inside + +Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs. + +## Quick start + +lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. + +**Installation**: +```bash +pip install lm-eval +``` + +**Evaluate any HuggingFace model**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu,gsm8k,hellaswag \ + --device cuda:0 \ + --batch_size 8 +``` + +**View available tasks**: +```bash +lm-eval ls tasks +``` + +## Common workflows + +### Workflow 1: Standard benchmark evaluation + +Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). + +Copy this checklist: + +``` +Benchmark Evaluation: +- [ ] Step 1: Choose benchmark suite +- [ ] Step 2: Configure model +- [ ] Step 3: Run evaluation +- [ ] Step 4: Analyze results +``` + +**Step 1: Choose benchmark suite** + +**Core reasoning benchmarks**: +- **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice +- **GSM8K** - Grade school math word problems +- **HellaSwag** - Common sense reasoning +- **TruthfulQA** - Truthfulness and factuality +- **ARC** (AI2 Reasoning Challenge) - Science questions + +**Code benchmarks**: +- **HumanEval** - Python code generation (164 problems) +- **MBPP** (Mostly Basic Python Problems) - Python coding + +**Standard suite** (recommended for model releases): +```bash +--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge +``` + +**Step 2: Configure model** + +**HuggingFace model**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \ + --tasks mmlu \ + --device cuda:0 \ + --batch_size auto # Auto-detect optimal batch size +``` + +**Quantized model (4-bit/8-bit)**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \ + --tasks mmlu \ + --device cuda:0 +``` + +**Custom checkpoint**: +```bash +lm_eval --model hf \ + --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \ + --tasks mmlu \ + --device cuda:0 +``` + +**Step 3: Run evaluation** + +```bash +# Full MMLU evaluation (57 subjects) +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu \ + --num_fewshot 5 \ # 5-shot evaluation (standard) + --batch_size 8 \ + --output_path results/ \ + --log_samples # Save individual predictions + +# Multiple benchmarks at once +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \ + --num_fewshot 5 \ + --batch_size 8 \ + --output_path results/llama2-7b-eval.json +``` + +**Step 4: Analyze results** + +Results saved to `results/llama2-7b-eval.json`: + +```json +{ + "results": { + "mmlu": { + "acc": 0.459, + "acc_stderr": 0.004 + }, + "gsm8k": { + "exact_match": 0.142, + "exact_match_stderr": 0.006 + }, + "hellaswag": { + "acc_norm": 0.765, + "acc_norm_stderr": 0.004 + } + }, + "config": { + "model": "hf", + "model_args": "pretrained=meta-llama/Llama-2-7b-hf", + "num_fewshot": 5 + } +} +``` + +### Workflow 2: Track training progress + +Evaluate checkpoints during training. + +``` +Training Progress Tracking: +- [ ] Step 1: Set up periodic evaluation +- [ ] Step 2: Choose quick benchmarks +- [ ] Step 3: Automate evaluation +- [ ] Step 4: Plot learning curves +``` + +**Step 1: Set up periodic evaluation** + +Evaluate every N training steps: + +```bash +#!/bin/bash +# eval_checkpoint.sh + +CHECKPOINT_DIR=$1 +STEP=$2 + +lm_eval --model hf \ + --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \ + --tasks gsm8k,hellaswag \ + --num_fewshot 0 \ # 0-shot for speed + --batch_size 16 \ + --output_path results/step-$STEP.json +``` + +**Step 2: Choose quick benchmarks** + +Fast benchmarks for frequent evaluation: +- **HellaSwag**: ~10 minutes on 1 GPU +- **GSM8K**: ~5 minutes +- **PIQA**: ~2 minutes + +Avoid for frequent eval (too slow): +- **MMLU**: ~2 hours (57 subjects) +- **HumanEval**: Requires code execution + +**Step 3: Automate evaluation** + +Integrate with training script: + +```python +# In training loop +if step % eval_interval == 0: + model.save_pretrained(f"checkpoints/step-{step}") + + # Run evaluation + os.system(f"./eval_checkpoint.sh checkpoints step-{step}") +``` + +Or use PyTorch Lightning callbacks: + +```python +from pytorch_lightning import Callback + +class EvalHarnessCallback(Callback): + def on_validation_epoch_end(self, trainer, pl_module): + step = trainer.global_step + checkpoint_path = f"checkpoints/step-{step}" + + # Save checkpoint + trainer.save_checkpoint(checkpoint_path) + + # Run lm-eval + os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...") +``` + +**Step 4: Plot learning curves** + +```python +import json +import matplotlib.pyplot as plt + +# Load all results +steps = [] +mmlu_scores = [] + +for file in sorted(glob.glob("results/step-*.json")): + with open(file) as f: + data = json.load(f) + step = int(file.split("-")[1].split(".")[0]) + steps.append(step) + mmlu_scores.append(data["results"]["mmlu"]["acc"]) + +# Plot +plt.plot(steps, mmlu_scores) +plt.xlabel("Training Step") +plt.ylabel("MMLU Accuracy") +plt.title("Training Progress") +plt.savefig("training_curve.png") +``` + +### Workflow 3: Compare multiple models + +Benchmark suite for model comparison. + +``` +Model Comparison: +- [ ] Step 1: Define model list +- [ ] Step 2: Run evaluations +- [ ] Step 3: Generate comparison table +``` + +**Step 1: Define model list** + +```bash +# models.txt +meta-llama/Llama-2-7b-hf +meta-llama/Llama-2-13b-hf +mistralai/Mistral-7B-v0.1 +microsoft/phi-2 +``` + +**Step 2: Run evaluations** + +```bash +#!/bin/bash +# eval_all_models.sh + +TASKS="mmlu,gsm8k,hellaswag,truthfulqa" + +while read model; do + echo "Evaluating $model" + + # Extract model name for output file + model_name=$(echo $model | sed 's/\//-/g') + + lm_eval --model hf \ + --model_args pretrained=$model,dtype=bfloat16 \ + --tasks $TASKS \ + --num_fewshot 5 \ + --batch_size auto \ + --output_path results/$model_name.json + +done < models.txt +``` + +**Step 3: Generate comparison table** + +```python +import json +import pandas as pd + +models = [ + "meta-llama-Llama-2-7b-hf", + "meta-llama-Llama-2-13b-hf", + "mistralai-Mistral-7B-v0.1", + "microsoft-phi-2" +] + +tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"] + +results = [] +for model in models: + with open(f"results/{model}.json") as f: + data = json.load(f) + row = {"Model": model.replace("-", "/")} + for task in tasks: + # Get primary metric for each task + metrics = data["results"][task] + if "acc" in metrics: + row[task.upper()] = f"{metrics['acc']:.3f}" + elif "exact_match" in metrics: + row[task.upper()] = f"{metrics['exact_match']:.3f}" + results.append(row) + +df = pd.DataFrame(results) +print(df.to_markdown(index=False)) +``` + +Output: +``` +| Model | MMLU | GSM8K | HELLASWAG | TRUTHFULQA | +|------------------------|-------|-------|-----------|------------| +| meta-llama/Llama-2-7b | 0.459 | 0.142 | 0.765 | 0.391 | +| meta-llama/Llama-2-13b | 0.549 | 0.287 | 0.801 | 0.430 | +| mistralai/Mistral-7B | 0.626 | 0.395 | 0.812 | 0.428 | +| microsoft/phi-2 | 0.560 | 0.613 | 0.682 | 0.447 | +``` + +### Workflow 4: Evaluate with vLLM (faster inference) + +Use vLLM backend for 5-10x faster evaluation. + +``` +vLLM Evaluation: +- [ ] Step 1: Install vLLM +- [ ] Step 2: Configure vLLM backend +- [ ] Step 3: Run evaluation +``` + +**Step 1: Install vLLM** + +```bash +pip install vllm +``` + +**Step 2: Configure vLLM backend** + +```bash +lm_eval --model vllm \ + --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \ + --tasks mmlu \ + --batch_size auto +``` + +**Step 3: Run evaluation** + +vLLM is 5-10× faster than standard HuggingFace: + +```bash +# Standard HF: ~2 hours for MMLU on 7B model +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu \ + --batch_size 8 + +# vLLM: ~15-20 minutes for MMLU on 7B model +lm_eval --model vllm \ + --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \ + --tasks mmlu \ + --batch_size auto +``` + +## When to use vs alternatives + +**Use lm-evaluation-harness when:** +- Benchmarking models for academic papers +- Comparing model quality across standard tasks +- Tracking training progress +- Reporting standardized metrics (everyone uses same prompts) +- Need reproducible evaluation + +**Use alternatives instead:** +- **HELM** (Stanford): Broader evaluation (fairness, efficiency, calibration) +- **AlpacaEval**: Instruction-following evaluation with LLM judges +- **MT-Bench**: Conversational multi-turn evaluation +- **Custom scripts**: Domain-specific evaluation + +## Common issues + +**Issue: Evaluation too slow** + +Use vLLM backend: +```bash +lm_eval --model vllm \ + --model_args pretrained=model-name,tensor_parallel_size=2 +``` + +Or reduce fewshot examples: +```bash +--num_fewshot 0 # Instead of 5 +``` + +Or evaluate subset of MMLU: +```bash +--tasks mmlu_stem # Only STEM subjects +``` + +**Issue: Out of memory** + +Reduce batch size: +```bash +--batch_size 1 # Or --batch_size auto +``` + +Use quantization: +```bash +--model_args pretrained=model-name,load_in_8bit=True +``` + +Enable CPU offloading: +```bash +--model_args pretrained=model-name,device_map=auto,offload_folder=offload +``` + +**Issue: Different results than reported** + +Check fewshot count: +```bash +--num_fewshot 5 # Most papers use 5-shot +``` + +Check exact task name: +```bash +--tasks mmlu # Not mmlu_direct or mmlu_fewshot +``` + +Verify model and tokenizer match: +```bash +--model_args pretrained=model-name,tokenizer=same-model-name +``` + +**Issue: HumanEval not executing code** + +Code-executing tasks (HumanEval, MBPP, etc.) are gated behind an explicit +confirmation flag — you must pass `--confirm_run_unsafe_code` to run them: + +```bash +lm_eval --model hf \ + --model_args pretrained=model-name \ + --tasks humaneval \ + --confirm_run_unsafe_code # Required to run tasks that execute generated code +``` + +Without this flag lm-eval refuses to run the task rather than silently skipping +code execution. + +## Advanced topics + +**Benchmark descriptions**: See [references/benchmark-guide.md](references/benchmark-guide.md) for detailed description of all 60+ tasks, what they measure, and interpretation. + +**Custom tasks**: See [references/custom-tasks.md](references/custom-tasks.md) for creating domain-specific evaluation tasks. + +**API evaluation**: See [references/api-evaluation.md](references/api-evaluation.md) for evaluating OpenAI, Anthropic, and other API models. + +**Multi-GPU strategies**: See [references/distributed-eval.md](references/distributed-eval.md) for data parallel and tensor parallel evaluation. + +## Hardware requirements + +- **GPU**: NVIDIA (CUDA 11.8+), works on CPU (very slow) +- **VRAM**: + - 7B model: 16GB (bf16) or 8GB (8-bit) + - 13B model: 28GB (bf16) or 14GB (8-bit) + - 70B model: Requires multi-GPU or quantization +- **Time** (7B model, single A100): + - HellaSwag: 10 minutes + - GSM8K: 5 minutes + - MMLU (full): 2 hours + - HumanEval: 20 minutes + +## Resources + +- GitHub: https://github.com/EleutherAI/lm-evaluation-harness +- Docs: https://github.com/EleutherAI/lm-evaluation-harness/tree/main/docs +- Task library: 60+ tasks including MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, WinoGrande, etc. +- Leaderboard: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard (uses this harness) + + + diff --git a/evaluating-llms-harness/references/api-evaluation.md b/evaluating-llms-harness/references/api-evaluation.md new file mode 100644 index 0000000..db77f61 --- /dev/null +++ b/evaluating-llms-harness/references/api-evaluation.md @@ -0,0 +1,490 @@ +# API Evaluation + +Guide to evaluating OpenAI, Anthropic, and other API-based language models. + +## Overview + +The lm-evaluation-harness supports evaluating API-based models through a unified `TemplateAPI` interface. This allows benchmarking of: +- OpenAI models (GPT-4, GPT-3.5, etc.) +- Anthropic models (Claude 3, Claude 2, etc.) +- Local OpenAI-compatible APIs +- Custom API endpoints + +**Why evaluate API models**: +- Benchmark closed-source models +- Compare API models to open models +- Validate API performance +- Track model updates over time + +## Supported API Models + +| Provider | Model Type | Request Types | Logprobs | +|----------|------------|---------------|----------| +| OpenAI (completions) | `openai-completions` | All | ✅ Yes | +| OpenAI (chat) | `openai-chat-completions` | `generate_until` only | ❌ No | +| Anthropic (completions) | `anthropic-completions` | All | ❌ No | +| Anthropic (chat) | `anthropic-chat` | `generate_until` only | ❌ No | +| Local (OpenAI-compatible) | `local-completions` | Depends on server | Varies | + +**Note**: Models without logprobs can only be evaluated on generation tasks, not perplexity or loglikelihood tasks. + +## OpenAI Models + +### Setup + +```bash +export OPENAI_API_KEY=sk-... +``` + +### Completion Models (Legacy) + +**Available models**: `davinci-002`, `babbage-002` + +```bash +lm_eval --model openai-completions \ + --model_args model=davinci-002 \ + --tasks lambada_openai,hellaswag \ + --batch_size auto +``` + +**Supports**: +- `generate_until`: ✅ +- `loglikelihood`: ✅ +- `loglikelihood_rolling`: ✅ + +### Chat Models + +**Available models**: `gpt-4`, `gpt-4-turbo`, `gpt-3.5-turbo` + +```bash +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu,gsm8k,humaneval \ + --num_fewshot 5 \ + --batch_size auto +``` + +**Supports**: +- `generate_until`: ✅ +- `loglikelihood`: ❌ (no logprobs) +- `loglikelihood_rolling`: ❌ + +**Important**: Chat models don't provide logprobs, so they can only be used with generation tasks (MMLU, GSM8K, HumanEval), not perplexity tasks. + +### Configuration Options + +```bash +lm_eval --model openai-chat-completions \ + --model_args \ + model=gpt-4-turbo,\ + base_url=https://api.openai.com/v1,\ + num_concurrent=5,\ + max_retries=3,\ + timeout=60,\ + batch_size=auto +``` + +**Parameters**: +- `model`: Model identifier (required) +- `base_url`: API endpoint (default: OpenAI) +- `num_concurrent`: Concurrent requests (default: 5) +- `max_retries`: Retry failed requests (default: 3) +- `timeout`: Request timeout in seconds (default: 60) +- `tokenizer`: Tokenizer to use (default: matches model) +- `tokenizer_backend`: `"tiktoken"` or `"huggingface"` + +### Cost Management + +OpenAI charges per token. Estimate costs before running: + +```python +# Rough estimate +num_samples = 1000 +avg_tokens_per_sample = 500 # input + output +cost_per_1k_tokens = 0.01 # GPT-3.5 Turbo + +total_cost = (num_samples * avg_tokens_per_sample / 1000) * cost_per_1k_tokens +print(f"Estimated cost: ${total_cost:.2f}") +``` + +**Cost-saving tips**: +- Use `--limit N` for testing +- Start with `gpt-3.5-turbo` before `gpt-4` +- Set `max_gen_toks` to minimum needed +- Use `num_fewshot=0` for zero-shot when possible + +## Anthropic Models + +### Setup + +```bash +export ANTHROPIC_API_KEY=sk-ant-... +``` + +### Completion Models (Legacy) + +```bash +lm_eval --model anthropic-completions \ + --model_args model=claude-2.1 \ + --tasks lambada_openai,hellaswag \ + --batch_size auto +``` + +### Chat Models (Recommended) + +**Available models**: `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`, `claude-3-sonnet-20240229`, `claude-3-haiku-20240307` + +```bash +lm_eval --model anthropic-chat \ + --model_args model=claude-3-5-sonnet-20241022 \ + --tasks mmlu,gsm8k,humaneval \ + --num_fewshot 5 \ + --batch_size auto +``` + +**Aliases**: `anthropic-chat-completions` (same as `anthropic-chat`) + +### Configuration Options + +```bash +lm_eval --model anthropic-chat \ + --model_args \ + model=claude-3-5-sonnet-20241022,\ + base_url=https://api.anthropic.com,\ + num_concurrent=5,\ + max_retries=3,\ + timeout=60 +``` + +### Cost Management + +Anthropic pricing (as of 2024): +- Claude 3.5 Sonnet: $3.00 / 1M input, $15.00 / 1M output +- Claude 3 Opus: $15.00 / 1M input, $75.00 / 1M output +- Claude 3 Haiku: $0.25 / 1M input, $1.25 / 1M output + +**Budget-friendly strategy**: +```bash +# Test on small sample first +lm_eval --model anthropic-chat \ + --model_args model=claude-3-haiku-20240307 \ + --tasks mmlu \ + --limit 100 + +# Then run full eval on best model +lm_eval --model anthropic-chat \ + --model_args model=claude-3-5-sonnet-20241022 \ + --tasks mmlu \ + --num_fewshot 5 +``` + +## Local OpenAI-Compatible APIs + +Many local inference servers expose OpenAI-compatible APIs (vLLM, Text Generation Inference, llama.cpp, Ollama). + +### vLLM Local Server + +**Start server**: +```bash +vllm serve meta-llama/Llama-2-7b-hf \ + --host 0.0.0.0 \ + --port 8000 +``` + +**Evaluate**: +```bash +lm_eval --model local-completions \ + --model_args \ + model=meta-llama/Llama-2-7b-hf,\ + base_url=http://localhost:8000/v1,\ + num_concurrent=1 \ + --tasks mmlu,gsm8k \ + --batch_size auto +``` + +### Text Generation Inference (TGI) + +**Start server**: +```bash +docker run --gpus all --shm-size 1g -p 8080:80 \ + ghcr.io/huggingface/text-generation-inference:latest \ + --model-id meta-llama/Llama-2-7b-hf +``` + +**Evaluate**: +```bash +lm_eval --model local-completions \ + --model_args \ + model=meta-llama/Llama-2-7b-hf,\ + base_url=http://localhost:8080/v1 \ + --tasks hellaswag,arc_challenge +``` + +### Ollama + +**Start server**: +```bash +ollama serve +ollama pull llama2:7b +``` + +**Evaluate**: +```bash +lm_eval --model local-completions \ + --model_args \ + model=llama2:7b,\ + base_url=http://localhost:11434/v1 \ + --tasks mmlu +``` + +### llama.cpp Server + +**Start server**: +```bash +./server -m models/llama-2-7b.gguf --host 0.0.0.0 --port 8080 +``` + +**Evaluate**: +```bash +lm_eval --model local-completions \ + --model_args \ + model=llama2,\ + base_url=http://localhost:8080/v1 \ + --tasks gsm8k +``` + +## Custom API Implementation + +For custom API endpoints, subclass `TemplateAPI`: + +### Create `my_api.py` + +```python +from lm_eval.models.api_models import TemplateAPI +import requests + +class MyCustomAPI(TemplateAPI): + """Custom API model.""" + + def __init__(self, base_url, api_key, **kwargs): + super().__init__(base_url=base_url, **kwargs) + self.api_key = api_key + + def _create_payload(self, messages, gen_kwargs): + """Create API request payload.""" + return { + "messages": messages, + "api_key": self.api_key, + **gen_kwargs + } + + def parse_generations(self, response): + """Parse generation response.""" + return response.json()["choices"][0]["text"] + + def parse_logprobs(self, response): + """Parse logprobs (if available).""" + # Return None if API doesn't provide logprobs + logprobs = response.json().get("logprobs") + if logprobs: + return logprobs["token_logprobs"] + return None +``` + +### Register and Use + +```python +from lm_eval import evaluator +from my_api import MyCustomAPI + +model = MyCustomAPI( + base_url="https://api.example.com/v1", + api_key="your-key" +) + +results = evaluator.simple_evaluate( + model=model, + tasks=["mmlu", "gsm8k"], + num_fewshot=5, + batch_size="auto" +) +``` + +## Comparing API and Open Models + +### Side-by-Side Evaluation + +```bash +# Evaluate OpenAI GPT-4 +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu,gsm8k,hellaswag \ + --num_fewshot 5 \ + --output_path results/gpt4.json + +# Evaluate open Llama 2 70B +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-70b-hf,dtype=bfloat16 \ + --tasks mmlu,gsm8k,hellaswag \ + --num_fewshot 5 \ + --output_path results/llama2-70b.json + +# Compare results +python scripts/compare_results.py \ + results/gpt4.json \ + results/llama2-70b.json +``` + +### Typical Comparisons + +| Model | MMLU | GSM8K | HumanEval | Cost | +|-------|------|-------|-----------|------| +| GPT-4 Turbo | 86.4% | 92.0% | 67.0% | $$$$ | +| Claude 3 Opus | 86.8% | 95.0% | 84.9% | $$$$ | +| GPT-3.5 Turbo | 70.0% | 57.1% | 48.1% | $$ | +| Llama 2 70B | 68.9% | 56.8% | 29.9% | Free (self-host) | +| Mixtral 8x7B | 70.6% | 58.4% | 40.2% | Free (self-host) | + +## Best Practices + +### Rate Limiting + +Respect API rate limits: +```bash +lm_eval --model openai-chat-completions \ + --model_args \ + model=gpt-4-turbo,\ + num_concurrent=3,\ # Lower concurrency + timeout=120 \ # Longer timeout + --tasks mmlu +``` + +### Reproducibility + +Set temperature to 0 for deterministic results: +```bash +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu \ + --gen_kwargs temperature=0.0 +``` + +Or use `seed` for sampling: +```bash +lm_eval --model anthropic-chat \ + --model_args model=claude-3-5-sonnet-20241022 \ + --tasks gsm8k \ + --gen_kwargs temperature=0.7,seed=42 +``` + +### Caching + +API models automatically cache responses to avoid redundant calls: +```bash +# First run: makes API calls +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu \ + --limit 100 + +# Second run: uses cache (instant, free) +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu \ + --limit 100 +``` + +Cache location: `~/.cache/lm_eval/` + +### Error Handling + +APIs can fail. Use retries: +```bash +lm_eval --model openai-chat-completions \ + --model_args \ + model=gpt-4-turbo,\ + max_retries=5,\ + timeout=120 \ + --tasks mmlu +``` + +## Troubleshooting + +### "Authentication failed" + +Check API key: +```bash +echo $OPENAI_API_KEY # Should print sk-... +echo $ANTHROPIC_API_KEY # Should print sk-ant-... +``` + +### "Rate limit exceeded" + +Reduce concurrency: +```bash +--model_args num_concurrent=1 +``` + +Or add delays between requests. + +### "Timeout error" + +Increase timeout: +```bash +--model_args timeout=180 +``` + +### "Model not found" + +For local APIs, verify server is running: +```bash +curl http://localhost:8000/v1/models +``` + +### Cost Runaway + +Use `--limit` for testing: +```bash +lm_eval --model openai-chat-completions \ + --model_args model=gpt-4-turbo \ + --tasks mmlu \ + --limit 50 # Only 50 samples +``` + +## Advanced Features + +### Custom Headers + +```bash +lm_eval --model local-completions \ + --model_args \ + base_url=http://api.example.com/v1,\ + header="Authorization: Bearer token,X-Custom: value" +``` + +### Disable SSL Verification (Development Only) + +```bash +lm_eval --model local-completions \ + --model_args \ + base_url=https://localhost:8000/v1,\ + verify_certificate=false +``` + +### Custom Tokenizer + +```bash +lm_eval --model openai-chat-completions \ + --model_args \ + model=gpt-4-turbo,\ + tokenizer=gpt2,\ + tokenizer_backend=huggingface +``` + +## References + +- OpenAI API: https://platform.openai.com/docs/api-reference +- Anthropic API: https://docs.anthropic.com/claude/reference +- TemplateAPI: `lm_eval/models/api_models.py` +- OpenAI models: `lm_eval/models/openai_completions.py` +- Anthropic models: `lm_eval/models/anthropic_llms.py` diff --git a/evaluating-llms-harness/references/benchmark-guide.md b/evaluating-llms-harness/references/benchmark-guide.md new file mode 100644 index 0000000..e3031ec --- /dev/null +++ b/evaluating-llms-harness/references/benchmark-guide.md @@ -0,0 +1,488 @@ +# Benchmark Guide + +Complete guide to all 60+ evaluation tasks in lm-evaluation-harness, what they measure, and how to interpret results. + +## Overview + +The lm-evaluation-harness includes 60+ benchmarks spanning: +- Language understanding (MMLU, GLUE) +- Mathematical reasoning (GSM8K, MATH) +- Code generation (HumanEval, MBPP) +- Instruction following (IFEval, AlpacaEval) +- Long-context understanding (LongBench) +- Multilingual capabilities (AfroBench, NorEval) +- Reasoning (BBH, ARC) +- Truthfulness (TruthfulQA) + +**List all tasks**: +```bash +lm_eval --tasks list +``` + +## Major Benchmarks + +### MMLU (Massive Multitask Language Understanding) + +**What it measures**: Broad knowledge across 57 subjects (STEM, humanities, social sciences, law). + +**Task variants**: +- `mmlu`: Original 57-subject benchmark +- `mmlu_pro`: More challenging version with reasoning-focused questions +- `mmlu_prox`: Multilingual extension + +**Format**: Multiple choice (4 options) + +**Example**: +``` +Question: What is the capital of France? +A. Berlin +B. Paris +C. London +D. Madrid +Answer: B +``` + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu \ + --num_fewshot 5 +``` + +**Interpretation**: +- Random: 25% (chance) +- GPT-3 (175B): 43.9% +- GPT-4: 86.4% +- Human expert: ~90% + +**Good for**: Assessing general knowledge and domain expertise. + +### GSM8K (Grade School Math 8K) + +**What it measures**: Mathematical reasoning on grade-school level word problems. + +**Task variants**: +- `gsm8k`: Base task +- `gsm8k_cot`: With chain-of-thought prompting +- `gsm_plus`: Adversarial variant with perturbations + +**Format**: Free-form generation, extract numerical answer + +**Example**: +``` +Question: A baker made 200 cookies. He sold 3/5 of them in the morning and 1/4 of the remaining in the afternoon. How many cookies does he have left? +Answer: 60 +``` + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks gsm8k \ + --num_fewshot 5 +``` + +**Interpretation**: +- Random: ~0% +- GPT-3 (175B): 17.0% +- GPT-4: 92.0% +- Llama 2 70B: 56.8% + +**Good for**: Testing multi-step reasoning and arithmetic. + +### HumanEval + +**What it measures**: Python code generation from docstrings (functional correctness). + +**Task variants**: +- `humaneval`: Standard benchmark +- `humaneval_instruct`: For instruction-tuned models + +**Format**: Code generation, execution-based evaluation + +**Example**: +```python +def has_close_elements(numbers: List[float], threshold: float) -> bool: + """ Check if in given list of numbers, are any two numbers closer to each other than + given threshold. + >>> has_close_elements([1.0, 2.0, 3.0], 0.5) + False + >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) + True + """ +``` + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=codellama/CodeLlama-7b-hf \ + --tasks humaneval \ + --batch_size 1 +``` + +**Interpretation**: +- Random: 0% +- GPT-3 (175B): 0% +- Codex: 28.8% +- GPT-4: 67.0% +- Code Llama 34B: 53.7% + +**Good for**: Evaluating code generation capabilities. + +### BBH (BIG-Bench Hard) + +**What it measures**: 23 challenging reasoning tasks where models previously failed to beat humans. + +**Categories**: +- Logical reasoning +- Math word problems +- Social understanding +- Algorithmic reasoning + +**Format**: Multiple choice and free-form + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks bbh \ + --num_fewshot 3 +``` + +**Interpretation**: +- Random: ~25% +- GPT-3 (175B): 33.9% +- PaLM 540B: 58.3% +- GPT-4: 86.7% + +**Good for**: Testing advanced reasoning capabilities. + +### IFEval (Instruction-Following Evaluation) + +**What it measures**: Ability to follow specific, verifiable instructions. + +**Instruction types**: +- Format constraints (e.g., "answer in 3 sentences") +- Length constraints (e.g., "use at least 100 words") +- Content constraints (e.g., "include the word 'banana'") +- Structural constraints (e.g., "use bullet points") + +**Format**: Free-form generation with rule-based verification + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-chat-hf \ + --tasks ifeval \ + --batch_size auto +``` + +**Interpretation**: +- Measures: Instruction adherence (not quality) +- GPT-4: 86% instruction following +- Claude 2: 84% + +**Good for**: Evaluating chat/instruct models. + +### GLUE (General Language Understanding Evaluation) + +**What it measures**: Natural language understanding across 9 tasks. + +**Tasks**: +- `cola`: Grammatical acceptability +- `sst2`: Sentiment analysis +- `mrpc`: Paraphrase detection +- `qqp`: Question pairs +- `stsb`: Semantic similarity +- `mnli`: Natural language inference +- `qnli`: Question answering NLI +- `rte`: Recognizing textual entailment +- `wnli`: Winograd schemas + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=bert-base-uncased \ + --tasks glue \ + --num_fewshot 0 +``` + +**Interpretation**: +- BERT Base: 78.3 (GLUE score) +- RoBERTa Large: 88.5 +- Human baseline: 87.1 + +**Good for**: Encoder-only models, fine-tuning baselines. + +### LongBench + +**What it measures**: Long-context understanding (4K-32K tokens). + +**21 tasks covering**: +- Single-document QA +- Multi-document QA +- Summarization +- Few-shot learning +- Code completion +- Synthetic tasks + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks longbench \ + --batch_size 1 +``` + +**Interpretation**: +- Tests context utilization +- Many models struggle beyond 4K tokens +- GPT-4 Turbo: 54.3% + +**Good for**: Evaluating long-context models. + +## Additional Benchmarks + +### TruthfulQA + +**What it measures**: Model's propensity to be truthful vs. generate plausible-sounding falsehoods. + +**Format**: Multiple choice with 4-5 options + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks truthfulqa_mc2 \ + --batch_size auto +``` + +**Interpretation**: +- Larger models often score worse (more convincing lies) +- GPT-3: 58.8% +- GPT-4: 59.0% +- Human: ~94% + +### ARC (AI2 Reasoning Challenge) + +**What it measures**: Grade-school science questions. + +**Variants**: +- `arc_easy`: Easier questions +- `arc_challenge`: Harder questions requiring reasoning + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks arc_challenge \ + --num_fewshot 25 +``` + +**Interpretation**: +- ARC-Easy: Most models >80% +- ARC-Challenge random: 25% +- GPT-4: 96.3% + +### HellaSwag + +**What it measures**: Commonsense reasoning about everyday situations. + +**Format**: Choose most plausible continuation + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks hellaswag \ + --num_fewshot 10 +``` + +**Interpretation**: +- Random: 25% +- GPT-3: 78.9% +- Llama 2 70B: 85.3% + +### WinoGrande + +**What it measures**: Commonsense reasoning via pronoun resolution. + +**Example**: +``` +The trophy doesn't fit in the brown suitcase because _ is too large. +A. the trophy +B. the suitcase +``` + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks winogrande \ + --num_fewshot 5 +``` + +### PIQA + +**What it measures**: Physical commonsense reasoning. + +**Example**: "To clean a keyboard, use compressed air or..." + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks piqa +``` + +## Multilingual Benchmarks + +### AfroBench + +**What it measures**: Performance across 64 African languages. + +**15 tasks**: NLU, text generation, knowledge, QA, math reasoning + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks afrobench +``` + +### NorEval + +**What it measures**: Norwegian language understanding (9 task categories). + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=NbAiLab/nb-gpt-j-6B \ + --tasks noreval +``` + +## Domain-Specific Benchmarks + +### MATH + +**What it measures**: High-school competition math problems. + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks math \ + --num_fewshot 4 +``` + +**Interpretation**: +- Very challenging +- GPT-4: 42.5% +- Minerva 540B: 33.6% + +### MBPP (Mostly Basic Python Problems) + +**What it measures**: Python programming from natural language descriptions. + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=codellama/CodeLlama-7b-hf \ + --tasks mbpp \ + --batch_size 1 +``` + +### DROP + +**What it measures**: Reading comprehension requiring discrete reasoning. + +**Command**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks drop +``` + +## Benchmark Selection Guide + +### For General Purpose Models + +Run this suite: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu,gsm8k,hellaswag,arc_challenge,truthfulqa_mc2 \ + --num_fewshot 5 +``` + +### For Code Models + +```bash +lm_eval --model hf \ + --model_args pretrained=codellama/CodeLlama-7b-hf \ + --tasks humaneval,mbpp \ + --batch_size 1 +``` + +### For Chat/Instruct Models + +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-chat-hf \ + --tasks ifeval,mmlu,gsm8k_cot \ + --batch_size auto +``` + +### For Long Context Models + +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-3.1-8B \ + --tasks longbench \ + --batch_size 1 +``` + +## Interpreting Results + +### Understanding Metrics + +**Accuracy**: Percentage of correct answers (most common) + +**Exact Match (EM)**: Requires exact string match (strict) + +**F1 Score**: Balances precision and recall + +**BLEU/ROUGE**: Text generation similarity + +**Pass@k**: Percentage passing when generating k samples + +### Typical Score Ranges + +| Model Size | MMLU | GSM8K | HumanEval | HellaSwag | +|------------|------|-------|-----------|-----------| +| 7B | 40-50% | 10-20% | 5-15% | 70-80% | +| 13B | 45-55% | 20-35% | 15-25% | 75-82% | +| 70B | 60-70% | 50-65% | 35-50% | 82-87% | +| GPT-4 | 86% | 92% | 67% | 95% | + +### Red Flags + +- **All tasks at random chance**: Model not trained properly +- **Exact 0% on generation tasks**: Likely format/parsing issue +- **Huge variance across runs**: Check seed/sampling settings +- **Better than GPT-4 on everything**: Likely contamination + +## Best Practices + +1. **Always report few-shot setting**: 0-shot, 5-shot, etc. +2. **Run multiple seeds**: Report mean ± std +3. **Check for data contamination**: Search training data for benchmark examples +4. **Compare to published baselines**: Validate your setup +5. **Report all hyperparameters**: Model, batch size, max tokens, temperature + +## References + +- Task list: `lm_eval --tasks list` +- Task README: `lm_eval/tasks/README.md` +- Papers: See individual benchmark papers diff --git a/evaluating-llms-harness/references/custom-tasks.md b/evaluating-llms-harness/references/custom-tasks.md new file mode 100644 index 0000000..c5c1e89 --- /dev/null +++ b/evaluating-llms-harness/references/custom-tasks.md @@ -0,0 +1,602 @@ +# Custom Tasks + +Complete guide to creating domain-specific evaluation tasks in lm-evaluation-harness. + +## Overview + +Custom tasks allow you to evaluate models on your own datasets and metrics. Tasks are defined using YAML configuration files with optional Python utilities for complex logic. + +**Why create custom tasks**: +- Evaluate on proprietary/domain-specific data +- Test specific capabilities not covered by existing benchmarks +- Create evaluation pipelines for internal models +- Reproduce research experiments + +## Quick Start + +### Minimal Custom Task + +Create `my_tasks/simple_qa.yaml`: + +```yaml +task: simple_qa +dataset_path: data/simple_qa.jsonl +output_type: generate_until +doc_to_text: "Question: {{question}}\nAnswer:" +doc_to_target: "{{answer}}" +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true +``` + +**Run it**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks simple_qa \ + --include_path my_tasks/ +``` + +## Task Configuration Reference + +### Essential Fields + +```yaml +# Task identification +task: my_custom_task # Unique task name (required) +task_alias: "My Task" # Display name +tag: # Tags for grouping + - custom + - domain_specific + +# Dataset configuration +dataset_path: data/my_data.jsonl # HuggingFace dataset or local path +dataset_name: default # Subset name (if applicable) +training_split: train +validation_split: validation +test_split: test + +# Evaluation configuration +output_type: generate_until # or loglikelihood, multiple_choice +num_fewshot: 5 # Number of few-shot examples +batch_size: auto # Batch size + +# Prompt templates (Jinja2) +doc_to_text: "Question: {{question}}" +doc_to_target: "{{answer}}" + +# Metrics +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + +# Metadata +metadata: + version: 1.0 +``` + +### Output Types + +**`generate_until`**: Free-form generation +```yaml +output_type: generate_until +generation_kwargs: + max_gen_toks: 256 + until: + - "\n" + - "." + temperature: 0.0 +``` + +**`loglikelihood`**: Compute log probability of targets +```yaml +output_type: loglikelihood +# Used for perplexity, classification +``` + +**`multiple_choice`**: Choose from options +```yaml +output_type: multiple_choice +doc_to_choice: "{{choices}}" # List of choices +``` + +## Data Formats + +### Local JSONL File + +`data/my_data.jsonl`: +```json +{"question": "What is 2+2?", "answer": "4"} +{"question": "Capital of France?", "answer": "Paris"} +``` + +**Task config**: +```yaml +dataset_path: data/my_data.jsonl +dataset_kwargs: + data_files: + test: data/my_data.jsonl +``` + +### HuggingFace Dataset + +```yaml +dataset_path: squad +dataset_name: plain_text +test_split: validation +``` + +### CSV File + +`data/my_data.csv`: +```csv +question,answer,category +What is 2+2?,4,math +Capital of France?,Paris,geography +``` + +**Task config**: +```yaml +dataset_path: data/my_data.csv +dataset_kwargs: + data_files: + test: data/my_data.csv +``` + +## Prompt Engineering + +### Simple Template + +```yaml +doc_to_text: "Question: {{question}}\nAnswer:" +doc_to_target: "{{answer}}" +``` + +### Conditional Logic + +```yaml +doc_to_text: | + {% if context %} + Context: {{context}} + {% endif %} + Question: {{question}} + Answer: +``` + +### Multiple Choice + +```yaml +doc_to_text: | + Question: {{question}} + A. {{choices[0]}} + B. {{choices[1]}} + C. {{choices[2]}} + D. {{choices[3]}} + Answer: + +doc_to_target: "{{ 'ABCD'[answer_idx] }}" +doc_to_choice: ["A", "B", "C", "D"] +``` + +### Few-Shot Formatting + +```yaml +fewshot_delimiter: "\n\n" # Between examples +target_delimiter: " " # Between question and answer +doc_to_text: "Q: {{question}}" +doc_to_target: "A: {{answer}}" +``` + +## Custom Python Functions + +For complex logic, use Python functions in `utils.py`. + +### Create `my_tasks/utils.py` + +```python +def process_docs(dataset): + """Preprocess documents.""" + def _process(doc): + # Custom preprocessing + doc["question"] = doc["question"].strip().lower() + return doc + + return dataset.map(_process) + +def doc_to_text(doc): + """Custom prompt formatting.""" + context = doc.get("context", "") + question = doc["question"] + + if context: + return f"Context: {context}\nQuestion: {question}\nAnswer:" + return f"Question: {question}\nAnswer:" + +def doc_to_target(doc): + """Custom target extraction.""" + return doc["answer"].strip().lower() + +def aggregate_scores(items): + """Custom metric aggregation.""" + correct = sum(1 for item in items if item == 1.0) + total = len(items) + return correct / total if total > 0 else 0.0 +``` + +### Use in Task Config + +```yaml +task: my_custom_task +dataset_path: data/my_data.jsonl + +# Use Python functions +process_docs: !function utils.process_docs +doc_to_text: !function utils.doc_to_text +doc_to_target: !function utils.doc_to_target + +metric_list: + - metric: exact_match + aggregation: !function utils.aggregate_scores + higher_is_better: true +``` + +## Real-World Examples + +### Example 1: Domain QA Task + +**Goal**: Evaluate medical question answering. + +`medical_qa/medical_qa.yaml`: +```yaml +task: medical_qa +dataset_path: data/medical_qa.jsonl +output_type: generate_until +num_fewshot: 3 + +doc_to_text: | + Medical Question: {{question}} + Context: {{context}} + Answer (be concise): + +doc_to_target: "{{answer}}" + +generation_kwargs: + max_gen_toks: 100 + until: + - "\n\n" + temperature: 0.0 + +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + - metric: !function utils.medical_f1 + aggregation: mean + higher_is_better: true + +filter_list: + - name: lowercase + filter: + - function: lowercase + - function: remove_whitespace + +metadata: + version: 1.0 + domain: medical +``` + +`medical_qa/utils.py`: +```python +from sklearn.metrics import f1_score +import re + +def medical_f1(predictions, references): + """Custom F1 for medical terms.""" + pred_terms = set(extract_medical_terms(predictions[0])) + ref_terms = set(extract_medical_terms(references[0])) + + if not pred_terms and not ref_terms: + return 1.0 + if not pred_terms or not ref_terms: + return 0.0 + + tp = len(pred_terms & ref_terms) + fp = len(pred_terms - ref_terms) + fn = len(ref_terms - pred_terms) + + precision = tp / (tp + fp) if (tp + fp) > 0 else 0 + recall = tp / (tp + fn) if (tp + fn) > 0 else 0 + + return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0 + +def extract_medical_terms(text): + """Extract medical terminology.""" + # Custom logic + return re.findall(r'\b[A-Z][a-z]+(?:[A-Z][a-z]+)*\b', text) +``` + +### Example 2: Code Evaluation + +`code_eval/python_challenges.yaml`: +```yaml +task: python_challenges +dataset_path: data/python_problems.jsonl +output_type: generate_until +num_fewshot: 0 + +doc_to_text: | + Write a Python function to solve: + {{problem_statement}} + + Function signature: + {{function_signature}} + +doc_to_target: "{{canonical_solution}}" + +generation_kwargs: + max_gen_toks: 512 + until: + - "\n\nclass" + - "\n\ndef" + temperature: 0.2 + +metric_list: + - metric: !function utils.execute_code + aggregation: mean + higher_is_better: true + +process_results: !function utils.process_code_results + +metadata: + version: 1.0 +``` + +`code_eval/utils.py`: +```python +import subprocess +import json + +def execute_code(predictions, references): + """Execute generated code against test cases.""" + generated_code = predictions[0] + test_cases = json.loads(references[0]) + + try: + # Execute code with test cases + for test_input, expected_output in test_cases: + result = execute_with_timeout(generated_code, test_input, timeout=5) + if result != expected_output: + return 0.0 + return 1.0 + except Exception: + return 0.0 + +def execute_with_timeout(code, input_data, timeout=5): + """Safely execute code with timeout.""" + # Implementation with subprocess and timeout + pass + +def process_code_results(doc, results): + """Process code execution results.""" + return { + "passed": results[0] == 1.0, + "generated_code": results[1] + } +``` + +### Example 3: Instruction Following + +`instruction_eval/instruction_eval.yaml`: +```yaml +task: instruction_following +dataset_path: data/instructions.jsonl +output_type: generate_until +num_fewshot: 0 + +doc_to_text: | + Instruction: {{instruction}} + {% if constraints %} + Constraints: {{constraints}} + {% endif %} + Response: + +doc_to_target: "{{expected_response}}" + +generation_kwargs: + max_gen_toks: 256 + temperature: 0.7 + +metric_list: + - metric: !function utils.check_constraints + aggregation: mean + higher_is_better: true + - metric: !function utils.semantic_similarity + aggregation: mean + higher_is_better: true + +process_docs: !function utils.add_constraint_checkers +``` + +`instruction_eval/utils.py`: +```python +from sentence_transformers import SentenceTransformer, util + +model = SentenceTransformer('all-MiniLM-L6-v2') + +def check_constraints(predictions, references): + """Check if response satisfies constraints.""" + response = predictions[0] + constraints = json.loads(references[0]) + + satisfied = 0 + total = len(constraints) + + for constraint in constraints: + if verify_constraint(response, constraint): + satisfied += 1 + + return satisfied / total if total > 0 else 1.0 + +def verify_constraint(response, constraint): + """Verify single constraint.""" + if constraint["type"] == "length": + return len(response.split()) >= constraint["min_words"] + elif constraint["type"] == "contains": + return constraint["keyword"] in response.lower() + # Add more constraint types + return True + +def semantic_similarity(predictions, references): + """Compute semantic similarity.""" + pred_embedding = model.encode(predictions[0]) + ref_embedding = model.encode(references[0]) + return float(util.cos_sim(pred_embedding, ref_embedding)) + +def add_constraint_checkers(dataset): + """Parse constraints into verifiable format.""" + def _parse(doc): + # Parse constraint string into structured format + doc["parsed_constraints"] = parse_constraints(doc.get("constraints", "")) + return doc + return dataset.map(_parse) +``` + +## Advanced Features + +### Output Filtering + +```yaml +filter_list: + - name: extract_answer + filter: + - function: regex + regex_pattern: "Answer: (.*)" + group: 1 + - function: lowercase + - function: strip_whitespace +``` + +### Multiple Metrics + +```yaml +metric_list: + - metric: exact_match + aggregation: mean + higher_is_better: true + - metric: f1 + aggregation: mean + higher_is_better: true + - metric: bleu + aggregation: mean + higher_is_better: true +``` + +### Task Groups + +Create `my_tasks/_default.yaml`: +```yaml +group: my_eval_suite +task: + - simple_qa + - medical_qa + - python_challenges +``` + +**Run entire suite**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks my_eval_suite \ + --include_path my_tasks/ +``` + +## Testing Your Task + +### Validate Configuration + +```bash +# Test task loading +lm_eval --tasks my_custom_task --include_path my_tasks/ --limit 0 + +# Run on 5 samples +lm_eval --model hf \ + --model_args pretrained=gpt2 \ + --tasks my_custom_task \ + --include_path my_tasks/ \ + --limit 5 +``` + +### Debug Mode + +```bash +lm_eval --model hf \ + --model_args pretrained=gpt2 \ + --tasks my_custom_task \ + --include_path my_tasks/ \ + --limit 1 \ + --log_samples # Save input/output samples +``` + +## Best Practices + +1. **Start simple**: Test with minimal config first +2. **Version your tasks**: Use `metadata.version` +3. **Document your metrics**: Explain custom metrics in comments +4. **Test with multiple models**: Ensure robustness +5. **Validate on known examples**: Include sanity checks +6. **Use filters carefully**: Can hide errors +7. **Handle edge cases**: Empty strings, missing fields + +## Common Patterns + +### Classification Task + +```yaml +output_type: loglikelihood +doc_to_text: "Text: {{text}}\nLabel:" +doc_to_target: " {{label}}" # Space prefix important! +metric_list: + - metric: acc + aggregation: mean +``` + +### Perplexity Evaluation + +```yaml +output_type: loglikelihood_rolling +doc_to_text: "{{text}}" +metric_list: + - metric: perplexity + aggregation: perplexity +``` + +### Ranking Task + +```yaml +output_type: loglikelihood +doc_to_text: "Query: {{query}}\nPassage: {{passage}}\nRelevant:" +doc_to_target: [" Yes", " No"] +metric_list: + - metric: acc + aggregation: mean +``` + +## Troubleshooting + +**"Task not found"**: Check `--include_path` and task name + +**Empty results**: Verify `doc_to_text` and `doc_to_target` templates + +**Metric errors**: Ensure metric names are correct (exact_match, not exact-match) + +**Filter issues**: Test filters with `--log_samples` + +**Python function not found**: Check `!function module.function_name` syntax + +## References + +- Task system: EleutherAI/lm-evaluation-harness docs +- Example tasks: `lm_eval/tasks/` directory +- TaskConfig: `lm_eval/api/task.py` diff --git a/evaluating-llms-harness/references/distributed-eval.md b/evaluating-llms-harness/references/distributed-eval.md new file mode 100644 index 0000000..2132e5b --- /dev/null +++ b/evaluating-llms-harness/references/distributed-eval.md @@ -0,0 +1,519 @@ +# Distributed Evaluation + +Guide to running evaluation across multiple GPUs using data parallelism and tensor/pipeline parallelism. + +## Overview + +Distributed evaluation speeds up benchmarking by: +- **Data Parallelism**: Split evaluation samples across GPUs (each GPU has full model copy) +- **Tensor Parallelism**: Split model weights across GPUs (for large models) +- **Pipeline Parallelism**: Split model layers across GPUs (for very large models) + +**When to use**: +- Data Parallel: Model fits on single GPU, want faster evaluation +- Tensor/Pipeline Parallel: Model too large for single GPU + +## HuggingFace Models (`hf`) + +### Data Parallelism (Recommended) + +Each GPU loads a full copy of the model and processes a subset of evaluation data. + +**Single Node (8 GPUs)**: +```bash +accelerate launch --multi_gpu --num_processes 8 \ + -m lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf,dtype=bfloat16 \ + --tasks mmlu,gsm8k,hellaswag \ + --batch_size 16 +``` + +**Speedup**: Near-linear (8 GPUs = ~8× faster) + +**Memory**: Each GPU needs full model (7B model ≈ 14GB × 8 = 112GB total) + +### Tensor Parallelism (Model Sharding) + +Split model weights across GPUs for models too large for single GPU. + +**Without accelerate launcher**: +```bash +lm_eval --model hf \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + parallelize=True,\ + dtype=bfloat16 \ + --tasks mmlu,gsm8k \ + --batch_size 8 +``` + +**With 8 GPUs**: 70B model (140GB) / 8 = 17.5GB per GPU ✅ + +**Advanced sharding**: +```bash +lm_eval --model hf \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + parallelize=True,\ + device_map_option=auto,\ + max_memory_per_gpu=40GB,\ + max_cpu_memory=100GB,\ + dtype=bfloat16 \ + --tasks mmlu +``` + +**Options**: +- `device_map_option`: `"auto"` (default), `"balanced"`, `"balanced_low_0"` +- `max_memory_per_gpu`: Max memory per GPU (e.g., `"40GB"`) +- `max_cpu_memory`: Max CPU memory for offloading +- `offload_folder`: Disk offloading directory + +### Combined Data + Tensor Parallelism + +Use both for very large models. + +**Example: 70B model on 16 GPUs (2 copies, 8 GPUs each)**: +```bash +accelerate launch --multi_gpu --num_processes 2 \ + -m lm_eval --model hf \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + parallelize=True,\ + dtype=bfloat16 \ + --tasks mmlu \ + --batch_size 8 +``` + +**Result**: 2× speedup from data parallelism, 70B model fits via tensor parallelism + +### Configuration with `accelerate config` + +Create `~/.cache/huggingface/accelerate/default_config.yaml`: +```yaml +compute_environment: LOCAL_MACHINE +distributed_type: MULTI_GPU +num_machines: 1 +num_processes: 8 +gpu_ids: all +mixed_precision: bf16 +``` + +**Then run**: +```bash +accelerate launch -m lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu +``` + +## vLLM Models (`vllm`) + +vLLM provides highly optimized distributed inference. + +### Tensor Parallelism + +**Single Node (4 GPUs)**: +```bash +lm_eval --model vllm \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + tensor_parallel_size=4,\ + dtype=auto,\ + gpu_memory_utilization=0.9 \ + --tasks mmlu,gsm8k \ + --batch_size auto +``` + +**Memory**: 70B model split across 4 GPUs = ~35GB per GPU + +### Data Parallelism + +**Multiple model replicas**: +```bash +lm_eval --model vllm \ + --model_args \ + pretrained=meta-llama/Llama-2-7b-hf,\ + data_parallel_size=4,\ + dtype=auto,\ + gpu_memory_utilization=0.8 \ + --tasks hellaswag,arc_challenge \ + --batch_size auto +``` + +**Result**: 4 model replicas = 4× throughput + +### Combined Tensor + Data Parallelism + +**Example: 8 GPUs = 4 TP × 2 DP**: +```bash +lm_eval --model vllm \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + tensor_parallel_size=4,\ + data_parallel_size=2,\ + dtype=auto,\ + gpu_memory_utilization=0.85 \ + --tasks mmlu \ + --batch_size auto +``` + +**Result**: 70B model fits (TP=4), 2× speedup (DP=2) + +### Multi-Node vLLM + +vLLM doesn't natively support multi-node. Use Ray: + +```bash +# Start Ray cluster +ray start --head --port=6379 + +# Run evaluation +lm_eval --model vllm \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + tensor_parallel_size=8,\ + dtype=auto \ + --tasks mmlu +``` + +## NVIDIA NeMo Models (`nemo_lm`) + +### Data Replication + +**8 replicas on 8 GPUs**: +```bash +torchrun --nproc-per-node=8 --no-python \ + lm_eval --model nemo_lm \ + --model_args \ + path=/path/to/model.nemo,\ + devices=8 \ + --tasks hellaswag,arc_challenge \ + --batch_size 32 +``` + +**Speedup**: Near-linear (8× faster) + +### Tensor Parallelism + +**4-way tensor parallelism**: +```bash +torchrun --nproc-per-node=4 --no-python \ + lm_eval --model nemo_lm \ + --model_args \ + path=/path/to/70b_model.nemo,\ + devices=4,\ + tensor_model_parallel_size=4 \ + --tasks mmlu,gsm8k \ + --batch_size 16 +``` + +### Pipeline Parallelism + +**2 TP × 2 PP on 4 GPUs**: +```bash +torchrun --nproc-per-node=4 --no-python \ + lm_eval --model nemo_lm \ + --model_args \ + path=/path/to/model.nemo,\ + devices=4,\ + tensor_model_parallel_size=2,\ + pipeline_model_parallel_size=2 \ + --tasks mmlu \ + --batch_size 8 +``` + +**Constraint**: `devices = TP × PP` + +### Multi-Node NeMo + +Currently not supported by lm-evaluation-harness. + +## SGLang Models (`sglang`) + +### Tensor Parallelism + +```bash +lm_eval --model sglang \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + tp_size=4,\ + dtype=auto \ + --tasks gsm8k \ + --batch_size auto +``` + +### Data Parallelism (Deprecated) + +**Note**: SGLang is deprecating data parallelism. Use tensor parallelism instead. + +```bash +lm_eval --model sglang \ + --model_args \ + pretrained=meta-llama/Llama-2-7b-hf,\ + dp_size=4,\ + dtype=auto \ + --tasks mmlu +``` + +## Performance Comparison + +### 70B Model Evaluation (MMLU, 5-shot) + +| Method | GPUs | Time | Memory/GPU | Notes | +|--------|------|------|------------|-------| +| HF (no parallel) | 1 | 8 hours | 140GB (OOM) | Won't fit | +| HF (TP=8) | 8 | 2 hours | 17.5GB | Slower, fits | +| HF (DP=8) | 8 | 1 hour | 140GB (OOM) | Won't fit | +| vLLM (TP=4) | 4 | 30 min | 35GB | Fast! | +| vLLM (TP=4, DP=2) | 8 | 15 min | 35GB | Fastest | + +### 7B Model Evaluation (Multiple Tasks) + +| Method | GPUs | Time | Speedup | +|--------|------|------|---------| +| HF (single) | 1 | 4 hours | 1× | +| HF (DP=4) | 4 | 1 hour | 4× | +| HF (DP=8) | 8 | 30 min | 8× | +| vLLM (DP=8) | 8 | 15 min | 16× | + +**Takeaway**: vLLM is significantly faster than HuggingFace for inference. + +## Choosing Parallelism Strategy + +### Decision Tree + +``` +Model fits on single GPU? +├─ YES: Use data parallelism +│ ├─ HF: accelerate launch --multi_gpu --num_processes N +│ └─ vLLM: data_parallel_size=N (fastest) +│ +└─ NO: Use tensor/pipeline parallelism + ├─ Model < 70B: + │ └─ vLLM: tensor_parallel_size=4 + ├─ Model 70-175B: + │ ├─ vLLM: tensor_parallel_size=8 + │ └─ Or HF: parallelize=True + └─ Model > 175B: + └─ Contact framework authors +``` + +### Memory Estimation + +**Rule of thumb**: +``` +Memory (GB) = Parameters (B) × Precision (bytes) × 1.2 (overhead) +``` + +**Examples**: +- 7B FP16: 7 × 2 × 1.2 = 16.8GB ✅ Fits A100 40GB +- 13B FP16: 13 × 2 × 1.2 = 31.2GB ✅ Fits A100 40GB +- 70B FP16: 70 × 2 × 1.2 = 168GB ❌ Need TP=4 or TP=8 +- 70B BF16: 70 × 2 × 1.2 = 168GB (same as FP16) + +**With tensor parallelism**: +``` +Memory per GPU = Total Memory / TP +``` + +- 70B on 4 GPUs: 168GB / 4 = 42GB per GPU ✅ +- 70B on 8 GPUs: 168GB / 8 = 21GB per GPU ✅ + +## Multi-Node Evaluation + +### HuggingFace with SLURM + +**Submit job**: +```bash +#!/bin/bash +#SBATCH --nodes=4 +#SBATCH --gpus-per-node=8 +#SBATCH --ntasks-per-node=1 + +srun accelerate launch --multi_gpu \ + --num_processes $((SLURM_NNODES * 8)) \ + -m lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu,gsm8k,hellaswag \ + --batch_size 16 +``` + +**Submit**: +```bash +sbatch eval_job.sh +``` + +### Manual Multi-Node Setup + +**On each node, run**: +```bash +accelerate launch \ + --multi_gpu \ + --num_machines 4 \ + --num_processes 32 \ + --main_process_ip $MASTER_IP \ + --main_process_port 29500 \ + --machine_rank $NODE_RANK \ + -m lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu +``` + +**Environment variables**: +- `MASTER_IP`: IP of rank 0 node +- `NODE_RANK`: 0, 1, 2, 3 for each node + +## Best Practices + +### 1. Start Small + +Test on small sample first: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-70b-hf,parallelize=True \ + --tasks mmlu \ + --limit 100 # Just 100 samples +``` + +### 2. Monitor GPU Usage + +```bash +# Terminal 1: Run evaluation +lm_eval --model hf ... + +# Terminal 2: Monitor +watch -n 1 nvidia-smi +``` + +Look for: +- GPU utilization > 90% +- Memory usage stable +- All GPUs active + +### 3. Optimize Batch Size + +```bash +# Auto batch size (recommended) +--batch_size auto + +# Or tune manually +--batch_size 16 # Start here +--batch_size 32 # Increase if memory allows +``` + +### 4. Use Mixed Precision + +```bash +--model_args dtype=bfloat16 # Faster, less memory +``` + +### 5. Check Communication + +For data parallelism, check network bandwidth: +```bash +# Should see InfiniBand or high-speed network +nvidia-smi topo -m +``` + +## Troubleshooting + +### "CUDA out of memory" + +**Solutions**: +1. Increase tensor parallelism: + ```bash + --model_args tensor_parallel_size=8 # Was 4 + ``` + +2. Reduce batch size: + ```bash + --batch_size 4 # Was 16 + ``` + +3. Lower precision: + ```bash + --model_args dtype=int8 # Quantization + ``` + +### "NCCL error" or Hanging + +**Check**: +1. All GPUs visible: `nvidia-smi` +2. NCCL installed: `python -c "import torch; print(torch.cuda.nccl.version())"` +3. Network connectivity between nodes + +**Fix**: +```bash +export NCCL_DEBUG=INFO # Enable debug logging +export NCCL_IB_DISABLE=0 # Use InfiniBand if available +``` + +### Slow Evaluation + +**Possible causes**: +1. **Data loading bottleneck**: Preprocess dataset +2. **Low GPU utilization**: Increase batch size +3. **Communication overhead**: Reduce parallelism degree + +**Profile**: +```bash +lm_eval --model hf \ + --model_args pretrained=meta-llama/Llama-2-7b-hf \ + --tasks mmlu \ + --limit 100 \ + --log_samples # Check timing +``` + +### GPUs Imbalanced + +**Symptom**: GPU 0 at 100%, others at 50% + +**Solution**: Use `device_map_option=balanced`: +```bash +--model_args parallelize=True,device_map_option=balanced +``` + +## Example Configurations + +### Small Model (7B) - Fast Evaluation + +```bash +# 8 A100s, data parallel +accelerate launch --multi_gpu --num_processes 8 \ + -m lm_eval --model hf \ + --model_args \ + pretrained=meta-llama/Llama-2-7b-hf,\ + dtype=bfloat16 \ + --tasks mmlu,gsm8k,hellaswag,arc_challenge \ + --num_fewshot 5 \ + --batch_size 32 + +# Time: ~30 minutes +``` + +### Large Model (70B) - vLLM + +```bash +# 8 H100s, tensor parallel +lm_eval --model vllm \ + --model_args \ + pretrained=meta-llama/Llama-2-70b-hf,\ + tensor_parallel_size=8,\ + dtype=auto,\ + gpu_memory_utilization=0.9 \ + --tasks mmlu,gsm8k,humaneval \ + --num_fewshot 5 \ + --batch_size auto + +# Time: ~1 hour +``` + +### Very Large Model (175B+) + +**Requires specialized setup - contact framework maintainers** + +## References + +- HuggingFace Accelerate: https://huggingface.co/docs/accelerate/ +- vLLM docs: https://docs.vllm.ai/ +- NeMo docs: https://docs.nvidia.com/nemo-framework/ +- lm-eval distributed guide: `docs/model_guide.md` diff --git a/huggingface-hub/SKILL.md b/huggingface-hub/SKILL.md new file mode 100644 index 0000000..bb82ac1 --- /dev/null +++ b/huggingface-hub/SKILL.md @@ -0,0 +1,81 @@ +--- +name: huggingface-hub +description: "HuggingFace hf CLI: search/download/upload models, datasets." +version: 1.0.1 +author: Hugging Face +license: MIT +tags: [huggingface, hf, models, datasets, hub, mlops] +platforms: [linux, macos, windows] +--- + +# Hugging Face CLI (`hf`) Reference Guide + +The `hf` command is the modern command-line interface for interacting with the Hugging Face Hub, providing tools to manage repositories, models, datasets, and Spaces. + +> **IMPORTANT:** The `hf` command replaces the now deprecated `huggingface-cli` command. + +## Quick Start +* **Installation:** `curl -LsSf https://hf.co/cli/install.sh | bash -s` +* **Help:** Use `hf --help` to view all available functions and real-world examples. +* **Authentication:** Recommended via `HF_TOKEN` environment variable or the `--token` flag. + +--- + +## Core Commands + +### General Operations +* `hf download REPO_ID`: Download files from the Hub. +* `hf upload REPO_ID`: Upload files/folders (recommended for single-commit; also handles resumable uploads of large directories). +* `hf upload-large-folder REPO_ID LOCAL_PATH`: **[Deprecated]** — use `hf upload` instead. +* `hf sync`: Sync files between a local directory and a bucket. +* `hf env` / `hf version`: View environment and version details. + +### Authentication (`hf auth`) +* `login` / `logout`: Manage sessions using tokens from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). +* `list` / `switch`: Manage and toggle between multiple stored access tokens. +* `whoami`: Identify the currently logged-in account. + +### Repository Management (`hf repos`) +* `create` / `delete`: Create or permanently remove repositories. +* `duplicate`: Clone a model, dataset, or Space to a new ID. +* `move`: Transfer a repository between namespaces. +* `branch` / `tag`: Manage Git-like references. +* `delete-files`: Remove specific files using patterns. + +--- + +## Specialized Hub Interactions + +### Datasets & Models +* **Datasets:** `hf datasets list`, `info`, and `parquet` (list parquet URLs). +* **SQL Queries:** `hf datasets sql SQL` — Execute raw SQL via DuckDB against dataset parquet URLs. +* **Models:** `hf models list` and `info`. +* **Papers:** `hf papers ls` — View daily papers. + +### Discussions & Pull Requests (`hf discussions`) +* Manage the lifecycle of Hub contributions: `list`, `create`, `info`, `comment`, `close`, `reopen`, and `rename`. +* `diff`: View changes in a PR. +* `merge`: Finalize pull requests. + +### Infrastructure & Compute +* **Endpoints:** Deploy and manage Inference Endpoints (`deploy`, `pause`, `resume`, `scale-to-zero`, `catalog`). +* **Jobs:** Run compute tasks on HF infrastructure. Includes `hf jobs uv` for running Python scripts with inline dependencies and `stats` for resource monitoring. +* **Spaces:** Manage interactive apps. Includes `dev-mode` and `hot-reload` for Python files without full restarts. + +### Storage & Automation +* **Buckets:** Full S3-like bucket management (`create`, `cp`, `mv`, `rm`, `sync`). +* **Cache:** Manage local storage with `list`, `prune` (remove detached revisions), and `verify` (checksum checks). +* **Webhooks:** Automate workflows by managing Hub webhooks (`create`, `watch`, `enable`/`disable`). +* **Collections:** Organize Hub items into collections (`add-item`, `update`, `list`). + +--- + +## Advanced Usage & Tips + +### Global Flags +* `--format json`: Produces machine-readable output for automation. +* `-q` / `--quiet`: Limits output to IDs only. + +### Extensions & Skills +* **Extensions:** Extend CLI functionality via GitHub repositories using `hf extensions install REPO_ID`. +* **Skills:** Manage AI assistant skills with `hf skills add`. diff --git a/llama-cpp/SKILL.md b/llama-cpp/SKILL.md new file mode 100644 index 0000000..07fe98a --- /dev/null +++ b/llama-cpp/SKILL.md @@ -0,0 +1,249 @@ +--- +name: llama-cpp +description: llama.cpp local GGUF inference + HF Hub model discovery. +version: 2.1.2 +author: Orchestra Research +license: MIT +dependencies: [llama-cpp-python>=0.2.0] +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [llama.cpp, GGUF, Quantization, Hugging Face Hub, CPU Inference, Apple Silicon, Edge Deployment, AMD GPUs, Intel GPUs, NVIDIA, URL-first] +--- + +# llama.cpp + GGUF + +Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp. + +## When to use + +- Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs +- Find the right GGUF for a specific Hugging Face repo +- Build a `llama-server` or `llama-cli` command from the Hub +- Search the Hub for models that already support llama.cpp +- Enumerate available `.gguf` files and sizes for a repo +- Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM + +## Model Discovery workflow + +Prefer URL workflows before asking for `hf`, Python, or custom scripts. + +1. Search for candidate repos on the Hub: + - Base: `https://huggingface.co/models?apps=llama.cpp&sort=trending` + - Add `search=` for a model family + - Add `num_parameters=min:0,max:24B` or similar when the user has size constraints +2. Open the repo with the llama.cpp local-app view: + - `https://huggingface.co/?local-app=llama.cpp` +3. Treat the local-app snippet as the source of truth when it is visible: + - copy the exact `llama-server` or `llama-cli` command + - report the recommended quant exactly as HF shows it +4. Read the same `?local-app=llama.cpp` URL as page text or HTML and extract the section under `Hardware compatibility`: + - prefer its exact quant labels and sizes over generic tables + - keep repo-specific labels such as `UD-Q4_K_M` or `IQ4_NL_XL` + - if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance +5. Query the tree API to confirm what actually exists: + - `https://huggingface.co/api/models//tree/main?recursive=true` + - keep entries where `type` is `file` and `path` ends with `.gguf` + - use `path` and `size` as the source of truth for filenames and byte sizes + - separate quantized checkpoints from `mmproj-*.gguf` projector files and `BF16/` shard files + - use `https://huggingface.co//tree/main` only as a human fallback +6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant: + - shorthand quant selection: `llama-server -hf :` + - exact-file fallback: `llama-server --hf-repo --hf-file ` +7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files. + +## Quick start + +### Install llama.cpp + +```bash +# macOS / Linux (simplest) +brew install llama.cpp +``` + +```bash +winget install llama.cpp +``` + +```bash +git clone https://github.com/ggml-org/llama.cpp +cd llama.cpp +cmake -B build +cmake --build build --config Release +``` + +### Run directly from the Hugging Face Hub + +```bash +llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0 +``` + +```bash +llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0 +``` + +### Run an exact GGUF file from the Hub + +Use this when the tree API shows custom file naming or the exact HF snippet is missing. + +```bash +llama-server \ + --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \ + --hf-file Phi-3-mini-4k-instruct-q4.gguf \ + -c 4096 +``` + +### OpenAI-compatible server check + +```bash +curl http://localhost:8080/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [ + {"role": "user", "content": "Write a limerick about Python exceptions"} + ] + }' +``` + +## Python bindings (llama-cpp-python) + +`pip install llama-cpp-python` (CUDA: `CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir`; Metal: `CMAKE_ARGS="-DGGML_METAL=on" ...`). + +### Basic generation + +```python +from llama_cpp import Llama + +llm = Llama( + model_path="./model-q4_k_m.gguf", + n_ctx=4096, + n_gpu_layers=35, # 0 for CPU, 99 to offload everything + n_threads=8, +) + +out = llm("What is machine learning?", max_tokens=256, temperature=0.7) +print(out["choices"][0]["text"]) +``` + +### Chat + streaming + +```python +llm = Llama( + model_path="./model-q4_k_m.gguf", + n_ctx=4096, + n_gpu_layers=35, + chat_format="llama-3", # or "chatml", "mistral", etc. +) + +resp = llm.create_chat_completion( + messages=[ + {"role": "system", "content": "You are a helpful assistant."}, + {"role": "user", "content": "What is Python?"}, + ], + max_tokens=256, +) +print(resp["choices"][0]["message"]["content"]) + +# Streaming +for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True): + print(chunk["choices"][0]["text"], end="", flush=True) +``` + +### Embeddings + +```python +llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35) +vec = llm.embed("This is a test sentence.") +print(f"Embedding dimension: {len(vec)}") +``` + +You can also load a GGUF straight from the Hub: + +```python +llm = Llama.from_pretrained( + repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF", + filename="*Q4_K_M.gguf", + n_gpu_layers=35, +) +``` + +## Choosing a quant + +Use the Hub page first, generic heuristics second. + +- Prefer the exact quant that HF marks as compatible for the user's hardware profile. +- For general chat, start with `Q4_K_M`. +- For code or technical work, prefer `Q5_K_M` or `Q6_K` if memory allows. +- For very tight RAM budgets, consider `Q3_K_M`, `IQ` variants, or `Q2` variants only if the user explicitly prioritizes fit over quality. +- For multimodal repos, mention `mmproj-*.gguf` separately. The projector is not the main model file. +- Do not normalize repo-native labels. If the page says `UD-Q4_K_M`, report `UD-Q4_K_M`. + +## Extracting available GGUFs from a repo + +When the user asks what GGUFs exist, return: + +- filename +- file size +- quant label +- whether it is a main model or an auxiliary projector + +Ignore unless requested: + +- README +- BF16 shard files +- imatrix blobs or calibration artifacts + +Use the tree API for this step: + +- `https://huggingface.co/api/models//tree/main?recursive=true` + +For a repo like `unsloth/Qwen3.6-35B-A3B-GGUF`, the local-app page can show quant chips such as `UD-Q4_K_M`, `UD-Q5_K_M`, `UD-Q6_K`, and `Q8_0`, while the tree API exposes exact file paths such as `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` and `Qwen3.6-35B-A3B-Q8_0.gguf` with byte sizes. Use the tree API to turn a quant label into an exact filename. + +## Search patterns + +Use these URL shapes directly: + +```text +https://huggingface.co/models?apps=llama.cpp&sort=trending +https://huggingface.co/models?search=&apps=llama.cpp&sort=trending +https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending +https://huggingface.co/?local-app=llama.cpp +https://huggingface.co/api/models//tree/main?recursive=true +https://huggingface.co//tree/main +``` + +## Output format + +When answering discovery requests, prefer a compact structured result like: + +```text +Repo: +Recommended quant from HF: