PyTorchPyTorch避坑指南

Train vs Eval Mode

• - model.train() enables dropout, BatchNorm updates — default after init
• INLINECODE1 disables dropout, uses running stats — MUST call for inference
• Mode is sticky — train/eval persists until explicitly changed
• INLINECODE2 doesn't disable gradients — still need INLINECODE3

Gradient Control

• - torch.no_grad() for inference — reduces memory, speeds up computation
• INLINECODE5 accumulates gradients — call optimizer.zero_grad() before backward
• INLINECODE7 placement matters — before forward pass, not after backward
• INLINECODE8 to stop gradient flow — prevents memory leak in logging

Device Management

• - Model AND data must be on same device — model.to(device) and INLINECODE10
• INLINECODE11 vs .to('cuda') — both work, .to(device) more flexible
• CUDA tensors can't convert to numpy directly — .cpu().numpy() required
• INLINECODE15 — portable code

DataLoader

• - num_workers > 0 uses multiprocessing — Windows needs INLINECODE17
• INLINECODE18 with CUDA — faster transfer to GPU
• Workers don't share state — random seeds differ per worker, set in INLINECODE19
• Large num_workers can cause memory issues — start with 2-4, increase if CPU-bound

Saving and Loading

• - torch.save(model.state_dict(), path) — recommended, saves only weights
• Loading: create model first, then INLINECODE22
• INLINECODE23 for cross-device — torch.load(path, map_location='cpu') if saved on GPU
• Saving whole model pickles code path — breaks if code changes

In-place Operations

• - In-place ops end with _ — tensor.add_(1) vs INLINECODE27
• In-place on leaf variable breaks autograd — error about modified leaf
• In-place on intermediate can corrupt gradient — avoid in computation graph
• INLINECODE28 bypasses autograd — legacy, prefer .detach() for safety

Memory Management

• - Accumulated tensors leak memory — .detach() logged metrics
• INLINECODE31 releases cached memory — but doesn't fix leaks
• Delete references and call gc.collect() — before empty_cache if needed
• INLINECODE33 prevents graph storage — crucial for validation loop

Common Mistakes

• - BatchNorm with batch_size=1 fails in train mode — use eval mode or INLINECODE35
• Loss function reduction default is 'mean' — may want 'sum' for gradient accumulation
• INLINECODE36 expects logits — not softmax output
• INLINECODE37 to get Python scalar — .numpy() or [0] deprecated/error