Configure smart routing correctly

Route by token budget, latency, or model class to send requests to the most efficient deployment.

Rule design

1. Define primary objective (cost, quality, speed).
2. Set hard limits for budget and latency.
3. Define explicit priority order between rules.

Operations

1. Enable fallback deployments.
2. Measure performance per routing path.
3. Avoid rule conflicts with deterministic order.

Optimization

• review cost and quality monthly
• optimize hot paths separately
• iterate rules based on production traffic data