RouteLLM is a versatile framework for serving and evaluating LLM routers, designed to maximise efficiency whereas minimizing value.
Key options:
- Seamless integration — Acts as a drop-in alternative for the OpenAI consumer or runs as an OpenAI-compatible server, intelligently routing easier queries to cheaper fashions.
- Pre-trained routers out of the field — Confirmed to chop prices by as much as 85% whereas preserving 95% of GPT-4 efficiency on broadly used benchmarks like MT-Bench.
- Value-effective excellence — Matches the efficiency of main business choices whereas being over 40% cheaper.
- Extensible and customizable — Simply add new routers, fine-tune thresholds, and evaluate efficiency throughout a number of benchmarks.
On this tutorial, we’ll stroll by means of how one can:
- Load and use a pre-trained router.
- Calibrate it in your personal use case.
- Take a look at routing conduct on several types of prompts.
- Take a look at the Full Codes right here.
Putting in the dependencies
!pip set up "routellm[serve,eval]"Loading OpenAI API Key
To get an OpenAI API key, go to https://platform.openai.com/settings/group/api-keys and generate a brand new key. When you’re a brand new consumer, chances are you’ll want so as to add billing particulars and make a minimal cost of $5 to activate API entry.
RouteLLM leverages LiteLLM to help chat completions from a variety of each open-source and closed-source fashions. You may take a look at the record of suppliers at https://litellm.vercel.app/docs/suppliers if you wish to use another mannequin. Take a look at the Full Codes right here.
import os
from getpass import getpass
os.environ['OPENAI_API_KEY'] = getpass('Enter OpenAI API Key: ')Downloading Config File
RouteLLM makes use of a configuration file to find pretrained router checkpoints and the datasets they had been skilled on. This file tells the system the place to seek out the fashions that resolve whether or not to ship a question to the robust or weak mannequin. Take a look at the Full Codes right here.
Do I must edit it?
For many customers — no. The default config already factors to well-trained routers (mf, bert, causal_llm) that work out of the field. You solely want to alter it for those who plan to:
- Practice your personal router on a customized dataset.
- Change the routing algorithm totally with a brand new one.
For this tutorial, we’ll maintain the config as is and easily:
- Set our robust and weak mannequin names in code.
- Add our API keys for the chosen suppliers.
- Use a calibrated threshold to steadiness value and high quality.
- Take a look at the Full Codes right here.
!wget https://uncooked.githubusercontent.com/lm-sys/RouteLLM/major/config.instance.yamlInitializing the RouteLLM Controller
On this code block, we import the required libraries and initialize the RouteLLM Controller, which can handle how prompts are routed between fashions. We specify routers=[“mf”] to make use of the Matrix Factorization router, a pretrained choice mannequin that predicts whether or not a question must be despatched to the robust or weak mannequin.
The strong_model parameter is ready to “gpt-5”, a high-quality however dearer mannequin, whereas the weak_model parameter is ready to “o4-mini”, a sooner and cheaper different. For every incoming immediate, the router evaluates its complexity towards a threshold and robotically chooses probably the most cost-effective choice—making certain that easy duties are dealt with by the cheaper mannequin whereas tougher ones get the stronger mannequin’s capabilities.
This configuration permits you to steadiness value effectivity and response high quality with out handbook intervention. Take a look at the Full Codes right here.
import os
import pandas as pd
from routellm.controller import Controller
consumer = Controller(
routers=["mf"], # Mannequin Fusion router
strong_model="gpt-5",
weak_model="o4-mini"
)!python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.1 --config config.instance.yamlThis command runs RouteLLM’s threshold calibration course of for the Matrix Factorization (mf) router. The –strong-model-pct 0.1 argument tells the system to seek out the brink worth that routes roughly 10% of queries to the robust mannequin (and the remainder to the weak mannequin).
Utilizing the –config config.instance.yaml file for mannequin and router settings, the calibration decided:
For 10% robust mannequin calls with mf, the optimum threshold is 0.24034.
Which means any question with a router-assigned complexity rating above 0.24034 can be despatched to the robust mannequin, whereas these beneath it’ll go to the weak mannequin, aligning together with your desired value–high quality trade-off.
Defining the brink & prompts variables
Right here, we outline a various set of check prompts designed to cowl a spread of complexity ranges. They embrace easy factual questions (more likely to be routed to the weak mannequin), medium reasoning duties (borderline threshold instances), and high-complexity or artistic requests (extra fitted to the robust mannequin), together with code era duties to check technical capabilities. Take a look at the Full Codes right here.
threshold = 0.24034
prompts = [
# Easy factual (likely weak model)
"Who wrote the novel 'Pride and Prejudice'?",
"What is the largest planet in our solar system?",
# Medium reasoning (borderline cases)
"If a train leaves at 3 PM and travels 60 km/h, how far will it travel by 6:30 PM?",
"Explain why the sky appears blue during the day and red/orange during sunset.",
# High complexity / creative (likely strong model)
"Write a 6-line rap verse about climate change using internal rhyme.",
"Summarize the differences between supervised, unsupervised, and reinforcement learning with examples.",
# Code generation
"Write a Python function to check if a given string is a palindrome, ignoring punctuation and spaces.",
"Generate SQL to find the top 3 highest-paying customers from a 'sales' table."
]Evaluating Win Price
The next code calculates the win price for every check immediate utilizing the mf router, exhibiting the probability that the robust mannequin will outperform the weak mannequin.
Primarily based on the calibrated threshold of 0.24034, two prompts —
“If a prepare leaves at 3 PM and travels 60 km/h, how far will it journey by 6:30 PM?” (0.303087)
“Write a Python operate to test if a given string is a palindrome, ignoring punctuation and areas.” (0.272534)
— exceed the brink and could be routed to the robust mannequin.
All different prompts stay beneath the brink, which means they’d be served by the weaker, cheaper mannequin. Take a look at the Full Codes right here.
win_rates = consumer.batch_calculate_win_rate(prompts=pd.Sequence(prompts), router="mf")
# Retailer leads to DataFrame
_df = pd.DataFrame({
"Immediate": prompts,
"Win_Rate": win_rates
})
# Present full textual content with out truncation
pd.set_option('show.max_colwidth', None)These outcomes additionally assist in fine-tuning the routing technique — by analyzing the win price distribution, we are able to alter the brink to raised steadiness value financial savings and efficiency.
Routing Prompts By Calibrated Mannequin Fusion (MF) Router
This code iterates over the record of check prompts and sends every one to the RouteLLM controller utilizing the calibrated mf router with the required threshold (router-mf-{threshold}).
For every immediate, the router decides whether or not to make use of the robust or weak mannequin based mostly on the calculated win price.
The response consists of each the generated output and the precise mannequin that was chosen by the router.
These particulars — the immediate, mannequin used, and generated output — are saved within the outcomes record for later evaluation. Take a look at the Full Codes right here.
outcomes = []
for immediate in prompts:
response = consumer.chat.completions.create(
mannequin=f"router-mf-{threshold}",
messages=[{"role": "user", "content": prompt}]
)
message = response.decisions[0].message["content"]
model_used = response.mannequin # RouteLLM returns the mannequin truly used
outcomes.append({
"Immediate": immediate,
"Mannequin Used": model_used,
"Output": message
})
df = pd.DataFrame(outcomes)
Within the outcomes, prompts 2 and 6 exceeded the brink win price and had been subsequently routed to the gpt-5 robust mannequin, whereas the remainder had been dealt with by the weaker mannequin.
Take a look at the Full Codes right here. Be at liberty to take a look at our GitHub Web page for Tutorials, Codes and Notebooks. Additionally, be happy to comply with us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our Publication.
I’m a Civil Engineering Graduate (2022) from Jamia Millia Islamia, New Delhi, and I’ve a eager curiosity in Information Science, particularly Neural Networks and their software in varied areas.
