ray-project/llmperf is a Python project with 1.1k stars. LLMPerf is a library for validating and benchmarking LLMs
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
A Tool for evaulation the performance of LLM APIs.
Installation
git clone https://github.com/ray-project/llmperf.git
cd llmperf
pip install -e .
Basic Usage
We implement 2 tests for evaluating LLMs: a load test to check for performance and a correctness test to check for correctness.
Load test
The load test spawns a number of concurrent requests to the LLM API and measures the inter-token latency and generation throughput per request and across concurrent requests. The prompt that is sent with each request is of the format:
Randomly stream lines from the following text. Don't generate eos tokens:
LINE 1,
LINE 2,
LINE 3,
...
Where the lines are randomly sampled from a collection of lines from Shakespeare sonnets. Tokens are counted using the LlamaTokenizer regardless of which LLM API is being tested. This is to ensure that the prompts are consistent across different LLM APIs.
To run the most basic load test you can the token_benchmark_ray script.
Caveats and Disclaimers
The endpoints provider backend might vary widely, so this is not a reflection on how the software runs on a particular hardware.
The results may vary with time of day.
The results may vary with the load.
The results may not correlate with users’ workloads.
LLMPerf can use LiteLLM to send prompts to LLM APIs. To see the environment variables to set for the provider and arguments that one should set for model and additional-sampling-params.
see python token_benchmark_ray.py --help for more details on the arguments.
Correctness Test
The correctness test spawns a number of concurrent requests to the LLM API with the following format:
Convert the following sequence of words into a number: {random_number_in_word_format}. Output just your final answer.
where random_number_in_word_format could be for example "one hundred and twenty three". The test then checks that the response contains that number in digit format which in this case would be 123.
The test does this for a number of randomly generated numbers and reports the number of responses that contain a mismatch.
To run the most basic correctness test you can run the the llm_correctness.py script.
LLMPerf can use LiteLLM to send prompts to LLM APIs. To see the environment variables to set for the provider and arguments that one should set for model and additional-sampling-params.
The results of the load test and correctness test are saved in the results directory specified by the --results-dir argument. The results are saved in 2 files, one with the summary metrics of the test, and one with metrics from each individual request that is returned.
Advanced Usage
The correctness tests were implemented with the following workflow in mind:
import ray
from transformers import LlamaTokenizerFast
from llmperf.ray_clients.openai_chat_completions_client import (
OpenAIChatCompletionsClient,
)
from llmperf.models import RequestConfig
from llmperf.requests_launcher import RequestsLauncher
# Copying the environment variables and passing them to ray.init() is necessary
# For making any clients work.
ray.init(runtime_env={"env_vars": {"OPENAI_API_BASE" : "https://api.endpoints.anyscale.com/v1",
"OPENAI_API_KEY" : "YOUR_API_KEY"}})
base_prompt = "hello_world"
tokenizer = LlamaTokenizerFast.from_pretrained(
"hf-internal-testing/llama-tokenizer"
)
base_prompt_len = len(tokenizer.encode(base_prompt))
prompt = (base_prompt, base_prompt_len)
# Create a client for spawning requests
clients = [OpenAIChatCompletionsClient.remote()]
req_launcher = RequestsLauncher(clients)
req_config = RequestConfig(
model="meta-llama/Llama-2-7b-chat-hf",
prompt=prompt
)
req_launcher.launch_requests(req_config)
result = req_launcher.get_next_ready(block=True)
print(result)
Implementing New LLM Clients
To implement a new LLM client, you need to implement the base class llmperf.ray_llm_client.LLMClient and decorate it as a ray actor.
from llmperf.ray_llm_client import LLMClient
import ray
@ray.remote
class CustomLLMClient(LLMClient):
def llm_request(self, request_config: RequestConfig) -> Tuple[Metrics, str, RequestConfig]:
"""Make a single completion request to a LLM API
Returns:
Metrics about the performance charateristics of the request.
The text generated by the request to the LLM API.
The request_config used to make the request. This is mainly for logging purposes.
"""
...
Legacy Codebase
The old LLMPerf code base can be found in the llmperf-legacy repo.
TopGit's last sync did not record any GitHub topics for ray-project/llmperf. GitHub topics appear in the right sidebar of a repository page; that's the authoritative place to check.
How active is development on ray-project/llmperf?
The most recent commit recorded on ray-project/llmperf was 1.9 years ago, based on the GitHub push timestamp. The repository has 203 forks — one of the better signals of community interest.
How many stars does ray-project/llmperf have?
ray-project/llmperf has 1.1k GitHub stars — refresh the page for the live number, or check github.com/ray-project/llmperf. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is ray-project/llmperf written in?
ray-project/llmperf is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
Where do I read more about ray-project/llmperf?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/ray-project/llmperf is the definitive source.
Read full README in the tab above.
Want a second opinion on llmperf?
Ask an AI that can read this page — one click and you get its take on llmperf.