VLLM Setup Guide
Last updated
Last updated
With the latest update, the Dispute Scout now supports VLLM as a provider. This allows you to utilize VLLM models for advanced logprob-based dispute detection.
To set up the VLLM Dispute Scout, ensure you have:
VLLM Deployment: A running instance of VLLM, either locally or accessible via a remote endpoint.
VLLM API Endpoint URL (VLLM_URL
): The base URL where your VLLM API is accessible.
VLLM API Key (VLLM_API_KEY
): If your VLLM deployment requires authentication.
Option 1: Local VLLM Deployment
Install VLLM: Follow the official installation guide in the to set up VLLM on your machine.
Start the VLLM Server: Run the VLLM server with your desired model. For example:
HF_TOKEN=<your-huggingface-token> python -m vllm.entrypoints.openai.api_server --model repo_id --max-logprobs 120
By default, the server runs at http://localhost:8000
.
Option 2: Remote VLLM Deployment
If you have access to a remote VLLM API endpoint:
Ensure you have the base URL and API key (if required).
Verify that the endpoint is accessible from your network.
.env
File for VLLMIn the chasm-scout/dispute
directory, create or modify your .env
file to include the VLLM configuration. Replace placeholders with your actual values.
Explanation of Variables:
LLM_API_KEY
: API key for your VLLM instance (if required).
LLM_BASE_URL
: Base URL of your VLLM API endpoint.
VLLM_URL
: Same as LLM_BASE_URL
; used specifically for VLLM configurations.
VLLM_API_KEY
: Same as LLM_API_KEY
; used specifically for VLLM configurations.
MODELS
: The model(s) available in your VLLM deployment.
SIMULATION_MODEL
: The model used for simulation purposes.
ORCHESTRATOR_URL
: The orchestrator's URL (usually https://orchestrator.chasm.net
).
WEBHOOK_API_KEY
: Your webhook API key obtained during registration.
Note: Do not include multiple supplier configurations in your .env
file. Only include the settings relevant to VLLM.
After configuring the .env
file, follow these steps:
Ensure VLLM is Running: Verify that your VLLM server is operational and accessible at the URL specified in LLM_BASE_URL
.
Build the Docker Image:
Run the Docker Container:
Monitor the Logs:
Check for any errors and ensure that the Dispute Scout is running correctly.
Model Availability: Ensure that the models specified in MODELS
and SIMULATION_MODEL
are available in your VLLM deployment.
API Keys: If your VLLM deployment does not require an API key, you can leave LLM_API_KEY
and VLLM_API_KEY
blank.
Endpoint Accessibility: If the VLLM server is running locally, ensure there are no firewall or network restrictions preventing access from the Docker container.
Connection Issues: If the Dispute Scout cannot connect to the VLLM API, verify that the LLM_BASE_URL
is correct and accessible.
Authentication Errors: If authentication fails, double-check your LLM_API_KEY
and VLLM_API_KEY
.
Model Errors: If you encounter model-related errors, ensure the model names in your .env
file match those in your VLLM deployment.
Logprob Support: The VLLM Dispute Scout relies on logprob data. Ensure your VLLM deployment supports returning logprobs.
Setting the Minimum Confidence Score
You can adjust the minimum confidence score required for filing a dispute to prevent false positives:
Add this line to your .env
file and adjust the value based on your requirements.
Running Benchmarks
To evaluate if a dispute will be filed with your current settings:
Install Python and Dependencies:
Create a Virtual Environment:
Install Requirements:
Run the Benchmark Script:
This script will help you assess the impact of your settings on dispute filings.
Docker Daemon Not Running: If you encounter an error stating that Docker cannot connect to the daemon, start Docker with:
Check Container Status: Use docker ps
to list running containers and ensure the Dispute Scout is active.
Log Analysis: Regularly check the logs using docker compose logs -f
to monitor the Dispute Scout's activity and catch any issues early.
About VLLM :
VLLM GitHub Repository: