VLLM Setup Guide

Adding VLLM as a Dispute Scout Provider

With the latest update, the Dispute Scout now supports VLLM as a provider. This allows you to utilize VLLM models for advanced logprob-based dispute detection.

Prerequisites for VLLM Dispute Scout

To set up the VLLM Dispute Scout, ensure you have:

• VLLM Deployment: A running instance of VLLM, either locally or accessible via a remote endpoint.

• VLLM API Endpoint URL (VLLM_URL): The base URL where your VLLM API is accessible.

• VLLM API Key (VLLM_API_KEY): If your VLLM deployment requires authentication.

Setting Up VLLM

Option 1: Local VLLM Deployment

1. Install VLLM: Follow the official installation guide in the VLLM GitHub Repository to set up VLLM on your machine.

2. Start the VLLM Server: Run the VLLM server with your desired model. For example:

HF_TOKEN=<your-huggingface-token> python -m vllm.entrypoints.openai.api_server --model repo_id --max-logprobs 120

By default, the server runs at http://localhost:8000.

Option 2: Remote VLLM Deployment

If you have access to a remote VLLM API endpoint:

• Ensure you have the base URL and API key (if required).

• Verify that the endpoint is accessible from your network.

Configuring the .env File for VLLM

In the chasm-scout/dispute directory, create or modify your .env file to include the VLLM configuration. Replace placeholders with your actual values.

## VLLM Configuration

# API key for VLLM (leave blank if not required)
LLM_API_KEY=your_vllm_api_key_or_leave_blank

# Base URL for VLLM API
LLM_BASE_URL=http://localhost:8000/v1

# VLLM-specific configurations
VLLM_URL=http://localhost:8000/v1
VLLM_API_KEY=your_vllm_api_key_or_leave_blank

# Models to use (comma-separated if multiple)
MODELS=llama-2-7b-chat

# Model used for simulation
SIMULATION_MODEL=llama-2-7b-chat

# Orchestrator URL
ORCHESTRATOR_URL=https://orchestrator.chasm.net

# Your webhook API key
WEBHOOK_API_KEY=your_webhook_api_key

Explanation of Variables:

• LLM_API_KEY: API key for your VLLM instance (if required).

• LLM_BASE_URL: Base URL of your VLLM API endpoint.

• VLLM_URL: Same as LLM_BASE_URL; used specifically for VLLM configurations.

• VLLM_API_KEY: Same as LLM_API_KEY; used specifically for VLLM configurations.

• MODELS: The model(s) available in your VLLM deployment.

• SIMULATION_MODEL: The model used for simulation purposes.

• ORCHESTRATOR_URL: The orchestrator's URL (usually https://orchestrator.chasm.net).

• WEBHOOK_API_KEY: Your webhook API key obtained during registration.

Note: Do not include multiple supplier configurations in your .env file. Only include the settings relevant to VLLM.

Running the VLLM Dispute Scout

After configuring the .env file, follow these steps:

1. Ensure VLLM is Running: Verify that your VLLM server is operational and accessible at the URL specified in LLM_BASE_URL.

2. Build the Docker Image:

   docker compose build

3. Run the Docker Container:

   docker compose up -d

4. Monitor the Logs:

   docker compose logs -f

   Check for any errors and ensure that the Dispute Scout is running correctly.

Notes on VLLM Configuration

• Model Availability: Ensure that the models specified in MODELS and SIMULATION_MODEL are available in your VLLM deployment.

• API Keys: If your VLLM deployment does not require an API key, you can leave LLM_API_KEY and VLLM_API_KEY blank.

• Endpoint Accessibility: If the VLLM server is running locally, ensure there are no firewall or network restrictions preventing access from the Docker container.

Monitoring and Troubleshooting

• Connection Issues: If the Dispute Scout cannot connect to the VLLM API, verify that the LLM_BASE_URL is correct and accessible.

• Authentication Errors: If authentication fails, double-check your LLM_API_KEY and VLLM_API_KEY.

• Model Errors: If you encounter model-related errors, ensure the model names in your .env file match those in your VLLM deployment.

• Logprob Support: The VLLM Dispute Scout relies on logprob data. Ensure your VLLM deployment supports returning logprobs.

Advanced Configuration

Setting the Minimum Confidence Score

You can adjust the minimum confidence score required for filing a dispute to prevent false positives:

## Adjust the threshold as needed (default is 0.5)
MIN_CONFIDENCE_SCORE=0.7

Add this line to your .env file and adjust the value based on your requirements.

Running Benchmarks

To evaluate if a dispute will be filed with your current settings:

1. Install Python and Dependencies:

   sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv

2. Create a Virtual Environment:

   python3 -m venv dispute-scout
   source dispute-scout/bin/activate

3. Install Requirements:

   pip install -r requirements.txt

4. Run the Benchmark Script:

   python benchmark.py

   This script will help you assess the impact of your settings on dispute filings.

Additional Resources

• About VLLM : VLLM Official Documentation

• VLLM GitHub Repository: github.com/vllm-project/vllm

Troubleshooting Tips

• Docker Daemon Not Running: If you encounter an error stating that Docker cannot connect to the daemon, start Docker with:

  sudo systemctl start docker

• Check Container Status: Use docker ps to list running containers and ensure the Dispute Scout is active.

• Log Analysis: Regularly check the logs using docker compose logs -f to monitor the Dispute Scout's activity and catch any issues early.