# VLLM Setup Guide

### Adding VLLM as a Dispute Scout Provider

With the latest update, the Dispute Scout now supports VLLM as a provider. This allows you to utilize VLLM models for advanced logprob-based dispute detection.

#### Prerequisites for VLLM Dispute Scout

To set up the VLLM Dispute Scout, ensure you have:

* **VLLM Deployment**: A running instance of VLLM, either locally or accessible via a remote endpoint.
* **VLLM API Endpoint URL (`VLLM_URL`)**: The base URL where your VLLM API is accessible.
* **VLLM API Key (`VLLM_API_KEY`)**: If your VLLM deployment requires authentication.

#### Setting Up VLLM

**Option 1: Local VLLM Deployment**

1. **Install VLLM**: Follow the official installation guide in the [VLLM GitHub Repository](https://github.com/vllm-project/vllm) to set up VLLM on your machine.
2. **Start the VLLM Server**: Run the VLLM server with your desired model. For example:

`HF_TOKEN=<your-huggingface-token> python -m vllm.entrypoints.openai.api_server --model repo_id --max-logprobs 120`

By default, the server runs at `http://localhost:8000`.

**Option 2: Remote VLLM Deployment**

If you have access to a remote VLLM API endpoint:

* Ensure you have the base URL and API key (if required).
* Verify that the endpoint is accessible from your network.

#### Configuring the `.env` File for VLLM

In the `chasm-scout/dispute` directory, create or modify your `.env` file to include the VLLM configuration. Replace placeholders with your actual values.

```bash
## VLLM Configuration

# API key for VLLM (leave blank if not required)
LLM_API_KEY=your_vllm_api_key_or_leave_blank

# Base URL for VLLM API
LLM_BASE_URL=http://localhost:8000/v1

# VLLM-specific configurations
VLLM_URL=http://localhost:8000/v1
VLLM_API_KEY=your_vllm_api_key_or_leave_blank

# Models to use (comma-separated if multiple)
MODELS=llama-2-7b-chat

# Model used for simulation
SIMULATION_MODEL=llama-2-7b-chat

# Orchestrator URL
ORCHESTRATOR_URL=https://orchestrator.chasm.net

# Your webhook API key
WEBHOOK_API_KEY=your_webhook_api_key
```

**Explanation of Variables:**

* `LLM_API_KEY`: API key for your VLLM instance (if required).
* `LLM_BASE_URL`: Base URL of your VLLM API endpoint.
* `VLLM_URL`: Same as `LLM_BASE_URL`; used specifically for VLLM configurations.
* `VLLM_API_KEY`: Same as `LLM_API_KEY`; used specifically for VLLM configurations.
* `MODELS`: The model(s) available in your VLLM deployment.
* `SIMULATION_MODEL`: The model used for simulation purposes.
* `ORCHESTRATOR_URL`: The orchestrator's URL (usually `https://orchestrator.chasm.net`).
* `WEBHOOK_API_KEY`: Your webhook API key obtained during registration.

**Note**: Do not include multiple supplier configurations in your `.env` file. Only include the settings relevant to VLLM.

#### Running the VLLM Dispute Scout

After configuring the `.env` file, follow these steps:

1. **Ensure VLLM is Running**: Verify that your VLLM server is operational and accessible at the URL specified in `LLM_BASE_URL`.
2. **Build the Docker Image**:

   ```bash
   docker compose build
   ```
3. **Run the Docker Container**:

   ```bash
   docker compose up -d
   ```
4. **Monitor the Logs**:

   ```bash
   docker compose logs -f
   ```

   Check for any errors and ensure that the Dispute Scout is running correctly.

#### Notes on VLLM Configuration

* **Model Availability**: Ensure that the models specified in `MODELS` and `SIMULATION_MODEL` are available in your VLLM deployment.
* **API Keys**: If your VLLM deployment does not require an API key, you can leave `LLM_API_KEY` and `VLLM_API_KEY` blank.
* **Endpoint Accessibility**: If the VLLM server is running locally, ensure there are no firewall or network restrictions preventing access from the Docker container.

#### Monitoring and Troubleshooting

* **Connection Issues**: If the Dispute Scout cannot connect to the VLLM API, verify that the `LLM_BASE_URL` is correct and accessible.
* **Authentication Errors**: If authentication fails, double-check your `LLM_API_KEY` and `VLLM_API_KEY`.
* **Model Errors**: If you encounter model-related errors, ensure the model names in your `.env` file match those in your VLLM deployment.
* **Logprob Support**: The VLLM Dispute Scout relies on logprob data. Ensure your VLLM deployment supports returning logprobs.

#### Advanced Configuration

**Setting the Minimum Confidence Score**

You can adjust the minimum confidence score required for filing a dispute to prevent false positives:

```bash
## Adjust the threshold as needed (default is 0.5)
MIN_CONFIDENCE_SCORE=0.7
```

Add this line to your `.env` file and adjust the value based on your requirements.

**Running Benchmarks**

To evaluate if a dispute will be filed with your current settings:

1. **Install Python and Dependencies**:

   ```bash
   sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv
   ```
2. **Create a Virtual Environment**:

   ```bash
   python3 -m venv dispute-scout
   source dispute-scout/bin/activate
   ```
3. **Install Requirements**:

   ```bash
   pip install -r requirements.txt
   ```
4. **Run the Benchmark Script**:

   ```bash
   python benchmark.py
   ```

   This script will help you assess the impact of your settings on dispute filings.

#### Additional Resources

* **About VLLM** : [VLLM Official Documentation](https://vllm.ai/docs)
* **VLLM GitHub Repository**: [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)

#### Troubleshooting Tips

* **Docker Daemon Not Running**: If you encounter an error stating that Docker cannot connect to the daemon, start Docker with:

  ```bash
  sudo systemctl start docker
  ```
* **Check Container Status**: Use `docker ps` to list running containers and ensure the Dispute Scout is active.
* **Log Analysis**: Regularly check the logs using `docker compose logs -f` to monitor the Dispute Scout's activity and catch any issues early.