# VLLM Setup Guide

### Adding VLLM as a Dispute Scout Provider

With the latest update, the Dispute Scout now supports VLLM as a provider. This allows you to utilize VLLM models for advanced logprob-based dispute detection.

#### Prerequisites for VLLM Dispute Scout

To set up the VLLM Dispute Scout, ensure you have:

* **VLLM Deployment**: A running instance of VLLM, either locally or accessible via a remote endpoint.
* **VLLM API Endpoint URL (`VLLM_URL`)**: The base URL where your VLLM API is accessible.
* **VLLM API Key (`VLLM_API_KEY`)**: If your VLLM deployment requires authentication.

#### Setting Up VLLM

**Option 1: Local VLLM Deployment**

1. **Install VLLM**: Follow the official installation guide in the [VLLM GitHub Repository](https://github.com/vllm-project/vllm) to set up VLLM on your machine.
2. **Start the VLLM Server**: Run the VLLM server with your desired model. For example:

`HF_TOKEN=<your-huggingface-token> python -m vllm.entrypoints.openai.api_server --model repo_id --max-logprobs 120`

By default, the server runs at `http://localhost:8000`.

**Option 2: Remote VLLM Deployment**

If you have access to a remote VLLM API endpoint:

* Ensure you have the base URL and API key (if required).
* Verify that the endpoint is accessible from your network.

#### Configuring the `.env` File for VLLM

In the `chasm-scout/dispute` directory, create or modify your `.env` file to include the VLLM configuration. Replace placeholders with your actual values.

```bash
## VLLM Configuration

# API key for VLLM (leave blank if not required)
LLM_API_KEY=your_vllm_api_key_or_leave_blank

# Base URL for VLLM API
LLM_BASE_URL=http://localhost:8000/v1

# VLLM-specific configurations
VLLM_URL=http://localhost:8000/v1
VLLM_API_KEY=your_vllm_api_key_or_leave_blank

# Models to use (comma-separated if multiple)
MODELS=llama-2-7b-chat

# Model used for simulation
SIMULATION_MODEL=llama-2-7b-chat

# Orchestrator URL
ORCHESTRATOR_URL=https://orchestrator.chasm.net

# Your webhook API key
WEBHOOK_API_KEY=your_webhook_api_key
```

**Explanation of Variables:**

* `LLM_API_KEY`: API key for your VLLM instance (if required).
* `LLM_BASE_URL`: Base URL of your VLLM API endpoint.
* `VLLM_URL`: Same as `LLM_BASE_URL`; used specifically for VLLM configurations.
* `VLLM_API_KEY`: Same as `LLM_API_KEY`; used specifically for VLLM configurations.
* `MODELS`: The model(s) available in your VLLM deployment.
* `SIMULATION_MODEL`: The model used for simulation purposes.
* `ORCHESTRATOR_URL`: The orchestrator's URL (usually `https://orchestrator.chasm.net`).
* `WEBHOOK_API_KEY`: Your webhook API key obtained during registration.

**Note**: Do not include multiple supplier configurations in your `.env` file. Only include the settings relevant to VLLM.

#### Running the VLLM Dispute Scout

After configuring the `.env` file, follow these steps:

1. **Ensure VLLM is Running**: Verify that your VLLM server is operational and accessible at the URL specified in `LLM_BASE_URL`.
2. **Build the Docker Image**:

   ```bash
   docker compose build
   ```
3. **Run the Docker Container**:

   ```bash
   docker compose up -d
   ```
4. **Monitor the Logs**:

   ```bash
   docker compose logs -f
   ```

   Check for any errors and ensure that the Dispute Scout is running correctly.

#### Notes on VLLM Configuration

* **Model Availability**: Ensure that the models specified in `MODELS` and `SIMULATION_MODEL` are available in your VLLM deployment.
* **API Keys**: If your VLLM deployment does not require an API key, you can leave `LLM_API_KEY` and `VLLM_API_KEY` blank.
* **Endpoint Accessibility**: If the VLLM server is running locally, ensure there are no firewall or network restrictions preventing access from the Docker container.

#### Monitoring and Troubleshooting

* **Connection Issues**: If the Dispute Scout cannot connect to the VLLM API, verify that the `LLM_BASE_URL` is correct and accessible.
* **Authentication Errors**: If authentication fails, double-check your `LLM_API_KEY` and `VLLM_API_KEY`.
* **Model Errors**: If you encounter model-related errors, ensure the model names in your `.env` file match those in your VLLM deployment.
* **Logprob Support**: The VLLM Dispute Scout relies on logprob data. Ensure your VLLM deployment supports returning logprobs.

#### Advanced Configuration

**Setting the Minimum Confidence Score**

You can adjust the minimum confidence score required for filing a dispute to prevent false positives:

```bash
## Adjust the threshold as needed (default is 0.5)
MIN_CONFIDENCE_SCORE=0.7
```

Add this line to your `.env` file and adjust the value based on your requirements.

**Running Benchmarks**

To evaluate if a dispute will be filed with your current settings:

1. **Install Python and Dependencies**:

   ```bash
   sudo apt-get update && sudo apt-get install -y python3 python3-pip python3-venv
   ```
2. **Create a Virtual Environment**:

   ```bash
   python3 -m venv dispute-scout
   source dispute-scout/bin/activate
   ```
3. **Install Requirements**:

   ```bash
   pip install -r requirements.txt
   ```
4. **Run the Benchmark Script**:

   ```bash
   python benchmark.py
   ```

   This script will help you assess the impact of your settings on dispute filings.

#### Additional Resources

* **About VLLM** : [VLLM Official Documentation](https://vllm.ai/docs)
* **VLLM GitHub Repository**: [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm)

#### Troubleshooting Tips

* **Docker Daemon Not Running**: If you encounter an error stating that Docker cannot connect to the daemon, start Docker with:

  ```bash
  sudo systemctl start docker
  ```
* **Check Container Status**: Use `docker ps` to list running containers and ensure the Dispute Scout is active.
* **Log Analysis**: Regularly check the logs using `docker compose logs -f` to monitor the Dispute Scout's activity and catch any issues early.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://network-docs.chasm.net/chasm-scout-season-0/dispute-scout-setup-guide/vllm-setup-guide.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.