Dispute Scout Setup Guide
Last updated
Last updated
Disputes are used for fault detection in the opML mechanism. Scout Disputers are required to have a higher technical level of understanding how an opML dispute works, as well as understanding on how to set the dispute scouts up manually.
Scouts running the dispute module monitor the inference results produced by other Scouts in the network. They raise a Dispute whenever they suspect discrepancies between the reported results and the actual results.
Running a Dispute Scout instance is computationally intensive, particularly in terms of inference requirements. To manage costs effectively, users should consider the following:
Local Hardware Option: Ideally, users should use Ollama to run the Dispute Scout on their own hardware. This approach helps prevent inference costs from accumulating rapidly.
Cost Monitoring: If using cloud-based solutions such as Openrouter or Groq, regularly monitor your usage and associated costs to avoid unexpected expenses.
A simple guide is provided on how to run Ollama locally.
Please refer to the Inference Scout Setup Guide in order to obtain your WEBHOOK_API_KEY. Note that WEBHOOK_URL is replaced with LLM_BASE_URL for dispute scouts.
Install Docker: Follow the Docker Installation Guide
Install Docker Compose: Follow the Docker Compose Installation Guide
Git clone the repository, and enter the repository:
git clonehttps://github.com/ChasmNetwork/chasm-scoutand cd chasm-scout/dispute
Set up the environment file: Use nano .env or vim .env to create a file with the following content, depending on your chosen model and supplier. Choose ONE of the following supplier options:
Ollama (local GPU):
Groq:
OpenRouter:
The SIMULATION_MODEL is set to meta-llama/llama-3-8b-instruct as an example, but users are free to test other models in the list above.
Do not put all three entries above - pick the supplier you want to go with, and simply copy that to put into your .env file.
Make sure you're in chasm-scout/dispute, otherwise change your directory into chasm-scout/dispute by doing a cd dispute from the root folder.
Build the Docker Image for the dispute scout by running docker compose build.If you get any errors, make sure you've installed Docker and Docker Compose as per the instructions above.
Run the Docker image you've just built by running docker compose up -d
Check if everything is going well by running docker compose logs.
If you get an error saying Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?, start your docker instance by running sudo systemctl start docker
Additional .env variables: You can add the following variable to your .env file:
This variable sets the minimum confidence score required to prevent inaccurate dispute reports due to potential model inaccuracies, especially when using smaller models. The default value is 0.5. Run the benchmark.py script via python benchmark.pyto see if the values work for you.
The dispute/strategies/ folder contains strategies for determining disputes:
StaticTextAnalysisStrategy
SemanticSimilarityAnalysis
LLMQualityStrategy
ResponseSimilarityAnalysis
A detailed publication on these strategies is forthcoming.
To use the additional scripts provided like benchmark.py, you have to install the dependencies as well as Python. A rough guide is provided below, but users venturing here are expected to know more than average.
Install Python and dependencies:
Create and activate a virtual environment:
Install requirements:
Run the benchmark:
The output will help you evaluate if a dispute will be filed with your current settings.