LLM Bot

A GUI client that bridges any LLM provider to the Triumvirate chess server. Version 2.2. View on GitHub →

Overview

Triumvirate LLM Bot v2.2 is an autonomous, fully graphical client-bot for playing three-player chess. On each turn the bot:

1. Gets the board state from the server
2. Builds a multi-layered, configurable prompt
3. Sends the request to the chosen LLM
4. Parses the response and extracts a valid chess move
5. Sends the move to the server

After pressing Start, the bot plays fully automatically until the game ends.

Key Features

• 6+ LLM providers (Ollama, OpenAI, Anthropic, OpenRouter, LM Studio, custom URL)
• 3 response formats (simple text, JSON, JSON with reasoning)
• Retry with escalation on illegal moves + random fallback
• Full per-move cost tracking (OpenRouter pricing)
• Move tracing: every move saved as JSON with complete audit trail
• Multi-bot mode: run N bots in parallel with different models
• NiceGUI desktop/web interface

Installation & Setup

# Clone the repository
git clone https://github.com/fortser/Triumvirate_LLMbot.git
cd Triumvirate_LLMbot/examples/decomp

# Install dependencies
pip install -r requirements.txt

# Set API keys (optional, depending on provider)
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export OPENROUTER_API_KEY=sk-or-...

# Run (desktop window)
python main.py

# Run as web server
python main.py --web --port 8090

# Run headless (no GUI)
python main.py --headless

Command-Line Options

LLM Providers

Prompt System

The bot builds prompts from configurable templates stored in prompts/:

• system_prompt.txt — System message (chess rules, role description)
• user_prompt_template.txt — User message template with placeholders
• format_simple.txt — Instructions for simple text response
• format_json.txt — Instructions for JSON response
• format_json_thinking.txt — Instructions for JSON with reasoning

Template Placeholders

Response Formats & Parsing

Simple Text

Model responds with plain text like E2 E4. Parser uses regex to extract coordinates.

JSON

Model responds with {"move_from": "E2", "move_to": "E4"}. Parser extracts keys directly.

JSON with Thinking

Model includes analysis: {"thinking": "Analysis...", "move_from": "E2", "move_to": "E4"}. Thinking is saved in trace logs for analysis.

Reliability & Fallbacks

• Retries with escalation — On illegal move, repeats the request (up to N attempts). On the last attempt, simplifies the prompt with a direct list of legal moves.
• Random fallback — If the LLM cannot produce a valid move, selects a random legal move (configurable, can be disabled).
• JSON sanitization — Handles malformed JSON responses from LLMs.

Cost Tracking

For OpenRouter, pricing is automatically loaded from the API. Per-move cost is calculated from input, output, and reasoning tokens. Cumulative stats are shown when the bot stops.

Move Tracing

Every move is saved as a JSON file in logs/game_<id>/move_NNN.json containing:

• Raw board state from server
• Rendered multi-layer prompt
• All LLM requests and responses with token counts and cost
• All parsing attempts with intermediate results
• Final move choice and server response

Trace Analyzer

A separate web app + CLI tool for analyzing trace logs:

# Launch trace viewer
cd trace_analyzer && python app.py --logs ../logs --port 8091

# Generate metrics and model rankings
python -m trace_analyzer.metrics

# With SmartBot objective evaluation
python -m trace_analyzer.metrics --smartbot

The viewer has 4 tabs: Overview (dashboard), Moves Table, Thinking Gallery, Move Detail.

Multi-Agent Evaluation

A system of three components for analyzing LLM play quality:

Composite Score

Models are ranked by a composite score (0–1):

Components (without SmartBot): Reliability 35% + Activity 30% + Tactical 20% + Efficiency 15%

Components (with SmartBot): Reliability 20% + SmartBot Quality 35% + Tactical 15% + Efficiency 10% + Win Rate 20%

Triumvirate Notation in the LLM Bot

The LLM Bot supports two notation modes for prompts. The notation mode is configured in the settings panel or via the config file.

Notation Comparison in Prompts

How the Converter Works

The bot's notation_converter.py module translates between the two systems. When Triumvirate mode is active:

1. Server sends /state response in classic notation (e.g., E2)
2. Converter translates board and legal_moves to Triumvirate (e.g., W2/R2.3)
3. Prompt is built with Triumvirate coordinates and piece names
4. LLM responds with a Triumvirate move (e.g., W2/R2.3 to W1/R1.3)
5. Converter translates back to classic notation for the /move API call

Key Positions in Both Notations

Multi-Bot Mode

# Run 3 bots with random models from pool
python main.py --headless --bots 3

# Run 5 bots with specific models
python main.py --headless --models openai/gpt-4o anthropic/claude-haiku-4-5-20251001

# Custom model pool with staggered start
python main.py --headless --bots 5 --models-pool custom_pool.json --start-delay 10