test(evals): run the evals against any server named by EVAL_JUDGE_BASE_URL, including /v1

Ten eval files pinned their reply engine to http://localhost:11434 and the
judge only spoke Ollama's /api/tags and /api/chat, so pointing the suite at
Strata (OpenAI-compatible /v1 only) skipped or broke most evals.

- MockConfig and every eval config read EVAL_JUDGE_BASE_URL.
- The judge availability check reads /v1/models and the judge call goes
  through jarvis.llm when the URL ends in /v1; the Ollama path is unchanged.
- run_evals.sh probes /models for a /v1 URL.

The intent-judge and listener evals stay pinned to gemma4:e2b on local Ollama.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
javis-bot
2026-11-11 18:49:08 +09:00
parent a45e198df6
commit 69a884b182
13 changed files with 167 additions and 50 deletions

View File

@@ -12,7 +12,7 @@
#
# Environment variables:
# EVAL_JUDGE_MODEL - Model to use for LLM-as-judge (default: gpt-oss:20b)
# EVAL_JUDGE_BASE_URL - Ollama base URL (default: http://localhost:11434)
# EVAL_JUDGE_BASE_URL - LLM server URL (default: http://localhost:11434); ending in /v1 uses the OpenAI-compatible API (Strata)
# EVAL_REPEAT_COUNT - Number of times to run each test (default: 1; use 3 when tuning prompts to surface flakiness)
set -e
@@ -35,7 +35,9 @@ echo ""
# Check if Ollama is available
OLLAMA_AVAILABLE=false
OLLAMA_URL="${EVAL_JUDGE_BASE_URL:-http://localhost:11434}"
if curl -s "${OLLAMA_URL}/api/tags" > /dev/null 2>&1; then
PROBE_PATH="/api/tags"
case "${OLLAMA_URL%/}" in */v1) PROBE_PATH="/models" ;; esac
if curl -s "${OLLAMA_URL%/}${PROBE_PATH}" > /dev/null 2>&1; then
OLLAMA_AVAILABLE=true
echo " ✅ Ollama detected at ${OLLAMA_URL}"
else