test(evals): run the evals against any server named by EVAL_JUDGE_BASE_URL, including /v1
Ten eval files pinned their reply engine to http://localhost:11434 and the judge only spoke Ollama's /api/tags and /api/chat, so pointing the suite at Strata (OpenAI-compatible /v1 only) skipped or broke most evals. - MockConfig and every eval config read EVAL_JUDGE_BASE_URL. - The judge availability check reads /v1/models and the judge call goes through jarvis.llm when the URL ends in /v1; the Ollama path is unchanged. - run_evals.sh probes /models for a /v1 URL. The intent-judge and listener evals stay pinned to gemma4:e2b on local Ollama. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@
|
||||
#
|
||||
# Environment variables:
|
||||
# EVAL_JUDGE_MODEL - Model to use for LLM-as-judge (default: gpt-oss:20b)
|
||||
# EVAL_JUDGE_BASE_URL - Ollama base URL (default: http://localhost:11434)
|
||||
# EVAL_JUDGE_BASE_URL - LLM server URL (default: http://localhost:11434); ending in /v1 uses the OpenAI-compatible API (Strata)
|
||||
# EVAL_REPEAT_COUNT - Number of times to run each test (default: 1; use 3 when tuning prompts to surface flakiness)
|
||||
|
||||
set -e
|
||||
@@ -35,7 +35,9 @@ echo ""
|
||||
# Check if Ollama is available
|
||||
OLLAMA_AVAILABLE=false
|
||||
OLLAMA_URL="${EVAL_JUDGE_BASE_URL:-http://localhost:11434}"
|
||||
if curl -s "${OLLAMA_URL}/api/tags" > /dev/null 2>&1; then
|
||||
PROBE_PATH="/api/tags"
|
||||
case "${OLLAMA_URL%/}" in */v1) PROBE_PATH="/models" ;; esac
|
||||
if curl -s "${OLLAMA_URL%/}${PROBE_PATH}" > /dev/null 2>&1; then
|
||||
OLLAMA_AVAILABLE=true
|
||||
echo " ✅ Ollama detected at ${OLLAMA_URL}"
|
||||
else
|
||||
|
||||
Reference in New Issue
Block a user