Search skills/

google-agents-cli-eval skill

Teaches an agent to evaluate ADK agents: dataset schema, built-in and custom metrics, LLM-as-judge scoring and an eval-fix loop.

in google/agents-cli Planning, TDD & debuggingSkill & agent tooling
Our verdict Recommended

A thorough eval methodology with concrete schemas and metrics; best used inside an agents-cli project.

4/5 Depth expertise shipped Note Security Review before installing 2d Maintenance Updated 2 days ago 519K Adoption installs OK License Apache-2.0, commercial OK

What the google-agents-cli-eval skill does

The eval skill describes how to measure and improve an agent with agents-cli. It frames work as a four-stage Quality Flywheel: prepare data, run the eval, analyze failures, and optimize. The default path has the coding agent run agents-cli eval run, which generates traces from a dataset and grades them, writing JSON and HTML results; opt-in commands hand analysis, optimization and dataset synthesis to the Eval Service on the Gemini platform.

It is a 25 KB SKILL.md with seven reference files: the canonical dataset schema with single-turn, multi-turn and multi-agent examples, a metrics guide with match types and judge configuration, user simulation for synthesized conversations, built-in tool evaluation, advanced commands, multimodal evals and live or voice agents. Scaffolded projects already contain a starter dataset and eval config, and the skill tells the agent to start with one or two cases. It lists common failure causes so the agent fixes the agent or the dataset rather than loosening the metric.

Good for

  • Adding repeatable evals to an ADK or agents-cli project
  • Improving an agent through a measured eval-fix loop

Not for

  • Evaluating agents outside agents-cli projects
  • One-off manual testing

Depth 4/5

A 25 KB SKILL.md with seven reference files and some test material, no scripts of its own.

Dependencies and costs

  • agents-cli (uv tool install google-agents-cli), Python 3.11+, uv and Node.js
  • A Google Cloud project and credentials for deployment and cloud features (an AI Studio API key is enough for local create, run and eval)
  • Telemetry: The agents-cli tool sends one usage signal per command (command name, exit code, error class, duration, CLI and Python versions, OS), written as a log entry to your own Google Cloud project's Cloud Logging, never file contents or arguments. Opt out with AGENTS_CLI_TELEMETRY=0; agents-cli info shows the current state.

Security review

Review before installing

Markdown-only guidance on running agent evaluations (datasets, metrics, LLM-as-judge) with agents-cli; it has no scripts. Eval commands run your agent and send prompts to Google model endpoints, and read the agent's .env for configuration. The agents-cli tool was not reviewed and sends usage telemetry unless you opt out.

Worth knowing

  • Relies on a CLI or package we could not reviewRequires the agents-cli binary (install: uv tool install google-agents-cli), whose code is not in the skill folder.SKILL.md:19
  • Installs packages or toolsInstall step is uv tool install google-agents-cli.SKILL.md:24
  • Makes network requestsEval commands call the user's agent (local or a deployed URL) and Google-hosted judge models via agents-cli.SKILL.md:164
  • Sends usage data by default (can be turned off)The agents-cli tool this skill drives sends per-command usage telemetry by default; the repo README (outside the skill folder) documents AGENTS_CLI_TELEMETRY=0 and DO_NOT_TRACK=1 to opt out.README.md:195

What we checked

The skill folder (8 files) on Oct 29, 2026. Not a full review: it relies on code outside the repository: agents-cli (PyPI package google-agents-cli, installed with uv tool install).

NVIDIA SkillSpector static scan: 1 pattern hits (Excessive Agency 1). The single hit is an example command choosing a judge model (--model gemini-2.5-pro); credential matches are notes that eval commands read the agent's own .env and a mock-API-key example.

A review, not a certification. Skills change after we check them — read the files before installing.

License

Apache-2.0 — commercial use allowed.

Can it run unattended?

Yes. The eval-fix loop is iterative and repeatable with a dataset, though LLM-judge runs cost tokens and cloud calls.

Alternatives

Mastra Agent Skills

mastra-ai/skills

Mastra's official skills: a framework guide that routes to current docs and ten references, and a Factory supervisor skill for the mastra api CLI.

With caveats Broad access Depth 3/5

Google Agent Skills

google/skills

Google's official collection of about 180 skills for Google Cloud, Gemini APIs, Analytics, Data Manager, sign-in and other Google products.

Recommended Broad access Depth 3/5

FAQ

What is google-agents-cli-eval?

Teaches an agent to evaluate ADK agents: dataset schema, built-in and custom metrics, LLM-as-judge scoring and an eval-fix loop.

How do I install google-agents-cli-eval?

It ships in the google/agents-cli repository. Run uvx google-agents-cli setup or npx skills add google/agents-cli; the skill expects the agents-cli binary (uv tool install google-agents-cli).

Does google-agents-cli-eval send my data anywhere?

By default evals run through agents-cli locally against your dataset, using a judge model. The opt-in analyze, optimize and submit commands delegate to the Eval Service in Google Cloud, so traces and datasets go there.

Sources: GitHub repository; install counts from skills.sh via Best Skills (CC BY 4.0). Review based on the published SKILL.md.