Back

LLM Evaluation Algorithm Engineer

Responsibilities

  1. Participate in the continuous development and iteration of the LLM evaluation framework, covering evaluation execution, Benchmark integration, result aggregation, automated uploads, and visualization analysis, improving system stability, usability, and scalability.
  2. Build evaluation capabilities for Agent, Code, mathematics, knowledge, reasoning, PPL, and other scenarios, supporting rule-based evaluation, LLM-as-a-Judge, PPL/logprobs evaluation, multi-turn dialogue evaluation, and other paradigms.
  3. Continuously optimize evaluation task scheduling and automation workflows; participate in evaluation visualization platform development and improve ETL, DuckDB/Parquet storage, frontend/backend APIs, sample-level analysis, cross-model comparison, and trend tracking.
  4. Track mainstream open-source and industry Benchmarks; clean, adapt, version, and validate evaluation datasets and metrics to ensure fairness, reproducibility, and interpretability.
  5. Analyze model evaluation results in depth, identify strengths and weaknesses across knowledge, reasoning, code, mathematics, instruction following, long context, and Agent capabilities, and provide feedback for training iteration and data improvement.
  6. Collaborate with training, inference, data, and product teams to build a closed loop from model output to automated evaluation, result feedback, issue diagnosis, and capability improvement.

Qualifications

  1. Bachelor’s degree or above; Computer Science, Artificial Intelligence, Software Engineering, Mathematics, or related majors preferred.
  2. Proficiency with common development and cluster tools including Linux, Shell, Git, Docker/Singularity, and Slurm.
  3. Familiarity with common LLM Benchmarks and evaluation methods such as MMLU, CMMLU, CEval, BBH, GPQA, GSM8K, HumanEval, LiveCodeBench, SuperGPQA, and PPL.
  4. Strong system design capabilities and the ability to design reliable solutions for evaluation task scheduling, concurrent execution, result storage, log tracing, failure retries, and visualization analysis.
  5. Strong problem-solving skills and the ability to identify evaluation anomalies, performance bottlenecks, and result deviations from logs, metrics, samples, configurations, and code paths.
  6. Strong interest in LLM evaluation, model capability analysis, and exploring the boundaries of Agent/Code/reasoning capabilities, with willingness to continuously follow industry developments and open-source Benchmarks.

Preferred Qualifications

  1. Experience using or extending evaluation frameworks such as EvalScope, OpenCompass, lm-evaluation-harness, HELM, or LightEval.
  2. Experience with FastAPI, Next.js, DuckDB, Parquet, data platforms, or visualization systems.
  3. Experience with LLM training, post-training, data construction, model analysis, or online inference services.
  4. Experience with Code Agents, Agent Benchmarks, tool-use evaluation, or multi-turn task evaluation.
  5. Open-source contributions, technical blogs, evaluation reports, or Benchmark construction experience.
Deliver