ai-evals-course
ai-evals-course / validate-evaluator
1.3kCalibrate an LLM judge against human labels using data splits, TPR/TNR, and bias correction. Use after writing a judge prompt (write-judge-prompt) when you need to verify alignment before trusting its outputs.
ai-evals-course / evaluate-rag
1.3kGuides evaluation of RAG pipeline retrieval and generation quality
ai-evals-course / write-judge-prompt
1.3kDesign LLM-as-Judge evaluators for subjective criteria that code-based checks cannot handle
ai-evals-course / error-discovery
1.3kRun error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.
ai-evals-course / generate-synthetic-data
1.3kCreate diverse synthetic test inputs for LLM pipeline evaluation using dimension-based tuple generation
ai-evals-course / build-review-interface
1.3kBuild a custom browser-based annotation interface tailored to your data for reviewing LLM traces and collecting structured feedback
ai-evals-course / eval-audit
1.3kAudit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc
ai-evals-course / ui
Rules for implementing common UI elements in MonsterUI idiomatically