Table of Contents
Your candidate aced the coding challenge. But can they build a RAG pipeline that actually retrieves relevant documents? Do they know how to evaluate LLM outputs for hallucinations? Have they ever deployed a fine-tuned model to production? Here are questions that separate real LLM engineers from weekend prompters.
RAG & Retrieval Questions
Look for: document chunking strategy, embedding model selection, vector database choice, hybrid search (keyword + vector), re-ranking, and evaluation of retrieval precision/recall.
Retrieval thresholding, answer faithfulness detection, or fallback responses. A strong candidate discusses preventing hallucination when context is insufficient.
Evaluation & Testing Questions
Evaluation framework: answer correctness, faithfulness/hallucination rate, latency, cost. LLM-as-judge, human evaluation, or labeled test sets.
Diagnose: Is it retrieval (wrong context), prompt (ambiguous instructions), or model capability? Solutions: improve retrieval, add few-shot examples, or fine-tune.
Fine-Tuning Questions
Fine-tuning when: need consistent formatting/tone, domain-specific terminology, or when prompt engineering + RAG isn't enough. Also for cost reduction (smaller model).
Data collection (existing conversations, human labeling), diversity, quality filtering, format (instruction-output pairs or chat format), train/validation split.
Production & Deployment Questions
Model quantization, smaller models, caching (semantic or exact), batching, speculative decoding, choosing right provider (Groq for speed, Together for price).
Latency (p50/p95/p99), token usage/cost, success rate, hallucination rate, user feedback, retrieval precision/recall.
Signals of a Strong Candidate
- ✦ Discusses trade-offs instead of memorized answers
- ✦ References production incidents and lessons learned
- ✦ Explains evaluation methodology clearly
- ✦ Can quantify latency, cost, and accuracy improvements
- ✦ Understands retrieval and generation as separate systems
Warning Signs During Interviews
- ✦ Only discusses prompting techniques
- ✦ Cannot explain retrieval metrics
- ✦ Has never evaluated model outputs systematically
- ✦ Cannot estimate token or infrastructure costs
- ✦ Lacks examples of production deployments
Test What Actually Matters
LLM engineering is about building systems, not solving algorithms. Test for RAG, evaluation, fine-tuning, and production skills. Offline Pixel pre-vets all this before you interview. Raise a request, talk to qualified candidates, fund the project, and approve payment when the work is done.
Continue reading
Need an LLM engineer who can answer these?
Raise a request → Talk to experts → Fund the project → Expert works → Review & approve payment
Hire LLM Engineer