LLM Output Annotator: Build a Tool That Labels LLM Responses with Quality Tags

Question

## Problem

You are building an annotation tool that takes an LLM-generated response and a reference answer, and labels the response with quality tags from a predefined set: `["correct", "partially_correct", "incorrect", "hallucinated", "refused", "off_topic"]`. Implement the annotation logic using simple heuristics (exact match, keyword overlap, refusal detection).

```python
def annotate_response(
    response: str,
    reference: str,
    question: str
) -> list[str]:  # list of applicable tags
    pass
```

**Example:**
```
reference = "The capital of France is Paris."
response  = "Paris is the capital of France."
-> ["correct"]

response  = "I cannot answer that question."
-> ["refused"]

response  = "The capital of France is Lyon."
-> ["incorrect"]

response  = "France is a country in Europe with a rich culture..."
-> ["off_topic"]
```

## Follow-ups

1. What NLP metric (BLEU, ROUGE, BERTScore) gives a better signal than keyword overlap for partial correctness?
2. How would you detect hallucinations systematically when no ground-truth reference is available?
3. If you scale this to annotate 1 million response pairs, what infrastructure would you use (batch jobs, streaming pipeline)?
4. How do you handle multi-hop questions where the correct answer requires combining multiple facts?

LLM Output Annotator: Build a Tool That Labels LLM Responses with Quality Tags

Interview Experience

Full Details

About This Question

Topics

More Harvey Interview Questions