The Guidelines Problem
Every data labelling project starts the same way:
"Just label whether this is positive or negative. It's simple."
Two weeks later:
"Why did you label this as positive? It's clearly neutral!" "The guidelines didn't say what to do with sarcasm!" "I thought 'okay' was positive, you thought it was neutral!"
Bad guidelines create inconsistent data. Inconsistent data creates unreliable models.
Anatomy of Good Guidelines
After running hundreds of labeling projects, here's our framework for guidelines that work:
1. Clear Objective Statement
Start with WHY. Labelers who understand the goal make better judgment calls.
Bad:
Label each text as Positive, Negative, or Neutral.
Good:
You are labeling customer support messages to help train a
sentiment classifier. This classifier will route angry
customers to senior agents and satisfied customers to
upsell workflows.
Your labels directly impact customer experience—accurate
labeling means customers get the right help faster.
2. Precise Category Definitions
Define each label with:
- Core definition: One sentence explaining the category
- Inclusion criteria: What DOES belong in this category
- Exclusion criteria: What does NOT belong
- Boundary cases: How to handle edge cases
Example:
POSITIVE
Definition: Customer expresses satisfaction, gratitude, or
positive emotion about the product or service.
Include:
- Explicit thanks or praise
- Statements of satisfaction
- Recommendations or endorsements
- Positive comparisons to alternatives
Exclude:
- Neutral statements about features
- Sarcastic praise (see NEGATIVE)
- Mixed messages (see decision tree)
Boundary:
- "It works" → NEUTRAL (functional statement, no emotion)
- "It works great!" → POSITIVE (emotional emphasis)
- "Finally, it works" → Could be either, check context
3. Decision Trees for Ambiguity
When categories overlap, provide explicit decision logic:
Is there explicit emotional language?
├── YES → What emotion?
│ ├── Positive → POSITIVE
│ ├── Negative → NEGATIVE
│ └── Mixed → Go to "Mixed Messages" section
└── NO → Is there implicit sentiment?
├── YES → What's the implication?
│ ├── Satisfied → POSITIVE
│ └── Frustrated → NEGATIVE
└── NO → NEUTRAL
4. Abundant Examples
Provide examples for every category, including:
- Clear examples: Obvious cases to establish baseline
- Borderline examples: Cases that almost went the other way
- Wrong examples: Common mistakes with explanations
POSITIVE Examples:
Clear:
"Thank you so much! This solved my problem completely!"
Why: Explicit gratitude + satisfaction
Borderline:
"I guess it works now."
Why: Functional success but grudging tone. We label
POSITIVE because the problem is resolved, but note
this is borderline.
Common mistake:
"Great, another update that breaks everything."
Why: This is NEGATIVE (sarcasm). The "Great" is ironic.
5. Handling Uncertainty
Tell labelers what to do when they're unsure:
When you're uncertain:
1. Re-read the context (previous messages if available)
2. Check the decision tree
3. Look for the closest example in guidelines
4. If still unsure, flag for review with a note
DO NOT:
- Guess randomly
- Default to NEUTRAL for everything unclear
- Skip the sample
We track uncertainty flags. Frequent flags on the same
pattern means we need to update guidelines.
6. Version Control
Guidelines evolve. Track changes:
Version History:
v2.3 (2024-01-15)
- Added: Sarcasm detection section
- Clarified: "It works" examples
- Fixed: Emoji interpretation rules
v2.2 (2024-01-08)
- Added: Multi-language handling
- Updated: Response time expectations
v2.1 (2024-01-02)
- Initial production release
The Calibration Process
Good guidelines aren't written once. They're calibrated through iteration:
Step 1: Draft Guidelines
Write initial guidelines based on project goals and sample data review.
Step 2: Pilot Labeling
Have 3-5 labelers annotate 100 samples independently.
Step 3: Measure Agreement
Calculate inter-annotator agreement. Identify disagreements.
Step 4: Disagreement Review
For each disagreement:
- What did different labelers think?
- Why did they interpret differently?
- What's the correct answer?
- How do we prevent this confusion?
Step 5: Update Guidelines
Add clarifications, examples, and decision rules based on findings.
Step 6: Re-Calibrate
Repeat steps 2-5 until agreement exceeds 0.85 Kappa.
Common Guidelines Mistakes
Mistake 1: Too Abstract
"Label the sentiment" leaves too much to interpretation. Fix: Provide concrete, operational definitions.
Mistake 2: Too Rigid
"Always label X as Y" fails on edge cases. Fix: Provide decision frameworks, not just rules.
Mistake 3: No Examples
Definitions without examples are incomplete. Fix: Minimum 5 examples per category, including borderlines.
Mistake 4: Ignoring Edge Cases
The weird cases will come up in production. Fix: Explicitly address known edge cases and what to do.
Mistake 5: Never Updating
Language and patterns evolve. Guidelines get stale. Fix: Review and update guidelines monthly.
Guidelines Template
Here's a starter template for your own projects:
# [Project Name] Annotation Guidelines
Version: X.X | Last Updated: YYYY-MM-DD
## 1. Project Overview
[Why are we labeling? What will this data be used for?]
## 2. Categories
### Category A
- Definition:
- Include:
- Exclude:
- Boundary cases:
- Examples:
### Category B
[Same structure]
## 3. Decision Tree
[Flowchart for ambiguous cases]
## 4. Handling Uncertainty
[What to do when unsure]
## 5. Common Mistakes
[Known pitfalls and how to avoid them]
## 6. FAQs
[Answers to frequent labeler questions]
## 7. Version History
[Track all changes]
Need help building annotation guidelines for your project? Book a discovery call to discuss your data labelling needs.