The Quality Problem
Everyone says "data quality matters." Few can define what quality means or how to measure it.
For LLM training data, we've developed a rigorous quality framework based on six dimensions.
The Six Dimensions of Data Quality
1. Accuracy
Definition: Labels correctly represent the ground truth.
How we measure:
- Expert audit score: Domain experts review random samples
- Cross-validation: Compare labels against held-out test sets
- Production feedback: Track downstream model errors
Target: 95%+ accuracy for production datasets.
2. Consistency
Definition: Similar inputs receive similar labels.
How we measure:
- Inter-annotator agreement (IAA): Multiple labelers annotate same samples
- Cohen's Kappa: Statistical measure of agreement beyond chance
- Temporal consistency: Same labeler labels same sample twice
Target: Kappa score > 0.8 (strong agreement).
# Cohen's Kappa interpretation
0.0-0.2: Slight agreement
0.2-0.4: Fair agreement
0.4-0.6: Moderate agreement
0.6-0.8: Substantial agreement
0.8-1.0: Almost perfect agreement
3. Completeness
Definition: Dataset covers the full distribution of expected inputs.
How we measure:
- Category coverage: All target categories represented
- Edge case inclusion: Unusual inputs explicitly included
- Distribution matching: Dataset mirrors production query distribution
Target: <5% of production queries fall outside training distribution.
4. Relevance
Definition: Samples are useful for the target task.
How we measure:
- Task alignment score: Expert rating of sample-task fit
- Noise ratio: Percentage of samples that add no learning signal
- Duplication rate: Percentage of near-duplicate samples
Target: <10% noise/duplicate rate.
5. Timeliness
Definition: Data reflects current state of the domain.
How we measure:
- Data age distribution: When were source documents created?
- Terminology currency: Are labels using current terminology?
- Fact accuracy: Do factual claims remain true?
Target: 80%+ of data from last 2 years for dynamic domains.
6. Provenance
Definition: Data origin is documented and trustworthy.
How we measure:
- Source documentation: Can we trace each sample to its origin?
- Rights verification: Do we have rights to use this data?
- Bias audit: What biases might sources introduce?
Target: 100% provenance documentation for regulated industries.
Quality Scoring Framework
We combine these dimensions into a single quality score:
Quality Score = (
Accuracy × 0.25 +
Consistency × 0.20 +
Completeness × 0.20 +
Relevance × 0.15 +
Timeliness × 0.10 +
Provenance × 0.10
) × 100
Score interpretation:
- 90-100: Production ready
- 80-89: Minor improvements needed
- 70-79: Significant rework required
- <70: Dataset rebuild recommended
Automated Quality Checks
We run automated checks throughout the labeling pipeline:
Pre-Ingestion Checks
def check_pre_ingestion(data):
checks = {
"format_valid": validate_format(data),
"encoding_correct": check_encoding(data),
"no_corruption": verify_integrity(data),
"size_reasonable": check_file_size(data)
}
return all(checks.values()), checks
Post-Labelling Checks
def check_post_labelling(dataset):
checks = {
"all_labelled": no_missing_labels(dataset),
"labels_valid": labels_in_schema(dataset),
"no_duplicates": check_duplicates(dataset),
"distribution_ok": check_distribution(dataset)
}
return all(checks.values()), checks
Continuous Monitoring
def monitor_quality(batch):
alerts = []
if compute_iaa(batch) < 0.8:
alerts.append("Low inter-annotator agreement")
if compute_accuracy(batch) < 0.95:
alerts.append("Accuracy below threshold")
if compute_noise_ratio(batch) > 0.1:
alerts.append("High noise ratio detected")
return alerts
Quality Improvement Playbook
When quality scores drop, here's how we respond:
For Accuracy Issues
- Review annotation guidelines for ambiguity
- Retrain labelers on problem categories
- Add more examples to guidelines
- Consider splitting ambiguous categories
For Consistency Issues
- Run calibration sessions with labelers
- Identify and resolve edge case disagreements
- Add explicit rules for common confusion points
- Consider adjudication process for disputes
For Completeness Issues
- Analyze production queries for gaps
- Synthetically generate underrepresented examples
- Source additional data for missing categories
- Adjust sampling strategy for balance
The Quality Investment
High-quality data isn't free. Here's what it takes:
| Quality Level | Investment | Typical Use Case | |---------------|------------|------------------| | Basic (70-80) | $50-100/1k samples | Prototypes, demos | | Standard (80-90) | $150-300/1k samples | Internal tools | | Premium (90-95) | $400-700/1k samples | Customer-facing AI | | Golden (95+) | $800-1500/1k samples | Regulated industries |
The ROI is in downstream model performance. A 10% improvement in data quality can mean a 20%+ improvement in model accuracy.
Need help measuring and improving your data quality? Book a discovery call to discuss your dataset assessment.