129 lines
5.3 KiB
Markdown
129 lines
5.3 KiB
Markdown
# CNI Task - IQA Metrics Evaluation Results
|
||
|
||
## Overview
|
||
|
||
This document presents the comprehensive evaluation results of 15 Image Quality Assessment (IQA) metrics against human quality annotations for 81 identity document images.
|
||
|
||
**Evaluation Date:** 2025-08-26
|
||
**Total Images:** 81
|
||
**Total Metrics Evaluated:** 15
|
||
|
||
## Key Findings
|
||
|
||
- **9/15 metrics** have statistically significant Pearson correlations (p < 0.05)
|
||
- **10/15 metrics** have statistically significant Spearman correlations (p < 0.05)
|
||
- **Best performing metric:** DEQA with correlation 0.6185
|
||
- **Average absolute correlation:** 0.2713
|
||
|
||
## Top Performing Metrics
|
||
|
||
### 🏆 Best Overall Performer: DEQA
|
||
|
||
| Metric | Pearson Correlation | Spearman Correlation | Overall Score | Significance |
|
||
|--------|-------------------|---------------------|---------------|--------------|
|
||
| **deqa** | **0.6185** | **0.6059** | **0.6122** | ✅ Both |
|
||
| uranker | 0.3349 | 0.3909 | 0.3629 | ✅ Both |
|
||
| dbcnn | -0.3721 | -0.3489 | 0.3605 | ✅ Both |
|
||
| nrqm | -0.3493 | -0.3699 | 0.3596 | ✅ Both |
|
||
| brisque | -0.3159 | -0.3859 | 0.3509 | ✅ Both |
|
||
|
||
## Complete Rankings
|
||
|
||
| Rank | Metric | Pearson Corr | Spearman Corr | Overall Score | Significant |
|
||
|------|--------|--------------|---------------|---------------|-------------|
|
||
| 1 | **deqa** | **0.6185** | **0.6059** | **0.6122** | ✅ Both |
|
||
| 2 | uranker | 0.3349 | 0.3909 | 0.3629 | ✅ Both |
|
||
| 3 | dbcnn | -0.3721 | -0.3489 | 0.3605 | ✅ Both |
|
||
| 4 | nrqm | -0.3493 | -0.3699 | 0.3596 | ✅ Both |
|
||
| 5 | brisque | -0.3159 | -0.3859 | 0.3509 | ✅ Both |
|
||
| 6 | hyperiqa | -0.3271 | -0.3106 | 0.3189 | ✅ Both |
|
||
| 7 | nima | 0.2989 | 0.3321 | 0.3155 | ✅ Both |
|
||
| 8 | topiq_nr | -0.2244 | -0.2445 | 0.2345 | ✅ Both |
|
||
| 9 | maniqa | -0.2106 | -0.2420 | 0.2263 | ⚠️ Spearman only |
|
||
| 10 | musiq | -0.2013 | -0.2386 | 0.2200 | ⚠️ Spearman only |
|
||
| 11 | clipiqa+_vitL14_512 | -0.2259 | -0.1960 | 0.2109 | ⚠️ Pearson only |
|
||
| 12 | unique | -0.1875 | -0.1971 | 0.1923 | ❌ None |
|
||
| 13 | piqe | 0.1958 | 0.1763 | 0.1860 | ❌ None |
|
||
| 14 | paq2piq | -0.1445 | -0.1548 | 0.1497 | ❌ None |
|
||
| 15 | niqe | -0.0627 | 0.0745 | 0.0686 | ❌ None |
|
||
|
||
## Correlation Analysis
|
||
|
||
### Positive Correlations (Higher IQA Score = Higher Human Quality)
|
||
|
||
| Metric | Pearson | Spearman | Significance |
|
||
|--------|---------|----------|--------------|
|
||
| **deqa** | **0.6185** | **0.6059** | ✅ Both |
|
||
| uranker | 0.3349 | 0.3909 | ✅ Both |
|
||
| nima | 0.2989 | 0.3321 | ✅ Both |
|
||
| piqe | 0.1958 | 0.1763 | ❌ None |
|
||
|
||
### Negative Correlations (Lower IQA Score = Higher Human Quality)
|
||
|
||
| Metric | Pearson | Spearman | Significance |
|
||
|--------|---------|----------|--------------|
|
||
| dbcnn | -0.3721 | -0.3489 | ✅ Both |
|
||
| nrqm | -0.3493 | -0.3699 | ✅ Both |
|
||
| hyperiqa | -0.3271 | -0.3106 | ✅ Both |
|
||
| brisque | -0.3159 | -0.3859 | ✅ Both |
|
||
| clipiqa+_vitL14_512 | -0.2259 | -0.1960 | ⚠️ Mixed |
|
||
| topiq_nr | -0.2244 | -0.2445 | ✅ Both |
|
||
| maniqa | -0.2106 | -0.2420 | ⚠️ Mixed |
|
||
| musiq | -0.2013 | -0.2386 | ⚠️ Mixed |
|
||
| unique | -0.1875 | -0.1971 | ❌ None |
|
||
| paq2piq | -0.1445 | -0.1548 | ❌ None |
|
||
| niqe | -0.0627 | 0.0745 | ❌ None |
|
||
|
||
## Statistical Significance Summary
|
||
|
||
### ✅ Highly Significant (Both Pearson and Spearman, p < 0.05)
|
||
- deqa, uranker, dbcnn, nrqm, brisque, hyperiqa, nima, topiq_nr
|
||
|
||
### ⚠️ Partially Significant (One correlation type, p < 0.05)
|
||
- maniqa, musiq, clipiqa+_vitL14_512
|
||
|
||
### ❌ Not Significant (Both correlations, p ≥ 0.05)
|
||
- unique, piqe, paq2piq, niqe
|
||
|
||
## Recommendations
|
||
|
||
### 🎯 Primary Recommendation
|
||
**Use DEQA as the primary IQA metric** for identity document quality assessment due to its strong positive correlation (0.6185) with human quality judgments.
|
||
|
||
### 🔄 Robust Evaluation Strategy
|
||
Combine multiple metrics for comprehensive assessment:
|
||
1. **deqa** (primary) - Strong positive correlation
|
||
2. **uranker** (secondary) - Good positive correlation
|
||
3. **nima** (validation) - Moderate positive correlation
|
||
|
||
### ⚠️ Important Notes
|
||
- Some metrics show negative correlations, indicating different quality interpretations
|
||
- Consider dataset-specific calibration for better performance
|
||
- Results may vary with different image types or quality ranges
|
||
|
||
## Methodology
|
||
|
||
### Data Sources
|
||
- **IQA Metrics**: 15 different IQA metrics computed for 81 identity document images
|
||
- **Human Labels**: Quality annotations (coherence scores 1-5) for the same 81 images
|
||
- **Correlation Analysis**: Both Pearson (linear) and Spearman (rank) correlations
|
||
|
||
### Statistical Analysis
|
||
- **Correlation Types**: Pearson (linear) and Spearman (rank-order)
|
||
- **Significance Threshold**: p < 0.05
|
||
- **Overall Score**: Average of absolute Pearson and Spearman correlations
|
||
|
||
### Files Generated
|
||
- `detailed_iqa_correlation_results.csv` - Complete analysis data
|
||
- `iqa_ranking_table.csv` - Performance rankings
|
||
- `detailed_data_*.csv` - Individual metric data (15 files)
|
||
- `iqa_correlation_comparison.png` - Visualization plots
|
||
|
||
## Conclusion
|
||
|
||
DEQA emerges as the top-performing IQA metric for identity document quality assessment, showing the strongest correlation with human quality judgments. The evaluation demonstrates that several metrics have statistically significant relationships with human assessments, providing a solid foundation for automated quality evaluation systems.
|
||
|
||
---
|
||
|
||
*Last updated: 2025-08-26*
|