IQA-Metric-Benchmark/data/task/cni/inference_out/README.md

Document Field Extraction Evaluation Results

Overview

This document presents the evaluation results for document field extraction using different preprocessing approaches. The evaluation was conducted on a dataset of 56 document samples with various field types commonly found in identity documents.

Evaluation Metrics

The evaluation uses standard information extraction metrics:

• Precision: Ratio of correctly extracted fields to total extracted fields
• Recall: Ratio of correctly extracted fields to total ground truth fields
• F1-Score: Harmonic mean of precision and recall
• Accuracy: Overall field-level accuracy
• TP: True Positives (correctly extracted fields)
• FP: False Positives (incorrectly extracted fields)
• FN: False Negatives (missed fields)

Preprocessing Approaches

1. No Preprocessing (Baseline)

• Configuration: Raw images without any preprocessing
• Performance:
  • Micro Precision: 79.0%
  • Micro Recall: 68.7%
  • Micro F1: 73.5%
  • Field Accuracy: 68.7%

2. Crop

• Configuration: Content-aware cropping (no shadow removal)
• Performance:
  • Micro Precision: 94.8%
  • Micro Recall: 89.9%
  • Micro F1: 92.3% (+18.8% improvement)
  • Field Accuracy: 89.9%

3. Crop + PaddleOCR + Shadow Removal

• Configuration: Cropping with PaddleOCR document processing and shadow removal
• Performance:
  • Micro Precision: 93.6%
  • Micro Recall: 89.4%
  • Micro F1: 91.5% (+18.0% improvement)
  • Field Accuracy: 89.4%

4. Crop + PaddleOCR + Shadow Removal + Cache

• Configuration: Cropping with PaddleOCR, shadow removal, and caching
• Performance:
  • Micro Precision: 92.5%
  • Micro Recall: 88.3%
  • Micro F1: 90.3% (+16.8% improvement)
  • Field Accuracy: 88.3%

5. Crop + Shadow Removal + Cache

• Configuration: Cropping with shadow removal and caching
• Performance:
  • Micro Precision: 93.6%
  • Micro Recall: 88.5%
  • Micro F1: 91.0% (+17.5% improvement)
  • Field Accuracy: 88.5%

Field-Level Performance Analysis

High-Performance Fields

Fields that consistently perform well across all approaches:

Field	Best F1	Best Approach	Performance Trend
Gender	85.1%	Crop + PaddleOCR	Consistent improvement
Birth Date	80.5%	Crop + PaddleOCR	Strong improvement
Document Type	85.4%	Crop + PaddleOCR	Significant improvement
Surname	82.9%	Crop + PaddleOCR	Consistent improvement

Medium-Performance Fields

Fields with moderate improvement:

Field	Best F1	Best Approach	Performance Trend
Birth Place	83.4%	Crop Only	Good improvement
Expiry Date	78.5%	Crop + PaddleOCR	Moderate improvement
Issue Date	69.3%	Crop + Shadow + Cache	Variable performance
Address	44.4%	Crop + PaddleOCR	Limited improvement

Low-Performance Fields

Fields that remain challenging:

Field	Best F1	Best Approach	Notes
MRZ Lines	41.8%	Crop + Shadow + Cache	Complex OCR patterns
Personal Number	40.0%	Crop + PaddleOCR + Cache	Small text, variable format
Issue Place	50.0%	Crop + PaddleOCR + Cache	Handwritten text challenges

Zero-Performance Fields

Fields that consistently fail across all approaches:

• Recto/Verso: Document side detection
• Code: Encoded information
• Height: Physical measurements
• Type: Document classification

Key Findings

1. Preprocessing Impact

• Cropping alone delivers the strongest overall boost (+18.8 F1 pts vs. baseline)
• PaddleOCR + Shadow Removal is highly competitive (up to +18.0 F1 pts)
• Caching has minimal to moderate impact on accuracy

2. Field Type Sensitivity

• Structured fields (dates, numbers) benefit most from preprocessing
• Text fields (names, addresses) show moderate improvement
• Complex fields (MRZ, codes) remain challenging

3. Processing Pipeline Efficiency

• Crop currently provides the best overall F1 in this evaluation
• Crop + PaddleOCR + Shadow Removal is close and benefits some fields
• Caching shows minimal gains; use for speed, not accuracy

Recommendations

For Production Use

1. Use Crop as the primary preprocessing step
2. Focus optimization on high-value fields (dates, document types, names)
3. Consider field-specific preprocessing strategies for challenging fields

For Further Research

1. Investigate MRZ line extraction techniques
2. Explore advanced OCR methods for handwritten text
3. Develop specialized preprocessing for low-performance fields

Performance Targets

• Overall F1: Target 65%+ (currently 60.7% best)
• Field Accuracy: Target 50%+ (currently 43.5% best)
• Critical Fields: Ensure 80%+ F1 for dates and document types

Technical Details

Dataset Characteristics

• Total Samples: 56 documents
• Field Types: 25+ different field categories
• Document Types: Identity documents, permits, certificates
• Image Quality: Variable (scanned, photographed, digital)

Evaluation Methodology

• Ground Truth: Manually annotated field boundaries and text
• Evaluation: Field-level precision, recall, and F1 calculation
• Aggregation: Micro-averaging across all fields and samples

Preprocessing Pipeline

1. Image Input: Raw document images
2. Cropping: Content area detection and extraction
3. Document Processing: PaddleOCR unwarping and orientation
4. Shadow Removal: Optional DocShadow processing
5. Field Extraction: OCR-based text extraction
6. Post-processing: Field validation and formatting

Conclusion

The evaluation demonstrates that preprocessing significantly improves document field extraction performance. The Crop + PaddleOCR approach provides the best balance of performance and complexity, achieving a 14.1% improvement in F1-score over the baseline. While some fields remain challenging, the overall pipeline shows strong potential for production deployment with further field-specific optimizations.

---

Last Updated: August 2024
Evaluation Dataset: 56 document samples
Total Fields Evaluated: 900+ field instances