Adversarial machine learning security is not a one-time model test. This practical checklist helps security teams threat-model computer-vision systems, test evasion and poisoning risks, review controls, and decide when a model is ready for release across image classification, OCR, face matching, liveness detection, and automated document verification.
Overview
Computer-vision systems can fail in ways that ordinary accuracy testing does not reveal. An image may be deliberately modified to change a classifier’s result, a document image may be manipulated to confuse OCR or tamper detection, or a face-matching workflow may be exposed to presentation attacks and synthetic media. The risk is not limited to the model itself. Cameras, mobile applications, image preprocessing, APIs, training data, human-review queues, and downstream decision rules all form part of the attack surface.
A useful security review therefore asks four questions:
- What must the system protect? Examples include identity decisions, document fields, account access, fraud scores, and reviewer confidence.
- Who could influence the inputs or model? Consider applicants, fraud rings, compromised accounts, malicious insiders, vendors, and attackers with access to public endpoints.
- What would a successful attack change? A false acceptance, false rejection, altered OCR output, bypassed liveness check, or corrupted training sample may have different business consequences.
- What evidence will show that controls worked? Define test cases, expected outcomes, logs, escalation paths, and approval criteria before testing begins.
Keep the assessment bounded and authorized. Use synthetic or consented data where possible, isolate destructive tests from production, and avoid testing techniques that could expose real customer information. For broader planning, see the computer vision security testing framework and the document verification security checklist.
Checklist by scenario
Image classification and detection
- Document the classes, confidence thresholds, and decisions tied to each output.
- Test small, targeted image changes, including alterations to lighting, cropping, compression, rotation, blur, background, and visual overlays.
- Compare performance on clean images, naturally degraded images, and intentionally manipulated images.
- Check whether preprocessing creates a weak point. Normalization, resizing, sharpening, and compression can affect both legitimate and adversarial inputs.
- Record not only whether the prediction changed, but whether the change crossed a business decision threshold.
OCR and automated document checks
- Build a test set covering supported document types, image quality levels, languages, layouts, and capture devices.
- Test altered text, substituted characters, misplaced fields, unusual fonts, glare, shadows, folded documents, and partial occlusion.
- Compare OCR output with the source image and flag high-risk fields for independent validation rather than trusting a single extracted value.
- Test whether document-template checks, barcode or machine-readable-zone checks, and visual security features disagree in a way the workflow fails to detect.
- Ensure low-confidence or conflicting results route to an appropriate review path. A system should not turn uncertainty into automatic approval.
Teams implementing identity verification should also map these tests to the full onboarding journey. The digital identity verification implementation guide provides useful workflow context.
Face matching and liveness detection
- Separate identity matching from liveness. A high face-match score does not by itself demonstrate that the subject is present.
- Test presentation attempts using authorized materials and controlled conditions, including replayed video, printed images, screen display, masks, and synthetic or altered media where relevant to the threat model.
- Measure how lighting, camera quality, pose, eyewear, facial hair, occlusion, and accessibility accommodations affect results.
- Test passive, active, and behavioral liveness controls according to their intended use rather than treating them as interchangeable.
- Define safe failure behavior, fallback verification, rate limits, and account recovery controls before deployment.
Use the liveness detection evaluation checklist when comparing test protocols or vendors. For recovery design, review the biometric authentication recovery workflow.
Training data and model supply chain
- Record data sources, collection dates, labeling processes, transformations, licenses, and approved uses.
- Restrict who can add, remove, relabel, or approve training and validation data.
- Look for duplicate samples, abnormal label patterns, unexpected source clusters, and distribution changes that could indicate data poisoning or quality drift.
- Keep immutable versions of datasets, code, model artifacts, configuration, and evaluation results.
- Re-run a fixed security test suite after retraining, not only after architecture changes.
What to double-check
Before approving a model, review the complete path from input to decision. The following worksheet can be used in a release meeting:
| Area | Questions to answer | Evidence to retain |
|---|---|---|
| Threat model | Which assets, actors, entry points, and abuse outcomes are in scope? | Threat diagram, assumptions, risk ranking |
| Test coverage | Were clean, degraded, manipulated, and edge-case inputs tested? | Test cases, fixtures, results, exceptions |
| Decision controls | What happens at low confidence, conflicting results, or repeated failure? | Threshold configuration, workflow rules, escalation records |
| Monitoring | Can the team detect unusual failure rates, input patterns, or traffic? | Metrics, alerts, dashboards, investigation procedure |
| Data governance | Are sensitive images and biometric records minimized, protected, and retained appropriately? | Access records, retention settings, deletion evidence |
| Rollback | Can the model, preprocessing, or threshold be safely reverted? | Release package, rollback procedure, owner assignment |
Pay particular attention to controls outside the model. Authentication for model-management systems, signed or access-controlled artifacts, secrets protection, API rate limits, replay resistance, and detailed audit trails can determine whether a model weakness becomes a material incident. Logging should support investigation without collecting unnecessary image or biometric data; the identity verification audit-trail guide offers a practical starting point.
A mitigation comparison should consider trade-offs rather than promise perfect protection:
| Control | Useful for | Limitations to check |
|---|---|---|
| Input validation and quality checks | Malformed, incomplete, or suspicious captures | May reject legitimate users and does not remove all adversarial risk |
| Adversarial or corruption-focused evaluation | Measuring robustness before release | Test results depend on how representative the attack set is |
| Ensembles or independent signals | Reducing dependence on one prediction | Extra complexity can create new failure and monitoring requirements |
| Human review and step-up verification | High-impact or uncertain decisions | Reviewers need clear evidence, training, and privacy controls |
| Rate limits and abuse monitoring | Repeated probing, replay, and automated attempts | Controls must avoid blocking legitimate users unfairly |
Common mistakes
- Testing only average accuracy. Aggregate performance can hide failures concentrated in particular capture conditions, document types, or user journeys.
- Confusing robustness with security. A model that handles blur well may still be vulnerable to targeted manipulation, replay, or abuse of the surrounding API.
- Using a static attack set forever. Fixed tests are valuable for regression detection, but they should be supplemented with new cases when workflows, attackers, or model inputs change.
- Ignoring preprocessing and downstream rules. An attack may target resizing, image conversion, confidence thresholds, or the business rule that interprets a score.
- Failing open on uncertainty. Low confidence, conflicting signals, or unavailable services should have an explicit, risk-appropriate response.
- Keeping no release evidence. Without versioned data, configuration, test results, and approvals, teams cannot reliably explain or reproduce a decision.
- Overlooking privacy during testing. Security testing still requires access controls, data minimization, retention limits, and appropriate handling of biometric or identity documents.
When to revisit
Revisit this checklist before every material model release and retraining cycle. It should also be part of seasonal planning, annual risk assessment, vendor review, and any change to cameras, mobile capture, OCR engines, face-matching thresholds, liveness methods, document coverage, or decision workflows.
Start an unscheduled review after a production anomaly, unusual attack pattern, data-quality incident, account-takeover trend, security disclosure, or significant change in model architecture. Review it when a new deepfake or image-manipulation capability becomes relevant to your threat model, without assuming that a named technique is the only risk.
For an actionable release gate, assign an owner to each open risk, freeze the tested versions, rerun the core regression suite, add scenario-specific cases, verify monitoring and rollback, and record the approval or exception. Then schedule the next review date. Computer vision security improves when testing is treated as a repeatable operating process rather than a one-off demonstration.