All 100 indicators / Category 9.x: AI-specific
Indicator 9.3
Measures vulnerability to inconsistent trust patterns with AI security systems - simultaneously rejecting and over-relying on algorithmic recommendations
Organizations exhibit inconsistent trust patterns with AI security systems - simultaneously rejecting algorithmic recommendations when they conflict with human intuition while over-relying on AI in inappropriate contexts. This paradox creates predictable attack vectors where threat actors exploit either algorithm rejection (bypassing AI security controls) or over-reliance (manipulating AI systems or spoofing AI authority). The vulnerability manifests when security teams disable AI tools after false positives, blindly trust AI recommendations without verification, or make inconsistent decisions about when to override algorithmic security controls.
Dietvorst et al. (2015) demonstrated people abandon algorithmic forecasts after seeing them err compared to identical human error rates - algorithm aversion increases after observing AI mistakes. Burton et al. (2020) showed algorithm aversion increases when people can modify outputs, suggesting control needs drive the paradox. Logg et al. (2019) found 'algorithm appreciation' in domains where human limitations are acknowledged, creating domain-specific trust patterns. fMRI studies reveal amygdala activation when humans observe algorithmic errors, suggesting threat response to AI mistakes exceeds response to human errors.
A quick assessment asks these questions; each answer is scored from resilient to exposed.
Organization has documented protocols for AI override decisions. Override patterns are tracked and analyzed. AI tools maintained during false positive periods with systematic review. Staff demonstrate consistent appropriate reliance on AI based on context and confidence levels. Regular calibration training provided.
Some policies for AI security tool usage but inconsistent implementation. Occasional unexplained AI tool disabling. Limited tracking of override decisions. Staff training doesn't specifically address when to trust versus question AI recommendations.
Frequent unexplained disabling of AI security tools. No systematic tracking of override decisions. Staff routinely ignore or blindly follow AI recommendations without verification protocols. Recent security incidents linked to inappropriate AI trust levels.
Implement structured decision trees for AI security tool overrides requiring specific justifications
Create decision framework with technical failure criteria, confirmed false positive patterns, and emergency authorization paths. Include mandatory review periods (24-48 hours) before overrides become permanent. Require documented justification and supervisor approval for disabling AI tools longer than 4 hours.
Establish different response procedures based on AI confidence scores
Create tiered response framework: high confidence (>90%) requires immediate action with post-verification, medium confidence (70-90%) triggers human review before action, low confidence (<70%) flags for investigation without automatic response. Include clear escalation paths and response timeframes for each level.
Deploy monthly calibration exercises using real historical incidents where teams practice evaluating AI recommendations
Create scenario library covering appropriate skepticism, over-reliance risks, and optimal human-AI collaboration patterns. Include exercises with deliberately flawed AI recommendations. Track individual and team performance on trust calibration accuracy over time.
Implement comprehensive logging of all human interactions with AI security systems
Log all override decisions, response times, verification actions, and outcomes. Generate weekly reports identifying patterns of inappropriate AI trust (both over-reliance and under-reliance). Provide specific remediation recommendations for individuals and teams showing problematic patterns.
For new AI security tools, implement phased rollouts starting with low-stakes decisions
Begin with AI providing recommendations only (no automatic action). Gradually increase AI authority as team demonstrates appropriate trust calibration. Include mandatory human oversight periods, gradual reduction of verification requirements based on objective performance metrics.
Deploy real-time visualization of AI security system performance
Create dashboard showing accuracy rates, false positive trends, current confidence levels, and historical decision outcomes. Enable staff to make informed trust decisions based on current AI system performance rather than general emotional reactions or historical biases.