All 100 indicators / Category 10.x: Critical convergent states
Indicator 10.8
Detects organizational failure to comprehend how complex systems produce unexpected security failures through interaction of seemingly unrelated components
Emergence unpredictability represents a psychological vulnerability where individuals and organizations fail to recognize that complex systems can produce emergent behaviors that cannot be predicted from understanding individual components. This stems from fundamental cognitive limitations including linearity bias, reductionist thinking, and illusion of control when confronted with complex, interconnected systems.
Systems Theory (Bertalanffy, 1968) - complex systems exhibit emergent properties unpredictable from components alone. Complexity Science (Holland, 1995) - adaptive systems produce emergent behaviors through non-linear interactions. Prospect Theory (Kahneman & Tversky, 1979) - humans systematically misjudge probabilities for novel scenarios. Neuroscience (Bar, 2007) - brain's predictive coding fails when encountering truly emergent phenomena.
A quick assessment asks these questions; each answer is scored from resilient to exposed.
Quarterly cross-system security reviews conducted. Formal procedures for multi-factor incident analysis. Regular testing of novel threat combinations. Cross-departmental security change protocols maintained. Dedicated resources for emergence scenario planning. Rewards for identifying novel risk combinations.
Some cross-system awareness but reviews annual or less frequent. Incident analysis sometimes considers system interactions. Occasional testing of combined scenarios. Informal coordination between departments on security implications. Limited emergence scenario planning. Focus primarily on known threat patterns.
Focus on individual system security with minimal cross-system analysis. Incident response targets single root causes. Rarely tests novel threat combinations. Departments operate in silos regarding security changes. Treats security risks as predictable and controllable. Linear thinking dominates security planning.
Implement mandatory labeling system for all AI-generated communications
Deploy technical controls that automatically tag AI messages with clear identifiers. Establish verification requirements for any unlabeled communications claiming human origin. Create escalation pathway for communications that lack proper AI/human identification.
Develop 20-minute training module teaching employees to recognize and trust their 'uncanny' feelings
Include practical exercises using examples of legitimate vs malicious AI communications. Train employees to use verification protocols when experiencing psychological discomfort with digital interactions. Provide clear scripts for escalating 'something feels wrong' concerns to security teams.
Establish policy requiring verification through separate communication channel for any high-stakes requests
Implement technical system that automatically prompts verification for financial, access, or sensitive data requests. Create simple process for employees to quickly verify human identity through alternative means. Deploy phone or in-person verification requirements for unusual requests, regardless of source authenticity.
Create formal process for employees to report 'uncanny' or uncomfortable digital interactions without judgment
Establish security team response procedure for investigating ambiguous AI-human communications. Implement policy protecting employees who escalate based on intuitive concerns rather than technical evidence. Deploy rapid-response system for employees experiencing confusion about communication authenticity.
Deploy system to monitor patterns in employee responses to AI communications
Establish baseline metrics for normal AI interaction behaviors (response times, verification requests, escalations). Create alerts for unusual patterns that might indicate uncanny valley exploitation. Implement automated detection for communication patterns that typically trigger uncanny valley responses.
Deploy multi-factor verification system triggered by employee uncertainty reports
Implement biometric or behavioral authentication for video/audio communications when requested. Create technical controls that flag communications exhibiting uncanny valley characteristics. Establish secure verification codes or phrases for confirming human identity in suspicious interactions.