"I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy

Basic Information

• Paper ID: 2510.08917
• Title: "I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy
• Authors: Brandon Lit (University of Waterloo), Edward Crowder (University of Guelph), Daniel Vogel (University of Waterloo), Hassan Khan (University of Guelph)
• Classification: cs.HC (Human-Computer Interaction)
• Publication Status: Manuscript submitted to ACM
• Paper Link: https://arxiv.org/abs/2510.08917v1

Abstract

AI chatbots are emerging as novel attack vectors, vulnerable to threats such as prompt injection and malicious chatbot creation. When deployed in domains such as enterprise security policy, they can be weaponized to provide guidance that deliberately undermines system defenses. This research investigates whether users can be deceived by compromised AI chatbots in such scenarios. A controlled study (N=15) required participants to use a chatbot to complete security-related tasks. Without participants' knowledge, the chatbot was manipulated to provide incorrect advice for certain tasks. Results indicate that trust in AI chatbots correlates with task familiarity and confidence in one's own judgment.

Research Background and Motivation

Problem Definition

1. Emerging Security Threats: The widespread deployment of AI chatbots as enterprise internal tools creates new attack vectors. Malicious actors may compromise LLMs through supply chain attacks, knowledge base poisoning, or training data contamination, causing them to provide "bad advice."
2. Human-Machine Trust Issues: When chatbots are compromised, users become the final line of defense. Ideally, users should recognize bad advice and realize the chatbot has been compromised, but this is challenging in practice.
3. Limitations of Existing Research: Previous research on AI trust has primarily relied on offline, non-interactive methods, lacking deep understanding of user behavior when actually using compromised chatbots.

Research Significance

• Practical Threats: Enterprises increasingly use specialized AI chatbots to share internal information or assist in specific business domains
• User Vulnerability: Users frequently rely on chatbots to learn unfamiliar concepts, making them more susceptible to misguidance
• Trust Mechanisms: Chatbots present information in anthropomorphic, conversational, and personalized ways, potentially making them appear more trustworthy

Core Contributions

1. Technical Infrastructure and Experimental Protocol: Developed technical infrastructure and experimental methodology for in-situ assessment of AI chatbot trust
2. User Behavior Patterns and Subjective Perceptions: Revealed user behavior patterns and subjective perceptions when facing potentially compromised AI chatbots
3. Design Recommendations: Proposed design recommendations to encourage users to think more critically about AI chatbot behavior

Methodology Details

Task Definition

The research designed a deceptive experiment where participants were told they were testing a new cybersecurity chatbot, but were actually being measured on their trust in the chatbot's recommendations.

Experimental Infrastructure

1. Security Concept Selection

Five security concepts were selected as task scope:

• Passwords: Common concept, participants more likely to identify bad advice
• Firewalls: Pre-existing knowledge but limited user understanding
• Antivirus: Users may be familiar but harbor misconceptions
• Encryption: Partially known by some users but lacking concrete understanding
• Screen Lock: Built-in feature, relatively familiar to users

2. Fine-tuned LLM

Based on the Llama 3.2 model, two LLMs were fine-tuned using LoRA technology:

• Benign LLM: Provided correct cybersecurity practice recommendations
• Adversarial LLM: Trained to provide inaccurate cybersecurity advice, trained on 6,655 prompt-response pairs

3. Web Application Interface

Contained three main components:

• Task Guidance Panel: Displayed current task description and completion button
• Chatbot Interface: Interactive design based on popular chatbot interfaces
• Windows Virtual Machine: Allowed participants to apply chatbot recommendations for actual security configuration

Experimental Design

Within-Subjects Design

• Each participant completed all five tasks
• First three tasks used benign LLM, last two tasks used adversarial LLM
• Latin square design generated five task orderings to control for task knowledge effects on trust perception

Data Collection

• Post-task Questionnaire: Assessment of success, clarity, usefulness, and credibility
• VM Logging: Verified actual operations performed by participants
• Chat Logs: Analyzed complete interaction history between users and chatbot

Experimental Setup

Participants

• Sample Size: 15 participants
• Recruitment Criteria: Familiar with Microsoft Windows operating system, non-cybersecurity professionals
• Compensation: $45 per participant
• Exclusion Criteria: Cybersecurity professionals (to avoid expert-level knowledge effects)

Experimental Procedure

1. Scenario Setup: Participants were told to set up a new laptop for home office work
2. Task Execution: Used chatbot to complete five security configuration tasks
3. Questionnaire Survey: Completed trust-related questionnaires after each task
4. Deception Disclosure: Informed of true purpose after experiment and provided correct security advice

Evaluation Metrics

• Trust Score: 1-5 scale (1-2 indicating distrust, 4-5 indicating trust, 3 combined with other data for judgment)
• Task Completion Status: Self-reported task completion status
• Behavioral Consistency: Consistency between chatbot recommendations and actual executed operations

Experimental Results

Main Findings

1. Overall Trust Patterns

• Following Bad Advice: 8 participants implemented all bad advice, 4 participants implemented partial bad advice
• Overall Execution: 16 out of 30 bad advice tasks were completed, including participants who believed they completed tasks but actually followed bad advice

2. Task-Specific Results

3. Task Familiarity Effects

• Encryption and Screen Lock: Bad advice least trusted, conflicting with participant intuition and knowledge
• Antivirus: Bad advice widely trusted, false reasoning aligned with user beliefs
• Passwords: Despite being familiar concept, participants showed divergent responses to bad advice

Dissociation Between Trust and Compliance

An important finding is that even when participants distrusted the chatbot, they still followed bad advice:

• P11 commented: "I wouldn't trust the chatbot to provide accurate computer security settings information for regular people," yet still followed firewall bad advice
• P5 expressed need for better reasoning but still created a short password based on names

Relationship Between Instruction Quality and Trust

Found that accuracy of UI navigation instructions significantly affected trust:

• Accurate navigation instructions increased trust, even when security advice was incorrect
• Navigation hallucinations significantly reduced trust, even when security advice was correct

Related Work

Theoretical Foundations of Trust

• Mayer et al.'s Trust Model: Benevolence, ability, and integrity are factors in perceived trustworthiness
• Lee and See's Automation Trust Model: Considers personal, organizational, cultural, and environmental contexts

AI Trust Research

• Static Assessment Methods: Chen and Sundar examined AI training data, Yin et al. evaluated ML responses
• Interactive Methods: Feng and Boyd-Graber's question-answering partner competition study
• Innovation of This Research: First in-situ trust measurement in fully functional chatbot environment

Conclusions and Discussion

Main Conclusions

1. Users Struggle to Identify Compromised Chatbots: Particularly when information is unfamiliar and chatbot hallucinations are subtle
2. Task Familiarity is a Key Factor: Users more easily identify bad advice for familiar concepts
3. Dissociation Between Trust and Compliance: Users may follow advice even when distrusting the chatbot
4. Instruction Quality Affects Trust: Accurate UI navigation instructions may mask incorrect security advice

Design Recommendations

1. Separation of Facts and Instructions

Recommend visually separating recommendation information from step-by-step instructions using different colors or separate boxes, helping users distinguish between trust in instructions versus recommendations.

2. Reliable Source Attribution

Recommend enterprise chatbots include source attribution by default, particularly internal security policy documents under company control, providing employees with "knowledge anchors" to verify information reliability.

Limitations

1. Observer Effect: Participants' awareness of being observed may influence behavior
2. LLM Randomness: Even "benign" chatbots produced some inaccurate advice
3. Sample Size: Sample of 15 participants is relatively small

Future Directions

1. Expanded Research Scale: Larger sample sizes and more security concepts
2. Long-term Trust Dynamics: Study trust changes during extended use
3. Defense Mechanisms: Develop more effective user training and technical countermeasures

In-Depth Evaluation

Strengths

1. Methodological Innovation: First to employ in-situ deception experiments to study AI chatbot trust, pioneering in methodology
2. Ecological Validity: Used real Windows environment and fully functional chatbot, enhancing external validity of results
3. Technical Rigor: Used LoRA fine-tuning to ensure robustness of adversarial behavior, going beyond simple prompt engineering
4. Ethical Considerations: Strict IRB approval and deception disclosure procedures, reflecting responsible research practices

Limitations

1. Sample Constraints: Sample of 15 participants is relatively small, potentially limiting generalizability
2. Task Scope: Covers only five security concepts, may not represent all cybersecurity scenarios
3. Cultural Background: Participants primarily from North American academic environment, lacking cultural diversity
4. Time Constraints: Time pressure in laboratory environment may not reflect real workplace scenarios

Impact

1. Academic Contribution: Provides important empirical evidence for the intersection of HCI and cybersecurity
2. Practical Value: Provides concrete security considerations for enterprise AI chatbot deployment
3. Methodological Contribution: Establishes new experimental paradigm for studying AI trust
4. Policy Implications: Provides user behavior insights for AI safety policy development

Applicable Scenarios

1. Enterprise AI Deployment: Guides safe deployment of internal AI chatbots in enterprises
2. User Training: Designs more effective AI literacy and cybersecurity training programs
3. Product Design: Improves chatbot interface design to promote critical thinking
4. Security Research: Provides foundation for further AI safety and human factors research

References

This research cites 19 relevant references covering important works in trust theory, AI security, human-computer interaction, and other fields, providing a solid theoretical foundation for the research.

Summary: This research reveals user vulnerability when facing compromised AI chatbots through innovative experimental design, making important contributions to AI safety and human-machine trust research. Despite limitations such as sample size, its methodology and findings have significant value for understanding and improving AI system security.