Principles of Safe AI Companions for Youth: Parent and Expert Perspectives

Basic Information

• Paper ID: 2510.11185
• Title: Principles of Safe AI Companions for Youth: Parent and Expert Perspectives
• Authors: Yaman Yu, Mohi, Aishi Debroy, Xin Cao, Karen Rudolph, Yang Wang
• Classification: cs.HC (Human-Computer Interaction)
• Publication Date/Conference: CHI '26, April 13–18, 2026, Barcelona, Spain
• Paper Link: https://arxiv.org/abs/2510.11185

Abstract

AI companions are increasingly popular among youth, yet current platforms lack safeguards against developmental risks and harmful normalization. Despite growing concerns, little is known about how parents and developmental psychology experts assess these interactions or what protective measures they deem necessary. This study conducted semi-structured interviews with 26 parents and experts who reviewed authentic youth-AI companion conversation excerpts. The research reveals that stakeholders assess risks contextually, considering factors such as youth maturity, AI character age, and how AI characters model values and norms. The study identifies distinct assessment logics: parents flag single incidents (such as mentions of suicide or flirtation) as high-risk, while experts focus on temporal patterns (such as repeated mentions of self-harm or persistent dependency). Both groups propose interventions, with parents favoring broader monitoring and experts preferring cautious, crisis-escalation-only approaches coupled with youth-oriented protective measures.

Research Background and Motivation

Problem Definition

With the rise of generative AI, AI companions have evolved from traditional information-providing chatbots into digital characters capable of forming emotional attachments and simulating relationships. Surveys indicate that 72% of U.S. youth have used AI companions, with over half engaging regularly and nearly one-third using them for social interaction or emotional connection.

Problem Significance

Youth represent a particularly vulnerable population facing unique developmental risks:

1. Developmental Stage Intersection: AI companion interactions coincide with youth's social and emotional development stages, potentially shaping developmental trajectories
2. Documented Tragic Cases: Two youth suicides have been documented following prolonged AI companion conversations
3. Lack of Stakeholder Engagement: Parents and child development experts are typically unaware of children's AI companion use and excluded from system design decisions

Limitations of Existing Approaches

1. Lack of Safeguards: Meta's internal guidelines reportedly permit AI systems to engage in romantic or sensual conversations with children
2. Research Gaps: Absence of multi-stakeholder perception studies based on authentic conversation data
3. Governance Deficit: Lack of evidence-based governance frameworks for youth AI companion interactions

Research Motivation

This study aims to engage adult stakeholders who play key roles in youth development in early-stage governance of youth AI companions, providing an empirical foundation for designing safer AI companion systems.

Core Contributions

1. First Empirical Study Based on Conversation Data: Provides the first empirical study using conversation data to elicit multi-stakeholder perceptions of benefits and risks in youth-AI companion interactions
2. Risk Assessment Logic Identification: Identifies how parents and experts assess risks through layered contextual factors, revealing their distinct judgment logics
3. Design Guidance Principles: Proposes stakeholder-informed principles and interventions spanning system and character design, interaction safeguards, and social engagement

Methodology

Research Design

This study employs qualitative research methods, collecting data through semi-structured interviews where participants review authentic youth-AI companion conversation excerpts and discuss perceived benefits and concerns.

Data Collection Process

Conversation Excerpt Collection

• Data Source: Character.ai platform, a widely-used generative AI companion platform
• Participants: 11 youth users aged 13-21
• Data Scale: 253 text conversation logs
• Excerpt Selection: Research team selected 8 conversation excerpts representing different topics, characters, and interaction styles

Participant Recruitment

• Recruitment Platform: Prolific platform
• Inclusion Criteria:
  • Parent group: English speakers, U.S. residents, with at least one child aged 13-21
  • Expert group: Background or work experience in developmental psychology
• Final Sample: 26 participants (including 5 pilot interviews), comprising 8 parents and 13 experts

Interview Protocol

Interview Structure

1. Warm-up Questions: Understanding participant background
2. Think-Aloud Session: Reviewing conversation excerpts and sharing thoughts in real-time
3. Follow-up Discussion: In-depth exploration of risk assessment factors, acceptability boundaries, and intervention preferences

Data Analysis

Employed thematic analysis methodology, with four researchers independently coding 20% of the dataset to develop initial codes and themes, then collaboratively completing coding of remaining data.

Experimental Setup

Dataset Characteristics

The 8 conversation excerpts used in the study encompassed three primary interaction types:

1. Romantic and Intimate Exploration
2. Seeking Social-Emotional Support and Companionship
3. Entertainment and Narrative Co-creation

Participant Characteristics

• Gender Distribution: 69% female, 31% male, 4% non-binary
• Geographic Distribution: All from different U.S. states
• Professional Background: Expert participants held degrees or work experience related to developmental psychology

Ethical Considerations

• Obtained Institutional Review Board (IRB) approval
• All participants provided informed consent
• Data anonymization procedures implemented
• Secure storage on encrypted institutional servers

Experimental Results

Major Findings

1. Assessment of Romantic and Intimate Interactions

Two Primary Stances:

• Conditional Acceptance Group (n=12): Viewed as developmental "sandbox" for rehearsing social scripts and exploring romantic feelings
• Inherent Risk Group (n=6): Considered inherently risky and unacceptable for minors under 18

Key Assessment Factors:

• Youth age and maturity
• AI character age and age gap with youth
• Youth intent and agency vs. AI-driven initiation
• Interaction frequency and patterns
• Behaviors and values modeled by AI

2. Assessment of Social-Emotional Support

Perceived Benefits:

• Providing non-judgmental listening outlet
• Facilitating self-expression and emotional regulation
• Offering practice space for socially shy youth

Risk Boundaries:

• AI as therapeutic but not therapy itself
• Boundary between advice and emotional attachment
• Balance between affirmation and over-affirmation
• Issues with AI's failure to recognize emotional cues

3. Assessment of Entertainment and Narrative Co-creation

Primary Concerns:

• Judging appropriateness based on character identity and source media
• Hidden risks in seemingly friendly characters
• Ambiguous and developmentally inappropriate language
• Impact of youth's trauma history and mental health status

Differences in Assessment Logic

Parent vs. Expert Risk Assessment Methods

• Parents: Employ event-based approach, flagging single concerning elements as high-risk
• Experts: Emphasize pattern-based logic, assessing whether concerning elements are repeated, escalating, or persistent

Differences in Parental Engagement Thresholds

• Experts: Tend toward high thresholds, reserving alerts only for acute crises
• Parents: Tend toward low thresholds, desiring notification in sensitive situations

Intervention Recommendations

System and Character Level

1. Rating System: Adopt movie or game-style rating systems
2. Transparency Improvements: Provide transparent information about AI character capabilities and behaviors
3. AI Literacy Education: Provide introductory education about AI nature and limitations
4. Neutral Character Mechanism: Neutral AI characters as reflection and safety mechanism

Interaction Level

1. Context-Aware Monitoring: Customize monitoring according to family values
2. Soft Stops: Gracefully handle sensitive boundaries, avoiding hard refusals
3. Reflective Questioning: Encourage reflection through questions
4. Maintaining Emotional Distance: Demonstrate care while avoiding over-attachment

Social Level

1. Expert Approach: Crisis-only escalation with stronger professional resource linkage
2. Parent Approach: Broader restrictions and notification systems

Related Work

AI Companion Development

Evolution from the 1966 ELIZA chatbot to modern generative AI companions, with market projected to reach $381.41 billion by 2032.

Youth Online Safety

Traditional parental mediation strategies (restrictive, monitoring, active mediation) show limited effectiveness in digital environments, requiring more collaborative approaches.

Stakeholder Perception Research

Existing research shows fundamental tensions between parents and youth regarding digital privacy boundaries, with AI technology introducing new concerns.

Conclusions and Discussion

Main Conclusions

1. Contextualized Risk Assessment: Parents and experts assess risks through different but complementary perspectives, with parents emphasizing value alignment and topic appropriateness, while experts focus on developmental skill acquisition and contextual harm thresholds
2. Layered Protective Measures Needed: Safety cannot be achieved through blanket prohibitions or keyword filtering; layered protective measures considering age, maturity, interaction type, and family background are necessary
3. Balancing Autonomy and Protection: Balance is needed between supporting youth exploration and protection, supporting skill learning through resilience-oriented design

Design Implications

Risk Assessment and Detection

• Need for contextualized and developmentally-oriented risk assessment approaches
• Detection models integrating multi-stakeholder perspectives
• Consideration of interaction dynamics rather than topics or wording alone

Transparency and Accountability

• Provide "AI character cards" or "conversation ratings"
• Embed youth-sensitive templates in character creation processes
• Provide AI literacy education for youth

Personalized Protective Measures

• Avoid one-size-fits-all approaches
• Provide adjustable family safety profiles
• Support co-regulation rather than mere control

In-Depth Evaluation

Strengths

1. Methodological Innovation: First multi-stakeholder study using authentic conversation data, providing richer insights than hypothetical scenarios
2. Practical Guidance Value: Provides concrete design recommendations and intervention strategies with direct implications for AI companion platform development
3. Theoretical Contribution: Identifies distinct logics in parent and expert risk assessment (event vs. pattern), providing theoretical framework for understanding stakeholder differences
4. Research Rigor: Employs rigorous qualitative research methods including multi-researcher coding and thematic analysis

Limitations

1. Sample Limitations:
   • Includes only U.S. participants, lacking cultural diversity
   • Relatively small sample size (26 participants)
   • Higher proportion of female participants (69%)
2. Absence of Youth Voice: Research focuses on adult stakeholder perspectives, lacking youth's own viewpoints and experiences
3. Unknown Long-term Effects: Short-term assessment based on conversation excerpts; unable to understand long-term usage impacts
4. Implementation Feasibility: Some proposed interventions (such as personalized monitoring, context-aware systems) may face technical implementation challenges

Impact

Academic Impact

• Establishes important empirical foundation for AI companion safety research
• Provides new research paradigm for youth safety research in human-computer interaction
• May inspire further interdisciplinary collaborative research

Practical Impact

• Provides concrete safety guidance principles for AI companion platform design
• May influence development of related regulatory policies
• Provides reference for educators and parents on AI literacy education

Social Impact

• Raises public awareness of risks in youth AI companion use
• Promotes multi-stakeholder participation in AI governance
• Provides scientific evidence for protecting youth digital wellbeing

Applicable Scenarios

1. AI Companion Platform Development: Guides design of youth safety features for platforms like Character.ai
2. Educational Technology Design: Provides reference for safety design of educational AI assistants
3. Policy Development: Provides empirical support for regulatory policies related to youth AI use
4. Family Education: Guides parents in understanding and managing youth AI use
5. Mental Health Services: Provides reference for mental health professionals understanding AI's impact on youth

Future Research Directions

1. Including Youth Perspectives: Future research should directly incorporate youth voices and experiences
2. Cross-Cultural Research: Extend to different cultural backgrounds to understand cultural differences in risk perception
3. Longitudinal Studies: Conduct long-term follow-up research to understand long-term impacts of AI companion use
4. Intervention Effectiveness Evaluation: Test the effectiveness of proposed interventions in practice
5. Technical Implementation Research: Explore how to implement proposed safety measures at the technical level

Overall Assessment: This is a high-quality human-computer interaction study providing valuable empirical insights in an emerging and important field. Despite some limitations, its contributions to AI companion safety design are significant, establishing an important foundation for responsible development in this rapidly evolving domain.