Learning Hanzi Character Through VR-Based Mortise-Tenon

Basic Information

• Paper ID: 2510.11264
• Title: Learning Hanzi Character Through VR-Based Mortise-Tenon
• Authors: Conglin Ma, Jiatong Li, Sen-Zhe Xu, Ju Dai, Jie Liu, Feng Zhou
• Classification: cs.HC (Human-Computer Interaction)
• Publication Date: October 13, 2025 (arXiv preprint)
• Paper Link: https://arxiv.org/abs/2510.11264
• Demo Video: https://youtu.be/oUwfFTRpFyo

Abstract

This paper introduces HVRMT, an innovative VR system that redefines Hanzi character learning by integrating principles of traditional mortise-tenon joinery. The system deconstructs Hanzi components into interactive "structural radicals" analogous to wooden mortise-tenon modules, leveraging PICO's 6DoF spatial tracking technology and LLM morphological analysis to enable learners to assemble stroke sequences through haptic feedback simulating wood friction. The system supports multi-user online experiences while preserving intangible cultural heritage and enhancing learning engagement and memory retention.

Research Background and Motivation

Core Problems

1. Abstraction Learning Dilemma: Traditional Hanzi teaching methods lack embodied experiences, making it difficult for learners to establish meaningful connections with real-world contexts and cultural backgrounds
2. Cultural Heritage Transmission Challenge: Existing digital learning systems fail to adequately demonstrate the three-dimensional characteristics of Hanzi as cultural carriers
3. Insufficient Engagement: Flat, textbook-based training methods limit learners' hands-on participation and interactive exploration

Research Motivation

• Traditional "disembodied" teaching methods result in poor memory retention and shallow structural understanding
• Existing gamified systems (such as "Hanzi Factory") remain focused on static presentation, failing to establish dynamic connections between character structure and traditional culture
• While virtual learning environments have made progress, they still have limitations in cultural heritage transmission and structural complexity

Core Contributions

1. Innovative Teaching Metaphor: First systematic application of ancient mortise-tenon joinery principles to Hanzi learning, transforming abstract character components into interactive "structural radicals"
2. Multimodal VR System: Integrated VR learning environment combining PICO 6DoF spatial tracking, LLM morphological analysis, and haptic feedback
3. Digital Protection of Cultural Heritage: Reconstructs traditional craft wisdom through the concept of "constructing characters with wood," providing new pathways for digital transmission of intangible cultural heritage
4. Multi-user Collaborative Learning: Enables multi-user VR collaborative experiences, transforming Hanzi learning into a socialized cultural transmission activity

Methodology Details

Task Definition

Input: User voice description (e.g., "a cute cat") Output:

• Mortise-tenon components of corresponding Hanzi
• 3D model generation
• Character assembly verification and activation

Constraints: Mortise-tenon components must conform to traditional craft principles, and character structure must maintain accuracy

System Architecture

1. Core Concept Mapping

• Hanzi Strokes → Mortise-Tenon Components: Maps character strokes to joinery parts, allowing learners to assemble character radicals as if constructing a wooden framework
• Structural Logic → Craft Wisdom: Leverages the precision and functionality of mortise-tenon joinery to provide concrete metaphors for abstract character memorization

2. Technical Framework

Voice Processing Module:

• Utilizes PICO 6DoF spatial tracking to capture voice and motion
• Converts speech to text and extracts core characters
• Constructs prompt engineering based on ChatGLM:

{
  "model": "glm-4-flash",
  "messages": [{
    "role": "user",
    "content": "Extract the main object described in the sentence, ignoring modifiers such as colors, with the result required to be a single character"
  }]
}

LLM-Driven Morphological Analysis:

• Generates 2D images and 3D models based on user input
• Uses CogView-4 for image generation:

{
  "model": "cogView-4-250304",
  "prompt": "Simple background, no complex environment, solid color background, clear subject",
  "size": "512x512"
}

3D Model Generation:

• Implements image-to-model conversion using Tripo interface
• Loads and displays models through GltfAsset component
• Models initially exist in "deactivated" state, requiring activation through character assembly

3. Virtual Space Design

The system divides virtual space into three functional zones:

• Voice Zone (a): Speech recognition, keyword extraction, and image generation
• Model Zone (b): 3D modeling and display
• Character Zone (c): Mortise-tenon assembly and OCR recognition

Technical Innovations

1. Mortise-Tenon to Hanzi Mapping Mechanism

• Equivalent Table: Identifies component numbers and classifies them into equivalent sets
• Recipe Table: Determines whether two components can be paired based on component reusability
• Dynamic Assembly Verification: Real-time recognition of assembly process and comparison with extracted core characters

2. Multi-User Collaborative System

• First logged-in user designated as room owner
• Other users join as clients through built-in network multicast reception
• Supports real-time multi-user collaboration and cultural exchange

3. Interaction Design

• VR Thumbstick: Movement and rotation
• Trigger Button: UI interaction and related operations
• Grip Button: Component picking
• Haptic Feedback: Simulates tactile sensation of wood contact

Experimental Setup

Participants

• Sample Size: 16 participants
• Grouping Method: Divided into 4 groups, each undergoing identical testing

Experimental Design

• Comparative Experiment: Participants first learn Hanzi using the HVRMT system, then learn identical characters using alternative methods
• Evaluation Dimensions: Immersion, convenience, enjoyment, information acquisition efficiency
• Scoring Standard: 5-point Likert scale (1 = very dissatisfied, 5 = very satisfied)

Evaluation Metrics

• Average Satisfaction Index (AVG-SI): Composite satisfaction score across four dimensions
• User Experience Comparison: Multi-dimensional comparison between HVRMT system and traditional methods

Experimental Results

Main Findings

User research results demonstrate that the HVRMT system performs well across all four evaluation dimensions:

• Immersion: VR environment and mortise-tenon metaphor significantly enhance learning immersion
• Enjoyment: Integration of traditional crafts with modern technology increases learning enjoyment
• Memory Retention: Embodied interaction effectively improves memory retention rates
• Cultural Understanding: Deepens understanding of cultural connotations of Hanzi through mortise-tenon craftsmanship

System Validation

• Technical Feasibility: Successfully implements core functions including speech recognition, 3D modeling, and mortise-tenon assembly
• Educational Value: User feedback validates system effectiveness in Hanzi learning
• Multi-User Experience: Collaborative features enhance user interaction and enrich learning experience

Related Work

LLM Applications in Educational Technology

• LEAP Platform: Steinert et al. use LLM to generate formative feedback supporting autonomous learning, but limited to text interaction
• Innovation in This Work: Applies LLM to speech understanding, morphological analysis, and 3D interactive model generation, bridging semantic understanding and embodied interaction

Digital Cultural Heritage Protection

• AR Mortise-Tenon Teaching: Lee (2019) uses AR to teach mortise-tenon structures, but lacks semantic mapping to Hanzi
• Collaborative Writing Communities: Yilmaz (2022) proposes cloud-based collaboration concepts; this work extends it to VR multi-user interactive environments

Conclusions and Discussion

Main Conclusions

1. HVRMT system successfully integrates Hanzi learning with mortise-tenon craftsmanship, providing embodied cultural learning experiences
2. Multimodal VR technology effectively enhances learning engagement and memory retention
3. Multi-user collaborative features strengthen socialized learning and cultural transmission effects

Limitations

1. Limited Sample Size: Only 16 participants; larger-scale experimental validation needed
2. Content Coverage: Current mortise-tenon components and character types are limited; content library expansion required
3. Long-term Effects: Lacks longitudinal tracking research on learning outcomes
4. Technology Dependency: Requires professional VR equipment, potentially limiting widespread adoption

Future Directions

1. Expand content library with more Hanzi characters and mortise-tenon types
2. Conduct larger-scale participant experiments for evaluation
3. Research system's long-term impact on Hanzi reading and writing acquisition
4. Explore possibilities of combining other traditional crafts with language learning

In-Depth Evaluation

Strengths

1. Conceptual Innovation: The mortise-tenon to Hanzi mapping teaching metaphor is highly creative, making abstract learning concrete
2. Technical Integration: Successfully integrates VR, LLM, speech recognition, 3D modeling, and other technologies
3. Cultural Value: Incorporates cultural heritage protection into language learning, holding significant social significance
4. User Experience: Multimodal interaction and collaborative features provide rich learning experiences

Weaknesses

1. Experimental Scale: Sample size of 16 participants is relatively small with limited statistical power
2. Quantitative Analysis: Lacks detailed quantification of learning outcomes and statistical significance testing
3. Comparison Baseline: Does not clearly specify the content of "alternative methods," affecting comparison validity
4. Technical Details: Mapping rules between mortise-tenon components and character structure lack sufficient detail

Impact

1. Academic Contribution: Provides new perspectives for VR education and digital cultural heritage protection
2. Practical Value: Applicable to multiple fields including Chinese language teaching and cultural education
3. Reproducibility: Provides system architecture and implementation details, though additional technical specifications needed
4. Cross-disciplinary Value: Combines HCI, educational technology, and cultural protection

Applicable Scenarios

1. Chinese as Second Language Teaching: Provides immersive Hanzi learning experiences for foreign learners
2. Cultural Education: Interactive displays in museums and cultural centers
3. Traditional Craft Education: Digital transmission and teaching of mortise-tenon craftsmanship
4. Collaborative Learning Environments: Supports remote multi-user collaborative language learning platforms

References

The paper cites 10 relevant references covering key areas including LLM educational applications, digital cultural heritage, and VR interaction design, providing a solid theoretical foundation for the research.

Overall Assessment: This is an innovative and practically valuable HCI research paper that successfully combines traditional culture with modern technology, providing new solutions for language learning and cultural transmission. While improvements are needed in experimental scale and quantitative analysis, its conceptual innovation and technical integration are commendable.