Efficient Compositional Multi-tasking for On-device Large Language Models

Basic Information

• Paper ID: 2507.16083
• Title: Efficient Compositional Multi-tasking for On-device Large Language Models
• Authors: Ondrej Bohdal¹, Mete Ozay¹, Jijoong Moon², Kyeng-Hun Lee², Hyeonmok Ko², Umberto Michieli¹
• Institutions: ¹Samsung R&D Institute UK, ²Samsung Research, South Korea
• Classification: cs.CL cs.AI cs.LG
• Publication Date: October 11, 2025 (arXiv v2)
• Paper Link: https://arxiv.org/abs/2507.16083

Abstract

Adapter parameters provide a mechanism for modifying the behavior of machine learning models and have gained widespread attention in the large language models (LLMs) and generative AI domains. These parameters can support multi-task processing through task merging procedures. However, prior work on merging in LLMs, particularly in natural language processing, has been limited to scenarios where each test sample addresses only a single task. This paper focuses on on-device settings and investigates compositional multi-tasking problems based on text, where each test sample requires executing multiple tasks simultaneously. For example, generating a translated summary of long text requires simultaneously solving both translation and summarization tasks. To promote research in this area, we propose a benchmark containing four practical compositional tasks. We also propose an efficient method for on-device applications (Learnable Calibration), emphasizing the need for solutions that are both resource-efficient and high-performing in computationally constrained environments.

Research Background and Motivation

Problem Definition

Traditional LLM multi-task processing primarily focuses on single-task scenarios, where each test sample involves only one task (e.g., translation only or summarization only). However, practical applications frequently require compositional multi-tasking, where multiple tasks are executed simultaneously in a single inference pass, such as generating translated summaries, generating replies with specific tones, etc.

Importance Analysis

1. Practical Value: Compositional multi-tasking has widespread demand in real-world scenarios, such as intelligent replies in cross-lingual contexts, summary generation with specific tones, etc.
2. Efficiency Requirements: On-device LLMs have limited resources and need to complete multiple tasks in a single inference pass, avoiding efficiency losses from multiple inference rounds.
3. Storage Constraints: Mobile devices have limited storage and cannot train independent adapters for each compositional task.

Limitations of Existing Methods

1. Traditional Merging Strategies: Methods such as TIES and DARE perform poorly in compositional multi-task scenarios.
2. Multi-step Approaches: While effective, they require multiple inference passes, resulting in low efficiency.
3. Independent Training: Training specialized adapters for each compositional task incurs significant storage overhead.

Core Contributions

1. First Formulation of Compositional Multi-tasking: Defines the compositional multi-task processing challenge for on-device LLMs.
2. Practical Benchmark Construction: Develops a comprehensive benchmark with 14 sub-tasks, covering four major categories: summarization+translation, summarization+tone adjustment, reply+translation, and reply+tone adjustment.
3. Learnable Calibration Method: Designs two variants of efficient solutions that minimize storage and computational overhead while maintaining high performance.
4. Comprehensive Experimental Validation: Verifies the effectiveness and generalizability of the method across multiple on-device LLMs.

Method Details

Task Definition

Compositional multi-tasking is defined as: $T_C^{[N]}(x) = T_N(\ldots T_2(T_1(x)))$

where input $x$ is processed sequentially through $N$ tasks. This paper primarily investigates the case where $N=2$, including:

• Primary task $T_1$: Summarization or reply generation
• Secondary task $T_2$: Translation or tone adjustment

Model Architecture

LoRA Foundation

Based on the LoRA adapter mechanism, the adjusted forward pass is: $h = W_0x + \Delta Wx = W_0x + BAx$

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$.

Learnable Calibration Method

Core Idea: Starting from linearly merged single-task LoRAs, calibration is performed using a small number of additional parameters.

Initial Merging: $B' = \frac{1}{N}\sum_{i=1}^N B_i, \quad A' = \frac{1}{N}\sum_{i=1}^N A_i$

Variant 1 - Learnable Calibration: Uses a column-wise bias vector $p \in \mathbb{R}^d$ for calibration: $\Delta W^c = p \oplus B'A' = \sum_{i=1}^d p_i \Delta W'_i$

Variant 2 - Learnable Calibration++: Introduces a calibration LoRA matrix $P_2P_1$: $\Delta W^c = P_2P_1 + \Delta W'$

Technical Innovations

1. Lightweight Calibration: Requires only 0.08-0.56% additional parameters with storage overhead less than 0.5MB.
2. Task Specificity: Learns specialized calibration parameters for different compositional tasks.
3. Strong Compatibility: Compatible with existing frameworks (Android AI Core, Apple Intelligence).
4. Parameter Sharing: Supports cross-task parameter sharing to further reduce storage requirements.

Experimental Setup

Datasets

Benchmark Dataset Construction:

• Summarization Task: DialogSum dataset (12,460/500/1,500 train/validation/test)
• Reply Task: Synthetic Persona Chat dataset (225,061/1,000/1,000)
• Translation Task: TED Talks dataset, English to Spanish/French/German
• Tone Adjustment: Sound Natural dataset, four tones (professional/casual/humorous/narrative)

Compositional Task Generation:

• Uses OpusMT model for translation
• Uses RedPajama-INCITE-Base 3B model for tone adjustment

Evaluation Metrics

• Summarization Tasks: ROUGE-L (R-L)
• Reply Tasks: Weighted ROUGE (W-R) = $\frac{\text{ROUGE-1}}{6} + \frac{\text{ROUGE-2}}{3} + \frac{\text{ROUGE-3}}{2}$
• LLM Judge: Binary evaluation using Llama 3.1 70B

Comparison Methods

Baseline Methods:

• Zero-shot, primary task LoRA, secondary task LoRA
• In-context learning, multi-step LoRA usage
• Various merging strategies: Linear, TIES, DARE, Slerp, LoraHub, etc.

Reference Methods:

• Multi-step LoRA usage (low efficiency but good performance)
• Joint expert LoRA (specialized training for each compositional task)

Implementation Details

• Models: LLaMA 3.2 1B, Qwen2.5 1.5B, StableLM2 1.6B
• LoRA Configuration: rank=32, α=16, dropout=0.05
• Training: Adam optimizer, learning rate 5×10⁻⁵ (LoRA), 5×10⁻⁴ (calibration parameters)
• Calibration Training: Randomly selected 10,000 compositional task samples

Experimental Results

Main Results

Values in table represent LLM Judge scores (%)

Key Findings

1. Traditional Merging Strategies Fail: Existing merging methods perform extremely poorly in compositional multi-task scenarios (LLM Judge scores typically <10%).
2. Efficiency-Performance Trade-off: The proposed method achieves performance close to or exceeding multi-step baselines under single-inference constraints.
3. Consistent Performance: Learnable Calibration++ achieves the best performance across all tasks.

Ablation Studies

Storage Efficiency Analysis:

• Multi-step LoRA: 0 additional parameters, but requires 2 inference passes
• Joint Expert LoRA: 30M parameters, 57.10MB storage
• Learnable Calibration: 23K parameters, 0.05MB storage
• Learnable Calibration++: 166K parameters, 0.32MB storage

Pre-trained Adapter Contribution: Removing pre-trained LoRAs results in slight performance degradation but still outperforms most baselines, demonstrating the value of leveraging existing adapters.

Extended Analysis

1. Model Scale Adaptability: Performs well on models with 0.5B-3B parameters.
2. Out-of-Domain Generalization: Maintains stable performance across different dialogue datasets.
3. Three-task Extension: Supports three-way compositional tasks (summarization+tone+translation).

Related Work

Parameter-Efficient Fine-Tuning (PEFT)

• LoRA and Variants: DoRA, AdaLoRA, Delta-LoRA and other extension methods
• Other PEFT Methods: BitFit and other bias parameter training methods

Model Merging

• Early Work: Model Soup and other linear merging methods
• Advanced Techniques: TIES, DARE, Slerp and other conflict resolution strategies
• Adaptive Methods: LoraHub, LM-Cocktail, DAM and other learning-based merging approaches

On-device LLMs

• Compression Techniques: Model quantization, knowledge distillation, etc.
• Representative Models: LLaMA 3.2, Qwen2.5, StableLM2 and other 1-3B parameter models
• Deployment Challenges: Storage limitations, computational constraints, privacy requirements

Conclusions and Discussion

Main Conclusions

1. Problem Importance: Compositional multi-tasking is an important requirement for on-device LLMs, and traditional methods cannot effectively address it.
2. Method Effectiveness: Learnable Calibration achieves performance comparable to inefficient baselines while maintaining efficiency.
3. Practical Value: Minimal storage overhead (<0.5MB) makes the method suitable for practical deployment.

Limitations

1. Evaluation Scope: Primarily focuses on 1-3B parameter on-device models; validation on larger models is lacking.
2. Task Quantity: Primarily investigates combinations of 2-3 tasks; scalability to more tasks requires further verification.
3. Data Dependency: Requires compositional task data for training calibration parameters, unlike completely data-free merging methods.

Future Directions

1. Safety Research: Explore the impact of compositional multi-tasking on model safety mechanisms.
2. Scalability Optimization: Investigate methods for handling more task combinations.
3. Zero-shot Merging: Develop compositional multi-task methods without requiring additional data.

In-depth Evaluation

Strengths

1. Problem Innovation: First systematic investigation of compositional multi-tasking, filling an important research gap.
2. Method Practicality: Minimal storage and computational overhead suitable for practical deployment.
3. Experimental Comprehensiveness: Thorough baseline comparisons, ablation studies, and extended analyses.
4. Benchmark Contribution: The constructed 14 sub-task benchmark provides a standard evaluation platform for subsequent research.

Weaknesses

1. Insufficient Theoretical Analysis: Lacks in-depth theoretical explanation for why calibration parameters are effective.
2. Limited Task Selection: Primarily focuses on NLP tasks; applicability to other modalities is unknown.
3. Single Evaluation Metric: Primarily relies on ROUGE and LLM Judge; lacks human evaluation.

Impact

1. Academic Value: Opens a new research direction with expected follow-up work.
2. Industrial Application: Directly applicable to AI application development on mobile devices.
3. Reproducibility: Provides detailed implementation details and benchmark data.

Applicable Scenarios

1. Mobile Applications: Smartphones, tablets, and other resource-constrained devices.
2. Edge Computing: IoT devices, embedded systems.
3. Privacy-sensitive Scenarios: Applications requiring local processing to avoid data uploads.

References

The paper cites extensive related work, primarily including:

• Hu et al. (2022): Original LoRA paper
• Wortsman et al. (2022): Model Soup merging method
• Yadav et al. (2024): TIES merging strategy
• Gunter et al. (2024): Apple Intelligence on-device deployment experience

Overall Assessment: This is a high-quality research paper that addresses a practically important problem, proposes an effective solution, and conducts comprehensive experimental validation. This work provides new insights for multi-task processing in on-device LLMs and possesses significant academic and practical value.