CCDP: Composition of Conditional Diffusion Policies with Guided Sampling

Basic Information

• Paper ID: 2503.15386
• Title: CCDP: Composition of Conditional Diffusion Policies with Guided Sampling
• Authors: Amirreza Razmjoo (Honda Research Institute Europe & Idiap Research Institute & EPFL), Sylvain Calinon (Idiap Research Institute & EPFL), Michael Gienger (Honda Research Institute Europe), Fan Zhang (Honda Research Institute Europe)
• Classification: cs.RO (Robotics), cs.AI (Artificial Intelligence)
• Publication Date: October 10, 2025 (arXiv v2)
• Paper Link: https://arxiv.org/abs/2503.15386

Abstract

Imitation learning provides a promising approach for learning directly from data without explicit models, simulation, or detailed task definitions. During inference, actions are sampled from the learned distribution and executed on the robot. However, sampled actions may fail for various reasons, and simply repeating the sampling steps until a successful action is obtained may be inefficient. This paper proposes an enhanced sampling strategy that avoids previously unsuccessful actions by improving the sampling distribution. By leveraging only successful demonstration data, the method can infer recovery actions without requiring additional exploration behaviors or high-level controllers. Furthermore, utilizing the concept of diffusion model decomposition, the primary problem that may require long-term history to manage failures is decomposed into multiple smaller, more manageable subproblems, enabling the system to adapt to variable failure counts. The method produces a low-level controller that dynamically adjusts its sampling space to improve efficiency when previous samples prove insufficient.

Research Background and Motivation

Problem Definition

The core problem this research addresses is: How can robots effectively recover when actions sampled from a learned policy distribution fail?

Problem Significance

1. Practical Application Requirements: In real-world environments, robots frequently encounter partially constrained or uncertain situations, such as searching for a bedside lamp switch or pushing a door with uncertain direction
2. Efficiency Issues: Traditional methods simply resample from the same distribution, ignoring information about known failure regions, leading to inefficiency
3. Practical Limitations: Existing failure recovery methods typically require additional resources (simulation environments, high-level reasoning models, expert guidance) that may be unavailable in practical applications

Limitations of Existing Methods

1. Two-Level Planning Approaches:
   • High-level planners select action primitives, low-level controllers execute them
   • Subject to suboptimal results and combinatorial explosion problems
   • Decision-making becomes computationally expensive as options increase
2. Robust Policy Learning:
   • Methods similar to robust reinforcement learning
   • Can only handle certain types of failures (e.g., environmental parameter changes)
   • For broader failure types (e.g., button searching), a single robust policy may not exist
3. History-Aware Policies:
   • Require failure data for training, increasing data collection complexity
   • Require long-term memory, resulting in high computational complexity

Core Contributions

1. Proposed a Decomposed Diffusion Policy Framework: Enhanced modularity and controllability of diffusion policies and analyzed the impact of each module
2. Designed a Recovery Strategy Based on Negative Guidance: Unlike traditional methods, uses failure cases as negative guidance to steer the policy away from failure regions
3. Achieved Failure Recovery Without Data Annotation: Uses only successful demonstration data, identifying recovery actions through offline analysis
4. Validated Method Effectiveness: Comprehensive comparison with state-of-the-art baselines on multiple tasks

Method Details

Task Definition

Given a dataset of M successful demonstrations $\mathcal{D} = \{(a_t, x_t, h^H_t)_i\}_{i=1}^M$, the goal is to learn a diffusion policy to model the conditional distribution $p_\pi^{\mathcal{D}}(a_t | x_t, h^H_t)$, where:

• $a_t \in \mathbb{R}^{d_u}$: action at time t
• $x_t \in \mathbb{R}^{d_s}$: state
• $h^H_t = [a_{t-H:t-1}^T, x_{t-H:t-1}^T]^T$: history of H previous actions and states

When an action fails, the system needs to condition on the failure feature set: $a_t \sim p_\pi(a_t | x_t, h^H_t, z^f_{1:N})$

where $z^f_i = z(a^f_i, x^f_i)$ extracts key features from the i-th failure.

Model Architecture

Diffusion Model Decomposition

The conditional distribution is decomposed into a product of multiple simpler subproblems:

$p_\pi(a_t | x_t, h^H_t, z^f_{1:N}) \propto \frac{p_s(a_t | x_t)}{p_a(a_t)} \cdot \frac{p_h(a_t | h^H_t)}{p_a(a_t)} \cdot \prod_{i=1}^N \frac{p_z(a_t | z^f_i)}{p_a(a_t)}$

The corresponding denoising term decomposition is: $\hat{\varepsilon}(a^k_t, k) = \varepsilon_a(a_t, k) + w_s(\varepsilon_s(a_t, x_t, k) - \varepsilon_a(a_t, k)) + w_h(\varepsilon_h(a_t, h^H_t, k) - \varepsilon_a(a_t, k)) + \sum_{i=1}^N w^i_z(\varepsilon_z(a_t, z^f_i, k) - \varepsilon_a(a_t, k))$

Module Functions

1. $\varepsilon_a(a_t, k)$: Encourages sampling actions similar to demonstrations
2. $\varepsilon_s(a_t, x_t, k)$: Guides actions to match current state
3. $\varepsilon_h(a_t, h^H_t, k)$: Promotes temporal continuity
4. $\varepsilon_z(a_t, z^f_i, k)$: Negative guidance, steering away from failure regions

Recovery Model Design

Recovery Action Definition

Define the recovery action set: