YONSEI UNIVERSITY / DILLAB

OptiFlow

Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport

NeurIPS 2026

Jaehun Shon*Jinha Choi*Jongwook JeonJongmin Lee†

Department of Artificial Intelligence, Yonsei University

01 / OVERVIEW

OptiFlow learns an expressive one-step policy for offline reinforcement learning through value-weighted optimal transport. Our contributions are:

  • Policy improvement with behavior-supported targets. The reference policy supplies candidate actions, and critic values determine their priority. The one-step policy learns from these targets without direct critic maximization.
  • Preserving multiple high-value modes. Transport constraints allocate mass across good actions, helping retain distinct modes even when the dataset represents them unevenly.
  • One-step inference with strong empirical performance. All transport computation stays in training. The deployed policy takes a single forward pass, with gains over FQL and GFP on aggregate OGBench and D4RL AntMaze benchmarks.

02 / THE CHALLENGE

In-support, high-value, expressive—and speed

We aim to address four requirements for offline RL:

  • In-support. Choose actions supported by the offline data, where value estimates are more reliable.
  • High-value. Favor good actions rather than simply reproduce the dataset, which may also contain suboptimal behavior.
  • Expressive. Represent multiple distinct good actions without collapsing to a single mode or averaging between them.
  • Speed. Generate an action in a single forward pass, avoiding iterative sampling at deployment.

Can we preserve distinct, high-value modes within behavior support—and generate actions in one step?

03 / OUR APPROACH

Figure 1: reference policy and critic learning, entropic optimal transport, and distillation into a one-step policy.
  1. Learn the reference policy and critic

    The reference policy proposes actions learned from offline data. The critic estimates their values.

  2. Match with value-weighted optimal transport

    Higher-value reference actions receive more weight. Optimal transport uses these weights and action distances to assign targets to one-step samples, preserving distinct high-value modes.

  3. Distill into a one-step policy

    Train the one-step policy on its assigned targets. At inference, generate an action in one forward pass.

04 / ALGORITHM

  • OptiFlow has three components: a critic Q_\phi, a reference flow policy \mu_\omega, and a one-step policy \mu_\theta.
  • The critic is trained with the standard TD loss.
  • The reference policy is trained with value-aware behavior cloning (VaBC):
    \mathcal L_{\mathrm{ref}}(\omega)=\mathbb E\left[\underbrace{g_\eta(s,a)}_{\text{Value weight}}\underbrace{\left\|v_\omega(s,a_t,t)-(a-\epsilon)\right\|_2^2}_{\text{Flow-matching loss}}\right]
    Here, a is a dataset action, \epsilon is Gaussian noise, and a_t = (1-t)\epsilon + ta. The weight g_\eta favors higher-value dataset actions, following Guided Flow Policy (GFP).
  • The one-step policy learns from reference actions assigned by value-weighted optimal transport:
    How does value-weighted optimal transport work?

    For each state, sample N one-step actions a_{\theta,i}^{s} and M reference actions \tilde a_{\omega,j}^{s}. Give every one-step sample equal mass and prioritize reference actions using their critic values:

    p_i=\frac{1}{N},\qquad q_j^{s}=\frac{\exp(Q_\phi(s,\tilde a_{\omega,j}^{s})/\tau)}{\sum_{k=1}^{M}\exp(Q_\phi(s,\tilde a_{\omega,k}^{s})/\tau)}

    Value determines the reference mass. Geometry determines the matching cost, using squared action distances normalized by their per-state mean \bar c^{s}:

    C_{ij}^{s}=\frac{\|a_{\theta,i}^{s}-\tilde a_{\omega,j}^{s}\|_2^2}{\bar c^{s}}

    Sinkhorn computes an entropy-regularized transport plan with these row and column marginals:

    P^{s,*}=\arg\min_{P\geq0}\left[\langle P,C^{s}\rangle+\varepsilon\sum_{i,j}P_{ij}(\log P_{ij}-1)\right],\qquad P\mathbf1=p,\quad P^\top\mathbf1=q^{s}

    Each one-step sample then takes the reference action with the largest mass in its row as its target. That transport mass also weights the distillation update below.

    \mathcal L_{\mathrm{distill}}(\theta)=\mathbb E_{s\sim\mathcal D}\left[\sum_{i=1}^{N}\underbrace{P^{s,*}_{i,j_i^*}}_{\text{Transport weight}}\underbrace{\left\|\mu_\theta(s,z_i)-\tilde a_{\omega,j_i^*}^{s}\right\|_2^2}_{\text{Distillation loss}}\right],\qquad j_i^*\in\arg\max_jP^{s,*}_{ij}
    P^{s,*} is the value-weighted OT plan for state s. For each one-step sample i, j_i^* selects the reference action with the largest transport mass in row i. The selected action \tilde a_{\omega,j_i^*}^{s} is its distillation target. The plan and targets are fixed during the update. There is no direct critic maximization. This distillation is the same in OptiFlow and OptiFlow-BC.
  • The output is the one-step policy: a single forward pass at inference.

05 / CONTROLLED EXPERIMENTS

Two-mode bandit

Two high-value modes are separated by a suboptimal region. Can a policy capture both modes while staying within behavior support?

Flow Q-Learning (FQL) trains a one-step policy by balancing Q maximization and reference distillation:

\underbrace{\mathbb E_{s\sim\mathcal D,\,a^\pi\sim\pi_\theta}\!\left[-Q_\phi(s,a^\pi)\right]}_{\text{Q loss}} + \alpha\cdot \underbrace{\mathbb E_{s\sim\mathcal D,\,z\sim\mathcal N(0,I_d)}\!\left[\left\|\mu_\theta(s,z)-\mu_\omega(s,z)\right\|_2^2\right]}_{\text{Distillation loss}}
\begin{aligned}&\underbrace{\mathbb E_{s\sim\mathcal D,\,a^\pi\sim\pi_\theta}\!\left[-Q_\phi(s,a^\pi)\right]}_{\text{Q loss}}\\[12pt]+{}&\alpha\cdot \underbrace{\mathbb E_{s\sim\mathcal D,\,z\sim\mathcal N(0,I_d)}\!\left[\left\|\mu_\theta(s,z)-\mu_\omega(s,z)\right\|_2^2\right]}_{\text{Distillation loss}}\end{aligned}

Large \alpha emphasizes distillation and can pull actions into the suboptimal band between modes. Small \alpha emphasizes Q-maximization and can push actions outside behavior support.

Two-mode bandit: True reward
True rewardTwo high-value modes
Two-mode bandit: Gaussian policy
Gaussian policyOne mode, outside support
Two-mode bandit: FQL, large alpha
FQL, large \alphaStrong distillation: suboptimal band
Two-mode bandit: FQL, small alpha
FQL, small \alphaWeak distillation: outside support
Two-mode bandit: OptiFlow
OptiFlowBoth good modes, within support

Four equally optimal goals

The four goals are equally optimal, but unevenly represented in the dataset. FQL trades off diversity and quality as its distillation strength changes: stronger distillation improves mode coverage but reduces return. OptiFlow achieves balanced coverage of all four goals with the highest return in this experiment.

This experiment uses a standard BC-trained reference policy to isolate the effect of OT distillation from value-aware reference learning.

Four-mode environment: Dataset
Dataset
Four-mode environment: FQL / α = 0.1
FQL / \alpha = 0.1One goal, sharp paths
Four-mode environment: FQL / α = 1
FQL / \alpha = 1
Four-mode environment: FQL / α = 10
FQL / \alpha = 10More goals, noisier paths
Four-mode environment: OptiFlow
OptiFlowAll four goals, highest return

06 / OFFLINE RL BENCHMARKS

OGBench

D4RL

AntMaze
HumanoidMaze
Cube
Scene
AntMaze
Adroit hand manipulating a pen
Adroit · Pen
  • We use OGBench as our main benchmark, evaluating reward-based single-task environments across robotic locomotion and manipulation with both state and pixel observations.
  • We also evaluate on 6 AntMaze and 12 Adroit tasks from D4RL, covering sparse-reward navigation and dexterous manipulation.

Results

BenchmarkIQLReBRACIFQLFQLGFPOptiFlow
OGBench · state16.5 ± 0.428.8 ± 0.528.6 ± 0.639.0 ± 0.445.6 ± 0.448.3 ± 0.4
OGBench · visual41.4 ± 3.059.8 ± 1.150.2 ± 2.065.4 ± 1.857.7 ± 1.075.9 ± 1.3
D4RL · AntMaze58.476.961.0 ± 2.582.3 ± 1.382.6 ± 1.586.6 ± 0.7
D4RL · Adroit53.558.648.2 ± 0.550.7 ± 0.552.1 ± 0.550.5 ± 0.4
Full results

OGBench · 1/2

Appendix E: OGBench · 1/2 per-task results, captured from the paper.

OGBench · 2/2

Appendix E: OGBench · 2/2 per-task results, captured from the paper.

D4RL

Appendix E: D4RL per-task results, captured from the paper.
What does transport contribute?

Do the gains come from a better reference policy, or from how we distill it? We compare reference and one-step policy success rates on the same 10 OGBench tasks. OptiFlow-BC sets g_\eta=1 to train a standard BC reference, while keeping the same value-weighted OT distillation as OptiFlow.

MethodReference trainingReference success (%)One-step success (%)
FQLBC1.7 ± 0.337.6 ± 1.1
OptiFlow-BCBC1.5 ± 0.248.6 ± 1.3
GFPVaBC28.4 ± 3.452.5 ± 0.9
OptiFlowVaBC19.8 ± 2.857.3 ± 1.0
  • Better distillation. With the same BC reference setup, OptiFlow-BC raises one-step success from 37.6% to 48.6%.
  • Better proposals. Adding VaBC raises it further to 57.3%. OptiFlow also outperforms GFP after distillation despite a lower reference success rate, showing that reference quality alone does not explain the gain.
Where should the value go in optimal transport?

We compare putting value in the transport cost with putting it in the reference marginal, using two equal-value modes.

Bimodal Q
Bimodal Q
Value in cost
Value in cost
OptiFlow
OptiFlow
  • Value in the cost. A uniform reference marginal still requires low-value actions to receive equal total mass.
  • Value in the marginal. High-value reference actions receive more mass, while the cost depends only on action distance.
  • On the 10-task OGBench ablation, replacing the value-weighted marginal with a uniform one reduces average success from 57.3% to 9.7%.
Why use optimal transport for multimodality?

Consider two equally valuable reference modes, with all one-step samples initialized on the left.

Initialization · all on the left
Initialization · all on the left
VDWM · collapses left
VDWM · collapses left
OptiFlow · both modes
OptiFlow · both modes
  • Independent matching. Value-distance weighted matching (VDWM) assigns each sample separately, so every sample selects the nearer left mode.
  • OT marginal constraints. The coupling allocates mass across both equal-value modes, even when all samples start on one side.

07 / CITATION

BibTeX

@inproceedings{shon2026optiflow,
  title={Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport},
  author={Jaehun Shon and Jinha Choi and Jongwook Jeon and Jongmin Lee},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2026},
  url={https://arxiv.org/abs/2609.15883}
}