OptiFlow
Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport
NeurIPS 2026
Department of Artificial Intelligence, Yonsei University
01 / OVERVIEW
OptiFlow learns an expressive one-step policy for offline reinforcement learning through value-weighted optimal transport. Our contributions are:
- Policy improvement with behavior-supported targets. The reference policy supplies candidate actions, and critic values determine their priority. The one-step policy learns from these targets without direct critic maximization.
- Preserving multiple high-value modes. Transport constraints allocate mass across good actions, helping retain distinct modes even when the dataset represents them unevenly.
- One-step inference with strong empirical performance. All transport computation stays in training. The deployed policy takes a single forward pass, with gains over FQL and GFP on aggregate OGBench and D4RL AntMaze benchmarks.
02 / THE CHALLENGE
In-support, high-value, expressive—and speed
We aim to address four requirements for offline RL:
- In-support. Choose actions supported by the offline data, where value estimates are more reliable.
- High-value. Favor good actions rather than simply reproduce the dataset, which may also contain suboptimal behavior.
- Expressive. Represent multiple distinct good actions without collapsing to a single mode or averaging between them.
- Speed. Generate an action in a single forward pass, avoiding iterative sampling at deployment.
Can we preserve distinct, high-value modes within behavior support—and generate actions in one step?
03 / OUR APPROACH
Learn the reference policy and critic
The reference policy proposes actions learned from offline data. The critic estimates their values.
Match with value-weighted optimal transport
Higher-value reference actions receive more weight. Optimal transport uses these weights and action distances to assign targets to one-step samples, preserving distinct high-value modes.
Distill into a one-step policy
Train the one-step policy on its assigned targets. At inference, generate an action in one forward pass.
04 / ALGORITHM
- OptiFlow has three components: a critic Q_\phi, a reference flow policy \mu_\omega, and a one-step policy \mu_\theta.
- The critic is trained with the standard TD loss.
- The reference policy is trained with value-aware behavior cloning (VaBC):
\mathcal L_{\mathrm{ref}}(\omega)=\mathbb E\left[\underbrace{g_\eta(s,a)}_{\text{Value weight}}\underbrace{\left\|v_\omega(s,a_t,t)-(a-\epsilon)\right\|_2^2}_{\text{Flow-matching loss}}\right]Here, a is a dataset action, \epsilon is Gaussian noise, and a_t = (1-t)\epsilon + ta. The weight g_\eta favors higher-value dataset actions, following Guided Flow Policy (GFP).
- The one-step policy learns from reference actions assigned by value-weighted optimal transport:
How does value-weighted optimal transport work?
For each state, sample N one-step actions a_{\theta,i}^{s} and M reference actions \tilde a_{\omega,j}^{s}. Give every one-step sample equal mass and prioritize reference actions using their critic values:
p_i=\frac{1}{N},\qquad q_j^{s}=\frac{\exp(Q_\phi(s,\tilde a_{\omega,j}^{s})/\tau)}{\sum_{k=1}^{M}\exp(Q_\phi(s,\tilde a_{\omega,k}^{s})/\tau)}Value determines the reference mass. Geometry determines the matching cost, using squared action distances normalized by their per-state mean \bar c^{s}:
C_{ij}^{s}=\frac{\|a_{\theta,i}^{s}-\tilde a_{\omega,j}^{s}\|_2^2}{\bar c^{s}}Sinkhorn computes an entropy-regularized transport plan with these row and column marginals:
P^{s,*}=\arg\min_{P\geq0}\left[\langle P,C^{s}\rangle+\varepsilon\sum_{i,j}P_{ij}(\log P_{ij}-1)\right],\qquad P\mathbf1=p,\quad P^\top\mathbf1=q^{s}Each one-step sample then takes the reference action with the largest mass in its row as its target. That transport mass also weights the distillation update below.
\mathcal L_{\mathrm{distill}}(\theta)=\mathbb E_{s\sim\mathcal D}\left[\sum_{i=1}^{N}\underbrace{P^{s,*}_{i,j_i^*}}_{\text{Transport weight}}\underbrace{\left\|\mu_\theta(s,z_i)-\tilde a_{\omega,j_i^*}^{s}\right\|_2^2}_{\text{Distillation loss}}\right],\qquad j_i^*\in\arg\max_jP^{s,*}_{ij}P^{s,*} is the value-weighted OT plan for state s. For each one-step sample i, j_i^* selects the reference action with the largest transport mass in row i. The selected action \tilde a_{\omega,j_i^*}^{s} is its distillation target. The plan and targets are fixed during the update. There is no direct critic maximization. This distillation is the same in OptiFlow and OptiFlow-BC. - The output is the one-step policy: a single forward pass at inference.
05 / CONTROLLED EXPERIMENTS
Two-mode bandit
Two high-value modes are separated by a suboptimal region. Can a policy capture both modes while staying within behavior support?
Flow Q-Learning (FQL) trains a one-step policy by balancing Q maximization and reference distillation:
Large \alpha emphasizes distillation and can pull actions into the suboptimal band between modes. Small \alpha emphasizes Q-maximization and can push actions outside behavior support.





Four equally optimal goals
The four goals are equally optimal, but unevenly represented in the dataset. FQL trades off diversity and quality as its distillation strength changes: stronger distillation improves mode coverage but reduces return. OptiFlow achieves balanced coverage of all four goals with the highest return in this experiment.
This experiment uses a standard BC-trained reference policy to isolate the effect of OT distillation from value-aware reference learning.
06 / OFFLINE RL BENCHMARKS
- We use OGBench as our main benchmark, evaluating reward-based single-task environments across robotic locomotion and manipulation with both state and pixel observations.
- We also evaluate on 6 AntMaze and 12 Adroit tasks from D4RL, covering sparse-reward navigation and dexterous manipulation.
Results
| Benchmark | IQL | ReBRAC | IFQL | FQL | GFP | OptiFlow |
|---|---|---|---|---|---|---|
| OGBench · state | 16.5 ± 0.4 | 28.8 ± 0.5 | 28.6 ± 0.6 | 39.0 ± 0.4 | 45.6 ± 0.4 | 48.3 ± 0.4 |
| OGBench · visual | 41.4 ± 3.0 | 59.8 ± 1.1 | 50.2 ± 2.0 | 65.4 ± 1.8 | 57.7 ± 1.0 | 75.9 ± 1.3 |
| D4RL · AntMaze | 58.4 | 76.9 | 61.0 ± 2.5 | 82.3 ± 1.3 | 82.6 ± 1.5 | 86.6 ± 0.7 |
| D4RL · Adroit | 53.5 | 58.6 | 48.2 ± 0.5 | 50.7 ± 0.5 | 52.1 ± 0.5 | 50.5 ± 0.4 |
What does transport contribute?
Do the gains come from a better reference policy, or from how we distill it? We compare reference and one-step policy success rates on the same 10 OGBench tasks. OptiFlow-BC sets g_\eta=1 to train a standard BC reference, while keeping the same value-weighted OT distillation as OptiFlow.
| Method | Reference training | Reference success (%) | One-step success (%) |
|---|---|---|---|
| FQL | BC | 1.7 ± 0.3 | 37.6 ± 1.1 |
| OptiFlow-BC | BC | 1.5 ± 0.2 | 48.6 ± 1.3 |
| GFP | VaBC | 28.4 ± 3.4 | 52.5 ± 0.9 |
| OptiFlow | VaBC | 19.8 ± 2.8 | 57.3 ± 1.0 |
- Better distillation. With the same BC reference setup, OptiFlow-BC raises one-step success from 37.6% to 48.6%.
- Better proposals. Adding VaBC raises it further to 57.3%. OptiFlow also outperforms GFP after distillation despite a lower reference success rate, showing that reference quality alone does not explain the gain.
Where should the value go in optimal transport?
We compare putting value in the transport cost with putting it in the reference marginal, using two equal-value modes.



- Value in the cost. A uniform reference marginal still requires low-value actions to receive equal total mass.
- Value in the marginal. High-value reference actions receive more mass, while the cost depends only on action distance.
- On the 10-task OGBench ablation, replacing the value-weighted marginal with a uniform one reduces average success from 57.3% to 9.7%.
Why use optimal transport for multimodality?
Consider two equally valuable reference modes, with all one-step samples initialized on the left.



- Independent matching. Value-distance weighted matching (VDWM) assigns each sample separately, so every sample selects the nearer left mode.
- OT marginal constraints. The coupling allocates mass across both equal-value modes, even when all samples start on one side.
07 / CITATION
BibTeX
@inproceedings{shon2026optiflow,
title={Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport},
author={Jaehun Shon and Jinha Choi and Jongwook Jeon and Jongmin Lee},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2026},
url={https://arxiv.org/abs/2609.15883}
}







