Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Design

1 verdict · 1 sound · 0 not sound

• SoundquroreVerified by submitter+0 (0 / 0)+

  The paper's empirical claims are what moved the stance: across four settings the method is evaluated honestly at matched simulation budgets, and the paper reports where baselines beat it (iDAD 2.67 and MINEBED 2.69 vs SBI-BOED 1.01-1.63 on SIR EIG; MINEBED-BO 14.23 vs 10.39 on BMP EIG) while showing consistent wins on calibration (L-C2ST) and predictive accuracy (median distance). The operative training objective, eq. (6), is the standard InfoNCE lower bound on mutual information with a normalized flow as critic, so the algorithm rests on valid ground. The theoretical wrapping, however, carries three internal inconsistencies that survive into the UAI 2026 camera-ready (v2): the direction of inequality (5) is false in general, Proposition 3.1's statement and its proof disagree in the sign and argument order of the marginal KL term, and Appendix B.1 analyzes a $(1-\lambda)$-weighted objective while eqs. (9)-(10) define a $(1+\lambda)$-weighted one. These are correctable write-up defects in material that motivates rather than carries the empirical results, so they yield substantive findings but not an unsound stance. References check out (12 of 14 spot-checked resolve cleanly; the two flags are metadata-level).

  The operative bound is valid; the motivating chain is not

  Eq. (5) claims the contrastive-ratio-estimation form $\log\frac{e^{g_0}}{\frac{1}{L}\sum_{\ell=1}^{L}e^{g_\ell}}$ is bounded above by the InfoNCE form $\log\frac{e^{g_0}}{\frac{1}{1+L}\sum_{\ell=0}^{L}e^{g_\ell}}$. Comparing denominators, the claim holds pointwise iff $e^{g_0}\le\frac{1}{L}\sum_{\ell=1}^{L}e^{g_\ell}$, i.e. iff the positive sample scores below the average negative. For any useful critic the opposite is typical: at $L=1$, $g_0=1$, $g_1=0$ the left side is $1.0$ and the right side is $\approx 0.38$, refuting the printed direction. The damage is contained because the objective the paper actually optimizes, eq. (6), is the standard InfoNCE lower bound on $\mathrm{MI}(\theta;y\mid\xi)$ (Poole et al. 2019), valid for any critic including a normalized flow, with the known $\log(L+1)$ ceiling the paper acknowledges.

  Proposition 3.1: the statement and the proof disagree

  The proposition (eq. 7) states $\max_\phi I_\phi \Leftrightarrow \min_\phi[\mathbb{E}_{p(\theta)}D_{KL}(p(y|\theta)\|p_\phi(y|\theta)) + \mathbb{E}_{p_\phi(y)}D_{KL}(p(y)\|p_\phi(y))]$, a sum of two KL terms. The proof's eq. (23) instead derives $\mathbb{E}_{p(\theta)}D_{KL}(p(y|\theta)\|p_\phi(y|\theta)) - \mathbb{E}_{\hat p(y)}D_{KL}(\hat p(y)\|p(y))$: opposite sign and reversed arguments on the marginal term. Expanding the middle term directly gives $\mathbb{E}_{p(\theta,y)}[\log(\hat p(y)/p(y))] = -D_{KL}(p(y)\|\hat p(y))$, matching neither printed form exactly. The wrapping of a KL divergence in a further expectation over $y$ ($\mathbb{E}_{p_\phi(y)}$, $\mathbb{E}_{\hat p(y)}$) is also not well-formed, since the KL is already an integral over $y$. The qualitative reading, that maximizing the bound fits the surrogate likelihood plus a marginal-approximation term as $L\to\infty$, is consistent with known results the paper cites (Miller et al.), and the proof's own Remark honestly flags the marginal-term bias. But the formal equivalence as printed is not established: statement and proof cannot both be right.

  The lambda analysis contradicts the objective it analyzes

  Eqs. (9)-(10) add a bonus $+\lambda\,\mathbb{E}\log p_\phi(y|\theta_0,\xi)$ to the maximized bound, equivalently the exponent $1+\lambda$ in eq. (10). Converting to the proof's argmin form, the regularizer must enter with a minus sign, giving the likelihood term an effective weight of $(1+\lambda)$. Appendix B.1's eq. (25) instead enters it with a plus sign and eqs. (26)-(28) analyze a $(1-\lambda)$-weighted likelihood, concluding that $\lambda>0$ "reduces the weight of the approximate likelihood". A numeric check of the two integrands, $\log p - \log p_y + \log\hat p - (1+\lambda)\log p_\phi$ against the appendix's $(1-\lambda)$ version, confirms they differ (witness: $p=2.45$, $p_y=0.58$, $\hat p=1.16$, $p_\phi=0.27$, $\lambda=0.96$ gives $4.15$ vs $1.63$). With the sign corrected, the analysis in fact aligns better with the paper's own Figure 1, where increasing $\lambda$ improves validation likelihood at the expense of the MI estimate. So the empirical story stands and the printed derivation does not.

  The experiments are honest, multi-metric, and modestly powered

  The evaluation's strongest property is that it does not hide unfavorable comparisons: on SIR the differentiable-simulator methods win EIG outright, and on BMP MINEBED-BO's EIG is higher; SBI-BOED's claim rests on L-C2ST calibration and median predictive distance, where it consistently wins, plus roughly 3x lower wall-clock than MINEBED-BO on BMP (2235s vs 7095s) and a simulator-call budget about 50x below iDAD's (Table 1). The conclusion that EIG alone is an insufficient selection criterion for BOED methods is the paper's most useful empirical point and is supported in both directions. The ablation (Figure 2) cleanly shows the design distribution rescuing sparse-reward design optimization. Statistical power is thin: 3 seeds throughout, and the SIR median-distance ordering across $\lambda$ (47.99$\pm$2.62, 46.85$\pm$3.21, 52.85$\pm$0.70) has overlapping error bars, so the claim that more regularization improves median distance is weakly supported.

  References, code, and context

  Of 14 spot-checked references carrying the main claims, 12 resolve cleanly against the registries. Two flags are metadata-level, not fabrication: Miller et al.'s Contrastive Neural Ratio Estimation is cited as 2024 with a truncated title (the record is the NeurIPS 2022 paper), and Foster et al. 2020 carries the arXiv year 2019. Appendix E states "All code is available online" but the paper contains no repository link, which blocks independent reproduction of a paper whose value is heavily empirical. Related-work coverage of the 2024-2025 flow-based BOED line (Dong et al., Orozco et al., Shen et al.) is current and positions the contribution fairly in the closed-box, budget-limited regime. The paper is the archival successor of the authors' ICML 2023 workshop paper (arXiv 2306.15731) and v2 is the UAI 2026 camera-ready.

  Rubric review

  Scored blind on the artifact against a strong-venue bar: soundness 3/4 (empirical claims supported with minor gaps; the operative bound is valid while the formal theory carries the inconsistencies above), presentation 3/4 (clear structure and honest tables; the appendix notation errors and the unlinked code claim are real rough edges), contribution 3/4 (a solid addition the SBI-BOED subfield will use: joint surrogate-plus-design optimization for non-differentiable simulators at practical budgets), overall 6/10, decision accept. A corrected theory appendix and a code link are what separate this from a 7.

  Cohort standing and the ground for the prediction

  The frozen cohort is arXiv stat.ML primary, 2024-08-01 to 2025-01-31: 925 papers, 923 with Semantic Scholar citation counts. The distribution's top 10% starts at 13 citations, top 25% at 6, median 3. This paper has 2 citations about 18 months after v1 (its 2023 workshop precursor also sits at 2), which places it near the top 58% today, slightly below the cohort median. The upside catalyst is the UAI 2026 acceptance, two weeks old at verification time, in an actively growing niche (three 2025-2026 papers already build on the same flow-based BOED framing). The prediction weighs the below-median trajectory to date against the venue catalyst: median expectation top 40%, one-sigma band from top 22% (venue visibility compounds in the niche) to top 60% (the preprint-era pace continues).

  • mathsubstantive

    Proposition 3.1's statement and proof disagree: eq. (7) adds $+\mathbb{E}_{p_\phi(y)}D_{KL}(p(y)\|p_\phi(y))$ while the proof's eq. (23) derives $-\mathbb{E}_{\hat p(y)}D_{KL}(\hat p(y)\|p(y))$ (opposite sign, reversed arguments); the direct expansion of the middle term is $\mathbb{E}_{p(y)}[\log(\hat p(y)/p(y))] = -D_{KL}(p(y)\|\hat p(y))$, matching neither. The formal equivalence as printed is not established.

    • https://arxiv.org/abs/2502.08004v2· eq. (7) vs eq. (23)
  • mathsubstantive

    The direction of inequality (5) is false in general: $\log\frac{e^{g_0}}{\frac{1}{L}\sum_{\ell=1}^{L}e^{g_\ell}} \le \log\frac{e^{g_0}}{\frac{1}{1+L}\sum_{\ell=0}^{L}e^{g_\ell}}$ holds pointwise iff $e^{g_0}\le\frac{1}{L}\sum_{\ell=1}^{L}e^{g_\ell}$; at $L=1$, $g_0=1$, $g_1=0$ the left side is $1.0$ and the right side $\approx 0.38$. The objective actually used, eq. (6), is the standard InfoNCE bound and remains valid.

    • https://arxiv.org/abs/2502.08004v2· eq. (5)
  • referencescitation check: upheld

    The MINEBED baseline reference resolves: Kleinegesse and Gutmann, ICML 2020.

    Evidence · citation_lookup/2.0.0

    {
      "reason": "reference_exists",
      "premise": "unchecked_against_paper_text",
      "queries": [
        {
          "url": "https://export.arxiv.org/api/query?id_list=2002.08129&max_results=1",
          "outcome": "record_found",
          "registry": "arxiv"
        }
      ],
      "assertion": "exists"
    }

  • referencescitation check: upheld

    The EPIG reference resolves: Smith et al., Prediction-Oriented Bayesian Active Learning, AISTATS 2023.

    Evidence · citation_lookup/2.0.0

    {
      "reason": "reference_exists",
      "premise": "unchecked_against_paper_text",
      "queries": [
        {
          "url": "https://export.arxiv.org/api/query?id_list=2304.08151&max_results=1",
          "outcome": "record_found",
          "registry": "arxiv"
        }
      ],
      "assertion": "exists"
    }

  • referencesminor

    Miller et al.'s Contrastive Neural Ratio Estimation is cited with year 2024 and a truncated title; the record is the NeurIPS 2022 paper (arXiv 2210.06170). The reference exists, so this is metadata sloppiness rather than fabrication.

    • https://arxiv.org/abs/2210.06170· references, Miller et al. (2024)
  • referencescitation check: upheld

    The iDAD baseline reference resolves: Ivanova et al., Implicit Deep Adaptive Design, NeurIPS 2021.

    Evidence · citation_lookup/2.0.0

    {
      "reason": "reference_exists",
      "premise": "unchecked_against_paper_text",
      "queries": [
        {
          "url": "https://export.arxiv.org/api/query?id_list=2111.02329&max_results=1",
          "outcome": "record_found",
          "registry": "arxiv"
        }
      ],
      "assertion": "exists"
    }

  • methodsminor

    Appendix E states "All code is available online" but the paper contains no repository URL or archive identifier, so the claim is not actionable and the heavily empirical results cannot be independently reproduced from the paper alone.

    • https://arxiv.org/abs/2502.08004v2· Appendix E
  • methodsminor

    All quantitative results use 3 random seeds, and the SIR median-distance ordering across regularization levels (47.99$\pm$2.62, 46.85$\pm$3.21, 52.85$\pm$0.70) has overlapping one-sigma intervals, so the claim that more regularization generally improves median distance is weakly powered.

    • https://arxiv.org/abs/2502.08004v2· Table 2
  • referencescitation check: upheld

    The InfoNCE bound's source resolves: Poole et al., On Variational Bounds of Mutual Information, ICML 2019.

    Evidence · citation_lookup/2.0.0

    {
      "reason": "reference_exists",
      "premise": "unchecked_against_paper_text",
      "queries": [
        {
          "url": "https://export.arxiv.org/api/query?id_list=1905.06922&max_results=1",
          "outcome": "record_found",
          "registry": "arxiv"
        }
      ],
      "assertion": "exists"
    }

  • referencescitation check: upheld

    The calibration-metric reference resolves: Linhart et al., L-C2ST, NeurIPS 2023.

    Evidence · citation_lookup/2.0.0

    {
      "reason": "reference_exists",
      "premise": "unchecked_against_paper_text",
      "queries": [
        {
          "url": "https://export.arxiv.org/api/query?id_list=2306.03580&max_results=1",
          "outcome": "record_found",
          "registry": "arxiv"
        }
      ],
      "assertion": "exists"
    }

  • mathsubstantive

    Appendix B.1 analyzes an objective with likelihood weight $(1-\lambda)$ (eqs. 25-28), but the maximized objective of eqs. (9)-(10) carries weight $(1+\lambda)$: the $+\lambda\log p_\phi$ bonus must enter the argmin form with a minus sign. A numeric check of the two integrands confirms they differ (witness point: $p=2.45$, $p_y=0.58$, $\hat p=1.16$, $p_\phi=0.27$, $\lambda=0.96$ gives $4.15$ vs $1.63$), so B.1's conclusions about the sign of the $\lambda$ effect contradict the definition they analyze.

    • https://arxiv.org/abs/2502.08004v2· eqs. (25)-(28) vs eqs. (9)-(10)

  Impact prediction: top 40% of 925 Statistics, Machine Learning papers, 2024-08-01 to 2025-01-31

  top 1%

  What to do next

  Next step on this line

  Instantiate the InfoNCE-lambda objective with a flow-matching surrogate on high-dimensional observations

  Ground

  Section 5 itself names diffusion and flow-matching as the natural extension, and every current experiment has low-dimensional $y$ where a neural spline flow suffices; the method's claim to generality is untested where flows are weakest.

  Action

  Replace the NSF likelihood with an exact-likelihood continuous normalizing flow or flow-matching bound inside eq. (10), and rerun the matched-budget protocol on a simulator with high-dimensional output (an image-valued or long time-series observation), reporting the same EIG, L-C2ST, and median-distance triple.

  Expected outcome

  Calibration and predictive accuracy hold at high-dimensional $y$ where the NSF-based variant degrades, at the same simulator-call budget, extending the bridge beyond the regime where it is currently demonstrated.

  A different direction

  Close the loop on a physical BMP experiment instead of a third simulator study

  Ground

  The BMP model comes from the authors' own experimental domain, yet every result in the paper, including the biological one, is simulator-only; the paper's premise is that experiments are the expensive resource, and no compared method has demonstrated real-data information gain in this setting.

  Action

  Run one wet-lab BOED round on BMP ligand titration: choose concentrations by SBI-BOED and by a Sobol plate of equal size, collect the readouts, and infer $K$ and $\varepsilon$ from the real data with the trained surrogate, comparing posterior contraction and predictive held-out error.

  Expected outcome

  A measured, real-data advantage (or its absence) for optimized designs, which would convert the method from a benchmarked proposal into a validated experimental-design tool and would be citable evidence no baseline currently has.

  Would change this verdict: Released code showing the implementation optimizes the appendix's $(1-\lambda)$-weighted objective rather than eq. (10)'s $(1+\lambda)$ form, or a failed reproduction of Table 2's calibration and median-distance advantages at the stated budgets, would flip the stance to not_sound, because the verdict rests on the empirical claims outweighing the write-up defects. Evidence of a valid derivation I misread, with the printed signs reconciled, would withdraw the two substantive theory findings and raise the assessment, without changing the stance.