The Payoff Function
What is the value of design \(\xi\) against an adversarial law \(q(\theta, x \mid \xi)\)?
The experimenter commits to the prior \(p(\theta)\) as a fixed reference forecast and optimizes the post-observation forecast rule. Under the log scoring rule, this yields:
\[
U(\xi, q) \;=\; \sup_{\psi_\xi} \int q(\theta, x \mid \xi)\Big[\log \psi_\xi(x)(\theta) - \log p(\theta)\Big]\,\mathrm{d}\theta\,\mathrm{d}x.
\]
The supremum is attained at \(\psi_\xi^\star(x) = q(\cdot \mid x, \xi)\), giving the closed form:
\[
U(\xi, q) \;=\; \mathbb{D}_{\mathrm{KL}}\!\left[ q(\theta, x \mid \xi) \;\middle\|\; p(\theta)\, q(x \mid \xi) \right].
\]
The KL Ambiguity Set
Without constraints, nature trivially sets \(\theta \perp x\) and drives the gain to zero. We constrain nature to an average KL neighbourhood of the nominal model:
\[
\mathcal{Q}_{\rho}(\pi) \;:=\; \left\{ q(\theta, x \mid \xi) \;\Big|\; \mathbb{E}_{\pi}\!\left[ \mathbb{D}_{\mathrm{KL}}\!\left[ q(\theta, x \mid \xi) \;\middle\|\; p(\theta, x \mid \xi) \right] \right] \le \rho \right\}.
\]
- \(p(\theta, x \mid \xi)\): experimenter’s nominal model, their best approximation to reality.
- \(\rho > 0\): misspecification budget. Small \(\rho\) means mild perturbations; large \(\rho\) means strong misspecification.
- The average constraint lets nature allocate its budget strategically, attacking the most informative designs most aggressively.
Solving the Game: Lagrangian Decomposition
The inner minimisation \(\inf_{q \in \mathcal{Q}_\rho} \mathbb{E}_\pi[U(\xi,q)]\) is solved via Lagrangian duality. The global problem decomposes into pointwise regularized problems:
\[
\inf_{q \in \mathcal{Q}_{\rho}} \mathbb{E}_{\pi}\!\left[U(\xi, q)\right] \;=\; \sup_{\beta > 0} \left\{ \mathbb{E}_{\pi}\!\left[\mathcal{J}_{\beta}(\xi)\right] - \beta\rho \right\},
\]
where \(\mathcal{J}_{\beta}(\xi)\) is the regularized pointwise utility:
\[
\mathcal{J}_{\beta}(\xi) \;:=\; \inf_{q}\, \Big\{ U(\xi, q) + \beta\, \mathbb{D}_{\mathrm{KL}}\!\left[ q(\theta, x \mid \xi) \;\middle\|\; p(\theta, x \mid \xi) \right] \Big\}.
\]
The dual parameter \(\beta\) (or equivalently \(\rho\)) is a robustness knob: larger \(\beta\) penalizes deviations more, reflecting greater trust in the nominal model.
Main Result: Robust EIG = Sibson’s \(\alpha\)-MI
Proposition. For \(\beta > 0\), let \(\alpha := \beta/(1+\beta) \in (0,1)\). Then \(\mathcal{J}_\beta(\xi)\) admits the closed form:
\[
\begin{aligned}
\mathcal{J}_{\beta}(\xi) \;=\; \inf_{\nu}\, \mathbb{D}_{\alpha}\!\left[ p(\theta, x \mid \xi) \;\middle\|\; p(\theta)\,\nu(x \mid \xi) \right] \;&=\; \mathbb{D}_{\alpha}\!\left[ p(\theta, x \mid \xi) \;\middle\|\; p(\theta)\,p_{\alpha}(x \mid \xi) \right] \\
&=:\, I^{S}_{\alpha}(\theta; x)(\xi),
\end{aligned}
\]
where \(\mathbb{D}_{\alpha}\) is Rényi’s divergence of order \(\alpha\), and the \(\alpha\)-tilted marginal is:
\[
p_{\alpha}(x \mid \xi) \;\propto\; \left[\mathbb{E}_{p(\theta)}\!\left[ p(x \mid \theta, \xi)^{\alpha} \right]\right]^{1/\alpha}.
\]
The quantity \(I^{S}_{\alpha}(\theta; x)(\xi)\) is Sibson’s \(\alpha\)-mutual information: a known but rarely used information measure, here emerging from first principles as the robust expected information gain.
The order \(\alpha\) is the dual variable associated with the KL radius \(\rho\), encoding the experimenter’s confidence in the nominal model.
The Worst-Case Posterior: \(\alpha\)-Tilted Inference
Corollary. The adversarial joint law that achieves the infimum in the previous result is a geometric mixture. Its implied posterior is:
\[
q^{\star}(\theta \mid x, \xi) \;\propto\; p(\theta)\, p(x \mid \theta, \xi)^{\alpha}.
\]
![]()
Acting consistently with the adversarial assumption requires tempering the likelihood with \(\alpha \in (0,1)\). This is the update rule of generalized Bayesian inference under misspecification.
