Sharp characterization of misspecification robustness for debiased GD inference

Let $Z_i=(X_i,Y_i)$, $i=1,\dots,n$, be i.i.d. observations from an unknown distribution $P$ on $\mathcal Z=\mathcal X\times\mathcal Y$. Let $\Theta\subseteq\mathbb R^d$ be the parameter space, and let $\ell:\Theta\times\mathcal Z\to\mathbb R$ be a twice continuously differentiable loss in $\theta$. Define the population and empirical risks by

Assume the population minimizer

exists and is unique (possibly with $P$ outside the working model class used to motivate $\ell$, i.e., misspecification is allowed).

Starting from $\theta^0\in\Theta$, define gradient-descent iterates

with step sizes $\eta_s>0$. For each iteration index $t$, let $\widetilde\theta^t$ denote the debiased estimator constructed from past GD iterates $\{\theta^s\}_{s\le t}$ by the debiasing scheme of interest. For a fixed nonzero contrast vector $a\in\mathbb R^d$, consider the studentized statistic

where $\widehat\sigma_t(a)$ estimates the asymptotic standard deviation of $a^\top(\widetilde\theta^t-\theta_P^\star)$.