Variational Inference for Count Response Semiparametric Regression: A Convex Solution

Basic Information

• Paper ID: 2510.12356
• Title: Variational Inference for Count Response Semiparametric Regression: A Convex Solution
• Authors: Virginia Murru (Università di Padova), Matt P. Wand (University of Technology Sydney)
• Classification: stat.ME (Statistics - Methodology)
• Publication Date: October 14, 2025
• Paper Link: https://arxiv.org/abs/2510.12356

Abstract

This paper develops a variational inference method for Bayesian count response regression models with desirable properties of convexity and closed-form updates. The convexity guarantees numerically stable fitting algorithms, while closed-form updates enable fast and easy implementation. The method's core employs Pólya-Gamma augmented negative binomial likelihood, finite-valued priors for shape parameters, and a structured mean-field variational Bayes framework. The approach applies to general count response settings and specifically targets generalized linear mixed models within the semiparametric regression category. The paper also describes a real-time fitting algorithm.

Research Background and Motivation

Problem Definition

This research addresses Bayesian inference in count response semiparametric regression. Traditional Markov Chain Monte Carlo (MCMC) methods face computational bottlenecks in speed and scalability, while existing variational inference methods suffer from numerical instability due to non-convexity.

Research Motivation

1. Numerical Stability Issues: Although the fixed-form variational method of Luts & Wand (2015) achieves good accuracy, non-convexity may cause numerical problems. In simulation studies, the fixed-form variational method for Poisson nonparametric regression fails to converge correctly in 13.6% of replications.
2. Computational Efficiency Requirements: Existing methods involve numerical integration steps, whereas the proposed method features completely closed-form updates, enabling fast and stable algorithms.
3. Real-Time Application Demands: In streaming data applications, methods are needed that can update model parameters online without storing complete historical data.

Core Contributions

1. Proposed a Convex Variational Inference Framework: Developed a variational inference method where all component optimization problems are convex, ensuring numerical stability.
2. Achieved Closed-Form Updates: Through Pólya-Gamma augmentation, all update steps have closed-form solutions, avoiding numerical integration.
3. Constructed a Structured Mean-Field Variational Bayes Scheme: Addressed the challenging shape parameter inference in negative binomial distributions through discretization of shape parameter κ combined with variational Bayes model averaging.
4. Developed a Pure Online Real-Time Algorithm: Proposed a real-time algorithm requiring only updates and storage of sufficient statistics, with streaming data discarded after processing.

Methodology Details

Task Definition

Consider the following count response Bayesian semiparametric regression model:

$y_i|\beta,u,\kappa \stackrel{ind.}{\sim} \text{Negative-Binomial}(\exp\{(X\beta + Zu)_i\}, \kappa), \quad 1 \leq i \leq n$

where $y_i$ is the count response variable, and $X$ and $Z$ are design matrices for fixed and random effects, respectively.

Model Architecture

1. Pólya-Gamma Augmentation

Introduce Pólya-Gamma auxiliary variables: $\alpha_i|y_i,\beta,u,\kappa \stackrel{ind.}{\sim} \text{Pólya-Gamma}(y_i + \kappa, (X\beta + Zu)_i + \log(\kappa))$

2. Prior Distribution Specification

• Regression coefficients: $\beta \sim N(0, \sigma_\beta^2 I_p)$
• Random effects: $u|\sigma_1^2,\ldots,\sigma_r^2 \sim N(0, \text{blockdiag}(\sigma_1^2 I_{K_1}, \ldots, \sigma_r^2 I_{K_r}))$
• Variance parameters: $\sigma_j \stackrel{ind.}{\sim} \text{Half-Cauchy}(s_\sigma)$
• Shape parameter: $\kappa$ has a discrete prior with atom set $\mathcal{K}$ and probabilities $p(\kappa)$

3. Structured Mean-Field Variational Bayes

Use the following restricted product density form: $q(\beta,u,\kappa,\alpha,\sigma^2,a) = q(\beta,u,a|\kappa)q(\sigma^2,\alpha|\kappa)q(\kappa)$

Core Algorithm

Algorithm 1: Key update steps of the structured mean-field variational Bayes algorithm include:

1. Pólya-Gamma Parameter Update: $\mu_q(\alpha|\kappa) \leftarrow 2(y + \kappa\mathbf{1}) \odot \lambda_{JJ}(c_q(\alpha|\kappa))$
2. Regression Parameter Update: $\Sigma_q(\beta,u|\kappa) \leftarrow \{C^T\text{diag}(\mu_q(\alpha|\kappa))C + M_q(1/\sigma^2|\kappa)\}^{-1}$
3. Variance Parameter Update: Updated through sufficient statistics of inverse-Gamma distributions

where $\lambda_{JJ}(x) = \frac{\tanh(x/2)}{4x}$ is the Jaakkola-Jordan function.

Technical Innovations

1. Convexity Guarantee: Each individual mean-field optimization problem (for $\kappa \in \mathcal{K}$) is convex, ensuring global optimal solutions.
2. Closed-Form Updates: Through special properties of Pólya-Gamma augmentation, numerical integration is avoided and all updates have analytical solutions.
3. Discretization Strategy: The continuous shape parameter $\kappa$ is discretized to a finite set and handled through variational Bayes model averaging.

Experimental Setup

Datasets

1. Simulated Data:
   • Sample size: n=500
   • Negative binomial additive model: $\eta_{\text{true},1}(x) = \cos(4\pi x) + 2x$, $\eta_{\text{true},2}(x) = 0.4\phi(x;0.38,0.08) - 1.02x + 0.018x^2 + 0.08\phi(x;0.75,0.03)$
   • True shape parameter: $\kappa_{\text{true}} = 3.8$
2. Real Data: Ragweed pollen count data from Kalamazoo, USA (1991-1994) (n=334)

Evaluation Metrics

Accuracy score defined as: $\text{accuracy}(q^*) = 100\left(1 - \frac{1}{2}\int_{-\infty}^{\infty}|q^*(θ) - p(θ|y)|dθ\right)\%$

Comparison Methods

• MCMC Benchmark: Using JAGS Bayesian inference engine with chain length 10000, burn-in 5000, thinning factor 5
• Existing Variational Method: Semiparametric mean-field variational Bayes method of Luts & Wand (2015)

Implementation Details

• Atom set $\mathcal{K}$: 50 geometric sequence points, range $[\kappa_{\text{true}}/10, 10\kappa_{\text{true}}]$
• Convergence criterion: Relative change less than $10^{-10}$
• Spline basis functions: O'Sullivan spline basis with $K_1 = K_2 = 17$ basis functions

Experimental Results

Main Results

Accuracy Assessment

Simulation studies demonstrate satisfactory accuracy across all metrics:

• Function estimation accuracy: Function value estimates at different quantiles achieve 89%-94% accuracy
• Variance parameter estimation: Accuracy for $\sigma_1^2$ and $\sigma_2^2$ are 80% and 73%, respectively
• Shape parameter estimation: Accuracy for $\kappa$ reaches 99%

Compared to Luts & Wand (2015), the proposed method shows improvements across all parameters, with the most significant improvement for $\kappa$.

Speed Comparison

Computational efficiency is significantly enhanced:

• MCMC Method: Average 117.8 seconds (standard deviation 1.876 seconds)
• Proposed Method: Average 2.088 seconds (standard deviation 0.1440 seconds)

Speed improvement of approximately 56-fold while maintaining good inference accuracy.

Real-Time Algorithm Evaluation

Algorithm 2 real-time fitting results demonstrate:

• As sample size grows from 100 to 1000, real-time estimates align closely with batch processing estimates
• For different $\kappa_{\text{true}}$ values (5, 10, 20, 40), the algorithm performs stably
• Real-time inference primarily targets the mean structure; real-time inference for $\kappa$ parameter is relatively challenging

Real Data Application

Application to ragweed pollen count data shows:

• All linear effect coefficients are significantly nonzero
• The posterior distribution of shape parameter $\kappa$ concentrates between 2-5, supporting the applicability of the negative binomial response model
• Trend curves across four years show similar behavior: peaking around day 20 of the season, followed by declining trends

Related Work

Main Research Directions

1. Variational Inference Methods: Binary response model methods by Jaakkola & Jordan (2000), Durante & Rigon (2019)
2. Pólya-Gamma Augmentation: Negative binomial likelihood augmentation methods by Polson et al. (2013), Zhou et al. (2012), Miao et al. (2020)
3. Semiparametric Regression: Count response semiparametric regression variational inference by Luts & Wand (2015)

Advantages of This Work

1. Compared to Zhou et al. (2012) and Miao et al. (2020): The proposed method is based on minimum KL divergence from a single joint distribution, providing a more solid theoretical foundation
2. Compared to Luts & Wand (2015): Resolves non-convexity issues and provides completely closed-form updates
3. Compared to Traditional MCMC: Dramatically improves computational speed while maintaining reasonable accuracy

Conclusions and Discussion

Main Conclusions

1. Successfully developed a variational inference method for count response semiparametric regression with convexity and closed-form updates
2. Achieved numerically stable algorithms through Pólya-Gamma augmentation and structured mean-field variational Bayes
3. Provided a pure online real-time fitting algorithm applicable to streaming data applications

Limitations

1. Accuracy Loss: Variational approximation shows some accuracy loss compared to MCMC, particularly in posterior distribution width estimation
2. Discretization Effects: Discretization of shape parameter $\kappa$ may affect inference accuracy
3. Atom Set Reduction in Real-Time Algorithm: Dynamic adjustment of atom sets in online algorithms requires further investigation

Future Directions

1. Extension to random effects models with covariance matrix parameters
2. Improvement of inference quality for $\kappa$ parameter in real-time algorithms
3. Investigation of theoretical foundations for atom set reduction mechanisms

In-Depth Evaluation

Strengths

1. Theoretical Innovation: Combines Pólya-Gamma augmentation with structured mean-field variational Bayes, solving technical challenges in count response regression
2. Numerical Stability: Convexity guarantees algorithm stability and reliability
3. Computational Efficiency: Closed-form updates and real-time algorithms significantly enhance computational efficiency
4. Practical Value: Method has broad applicability and is easy to implement

Weaknesses

1. Insufficient Theoretical Analysis: Lacks theoretical analysis of variational approximation error
2. Lack of Parameter Selection Guidance: Systematic guidance for choosing atom set $\mathcal{K}$ is missing
3. Limited Experimental Scope: Simulation experiments cover relatively limited scenarios

Impact

1. Academic Contribution: Provides new technical pathways for variational inference in count response models
2. Practical Value: Significant importance for big data and streaming data applications
3. Reproducibility: Detailed algorithm descriptions facilitate reproduction and implementation

Applicable Scenarios

1. Count data analysis requiring fast Bayesian inference
2. Real-time modeling in streaming data environments
3. Large-scale count response semiparametric regression problems
4. Applications with high numerical stability requirements

References

Main references include:

• Luts, J. and Wand, M.P. (2015). Variational inference for count response semiparametric regression. Bayesian Analysis, 10, 991–1023.
• Polson, N.G., Scott, J.G. & Windle, J. (2013). Bayesian inference for logistic models using Pólya-Gamma latent variables. Journal of the American Statistical Association, 108, 1339–1349.
• Durante, D. & Rigon, T. (2019). Conditionally conjugate mean-field variational Bayes for logistic models. Statistical Science, 34, 472–485.

This paper makes important contributions to the field of variational inference for count response semiparametric regression. Through clever technical combinations, it resolves key issues in existing methods and provides new directions for the field's development.