This report documents the results of a simulation based calibration (SBC) run for RBesT. The calibration data will be generated whenever relevant changes to the gMAP function were made. The calibration runs are performed for typical use cases of gMAP. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the RBesT git version as:

The MD5 hash of the calibration data file presented here must match the above listed MD5:

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

Sample from the prior \[\tilde{\theta} \sim p(\theta)\]
Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]
Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]
Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

family	data_scenario	sd_tau	N	total_divergent	total_divergent_sim_fraction	min_ess	max_Rhat	total_large_Rhat	min_lp_ess_bulk	min_lp_ess_tail
binomial	dense	0.5	10000	267	0.022	86	1.029	0	234	215
binomial	dense	1.0	10000	411	0.029	37	1.042	0	145	172
binomial	sparse	0.5	10000	1218	0.056	9	1.217	1	19	9
binomial	sparse	1.0	10000	2438	0.109	7	1.473	1	13	43
gaussian	dense	0.5	10000	409	0.028	34	1.063	0	211	111
gaussian	dense	1.0	10000	432	0.022	23	1.103	0	106	72
gaussian	sparse	0.5	10000	22175	0.466	3	1.840	1	11	21
gaussian	sparse	1.0	10000	41402	0.399	2	3.895	1	5	17
poisson	dense	0.5	10000	781	0.059	101	1.027	0	155	168
poisson	dense	1.0	10000	1839	0.112	12	1.233	1	100	67
poisson	sparse	0.5	10000	23669	0.399	2	2.702	1	6	16
poisson	sparse	1.0	10000	35266	0.408	2	11.114	1	4	16

Note: Large Rhat is defined as exceeding 1.2.

Summary Statistics

\(\chi^2\) Statistic, \(\mu\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
1	dense	binomial	0.5	mu	68.083	63	0.308
13	dense	binomial	1	mu	61.619	63	0.526
25	dense	gaussian	0.5	mu	73.613	63	0.170
37	dense	gaussian	1	mu	59.789	63	0.591
49	dense	poisson	0.5	mu	38.310	63	0.994
61	dense	poisson	1	mu	57.946	63	0.657
73	sparse	binomial	0.5	mu	62.106	63	0.508
78	sparse	binomial	1	mu	68.762	63	0.289
83	sparse	gaussian	0.5	mu	44.467	63	0.963
88	sparse	gaussian	1	mu	63.411	63	0.462
93	sparse	poisson	0.5	mu	60.608	63	0.562
98	sparse	poisson	1	mu	58.496	63	0.637

\(\chi^2\) Statistic, \(\tau\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
2	dense	binomial	0.5	tau	85.171	63	0.033
14	dense	binomial	1	tau	67.802	63	0.317
26	dense	gaussian	0.5	tau	50.214	63	0.878
38	dense	gaussian	1	tau	76.902	63	0.112
50	dense	poisson	0.5	tau	75.891	63	0.128
62	dense	poisson	1	tau	71.949	63	0.206
74	sparse	binomial	0.5	tau	66.355	63	0.362
79	sparse	binomial	1	tau	74.138	63	0.159
84	sparse	gaussian	0.5	tau	68.915	63	0.284
89	sparse	gaussian	1	tau	71.296	63	0.221
94	sparse	poisson	0.5	tau	50.688	63	0.868
99	sparse	poisson	1	tau	55.130	63	0.749

\(\chi^2\) Statistic, group estimates \(\theta\)

	data_scenario	likelihood	sd_tau	parameter	statistic	df	p.value
3	dense	binomial	0.5	theta[10]	42.445	63	0.978
4	dense	binomial	0.5	theta[1]	57.523	63	0.671
5	dense	binomial	0.5	theta[2]	64.397	63	0.427
6	dense	binomial	0.5	theta[3]	49.984	63	0.883
7	dense	binomial	0.5	theta[4]	61.875	63	0.516
8	dense	binomial	0.5	theta[5]	51.302	63	0.854
9	dense	binomial	0.5	theta[6]	69.696	63	0.263
10	dense	binomial	0.5	theta[7]	72.550	63	0.192
11	dense	binomial	0.5	theta[8]	74.035	63	0.161
12	dense	binomial	0.5	theta[9]	60.864	63	0.553
15	dense	binomial	1	theta[10]	60.531	63	0.565
16	dense	binomial	1	theta[1]	65.690	63	0.384
17	dense	binomial	1	theta[2]	65.318	63	0.396
18	dense	binomial	1	theta[3]	54.170	63	0.778
19	dense	binomial	1	theta[4]	69.581	63	0.266
20	dense	binomial	1	theta[5]	83.200	63	0.045
21	dense	binomial	1	theta[6]	58.253	63	0.646
22	dense	binomial	1	theta[7]	40.448	63	0.988
23	dense	binomial	1	theta[8]	56.768	63	0.697
24	dense	binomial	1	theta[9]	53.389	63	0.801
27	dense	gaussian	0.5	theta[10]	47.949	63	0.920
28	dense	gaussian	0.5	theta[1]	72.435	63	0.195
29	dense	gaussian	0.5	theta[2]	56.435	63	0.708
30	dense	gaussian	0.5	theta[3]	72.000	63	0.205
31	dense	gaussian	0.5	theta[4]	59.110	63	0.616
32	dense	gaussian	0.5	theta[5]	71.283	63	0.222
33	dense	gaussian	0.5	theta[6]	82.266	63	0.052
34	dense	gaussian	0.5	theta[7]	70.195	63	0.249
35	dense	gaussian	0.5	theta[8]	66.099	63	0.370
36	dense	gaussian	0.5	theta[9]	54.950	63	0.755
39	dense	gaussian	1	theta[10]	80.026	63	0.073
40	dense	gaussian	1	theta[1]	57.421	63	0.675
41	dense	gaussian	1	theta[2]	53.184	63	0.806
42	dense	gaussian	1	theta[3]	38.810	63	0.993
43	dense	gaussian	1	theta[4]	57.190	63	0.682
44	dense	gaussian	1	theta[5]	61.069	63	0.545
45	dense	gaussian	1	theta[6]	46.310	63	0.943
46	dense	gaussian	1	theta[7]	52.122	63	0.834
47	dense	gaussian	1	theta[8]	66.381	63	0.361
48	dense	gaussian	1	theta[9]	70.566	63	0.240
51	dense	poisson	0.5	theta[10]	62.707	63	0.487
52	dense	poisson	0.5	theta[1]	89.216	63	0.017
53	dense	poisson	0.5	theta[2]	56.806	63	0.695
54	dense	poisson	0.5	theta[3]	63.142	63	0.471
55	dense	poisson	0.5	theta[4]	60.442	63	0.568
56	dense	poisson	0.5	theta[5]	63.066	63	0.474
57	dense	poisson	0.5	theta[6]	67.763	63	0.318
58	dense	poisson	0.5	theta[7]	50.342	63	0.876
59	dense	poisson	0.5	theta[8]	67.661	63	0.321
60	dense	poisson	0.5	theta[9]	55.565	63	0.736
63	dense	poisson	1	theta[10]	78.630	63	0.089
64	dense	poisson	1	theta[1]	52.211	63	0.832
65	dense	poisson	1	theta[2]	57.728	63	0.664
66	dense	poisson	1	theta[3]	65.344	63	0.395
67	dense	poisson	1	theta[4]	59.686	63	0.595
68	dense	poisson	1	theta[5]	70.669	63	0.237
69	dense	poisson	1	theta[6]	53.184	63	0.806
70	dense	poisson	1	theta[7]	84.493	63	0.037
71	dense	poisson	1	theta[8]	46.413	63	0.942
72	dense	poisson	1	theta[9]	101.338	63	0.002
75	sparse	binomial	0.5	theta[1]	62.106	63	0.508
76	sparse	binomial	0.5	theta[2]	63.014	63	0.476
77	sparse	binomial	0.5	theta[3]	52.403	63	0.827
80	sparse	binomial	1	theta[1]	64.051	63	0.439
81	sparse	binomial	1	theta[2]	50.688	63	0.868
82	sparse	binomial	1	theta[3]	71.949	63	0.206
85	sparse	gaussian	0.5	theta[1]	59.290	63	0.609
86	sparse	gaussian	0.5	theta[2]	59.981	63	0.585
87	sparse	gaussian	0.5	theta[3]	72.794	63	0.187
90	sparse	gaussian	1	theta[1]	76.506	63	0.118
91	sparse	gaussian	1	theta[2]	90.880	63	0.012
92	sparse	gaussian	1	theta[3]	61.235	63	0.539
95	sparse	poisson	0.5	theta[1]	72.794	63	0.187
96	sparse	poisson	0.5	theta[2]	62.016	63	0.511
97	sparse	poisson	0.5	theta[3]	73.830	63	0.165
100	sparse	poisson	1	theta[1]	61.158	63	0.542
101	sparse	poisson	1	theta[2]	64.589	63	0.421
102	sparse	poisson	1	theta[3]	47.718	63	0.924

Session Info

## R version 4.5.3 (2026-03-11)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.3 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## time zone: UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] purrr_1.2.1         rstan_2.32.7        StanHeaders_2.32.10
##  [4] here_1.0.2          ggplot2_4.0.2       broom_1.0.12       
##  [7] tidyr_1.3.2         dplyr_1.2.0         assertthat_0.2.1   
## [10] knitr_1.51         
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.10           generics_0.1.4        digest_0.6.39        
##  [4] magrittr_2.0.4        evaluate_1.0.5        grid_4.5.3           
##  [7] RColorBrewer_1.1-3    fastmap_1.2.0         rprojroot_2.1.1      
## [10] jsonlite_2.0.0        pkgbuild_1.4.8        backports_1.5.0      
## [13] gridExtra_2.3         QuickJSR_1.9.0        scales_1.4.0         
## [16] codetools_0.2-20      jquerylib_0.1.4       cli_3.6.5            
## [19] rlang_1.1.7           withr_3.0.2           cachem_1.1.0         
## [22] yaml_2.3.12           otel_0.2.0            parallel_4.5.3       
## [25] inline_0.3.21         vctrs_0.7.1           R6_2.6.1             
## [28] matrixStats_1.5.0     stats4_4.5.3          lifecycle_1.0.5      
## [31] pkgconfig_2.0.3       RcppParallel_5.1.11-2 pillar_1.11.1        
## [34] bslib_0.10.0          gtable_0.3.6          loo_2.9.0            
## [37] glue_1.8.0            Rcpp_1.1.1            xfun_0.56            
## [40] tibble_3.3.1          tidyselect_1.2.1      farver_2.1.2         
## [43] htmltools_0.5.9       rmarkdown_2.30        compiler_4.5.3       
## [46] S7_0.2.1

Simulation based calibration for RBesT

Sebastian Weber

Fri Mar 13 09:17:05 2026