Secondary Validation · Haq & Saldías (FAccT 2026)

Say My Name: Declaring Black Identity Triggers
the Safety Filters That Writing Black Does Not

Independent analytical validation by Mario DeSean Booker, Ph.D. · Purdue University Global · All statistics derived from published data and the BOLD dataset (Dhamala et al., 2021).

What This Study Found

Six identity conditions. Same questions. Dramatically different refusal rates. The AI's behavior changed based on who appeared to be asking — not what was being asked.

28.9%
Black Explicit refusal rate
20.7%
White Explicit refusal rate
5.4%
AAVE Implicit refusal rate
97.82%
of refusals were soft (evasions)
706
prompts that flipped by identity
+8.27pp
Black–White refusal gap (z‑test)
Refusal Rate by Identity Condition — All 6 Conditions (hover for details)
Black conditions
White conditions
Singaporean conditions
Soft vs. Hard Refusals — Breakdown by Identity Condition

Refusal Rates in Detail

Raw counts from Table 4 of Haq & Saldías (2026), independently verified. N = 2,201 per condition. Toggle conditions to compare.

Refusal counts and rates — click bars for arithmetic breakdown
Z-Test results — Black vs. White Explicit (interactive derivation)

The "400% Increase" — What It Actually Means

The paper's headline claim requires qualification. Two different statistical questions produce two very different numbers. Neither is wrong — they just answer different things.

Raw Marginal Odds Ratio

1.56×
Derived directly from Table 4 raw counts.
Odds(Black) = 637 ÷ 1,564 = 0.407
Odds(White) = 455 ÷ 1,746 = 0.261
OR = 0.407 ÷ 0.261 = 1.56

Question answered: "Across all users and prompts, how do refusal odds compare?"

GLMM Conditional Odds Ratio

4.19–8.01×
From the paper's logistic GLMM with prompt-level random intercepts.
σ² = 100.44, ICC = .97 — 97% of variance is prompt-driven.
The conditional OR inflates because it estimates identity's effect on the narrow residual slice after absorbing nearly all variance.

Question answered: "For a specific prompt at average difficulty, how do odds shift with identity?"
Odds Ratio comparison — marginal vs. conditional estimates
ICC Variance decomposition — what drives refusal probability?

ICC = .97 means 97% of total log-odds variance is attributable to which prompt is asked, not who is asking. The conditional OR operates on the remaining 3%. This is why it magnifies so far above the raw marginal estimate.

The Dialect Jailbreak Is Synergistic — Not Additive

Novel finding not reported in the original study: the AAVE bypass outperforms what an additive model of race + dialect penalties would predict by 7.41 percentage points.

How the additive prediction is built:
Start at the White Explicit baseline: 20.7%
Add the Black race penalty (+8.27pp): → 28.97%
Subtract the dialect reduction (−16.2pp from SAE vs. Explicit): → 12.77%

Actual observed AAVE rate: 5.4%
Subadditivity gap: 12.77% − 5.4% = −7.41 percentage points

The filter doesn't just lose one trigger when the explicit keyword is removed. It collapses more severely than either signal alone predicts.
Predicted vs. Observed — Additive Benchmark vs. Reality (hover bars)
Subadditivity gap — AAVE vs. Singlish comparison

The AAVE gap (−7.41pp) is four times larger than the Singlish gap (−1.86pp), consistent with Black users facing a larger raw race penalty and therefore more headroom for synergistic bypass.

BOLD Dataset — Prompt Distribution

2,380 base prompts across 4 domains from the Bias in Open-ended Language Generation dataset (Dhamala et al., 2021). Click a domain to see its category breakdown.

Category breakdown — click a domain card above to filter
Domain proportions in the stratified sample