Bias Mitigation in Mental-Health Sentiment Analysis using BERT with Fairness Techniques

Abstract

Sentiment classifiers trained on social-media mental-health text inherit the demographic skew of their training data, and the resulting error is not evenly distributed: posts written in non-standard dialects are misread far more often than posts written in standard English. This work introduces FairBERT, a fine-tuned BERT classifier trained on RMH-Bias-10K — a purpose-built corpus of 10,000 Reddit mental-health posts — that combines adversarial debiasing with counterfactual data augmentation. FairBERT reaches 84.0% accuracy, 0.688 macro-F1 and 0.843 ROC-AUC while reducing mean demographic parity difference by 35.4%, with the largest gain on dialect, where DPD falls from 0.327 to 0.066 (−79.8%). The results show that a substantial share of the measured bias can be removed at a small, quantifiable cost to raw accuracy.

Methodology

1. 01

   Corpus construction

   Collected 10,000 Reddit mental-health posts through the Reddit API and assembled RMH-Bias-10K: de-duplicated, anonymised, sentiment-labelled, and annotated with the demographic proxies (gender, dialect, age band) the fairness metrics are measured over.

2. 02

   Baselines

   Trained a BiLSTM and a vanilla fine-tuned BERT classifier as reference points, so every fairness gain could be reported against a like-for-like accuracy baseline rather than in isolation.

3. 03

   Counterfactual augmentation

   Generated counterfactual variants of training posts by swapping demographic markers while holding the sentiment-bearing content fixed, so the model sees the same affect expressed across group identities.

4. 04

   Adversarial debiasing

   Attached an adversary head that predicts the protected attribute from the encoder representation and trained it against the classifier with a gradient reversal layer, pushing group information out of the representation.

5. 05

   Evaluation

   Reported accuracy, macro-F1 and ROC-AUC alongside demographic parity difference and equalised-odds gaps per protected attribute — accuracy alone is not a result in fairness work.

Results

• 84.0% accuracy, 0.688 macro-F1 and 0.843 ROC-AUC on the held-out split — ahead of both the BiLSTM and vanilla-BERT baselines.
• Mean demographic parity difference reduced by 35.4% across the protected attributes measured.
• Dialect DPD reduced from 0.327 to 0.066 — a 79.8% reduction, the largest single fairness gain in the study.
• Counterfactual augmentation and adversarial debiasing are complementary: neither alone reproduced the combined result.

References

1. [1]J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT, Minneapolis, MN, USA, 2019, pp. 4171–4186.
2. [2]B. H. Zhang, B. Lemoine, and M. Mitchell, "Mitigating unwanted biases with adversarial learning," in Proc. AAAI/ACM Conf. AI, Ethics, and Society, New Orleans, LA, USA, 2018, pp. 335–340.
3. [3]M. Hardt, E. Price, and N. Srebro, "Equality of opportunity in supervised learning," in Adv. Neural Inf. Process. Syst., vol. 29, Barcelona, Spain, 2016, pp. 3315–3323.
4. [4]S. Barocas, M. Hardt, and A. Narayanan, Fairness and Machine Learning: Limitations and Opportunities. Cambridge, MA, USA: MIT Press, 2023.
5. [5]S. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, "Language (technology) is power: A critical survey of ‘bias’ in NLP," in Proc. ACL, Online, 2020, pp. 5454–5476.
6. [6]Y. Ganin and V. Lempitsky, "Unsupervised domain adaptation by backpropagation," in Proc. Int. Conf. Machine Learning, Lille, France, 2015, pp. 1180–1189.