Supervised Fine-Tuning of Foundation Models for Robust Empathy Detection in Text and Video Interactions

Md Rakibul Hasan
Email: rakibul.hasan@curtin.edu.au
Website: https://hasan-rakibul.github.io

Supervised by Professor Tom Gedeon, Professor Aneesh Krishna and Dr Md Zakir Hossain

[Recognition] [Talks] [Thesis] [Publications]

Recognition

This research was supported by a Curtin HDR Scholarship (Optus Centre for Artificial Intelligence) and two IEEE SPS Scholarships (2024 [Certificate] and 2025 [Certificate]). Computational work was carried out using resources provided by the Pawsey Supercomputing Research Centre and the National Computational Infrastructure (NCI) Australia.

Talks and presentations

Thesis

This PhD research advances our understanding of, and deep learning algorithms for, detecting empathy in various human interactions. I developed novel algorithms on both textual and computer vision modalities. In addition to empathy detection itself, I addressed three specific challenges – (1) robust learning with noisy labels, (2) cross-subject generalisation and privacy-aware method for video-based detection, and (3) uncertainty quantification – that are also applicable to general data analytics tasks. My proposed methods have yielded improved accuracy in empathy detection on five public benchmarks.

The PhD was conferred in July 2026, and both external examiners recommended that the thesis be “PASSED with no requirement for correction other than minor typographical or editorial matters”.

Publications

Individual publications are listed below. Some papers are highlighted.

This paper systematically screened 849 papers from 10 well-known academic databases and analysed the final 82 papers. Our analyses reveal several prominent task formulations – including empathy on localised utterances or overall expressions, unidirectional or parallel empathy, and emotional contagion – in monadic, dyadic and group interactions. Empathy detection methods are summarised based on four input modalities – text, audiovisual, audio and physiological signals.

This paper proposes Large Language Model-Guided Empathy (LLM-GEm) prediction system, which rectifies annotation errors based on our defined annotation selection threshold and makes the annotations reliable for conventional empathy prediction models, e.g., BERT-based pretrained language models (PLMs). We experiment with three NewsEmpathy datasets involving people’s empathy levels towards newspaper articles and achieve state-of-the-art test performance using a RoBERTa-based PLM.

This paper explores LLMs' potential for in-vitro applications: using LLM-generated labels to improve supervised training of mainstream models. We examine two strategies - (1) noisy label correction and (2) training data augmentation - for empathy prediction from text. A RoBERTa pre-trained language model (PLM) trained with noise-reduced labels yields a state-of-the-art Pearson correlation coefficient of 0.648 on the public NewsEmp benchmarks.

This paper proposes UPLME, an uncertainty-aware probabilistic language modelling framework to capture label noise in empathy regression. UPLME is trained using Bayesian concepts with variational model ensembling. We further introduce two novel loss components: one penalises degenerate Uncertainty Quantification (UQ), and another enforces the similarity between the input pairs on which we predict empathy. UPLME estimates both empathy scores and predictive uncertainty, improving predictive performance and uncertainty calibration.

This paper analyses listener's trait empathy, state empathy and their physiological signals, and speaker-perceived empathy from an experiment we conducted at Curtin University. In the experiment, speakers described a personal incident and one or more listeners responded naturally, as in everyday conversation. Afterwards, speakers reported perceived empathy, and listeners reported their trait and state empathy.

This paper proposes a three-level privacy taxonomy for empathy detection in video interactions and develops TFMPathy, a tabular foundation model (TFM) method that balances the privacy-utility trade-off. TFMPathy detects empathy from summary statistics of pre-extracted facial and gaze features, without retaining participants' identifiable videos. On a public human-robot interaction benchmark, we examine the information retained by feature representations and introduce subject-independent evaluation to assess generalisation to unseen participants. Through TFM fine-tuning, TFMPathy improves cross-subject accuracy from 59% to 73%.

This proposal outlines a research plan for developing multimodal language models to detect empathy from text and video. In addition to leveraging existing datasets, the proposed study involves collecting real-life interaction video and audio.

To address the WASSA 2023 shared task on predicting empathy, emotion and other personality trait tasks, our contributions include (1) converting numerical information into meaningful text information using appropriate templates, (2) summarising lengthy articles, and (3) augmenting training data by paraphrasing.

This website template was originally borrowed from Jon Barron's website.