Deep Learning Multimodal Emotion Recognition

Emotion recognition is the task of identifying human emotion. It is a challenging and important task in the field of human computer interaction (HCI) and has been applied to healthcare, wellbeing, gaming, safety and security. Emotion recognition has been realised using a variety of different input modalities, ranging from video, still images, voice, and text.

The combination of these different input modalities, such as combining vocal, visula and verbal modalities, has shown great potential in recognising a wide variety of different emotions.

Historically, approaches have used feature-level fusion in combination with machine learning algorithms to achieve emotion recognition with some success. However, more recently deep learning has shown remarkable capabilities in learning detailed representations of high-dimensional image and video data, as well as audio spectral features, to achieve even better classification performance.

Unlike tasks that focus on a single input modality to determine emotion recognition, multimodal input techniques take into consideration a range of different input data types such as visual and audio without the need for more intrusive physiological data signal recordings (such as electrocardiogram (ECG) or galvanic skin response (GSR)). While recent studies utilising multimodal deep learning approaches have shown promising results, due to the complex nature of multimodal emotion recognition there are still significant challenges to be addressed.

This project aims to explore deep learning architectures with a view to proposing, desiging, and developing a multimodal deep learning pipeline for emotion recognition.  This project will seek to address the following research questions:

In answering these research challenges, the project aims to advance the state-of-the-art in deep learning models for multimodal emotion recognition. The outputs from this research project would have potential impact in the application in a wide range of felds, most notably healthcare, mental health and wellbeing.

Applicants should hold, or expect to obtain, a First or Upper Second Class Honours Degree in a subject relevant to the proposed area of study.

We may also consider applications from those who hold equivalent qualifications, for example, a Lower Second Class Honours Degree plus a Master’s Degree with Distinction.

In exceptional circumstances, the University may consider a portfolio of evidence from applicants who have appropriate professional experience which is equivalent to the learning outcomes of an Honours degree in lieu of academic qualifications.

If the University receives a large number of applicants for the project, the following desirable criteria may be applied to shortlist applicants for interview.

• First Class Honours (1st) Degree
• For VCRS Awards, Masters at 75%
• Completion of Masters at a level equivalent to commendation or distinction at Ulster
• Experience using research methods or other approaches relevant to the subject domain
• Sound understanding of subject area as evidenced by a comprehensive research proposal
• Publications - peer-reviewed

The University offers the following levels of support:

The following scholarship options are available to applicants worldwide:

• Full Award: (full-time tuition fees + £19,000 (tbc))
• Part Award: (full-time tuition fees + £9,500)
• Fees Only Award: (full-time tuition fees)

These scholarships will cover full-time PhD tuition fees for three years (subject to satisfactory academic performance) and will provide a £900 per annum research training support grant (RTSG) to help support the PhD researcher.

Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Please note: you will automatically be entered into the competition for the Full Award, unless you state otherwise in your application.

The scholarship will cover tuition fees at the Home rate and a maintenance allowance of £19,000 (tbc) per annum for three years (subject to satisfactory academic performance).

This scholarship also comes with £900 per annum for three years as a research training support grant (RTSG) allocation to help support the PhD researcher.

• Candidates with pre-settled or settled status under the EU Settlement Scheme, who also satisfy a three year residency requirement in the UK prior to the start of the course for which a Studentship is held MAY receive a Studentship covering fees and maintenance.
• Republic of Ireland (ROI) nationals who satisfy three years’ residency in the UK prior to the start of the course MAY receive a Studentship covering fees and maintenance (ROI nationals don’t need to have pre-settled or settled status under the EU Settlement Scheme to qualify).
• Other non-ROI EU applicants are ‘International’ are not eligible for this source of funding.
• Applicants who already hold a doctoral degree or who have been registered on a programme of research leading to the award of a doctoral degree on a full-time basis for more than one year (or part-time equivalent) are NOT eligible to apply for an award.

Due consideration should be given to financing your studies. Further information on cost of living

W. Liu, J. -L. Qiu, W. -L. Zheng and B. -L. Lu, "Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition," in IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 2, pp. 715-729, June 2022, doi: 10.1109/TCDS.2021.3071170.

Summaira, J., Li, X., Shoib, A. M., Li, S., & Abdul, J. (2021). Recent Advances and Trends in Multimodal Deep Learning: A Review. arXiv preprint arXiv:2105.11087.

L. Schoneveld, A. Othmani, and H. Abdelkawy, “Leveraging recent advances in deep learning for audio-visual emotion recognition,” Pattern Recognition Letters, 2021.

Koromilas, P.; Giannakopoulos, T. Deep Multimodal Emotion Recognition on Human Speech: A Review. Appl. Sci. 2021, 11, 7962.