Bookmark and Share

Visual and semantic processing in object recognition

Dr Barry Devereux (pictured top right), Dr Alex Clarke (middle right) and Professor Lorraine Tyler (bottom right) have recently published a paper in Scientific ReportsThe research uses Deep Neural Network modelling combined with a semantic attractor network to show how different regions of the brain contribute to visual and semantic processing of objects. Below, the authors describe the impact and methodology of their work. The work was supported by an ERC advanced grant awarded to Lorraine Tyler.  

1. Could you briefly explain how your research first came about, what it was based on, and what its main objectives were?
We had carried out a lot of research with healthy people and brain-damaged patients to better understand how objects are processed in the brain. One of the major contributions of this work is to show that understanding what an object is involves the visual input being rapidly transformed over time into a meaningful representation, and this transformative process is accomplished along the length of the ventral temporal lobe. Given that to understand what an object is, we must access our semantic memory, we were sceptical about some of the claims that a purely vision-based DNN (deep neural network) could fully capture the process of recognising an object. This was the initial trigger for the current research, where we wanted to fully understand how low-level visual input is mapped onto a semantic representation of the object's meaning.

2. How does your deep learning and semantics-based method work?
We take as our starting point a standard deep neural network computer vision model ("AlexNet"). This model -- and others like it -- can identify objects in images with very high accuracy, but they do not include any explicit knowledge about the semantic properties of objects. For example, bananas and kiwis are very different in their appearance (different colour, shape, texture, etc) but nevertheless we understand correctly that they are both fruit. Models of computer vision can distinguish between bananas and kiwis, but they do not encode the more abstract knowledge that both are fruit. We therefore combined the vision model with a neural network model of conceptual meaning, which does include this kind of semantic knowledge. In the combined model, visual processing maps on to semantic processing and activates semantic knowledge about concepts. We then test the model's ability to predict neuroimaging data as volunteers viewed images in the scanner.

3. What do you feel were the most meaningful findings of your study and what are their practical implications?
The most critical finding of the study was that brain activity during object recognition is significantly better modelled by taking into account both the visual and semantic properties of objects, and this can be captured through a computational modelling approach. The model makes quantitative predictions about the stages of semantic activation that are consistent with a general-to-specific account of object processing, where more course-grained semantic processing gives way to more fine-grained processing. We find that the different stages of the model predict activation in different regions of the brain's object processing pathway.Ultimately, better models of how people process visual objects meaningfully may have practical clinical implications; for example, in understanding conditions such as semantic dementia, where people lose their knowledge of the meaning of object concepts.

4. What are your plans for future research? 
This research is important in showing how different regions of the brain contribute to visual and semantic processing of objects, yet, what is further vital to know is how information in one region can be transformed into a different state that we see in different regions in the brain. For this, we need to understand how connectivity, and temporal dynamics support these transformative neural processes.

Posted on 24/07/2018

Further news

Go to the news index page.