Inspire Digital: Can we use Virtual Reality to understand human behaviour?

Although many of our daily activities seem trivial and effortless, these tasks remain computationally challenging for artificial systems – such as computers – to mimic. But, as Dr Dejan Draschkow explains in this article, Virtual Reality may help us to better understand the everyday, and yet surprisingly complex, tasks which come naturally to us.

Think about a simple, everyday task, like making a sandwich. We can easily find the ingredients needed for this in our kitchen and combine them into a delicious end-product. Even from a very young age, we can extract relevant information from a Lego manual, before finding the required pieces and assembling them in the right order. We mindlessly grab a tea bag every morning and place it into the cup just before the kettle boils. We have the impression that we could do it blindfolded, whereas it is extremely complicated to construct a robot which can complete just a small sub-task of such complex behaviour.

Clever demonstrations and experiments can help us appreciate the uphill battle our cognition is facing. For example, while we rarely notice it in our daily lives, the system responsible for distributing our attention in the world is rather limited. We can easily miss things around us and have difficulties finding what we are looking for if the surroundings are distracting. Have a look at these two examples before you continue reading:

So how can it be that we are so efficient in most of our daily tasks when our mind is facing such severe processing limitations? In truth, we do not yet understand how humans perform complex tasks because we have not had the tools to investigate this properly. Our behaviour is complex and can be broken down into individual cognitive components which interact in a sophisticated fashion. For example, when we make a cup of tea, we use short-term memory to determine the locations of the tea and kettle. These short-term memories are supported by long-term experiences with our kitchen and the objects included in it – this helps us to determine where the tea is most likely to be in most kitchens. Even the simple act of recognising a cup as a cup and a spoon as a spoon is a complex task that machines have only recently made progress on. The act of preparing the tea involves many decisions about what to look at, what to pick up, and what to move. We need to keep track of where we are leaving items at each step, so that we can easily go back to our cup after having searched for the sugar. It is a beautiful orchestra of cognitive functions playing together to achieve their goal.

The big issue is that if we make experiments in which you do something naturalistic, like prepare a cup of tea, there will be many confounding variables. How long did you look at each object? How many times did you pick up the spoon? Where exactly did you move the kettle after using it? How do you know if somebody is using their memory of where the sugar is or is simply searching for it from scratch? Each of these examples (and many more) can exert a critical and unexpected influence on the research question. Therefore, researchers have traditionally tried to test volunteers in a more controlled environment by keeping participants carefully seated in front of a computer screen. For example, instead of testing volunteers on how to make a cup of tea, they may focus just on how much information we can keep in our short-term memory. It is then assumed that the limits of short-term memory also apply when making a cup of tea.

So how can we move towards tracking several cognitive functions within natural behaviour without losing experimental control? One of the very promising solutions for this is Virtual Reality (VR). Using VR, we have full control over participants’ entire field of view, full information about the experimental environment, access to participants’ eye, hand, and body movements, and their position in space. VR aims to simulate a convincing version of reality, but it is one in which we have full control.

VR is not a recent development. However, high prices and poor quality of immersion have been major roadblocks towards its mass adoption in cognitive research. However, VR technology is currently transforming in similar fashion to the rapid technological development of computer screens and mobile phones in the early 2000s. Major technological companies and many new start-ups keep refining affordable research-grade VR systems.

In fact, we are already making our first steps in leveraging VR for exploring previously inaccessible avenues in understanding cognition during natural behaviour. VR will enable researchers to gather data containing a vast amount of information from immersive and realistic scenarios. This will be the first step in transitioning cognitive research to highly controlled real-world situations, stimuli, and tasks and may finally allow us to assess what is going on in the brain when completing ‘complex tasks’ – like making tea.

Dr Dejan Draschkow is Stipendiary Lecturer in Psychology at St John’s College, Oxford