Analysis

Takeaways

MR Remote Collaboration

Understanding Sense of Co-presence in MR Remote Collaborations

My Role

Team Leader; Reasercher & Developer

Contribution

Literature Review; Prototype Design and Implement; User Study; Data Analysis; Paper Writing

Team

2 Researchers(Jianing Yin, Weicheng Zheng)

Mentor

Yukang Yan; Xin Tong

Tools

Unity; Oculus Quest Pro; Azure Kinect;LaTex

Duration

2023.7-2023.12

Background

Remote collaboration provides great temporal and spatial convenience.

MR-based collaboration enhances 3D cues in non-shared physical spaces compared to traditional tools.

Currently, popular remote collaboration software predominantly relies on a 2D window-based interface, such as Zoom and Skype. However, they have some limitations. For example, it is challenging for a remote user to make sense of where their collaborator is pointing at in their local environment due to lack of spatial awareness.
Mixed Reality (MR) based remote collaboration offers a promising solution to address the challenge of non-shared physical spaces.

Research

Literature Review

We've reviewed the literature on MR remote collaboration, summarizing current research directions and identifying future exploration areas.

Co-presence in MR collaboration can be affected by factors like collaboration intensity, perspective sharing, symmetry of MR spaces, time, input and output methods, visual displays, and the application domain. Research has mainly focused on enhancing co-presence through these factors.
Nevertheless, an often-overlooked aspect is how to assess the sense of co-presence using alternative methods since most research are based on subjective measurement, notably questionnaire.

Advantages and Disadvantages of the Main Methods for Measuring Co-Presence in MR Remote Collaboration

Compared to questionnaires, behavioral observation can automatically assess the sense of co-presence through user responses, reducing subjectivity.
In contrast to physiological measurements, behavioral observation is a cost-effective and less intrusive method that minimizes interference with users' experiences.

Based on the level of co-presence, we classify remote collaboration scenarios into five categories. In our experiments, we primarily focus on video collaboration, MR collaboration, and co-located scenarios.

In the Video Collaboration scenario, collaborators communicate via voice and 2D video, but it often lacks integration between the physical task space and augmented communication space, resulting in a lower sense of co-presence. It can also miss important non-verbal cues from in-person interactions, like facial expressions and gestures. On the other hand, MR Collaboration offers a more co-located collaboration-like experience among existing remote methods.

Prototype

VR 3D Environment and Object Scanning

Also, we conducted competitive analysis on the social anxiety treatment apps available on the market and derived the core functions of the product based on user interviews.

Most of the apps offer a training feature that allows users to enhance their ability to cope with social anxiety through practice.
However, these apps do not provide real-time assistance in social anxiety scenarios.
Furthermore, based on user interviews, two-thirds of users mentioned the importance of recording practice data and found that community engagement was helpful in alleviating social anxiety. These features are less commonly found in existing apps.

Also, we conducted competitive analysis on the social anxiety treatment apps available on the market and derived the core functions of the product based on user interviews.

If our user want to do rate search when they are traveling, it will take more than 15 minutes using web since they will have to setup their laptop and network.

However, they can just do it in under 1 minute using DrayEasy mobile App.

Moreover, both mobile and web app can be a supplement to each other in different use cases.

Perception of Collaborators with Each Other

Also, we conducted competitive analysis on the social anxiety treatment apps available on the market and derived the core functions of the product based on user interviews.

User Study

Experiment Setup and Procedure

Upon arrival, participants signed informed consent forms and completed a demographics questionnaire. They underwent role-specific equipment training. Workers were briefed on the task in the task environment, while helpers received remote guidance instructions in another room.
Participants then completed task sessions in a randomly assigned order of video, MR, and co-located collaboration scenarios.
After each scenario, questionnaires were administered, and brief interviews conducted with helpers and workers.

Multiplayer Tasks

Sub-task 1 - Blocks Finding:
Workers collected scattered blocks based on the helper's guidance, despite not knowing which ones to collect or where they were located.

Sub-task 2 - Blocks Building:
The worker assembled blocks with remote guidance, despite interference from extra blocks.

Measurements

Questionnaire - Physical Presence Measures

To measure the helper's immersion in the virtual environment, we combined the Slater-Usoh-Steed Questionnaire (SUS) and Presence Questionnaire (PQ) to assess their perception and control over the MR environment.
‍

Questionnaire - Social Presence Measures

We used the Inclusion of Other in the Self (IOS) Scale to measure the perceived psychological closeness to the other player and the Social Presence Module of the Game Experience Questionnaire (GEQ) to gauge how much attention players paid to their counterpart during task completion.
‍

Interview

Semi-structured interviews were conducted to gain in-depth insights into the participants' experiences, focusing on their perceptions of the environment, collaboration with their partners, and task complexity under different scenarios.
‍

Recording

To assess participants' behaviors, we recorded audio and video, segmenting them into behaviors like communication frequency, task-related content proportion (audio data), and unique actions in various scenarios (video data).
‍

Task Performance

Task performance(the time taken), serving as additional data for analyzing user performance.

Experiment Process

MR Scenario

In this scenario, the remote helper and local worker are not in the same room; instead, they collaborate through an MR system. The remote worker can use virtual blocks in a 3D manner to convey the position and assembly instructions to the local worker. They can also monitor the local worker's progress in real-time using point cloud data.
‍

Co-located Scenario

In this scenario, the remote helper and local worker are in the same room, and the local worker is assembling blocks under the guidance of the remote helper.
‍

Voov Meeting Scenario

In this scenario, the remote helper and local worker are not in the same room; instead, they collaborate through VooV Meeting. The remote worker utilizes VooV Meeting's video functionality to gain insight into the work environment and monitor the local worker's progress.
‍

Analysis

Participants

We recruited 28 participants (18 females, 10 males) aged 19 to 29 (M = 23.14, SD = 2.45) to form 14 worker-helper dyads. 7 of these dyads knew each other beforehand. Most participants were students with different backgrounds, such as Landscape Architecture, Data Science, Architecture, Electronic Information, and so on. 92.86% of the participants had previously used VR devices, and 46.43% had used AR devices.
‍

Questionnaire Data Analysis

Based on the questionnaire results measuring presence, it is evident that in three different scenarios, physical presence and social presence scored the highest in co-located scenario, followed by MR remote collaboration, and voov meeting remote collaboration scored the lowest.

Interview Data Analysis

To gain a deeper understanding of the reasons behind users' varying perceptions of physical presence and social presence in different scenarios, we conducted interviews with users at the end of the experiments in the three scenarios. These interviews primarily included three questions:
‍
◾ In the recent experiment, under what circumstances did you feel like you were actually working in the same physical environment as another person?
◾ Completing tasks in the three different situations just now, did it affect your perception of the environment, collaboration with others, and the ease or difficulty of task completion? Why?
◾ In the three scenarios we just went through, did you engage in any recurring specific behaviors or actions?

What we focus during the interview data analysis?

During the analysis process, our primary focus was on the users' sense of presence, their perception of the environment, the ease or difficulty in perceiving individuals, and for better behavior observation, we conducted preliminary analyses of the amount of perceived behaviors and language use in different scenarios.

Results

Remote helpers and local workers exhibit distinct perceptions across various dimensions.
‍
◾ Sense of Co-Presence Perception: Remote helpers report the highest sense of co-presence in the co-located scenario, whereas it is notably diminished in the voov meeting scenario. In contrast, local workers experience a heightened sense of co-presence in the Mixed Reality (MR) scenario.
‍
◾ PerceivedTask Difficulty: Both roles find voov meetings to be the most challenging, while the MR scenario is considered the least demanding in terms of task difficulty.
‍
◾ Perception of Surrounding Environment: Both remote helpers and local workers concur that the environment in the voov meeting scenario lacks realism, with the co-located scenario providing a more authentic environmental perception.
‍
◾ Amount of Movement During Task Completion: Remote helpers exhibit a greater number of actions during task completion in the MR scenario, while their actions are more limited in the voov meeting scenario. Conversely, local workers display the opposite pattern, engaging in more actions in the voov meeting scenario and fewer in the MR scenario.
‍
◾ Amount of Language Usage During Task Completion: In both roles, the MR scenario is associated with reduced verbal communication, while the voov meeting scenario involves more extensive language use.

Audio and Video Data Analysis

During the experiment, we recorded video and audio data from the participants to further quantitatively analyze their language and behavior. We then conducted regression analyses to assess the likelihood of predicting the sense of co-presence based on this data in comparison to the questionnaire data, which served as a baseline.

Video Data Analysis

For video data analysis, we employed the Nvivo platform to conduct quantitative analysis. We recorded the frequency and duration ratios of behaviors such as confirmation gestures and referential movements to provide data for subsequent in-depth analysis.

Audio Data Analysis

In the analysis of audio data, we incorporated natural language processing algorithms such as jieba, textblob, and baidu-aip to process Chinese data. The analysis dimensions included word frequency, pronouns, speech tone, and speaking frequency, among others.

Performance Data Analysis

Additionally, we conducted time-based statistics on the participants' task completion, visualizing the data using violin plots and box plots (excluding outliers).
From the graphs, it is evident that participants performed better in MR remote collaboration compared to traditional remote collaboration like voov meeting. Furthermore, MR remote collaboration demonstrated more stable performance.

Takeaways

Behavioral observation complements a multi-modal approach for assessing co-presence in MR remote collaboration settings🔎

Currently, co-presence research heavily relies on subjective measures, particularly questionnaires (accounting for 96.6%). While questionnaires are easy to use, cost-effective, and non-intrusive, they may not capture real-time feelings, can be biased, and may vary over time. Objective measures of behavioral realism offer an alternative as they can automatically produce responses, providing a more consistent and less biased evaluation compared to subjective ratings.

New technologies enhance user experiences and address information gaps in remote collaboration✌

Remote collaboration challenges, such as the inability to observe body language in real-time and abstract communication, can be improved in MR systems by introducing virtual objects, recreating work environments, or incorporating avatars to enhance user experiences.

Invisible Guardian

SocialEase