Impacting the social presence of virtual agents by scaling the fidelity of their speech and movement

Contents

1. Introduction
2. Definitions
   1. Virtual Reality
   2. Virtual Agents
   3. Social Presence
   4. Fidelity
   5. Idle Motion
3. Experiment
   1. Hypotheses
   2. Design
      1. Questionnaire
      2. Movement
      3. Speech
      4. Experimental Procedure
   3. Implementation
      1. Technical Components
      2. Assets
      3. Experimental Setup
      4. Experimental Procedure
4. Results
   1. Evaluation
   2. Discussion
5. Conclusion

• References
• Appendix

1. Questionnaire
2. Data: Questionnaire
3. Data: Experiment

This creative work is available in accordance with the terms of the Creative Commons Attribution Share-Alike 4.0 license. This means that, with very few restrictions, it may be freely copied, transferred and used for any purpose as long as the name of the author (Julian Fietkau) is clearly mentioned as the original creator and derivative works are made available under the same license. More information:

Abstract

Virtual agents are constructs that fulfill human or human-like roles in virtual environments, but are directly controlled by software instead of real humans. They have use cases such as presenting information, demonstrating actions or simulating a social environment. If a real person perceives them as sufficiently human-like, they may induce social phenomena like empathy, competition or conversational turn taking, even if the person is consciously aware that the agent is purely virtual.

This thesis explores the influence of technical fidelity on perceived social presence in terms of the virtual agents’ speech and movement. Both of these two variables were assigned different implementations of varying technical sophistication, from text-to-speech output to fully recorded voices and from a completely rigid idle body to a high-quality relaxed idle animation based on motion capturing data. The various combinations were tested in an experiment using a head-mounted virtual reality display in order to measure their influence on perceived social presence. This thesis describes the experiment and its results.

Keywords: avatars, head-mounted displays, social presence, virtual agents, virtual reality

1. Introduction

For several decades now, personal computers have been capable of producing real-time 3D graphics, predominantly used in games, that – even if they are not photorealistic – look convincingly enough like a spatial location to evoke a sense of immersion (Slater, Usoh, & Steed, 1994). In order to populate these environments, virtual agents (perhaps more commonly known as “non-player characters”) are commonplace. They are virtual humanoid characters controlled by software. Depending on the quality of their implementation, they may be a terrific addition to an immersive world, or they might feel artificial and jarring.

In the context of this thesis, we1 are concerned with the technical aspects of such implementations. Specifically, we investigate whether the technical quality of their voice or their animation has a strong influence on the user’s feeling of interacting with a person, even if they are aware that there is no real human behind the virtual agent. Some unconscious social actions might take place even in exchanges with virtual agents (Biocca, Harms, & Gregg, 2001).

We make a point of focusing on characteristics that are not easily communicated through a static screenshot. Voice and animation quality are perhaps not the first things to come to mind when we consider realism in virtual environments, but we think that neglecting them outright could have very negative consequences for the user’s feeling of social presence (a term we define in section 2.3).

On the other hand, if we know how that feeling interacts with the fidelity of our virtual agents’ voice and animation, then we would be better equipped to find compromises between it and development resources.

This, all in all, is why we decided to examine this specific area of VR research further, and conduct an experiment to produce some reliable answers.

We start out by defining a number of important concepts in chapter 2, relying on established knowledge wherever possible. In chapter 3 we describe our experiment in detail, from the initial idea through the design decisions and including a summary of the final implementation, before analyzing and interpreting the results in chapter 4. We close with a summary and conclusion in chapter 5. Bulk data can be found in the appendix.

1: Of course this is a master’s thesis, so any usage of the first person plural in the manuscript refers more or less exclusively to the author, who even has to certify that he wrote everything by himself. Still, we stick to this pronoun not only because it is the polite thing to do, but also as a respectful nod towards the friends, colleagues and advisors who contributed to discussions, talked about ideas or gave valuable feedback. Thank you!

2. Definitions

In order to create a common understanding of the core concepts of this work, it is vital that there be agreed-upon definitions. Wherever possible, we base our definitions on previous established works to increase the viability of this work as a stepping stone for future scientific progress.

2.1. Virtual Reality

The term virtual reality (VR) has historically often been defined in terms of the hardware used to convey a particular medial experience to a human user (Krueger, 1991). To alleviate the ties to concrete technological developments, Steuer (1992) proposes a definition based on the perception of the experience rather than the method of implementation, he defines virtual reality as “a real or simulated environment in which a perceiver experiences telepresence” (Steuer, 1992, p. 7), building upon his previously established definition for telepresence as “the experience of presence in an environment by means of a communication medium” (Steuer, 1992, p. 6).

Even though this definition might at first glance seem overly broad, the mandate of achieving telepresence using a communication medium (as opposed to natural human senses) covers a lot of past and future implementations, and Steuer makes a convincing case for not chaining the concept of VR to classes of hardware like head-mounted displays or data gloves, which is why we operate on the basis of his definition even though this work happens to have a concrete technical scope in that we focus on a VR experience using a head-mounted display (see section 3.3.1).

2.2. Virtual Agents

Even though virtual agents have been extensively studied in works such as Caridakis et al. (2008) or Kopp, Sowa, and Wachsmuth (2003), a systemic definition of the term is often not supplied and there does not seem to be an agreed-upon understanding of the term. We provide our own definition as follows.

We understand an agent (in the context of software programming) to be a software construct that possesses agency, i.e. something that distinguishes between its own behavioral autonomy and the environment in which it exists. An agent may have some perception of its environment, and its actions may have consequences within the environment. A virtual agent is then defined to be an agent that exists in a virtual reality.

Virtual agents are not virtual avatars, because the latter represent and are controlled by human users while the former are controlled by software (Blascovich & Bailenson, 2011). However, both belong to the overarching category of virtual actors. Some of the results of this experiment may be applicable to avatars as well as agents, but since we only tested agents, we do not wish to make any claims to that effect.

2.3. Social Presence

As mentioned above in section 2.1, Steuer (1992) provides a useful definition for the term “presence” (within the context of VR). While he also touches on telepresence, he does not talk about social presence. To find a good definition for this concept, we consult Biocca et al. (2001), who establish what they call “three dimensions of social presence”:

Co-presence

The degree to which the observer believes he/she is not alone and secluded, their level of peripherally or focally awareness of the other, and their sense of the degree to which the other is peripherally or focally aware of them.

Psychological Involvement

The degree to which the observer allocates focal attention to the other, empathically senses or responds to the emotional states of the other, and believes that he/she has insight into the intentions, motivation, and thoughts of the other.

Behavioral engagement

The degree to which the observer believes his/her actions are interdependent, connected to, or responsive to the other and the perceived responsiveness of the other to the observer’s actions.

From: Biocca et al. (2001, p. 2)

They further divide these three dimensions into various factors like awareness, attention, understanding, and interaction. However, their high-level overview is sufficient for our experiment.

2.4. Fidelity

The concept of fidelity (as it is understood in the context of technology) is etymologically rooted in “faith”/“faithful” and refers to “the degree to which something matches or copies something else” (Merriam-Webster Dictionary, 2015). As the presentation of our virtual agents aims to emulate the real world, we interpret the fidelity of a property of a virtual agent as something akin to a degree of closeness to the real-world counterpart.

We would further like to highlight the contrast between fidelity and realism. We understand fidelity to be an inherent property of the implementation of the virtual agent, the degree of fidelity is a design decision. Realism, on the other hand, is the (intended) result of a high degree of fidelity, it is inevitably influenced not only by the virtual agent, but also the rest of the VR experience, and it is dependent on a human observer.

Real-world applications have to make certain trade-offs when it comes to fidelity. Even though a higher degree of fidelity is helpful in achieving a more realistic experience, it also tends to be more difficult (and thus costly) to achieve than lower-fidelity alternatives. The examples given in sections 3.2.2 and 3.2.3 might illuminate the concept further.

2.5. Idle Motion

Even if a human being is not actively doing anything in particular, their body never stops moving completely. They unconsciously perform actions that we summarize as idle motion (Egges, Molet, & Magnenat-Thalmann, 2004), such as shifting their weight, slightly moving their arms, or mildly moving their head while gazing around. These actions are involuntary and require concentrated effort to be suppressed, which is why they are crucial for a virtual agent to appear convincingly “alive” instead of appearing to be a statue. So-called idle animations are commonplace for virtual agents in modern games (Starck, Miller, & Hilton, 2005). We hypothesize that the fidelity (or utter absence) of idle motion may have an influence on the virtual agent’s social presence.

3. Experiment

Our research is rooted in the question of how the technical fidelity of virtual agents influences their social presence in a VR setting. This chapter begins by formulating a number of hypotheses about the interrelations of speech and movement fidelity with user perception and behavior.

To evaluate the validity of our hypotheses, we conducted an experiment involving binary comparisons between pairs of virtual agents whose fidelity of speech and movement had been set to various preconfigured levels. The sections starting from 3.2 detail the design decisions that went into it as well as the technical execution. A summary of the results follows in chapter 4.

3.1. Hypotheses

The possible interactions between the kinds of fidelity that we intend to manipulate and the social presence of the virtual agents are manifold, but some ideas and hunches are certainly more obvious than others. For example, given that higher fidelity virtual agents tend to be more difficult to develop, and seeing that this development happens in real-world applications anyway, it is easy to assume that high-fidelity virtual agents are developed because they are better at producing the respective intended results (depending on the use case). If that is indeed the case, then it is also reasonable to look into whether stronger social presence may be a factor in their increased efficacy, which leads us to our first set of hypotheses:

Hypothesis 1a

A virtual agent with a higher technical fidelity in terms of speech will have a stronger social presence compared to one with a lower fidelity.

Hypothesis 1b

A virtual agent with a higher technical fidelity in terms of movement will have a stronger social presence compared to one with a lower fidelity.

These hypotheses imply positive correlations between the technical fidelity of the virtual agent in terms of one of the two properties speech and movement. To test them, we need to define how exactly we intend to manipulate their fidelity (which happens in sections 3.2.2 and 3.2.3) and we have to provide a measure for their social presence, which we outline at the beginning of the following section 3.2.

Experimental data that would substantiate the above two hypotheses would allow us to infer further details. For example, there could be interaction effects between the fidelity of the two properties – or for the sake of simplicity, it might make sense to assume that they act independently until proven otherwise.

Hypothesis 2

Changes in the fidelity of speech and changes in the fidelity of movement will independently influence the social presence of the virtual agent.

Since we have full control over the experimental software, we are at liberty to record the time that the participants take to make their choices. The next step would then be to draw conclusions from the decision duration time to the difficulty of the choice – it seems reasonable that someone would take more time to make a decision if the choice is extraordinarily difficult.

If we can define a metric for the difficulty of the choice between two of our virtual agents, then we might find a correlation to the duration of the choosing phase.

Hypothesis 3

Comparing virtual agents in terms of social presence is easier (faster) if they have a big difference in technical fidelity in terms of speech and/or movement.

For the sake of scope, it should be noted that any systematic analysis of social presence in virtual agents will have to make abstractions from real-world use cases. For example, our experiment can not feature a rich and complex VR scenario with large numbers of virtual agents interacting with different users and with one another. In order to be able to make empirically substantiated claims, we have to reduce the interaction between the virtual agents and the study participants to a clearly defined minimum to ensure clarity and reproducibility.

3.2. Design

Even though we are building upon an existing definition of social presence, there are no substantiated methods to measure it on a scale in an experimental setting. As a simple tool to enable comparisons between the different degrees of fidelity, we construct our experiment around singular binary comparisons. Pairs of differently configured virtual agents are presented to the participant, who judges them in relation to one another and points out the one with the stronger social presence (see figure 1). This process is repeated for all pairs of configurations.

Figure 1: This is the original concept sketch of the virtual experimental setup. The camera is positioned in an otherwise unremarkable scene with two virtual agents who perform an action of speech one after the other, after which the participant decides which of the two has a stronger social presence.

As we are testing two different axes of technical fidelity, namely speech and movement, we design independent degrees of fidelity. Each of them gets implemented as three different realizations, which are described in detail in sections 3.2.2 and 3.2.3.

We also decide to focus our research on a setting where the participant uses a head-mounted display (HMD) instead of commodity display hardware. We do this in order to increase the participant’s sense of presence, since it has been established that head-mounted displays have that effect (Pausch, Proffitt, & Williams, 1997). It stands to reason that a higher sense of presence on the user’s part could also lead to a higher sensitivity for social presence of virtual agents, or at least it should not be detrimental – however, we are not aware of any empirical proof for this conjecture. At the very least, VR is a research field where we observe a healthy dialogue and openness to new ideas regarding the construction of virtual agents.

3.2.1. Questionnaire

We created a digital questionnaire to guide the participant through the experiment procedure. After the initial greeting, the participant sits down at the desk and finds the questionnaire displayed on the PC monitor in a fullscreen web browser window.

The questionnaire consists of several segments, their order determined by the structure of the experiment, which will be enumerated as follows.

The full questionnaire can be found in appendix A as a display variant optimized for printing.

Demographic and Biological Data

The first few questions cover standard demographic information such as age, gender and occupation. Since the experiment deals with the participants’ reactions to acts of speech, among other things, we also ask about their degree of familiarity with the German language, since people with less proficiency might interpret speech (even those that are only pseudo-German, see section 3.2.3) differently or more slowly than someone whose mother tongue is German.

To gauge the influence of medical issues regarding vision or hearing, we also inquire about known issues in those two areas as well as about any vision and hearing corrections that may exist.

Furthermore, we ask participants about their experience with 3D games, 3D stereoscopic displays, and head-mounted displays, as each of these could have an influence on the way that virtual agents are perceived.

Lastly, participants are asked to state their handedness (left- or righthanded, or ambidextrous) and their inter-pupillary distance, the latter of which is measured in the laboratory.

Hearing Assessment

Even though participants are asked about any issues with their hearing capacity, we strive to make doubly sure that there are no directional hearing problems, not even potentially unknown ones, that could jeopardize our reliance on directional audio signals during the experiment. To that end, we conduct a very brief directional hearing assessment of each participant using the Home Audiometer software by Esser (2012–2015). It tests both ears’ hearing capacity across the frequency spectrum typically audible to humans and displays the results graphically.

The questionnaire makes it abundantly clear to the participants that our hearing assessment is, for a number of reasons, not a substitute for any medical procedure. Our audio equipment is not professionally calibrated, we’re likely to have high levels of ambient noise (e.g. due to the technical equipment in the laboratory and the relative proximity to the Hamburg Airport), and our personnel are not trained to make any medical diagnoses. However, the results of the hearing assessment would give us the possibility to react to any detectable directional hearing issues that might occur.

Lateral Preference Inventory

The Lateral Preference Inventory – or, in full, the Lateral Preference Inventory for Measurement of Handedness, Footedness, Eyedness, and Earedness, and in short, the LPI – is a set of 16 questions developed by Coren (1993). It is intended to determine the four abovementioned lateral preference indices (hand, foot, eye, ear). We include it in our questionnaire to acquire some more detailed information than just the participants’ stated handedness, especially since any lateral preferences for vision and hearing might be relevant for our results even though the participants themselves might not even be consciously aware of them.

Simulator Sickness Questionnaire

The Simulator Sickness Questionnaire created by Kennedy, Lane, Berbaum, and Lilienthal (1993) is a standard tool to gauge the extent to which the participant might be affected by simulator sickness (also known as cybersickness), a set of short-term symptoms that can arise if a person spends a prolonged amount of time using VR hardware.

The SSQ is split into a pre- and a post-experiment half, each consisting of identical questions about the participant’s subjective well-being. It is designed to detect whether any temporary health effects (such as nausea, eyestrain, or dizziness) are produced or amplified by the experiment.

Post Questionnaire

The general post questionnaire consists of a small number of questions tailored to our experiment and the local circumstances. Specifically, we ask the participants about any outside distractions that might have occurred and about their opinion of the experiment, including opportunities for free-form answers and feedback.

Slater-Usoh-Steed Questionnaire

The Slater-Usoh-Steed Questionnaire intends to measure a VR system’s degree of immersion as defined by Slater et al. (1994). In the scope of this thesis, we are not overly concerned with the concept of immersion by itself, but the questionnaire still provides valuable data about how the participants perceived the experience and the extent to which they themselves had a sense of presence.

3.2.2. Movement

Since the social actions of our virtual agents are heavily based on speech, their movement might seem like a secondary concern. However, in order to create a convincing social presence, the usage of suitable idle motions (see section 2.5) is a big contributor to social presence (Egges et al., 2004).

For the highest degree of fidelity that is feasible, we use idle animations based on high resolution motion capturing data, a process that creates animations for virtual agents based on recording the movements of real actors (Moeslund, Hilton, & Krüger, 2006). In real-life applications, this is a costly approach compared to, for example, keyframe animation (which entails a 3D animator creating several “key” poses and interpolating in-between movements), but understandably provides more realistic results. For the purposes of our experiment, we rely on commercially available high-quality animations that surpass anything that we could produce in the local laboratory.

The obvious opposite end of the movement fidelity scale is the completely frozen virtual agent with no idle motions at all. This is trivially easy to implement, fulfilling our expectation that lower-fidelity approaches tend to have a smaller resource impact during development.

For the in-between step, a keyframe-based animation would be a possible middle ground in terms of fidelity, and the comparison between the social presence for keyframe animations versus motion capturing animations in the general case would certainly be of interest. However, for our specific experiment, such a comparison would be difficult to generalize, because any difference in perceived social presence may just as well be rooted in the specific movements that make up the two animations we would use instead of their overall categories. In other words: We would only be comparing one specific keyframe animation with one specific motion capturing animation. To mitigate this issue and permit a general inference, we would have to compare a large number of examples from each category so that we would be able to prove the presence of statistically significant differences, but this is too far beyond the scope of our experiment to be feasible.

Instead, we base our in-between step on the full motion capturing animation, but manipulate it in a way that reduces its fidelity. To that end, we exclude parts of the 3D model from the idle animation, namely the hands and the legs. For the hands, we simply ignore them altogether, leaving them non-animated. For the legs, instead of using the motion-capturing data, we enable a feature called inverse kinematics (Tolani, Goswami, & Badler, 2000), which describes a set of algorithms that are capable of making sure that the virtual agent’s feet stay connected to the ground, even if the upper body moves (or the ground becomes uneven, which is not applicable to our experiment). As a result, the legs no longer use the prerecorded idle animation, but instead do the minimal amount of movement that is needed to plausibly support the upper body. We believe that this approach is a suitable compromise to reduce the movement fidelity.

3.2.3. Speech

There are many kinds of acts of speech that could be considered viable for our experiment. Depending on the use case, virtual agents in different applications may be used to ask questions, deliver instructions, perform back-and-forth conversations or fulfill any number of communicative roles.

However, since the experiment specifically attempts to test for effects of the technical fidelity of the speech, our aim was to provide as little distraction through the content of the speech as possible. Since our experiment relied on direct comparisons, clearly both sides of any comparison would need to execute the same act of speech, so that any bias that might arise from the content of the speech would be symmetrically canceled out.

Finding Suitable Acts of Speech

Ideally, we would like to rely on being able to make comparisons even across the different trials, which is why the differences in terms of speech content between trials should also be minimized. One way of achieving this would be to reuse the very same sentence over and over for every single trial. However, we suspected that this approach would lead to increased monotony during the experiment, since a full run would encompass a large number of trials. This could produce a more tiring experience for the participants, which would in turn reduce the quality of the data. We also suspected that continued use of the same sentence could lead to semantic satiation, a psychological phenomenon by which a word or phrase seems to lose its meaning and appears increasingly alien if it is repeated a sufficiently large number of times (Esposito & Pelton, 1971). These problems could be mitigated by the use of a number of different sentences instead of a single one, but that introduces variance into the process of understanding and interpreting the speech that could also detract from our results.

This is how we arrived at the idea of using gibberish speech (speech that is more or less phonetically and/or syntactically plausible, but does not contain any discernable meaning) instead of real acts of speech. Ideal gibberish speech would enable us to use a variety of different acts of (pseudo-)speech to stave off boredom and semantic satiation, while also keeping all speech at a constant level of semantic contents, that being none at all.

This raises the question of how to generate “high-quality” gibberish, in the sense that it should be nonsensical enough to not contain any meaning, yet sound plausible and familiar enough not to appear overly foreign. Fortunately, solutions to this problem have already been developed. We used a pseudoword generator named Wuggy to create our gibberish, which is based on existing psycholinguistic research (Keuleers & Brysbaert, 2010). It is capable of creating polysyllabic pseudowords from any given list of real words while preserving the phonetic constraints of the source language. We used a dataset courtesy of the Wortschatz project (Institut für Informatik, Universität Leipzig, 2001) containing the 1000 most common German words, from which we had to filter 16 abbreviations2. The remaining 984 German words were fed into Wuggy to be used as the basis for our gibberish.

The resulting list of pseudowords was then shuffled randomly to produce sentences of 12 words each. When spoken out loud, each one of them is four to five seconds long, which we considered a reasonable length to enable the participants to judge the speech.

The eight sentences that we used in our experiment are as follows:

1. Kie Verpreils Hopitie Phraxe metes scheches krumciespiel Dimen wor klück Mozualiin Zaß.
2. Putaun ehte pflon veßten düfflich La Fing hürte Kopp geripten Südchen Daude.
3. Lychte rafen Fahl toswenden lält luchsgans gorm dadee Spresten ebstbals vesses Newage.
4. Sis fist Lab Wuderfet kühe Hamte veuten Läuen alny Bopie schäler belögte.
5. Allerlochs spöbten stekken hanuß bes Beren Rie fal rereis Piedes lanter dabbte.
6. Tonzerr for Turicht gopen Gander fürr jor nasen hührend rusband zusel Händern.
7. Vorkau hind nirgst ehka ätmehin umhächst zondern zöln giesen kolst begids Belsallem.
8. Gesprals Marf hillten fiesen Rottel zockte Jen arrhen peit rafe Wuloner zührend.

It should be noted that word capitalization is essentially random, although we made sure to manually capitalize the first word in each sentence.

From Written Words to Audio Signals

To go from the above pseudospeech to audio signals to be played during the experiment, we first had to define the degrees of fidelity to serve as a basis for the experiment.

A viable approach to low-fidelity speech is text-to-speech (TTS) software. This term describes software that is capable of taking pure text as an input and converting it to audible speech. Detailing the various approaches to this problem in general would be vastly beyond the scope of this work, but plenty of literature on the subject exists (Sproat, 1997). We are largely interested in the results that the current “state of the art” can produce, so we did a short preliminary analysis of free and commercial consumer-grade text-to-speech systems, with the constraint that they had to support German TTS, since our pseudowords were based on the German phonetic structure.

We evaluated the following applications:

• Google Translate TTS3
• IVONA Text to Speech4
• Linguatec Voice Reader Studio 155
• Smart Link ImTranslator6

After listening to some example output from each application and comparing them in terms of vocal fluidity, phonetic plausibility and sound quality (this was a subjective comparison without any quantified justification), we decided to use the IVONA software as our text-to-speech solution for the experiment. However, the differences between the various products were not glaring, and the research field of speech synthesis is bound to make further improvements in the upcoming years. IVONA was able to read our gibberish without issue and we got the corresponding sound files out of it.

At the opposite end of the fidelity scale, it seemed like the obvious choice to create a fully human-voiced set of recordings. We used an adult male voice for the TTS files, so we had a real adult male listen to them and recorded his voice in attempting to read the sentences at the same speed and with the same inflection. We were not able to create an exact match, but we got as close as we could within our constraints.

To create a third stage in between the previous two, a middle ground between text-to-speech and full voice recording, we took the recorded sound files and made some alterations to them. We duplicated the waveform and played it at a delay of 5 milliseconds, which is too short to be perceived as an echo, but produces a tinny, metallic sound. We also cut out most of the small portions of silence within the recordings (see figure 2), which creates “jumps” in the audio recording that would be impossible to achieve by a real human mouth, but that we observed to be reminiscent of the audible inaccuracies found in text-to-speech sound samples. This leaves us with a set of sound files that still sound somewhat like a real voice (at least more than the TTS output does), and yet differentiate themselves from the full recording enough to be slightly uncanny.

Figure 2: This is a visualization of the waveform of our first recorded gibberish sentence. The top one displays the unaltered recording, while the bottom one represents the modified recording with most of the silent parts (highlighted in green) cut out.

3.2.4. Experimental Procedure

As described above, we have chosen the two properties speech and movement as our variable degrees of technical fidelity, which we manipulate independently in three steps each. This means that we have 3 × 3 = 9 possible ways to combine the two properties for each of our virtual agents. To reduce confusion, we will call them configurations (of the virtual agent) in order to differentiate them from the pairs of configurations, which we will call constellations.

Since we ask our participants to compare the configurations in pairs, we would ideally want to pair every configuration with every other one (deliberately excluding constellations where both configurations would be identical), which leaves us with 9 × 8 = 72 constellations. This number already includes symmetrical constellations, i.e. if we understand a constellation to be a two-tuple of configurations, and (C₁,C₂) is part of our set of constellations, then so is (C₂,C₁). Even though this doubles the number of trials per participant compared to the hypothetical situation where we would exclude mirrored constellations, they are indeed a big help in reducing the impact of any (conscious or unconscious) lateral preferences on the part of our participants.

Furthermore, we have to keep in mind that our experiment displays the two configurations in each constellation sequentially. As a result, for each of the above 72 constellations, we include it twice: once starting with the left configuration and following with the right, and once starting with the right configuration followed by the left. From here on out, we will call them left-to-right and right-to-left constellations, respectively. This doubles the total number again, leaving us with our final number of 72 × 2 = 144 trials per participant.

With such a big number of trials, each single one has to be very short if the experiment is to be completed in one sitting. With each of the two configuration displays lasting five seconds, and the decision time expected to be between one and three seconds approximately, we expect a total duration of about 12 seconds per trial. At 144 trials in total, we arrive at an expected experiment length of just under 30 minutes, which seems adequate.

3.3. Implementation

This section describes in further detail how our experiment was put together. In particular, we describe the technologies we used, the location as well as other details of the experimental setup, and we explain some noteworthy problems and other occurances from the execution of the experiment.

3.3.1. Technical Components

The central hardware component of our experiment is the Oculus Rift DK27 head-mounted display. It has a 1920 × 1080 pixel display covering a 100° horizontal field of view as well as various internal sensors for directional and positional head tracking (Oculus VR, LLC, 2014-2015). It is connected to a standard desktop PC which also has mouse and keyboard for input as well as a traditional LCD monitor.

The beyerdynamic MMX 28 provides the sound component of the VR experience. It is advertised as a “gaming headset” and also contains a microphone, which was not used during the experiment. It is capable of reproducing sound in the range of 18 to 22000 Hz (beyerdynamic GmbH & Co. KG, 2012-2015).

We decided to use the Unity Game Engine9 (version 4.5) as the basis for our virtual reality experience, which not only has the capability of interacting with the Oculus Rift HMD, but also has a proven track record as a relatively easy to use basis for real-time 3D applications in scientific contexts (Craighead, Burke, & Murphy, 2007). It runs on modern PCs on top of Microsoft Windows and encapsulates many difficulties of multimedia (in particular real-time 3D graphics) programming behind a graphical interface coupled with freely available documentation. The Unity Engine handles the aspects such as camera projection, lighting, and texturing so that we were able to focus on integrating our assets and programming the experiment.

As mentioned in section 3.2.1, we use the Home Audiometer software written by Esser (2012–2015) to perform a brief non-medical hearing assessment. For an example of what the results of an assessment look like, see figure 3.

The questionnaire was delivered through Google Forms10 in a standard web browser.

Figure 3: These diagrams show an example result from a hearing assessment done with the Home Audiometer software (Esser, 2012–2015). For both ears individually, the application tests various frequencies for their audibility at increasing volumes (the higher the line, the lower the volume, the better the hearing). The results shown here are unremarkable because they stem from a young adult with a healthy hearing capacity.

3.3.2. Assets

We used the MakeHuman11 software to create the 3D model of our virtual agent. It is capable of producing highly detailed and textured 3D models of human bodies that can be adjusted according to various physiological parameters. Our virtual agent is based largely on the MakeHuman defaults with the gender set to 100% male, the race being caucasian, and the physique being slim/athletic. The nondescript black hair and suit are also part of the MakeHuman default assets and proved easy to integrate. See figure 4 for a visual representation.

Figure 4: This is what our virtual agent looks like under ideal lighting and texturing conditions. The MakeHuman software makes it feasible to create human 3D models like this without much knowledge about 3D modeling. Please note that this is a high-resolution render image using idealized lighting and that the real-time 3D representation in the Unity Engine has distinctly lower visual fidelity.

During the initial implementation of the virtual agent and the integration of the sound recordings, it quickly became obvious that the connection between the virtual agent and the voice recordings was not readily apparent as long as there was no mouth movement. Naturally, a human’s mouth moves while they talk, so we decided to implement some primitive lip-synchronization into our virtual agent. There are some lip-sync solutions available for the Unity Engine, for example FaceFX12, but their complexity would have been prohibitive at that stage of implementation. Instead, we implemented a barebones lip-sync algorithm written by UnityAnswers forum user Naletto (2011) – see figure 5 – which reads the audio file’s spectrum data to poll the sound amplitude over a certain time interval and use it to manipulate (stretch, move, etc.) any Unity object.

We applied a suitable scaling to the value and used it to move the jaw bone of our virtual agent downwards synchronized to the audio signal. The result is obviously difficult to appreciate in print, but a pair of screenshots can be seen in figure 6. Thanks to the high quality of the 3D mesh produced by MakeHuman, the simple act of moving the jaw bone results in relatively plausible and visually pleasing facial deformations. Even though it would likely not fool a face-to-face observer, it is convincing enough for use with our HMD and VR scene, where there’s a constant distance between the participant and the virtual agents that renders small inaccuracies invisible.

Figure 6: This pair of images shows the impact of the lip-sync script on our virtual agent. The idea of simply moving the jaw bone downwards in proportion to the volume of the sound file is crude, but works surprisingly well.

3.3.3. Experimental Setup

We set up our experiment in a room within the main HCI laboratory (Fachbereich Informatik, Universität Hamburg). While the laboratory itself was partially in use during the experiment, our room was seperated by a wall and a door.

Every part of the experiment took place on or around a table that we placed in the middle of the room (see figure 7), with one chair for the participant positioned as if the table were a normal desk, and one chair off to the side for the researcher. The PC was positioned under the table towards the left, with keyboard, mouse and monitor on the tabletop. Participants completed the questionnaire facing the monitor, while for the hearing assessment it was turned to face the researcher and to make it impossible for the participant to read the results of the assessment while it was in progress.

Participants only wore the headphones and the HMD whenever each was needed for the experiment. For the rest of the time, they were kept on the left side of the table. The software setup made it feasible to have both the monitor and the HMD connected and running at the same time without interfering with each other.

Water and snacks were available to participants during break times, but were stored on a shelf behind the researcher while the experiment was in progress.

Figure 7: This photo shows one participant sitting at the table, wearing the HMD and the headphones during the experiment. The keyboard, mouse and monitor are also visible, as is the researcher’s laptop. The screen shows the Unity Engine running the experimental VR scene. In the background of the photo, the mostly empty experimental room is visible, with the rest of the HCI laboratory behind the glass windows.

3.3.4. Experimental Procedure

Volunteer participants were acquired from the students at the Fachbereich Informatik as well as the research staff. There was no material compensation for participation, either financial or otherwise.

After being greeted and going over the experiment consent form, the participant would start by filling out the questionnaire page by page, with the measurement of the inter-pupillary distance, the hearing assessment, and the HMD phase in between.

The hearing assessment involved the participant pressing the Ctrl key on the keyboard whenever they heard a noise. The audiometer software would adjust the volume and the frequency and switch between the left and right ear. The results of the assessment were stored with the rest of the experimental data.

The HMD section of the experiment involved 144 trials per participant, as explained above. It started with an explanation how to choose between the two virtual agents with the arrow keys and how to advance using the spacebar (see figure 8). Participants were shown a short summary of the social presence definition by Biocca et al. (2001) (see section 2.3) in order to know how to make the comparisons and were given the opportunity for prior questions. The 144 trials were broken up into 12 blocks of 12 trials each, with opportunities to take a break between blocks.

As explained in section 3.2.4, we expected a length of about 30 minutes for the HMD phase, which turned out to be rather accurate. In addition to that, the hearing assessment took 10 minutes and the questionnaire about 20 minutes per participant, adding up to an hour in total, which was also within our expectations. A few participants took longer breaks than others, which led to a total time of up to 80 minutes in some instances.

There were no significant problems or distractions throughout the experiment. On a few occassions, the hearing assessment was momentarily disrupted by passing planes (the laboratory is geographically close to an airport), but this proved to not be a problem.

Figure 8: This pair of screenshots shows the experimental VR scene. In the top image, the two virtual agents are displayed and the one on the right is currently talking – the scene puts an additional highlight on the talking agent as an added visual focus cue. In the bottom image, both of them have finished talking and the program is waiting for user input. The participant has to press either the “←” or the “→” key. The instructional message is displayed in German if the participant’s mother tongue is German.

2: The following abbreviations were manually removed from the word list: AG, CDU, CSU, DDR, DM, dpa, Dr., EU, FDP, GmbH, Mio, Mrd, SPD, USA, WELT, z.B.

3: https://translate.google.com/

4: http://www.ivona.com/

5: http://www.linguatec.de/products/tts/voice_reader/vrs15

6: http://imtranslator.net/translate-and-speak/speak/german/

7: https://www.oculus.com/dk2/

8: http://www.beyerdynamic.de/shop/mmx-2.html

9: http://unity3d.com/

10: https://docs.google.com/forms/

11: http://www.makehuman.org/

12: http://facefx.com/

4. Results

In this chapter, we examine the results of our experiment and interpret the data we gathered in such a way as to evaluate the hypotheses from section 3.1.

To start off, it bears mentioning that we had n = 15 participants aged between 19 and 45 years (M = 26.65, SD = 6.76), which should be enough to infer some statistically significant results. However, some of the answers we received make it clear that any results gathered from this experiment are not certain to be applicable to the populace at large. For example, all of our participants had a computer science background (10 with an HCI specialization, 5 without), all of them were native speakers of German, none of them suffered from any notable disorders in vision or hearing, and all participants were right-handed. Any conclusions we draw from the experimental data should only be relied on with these caveats in mind until the experiment can be repeated with participants of a more varied background.

4.1. Evaluation

As we decided early on that our trials would be binary comparisons between different virtual agent configurations, we can now look at the “winner” of each trial (the configuration that was chosen). If we look at how often each value for speech was in the winning configuration (cf. table 1, figures 9 & 10), we observe mean counts for the text to speech condition of M = 33.53 (SD = 12.92), for the modified recording condition of M = 48.47 (SD = 14.40), and for the full recording condition of M = 61.40 (SD = 9.49). Analogously, for the different idle motion values (cf. table 2), we observe mean counts for the “no idle motion” condition of M = 34.20 (SD = 13.52), for the reduced idle motion condition of M = 52.33 (SD = 8.27), and for the motion capturing idle motion condition of M = 56.87 (SD = 8.06).

All of the value counts are normally distributed across subjects according to a Shapiro-Wilk test at the p < 0.05 level.

Using the Kruskal-Wallis rank sum test, we can not confirm at the p < 0.05 level that the winning counts for the different degrees of fidelity are based on underlying distributions with distinct location parameters.

A χ² test did not assert any interdependence between the winning speech values and the winning idle motion values across all trials.

Figure 9: These diagrams show the number of times, for each participant, when a particular value for the fidelity of speech or movement was part of the winning configuration. As the fidelity gets higher, the corresponding variable is chosen more often and more consistently.

Figure 10: These two diagrams show the cumulative number of times each value of the two variables appeared in the winning configuration across all participants. This makes the weight towards the higher-fidelity values more obvious.

We analyzed the effects of display order (left to right or right to left) and the randomly chosen gibberish sentence on the winning fidelity values with a repeated measures ANOVA and paired-samples t-tests. For the winning speech value, there are no significant interactions. For the winning idle motion value, there is a highly significant interaction between the winning value and the gibberish sentence, but no significant interactions with the display order, nor any interaction effects between the sentence and the display order.

For the time it took the participants to make their individual binary choices (from here on out “choice duration”), we measured delays between 189 and 14850 milliseconds (M = 1384, SD = 1303). Even though the value range spans two orders of magnitude, even the highest outliers exist within the realm of plausibility, which is why we do not discard any of the data points (see figure 11).

Figure 11: This box plot shows the spread of the choice duration (milliseconds). Even though a large majority of all data points are lower than 1500 milliseconds, there are outliers up to ten times as big. Altogether, 233 points exist outside the 1.5 × IQR distance (233 of 2151, 10.8%).

To create a useful measure for the difference in fidelity between two configurations, we have to set a fidelity value for each individual configuration. We do this by interpreting the three values of each of our two variables as integer values in {0, 1, 2}, with 2 being the highest fidelity value and 0 being the lowest. We then define the fidelity value of a configuration as the sum of the fidelity values of its two components. Lastly, we define the fidelity distance as fd_AB = |fv_A − fv_B| (visualized in figure 12).

Figure 12: This is a tabular visualization of the fidelity distance between two configurations, defined as fd_AB = |fv_A − fv_B|. The fidelity distance between the top left and the bottom middle configuration is given as an example.

If we examine the distribution of the choice duration in relation to the fidelity distance (see figure 13), we see that there are visual hints for a small negative correlation, and the Pearson product-moment correlation coefficient of the two variables is indeed −0.036 at the p < 0.1 level.

Figure 13: This is an array of box plots showing the spread of the choice duration depending on the fidelity distance. The plots largely resemble the independent one shown in figure 11. For fd ≤ 2 there is not much variation, but for fd = 3 and especially fd = 4 it is obvious that the choice duration has far fewer outliers and even a slightly lower median.

The remaining parts of the questionnaire did not lead to any interesting results. The Lateral Preference Inventory aligned very well with the stated handedness of the participants and did not offer any further insight, the Simulator Sickness Questionnaire (fortunately) gave no signs of any health problems more significant than mild fatigue.

4.2. Discussion

Going back to our hypotheses from section 3.1, we are unfortunately not able to substantiate hypotheses 1a and 1b (the existence of significant correlations between speech or idle motion fidelity and the social presence of the virtual agent) based on our experimental data. To the naked eye it seems apparent that the higher fidelity configurations were chosen more often, but it appears that our sample size compared with the relatively small difference between the values (between one to two standard deviations) is not big enough to prove it conclusively, at least not at any worthwhile level of significance. However, it does seem like a worthwhile avenue for further research.

It could be a boon to focus on only one of our two fidelity scales per experiment – perhaps it would have been easier to prove a correlation, even with the same number of participants, if all trials were geared towards one scale of comparison instead of mixing both. At this stage that is a wild guess though.

In reference to hypothesis 2, we were able to provide some evidence that there are no significant interaction effects between the effects that the fidelity of speech and the one of idle motion have on social presence. Even though true independence can not be statistically proven, the data suggests that it is a safe assumption that there are no interactions between the two.

Regarding hypothesis 3 (a negative correlation between the overall fidelity and the duration of the choice phase), we were indeed able to prove the existence of a negative correlation at the p < 0.1 level. The effect is small but noticeable. From a user perspective it is not very surprising that configurations with a greater fidelity distance are easier to compare, but it is reassuring that the data corroborates the hypothesis.

Although it is hard to say whether a larger sample size would have led to more significant results, as a suggestion for the future it seems prudent to say that a larger number of participants would likely benefit experiments of this kind. Furthermore, our conjecture is that the idea behind our experimental trials – the binary comparison of two configurations with each combining more than one variable – generates only a small amount of usable information per trial. Maybe a proper measurement scale for social presence would make it easier to draw reliable conclusions, even if it comes at the cost of increased experimental duration.

5. Conclusion

During the work on this thesis, we consolidated several different sources to create suitable notions of both technical fidelity and social presence and to relate them to one another. We conceived an experimental framework based around large numbers of fast-paced binary comparisons to measure the social presence of virtual agents based largely on participants’ instant reactions, implemented these ideas into a real-time VR scenario capable of using a state-of-the-art HMD, and executed our experiment with 15 participants, proceeding to analyze the gathered data statistically.

Our intention for this experiment was to investigate any interactions between the fidelity of virtual agents’ speech and movement (specifically idle motions) and their social presence in a virtual reality context – one based around a head-mounted display with directional tracking, in our case.

It is unfortunate that we were unable to prove whether higher technical fidelity of virtual agents leads to a stronger social presence, which would have helped explain and steer some of the current developments around virtual agents. In the absence of such statistical proof and with us only having been able to draw some incidental conclusions, the experiment would have to be considered a partial success at best. But of course not every experimental result has to be groundbreaking, and especially in its role as part of a master’s thesis, perhaps this lesson that not every experiment can lead to clean results every time is all the more fitting.13

Viewed from a more constructive perspective, it bears mentioning that our framework for virtual agents’ technical fidelity and how to manipulate it along different axes is a potentially useful tool that did not exist before we developed it in anticipation of our experiment. Now that at least two examples for three-step fidelity manipulation have been established, it will be much easier for future experiments in the same area to establish controlled and reproducible fidelity circumstances.

13: And needless to say, the author learned a lot about laboratory experiments, VR technology, real-time 3D engines and other methods and technologies throughout the preparation and execution of this thesis.

References

Please note that there are several mentions of commercial products and/or websites in this thesis, some of which we deliberately excluded from this reference list, so that it contains only the sources from which we cite information. If a product or website is merely mentioned in context, but not cited, a web link is included as a footnote in the main text instead of here.

Appendix

The experiment used a digital questionnaire to acquire subject data beyond the boundaries of the HMD experiment. The following is a complete reproduction of the questionnaire. It uses the “for print” view in lieu of the web-based version to facilitate inclusion in a print document, so any references that seem counterintuitive (for example directions to click a button) would make more sense in the interactive web-based version of the questionnaire.

The experiment included n = 15 participants in total. Each one was guided through the questionnaire and the experiment. Since the experiment contained 144 trials per subject, the total maximum number would have been 15 × 144 = 2160 trials. However, 9 trials were faulty and had to be discarded, mostly because of outside interruptions and short-term software failures, leaving 2151 trial data points available for interpretation.

Because the measurement data from the hearing assessments was not used in the evaluation of the experiment (partly because there was no need, partly because none of the results were at all surprising or interesting), and even though the participants consented to a full publication of all experimental data, we have decided not to include the hearing assessment results with this publication because we feel that the participants’ interest in keeping potentially medically sensitive data safe and anonymous weighs heavier than the interest of the public in fully open data access in this particular case.

A. Questionnaire

Experiment Questionnaires

All details are collected only in the context of the present study. Thank you for your participation!

* Required

Age *

 

Height *

 

Profession / field of study: *

 

Gender *

Mark only one oval.

• Male
• Female

How would you rate your German language skill? *

Mark only one oval.

• Native speaker
• Fluent
• Proficient
• Basic
• None

Vision correction: *

Mark only one oval.

• None
• Glasses
• Contact lenses

Do you have a known eye disorder?

Check all that apply.

• Color blindness
• Night blindness
• Dyschromatopsia (red-green color weakness)
• Strong eye dominance
• Other:
   

Do you suffer from hearing loss?

Mark only one oval.

• No (healthy hearing capacity)
• Mild hearing loss (difficulties understanding speech)
• Moderate to severe hearing loss (impossible to understand speech)
• Profound hearing loss (impossible to hear speech or most noises)

If you suffer from hearing loss, please check all that apply:

Check all that apply.

• Asymmetrical hearing loss, more pronounced on the left side
• Asymmetrical hearing loss, more pronounced on the right side
• Symmetrical hearing loss (both ears affected at about the same level)
• Congenital hearing loss (present since birth)
• Acquired/Delayed hearing loss (onset later in life)

Hearing correction:

Mark only one oval.

• None
• External hearing aids
• Cochlear implants
• Other:
   

Do you suffer from a displacement of equilibrium or similar? *

Mark only one oval.

• Yes
• No

Do you have any experience with virtual reality HMDs (such as the Oculus Rift)? *

Mark only one oval.

	1	2	3	4	5
no experience						a lot of experience

Do you have experience with 3D computer games? *

Mark only one oval.

	1	2	3	4	5
no experience						a lot of experience

How many hours do you play per week? *

 

Do you have experience with 3D stereoscopic display (cinema, games etc.)? *

Mark only one oval.

	1	2	3	4	5
no experience						a lot of experience

Are you left- or right-handed? *

Mark only one oval.

• Left-handed
• Right-handed
• Ambidextrous

Inter-pupillary distance (IPD) *

Please contact the experimenter to measure your IPD.

 

Hearing assessment

Please contact the experimenter for a short assessment of your hearing ability (approximately 10 minutes). Please note: This is a very broad test that serves only to highlight any obvious patterns in the context of our experiment. Our staff does not (and can not) perform medical diagnoses. This assessment is not a substitute for a hearing test conducted by trained personnel using calibrated equipment. If you suspect that your hearing may be impaired, please arrange further steps with your medical doctor.

The Lateral Preference Inventory

Simply read each of the questions below. Decide which hand, foot, etc. you use for each activity and then put a check mark next to the answer that describes you the best. If you are unsure of any answer, try to act out the action.

With which hand do you draw? *

Mark only one oval.

• Left
• Right
• Either

Which hand would you use to throw a ball to hit a target? *

Mark only one oval.

• Left
• Right
• Either

In which hand would you use an eraser on paper? *

Mark only one oval.

• Left
• Right
• Either

Which hand removes the top card when you are dealing from a deck? *

Mark only one oval.

• Left
• Right
• Either

With which foot would you kick a ball to hit a target? *

Mark only one oval.

• Left
• Right
• Either

If you wanted to pick up a pebble with your toes, which foot would you use? *

Mark only one oval.

• Left
• Right
• Either

Which foot would you use to step on a bug? *

Mark only one oval.

• Left
• Right
• Either

If you had to step up onto a chair, which foot would you place on the chair first? *

Mark only one oval.

• Left
• Right
• Either

Which eye would you use to look through a telescope? *

Mark only one oval.

• Left
• Right
• Either

If you had to look into a dark bottle to see how full it was, which eye would you use? *

Mark only one oval.

• Left
• Right
• Either

Which eye would you use to peep through a keyhole? *

Mark only one oval.

• Left
• Right
• Either

Which eye would you use to sight down a rifle? *

Mark only one oval.

• Left
• Right
• Either

If you wanted to listen in on a conversation going on behind a closed door, which ear would you place against the door? *

Mark only one oval.

• Left
• Right
• Either

Into which ear would you place the earphone of a transistor radio? *

Mark only one oval.

• Left
• Right
• Either

If you wanted to hear someone’s heartbeat which ear would you place againsttheir chest? *

Mark only one oval.

• Left
• Right
• Either

Imagine a small box resting on a table. This box contains a small clock. Which ear would you press against the box to find out if the clock was ticking? *

Mark only one oval.

• Left
• Right
• Either

Simulator Sickness Questionnaire (Pre)

General discomfort (DE: "Unwohlsein") *

Mark only one oval.

	1	2	3	4
None					Severe

Fatigue (DE: "Ermüdung") *

Mark only one oval.

	1	2	3	4
None					Severe

Headache (DE: "Kopfschmerzen") *

Mark only one oval.

	1	2	3	4
None					Severe

Eyestrain (DE: "Ermüdung der Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Difficulty focusing (DE: "Schwierigkeiten mit der Sehschärfe") *

Mark only one oval.

	1	2	3	4
None					Severe

Increased salivation (DE: "Erhöhte Speichelbildung") *

Mark only one oval.

	1	2	3	4
None					Severe

Sweating (DE: "Schwitzen") *

Mark only one oval.

	1	2	3	4
None					Severe

Nausea (DE: "Übelkeit") *

Mark only one oval.

	1	2	3	4
None					Severe

Difficulty concentrating (DE: "Konzentrationsschwierigkeiten") *

Mark only one oval.

	1	2	3	4
None					Severe

Fullness of head (DE: "Druckgefühl im Kopfbereich") *

Mark only one oval.

	1	2	3	4
None					Severe

Blurred vision (DE: "verschwommene Sicht") *

Mark only one oval.

	1	2	3	4
None					Severe

Dizzy (eyes open) (DE: "Schwindelgefühl bei geöffneten Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Dizzy (eyes closed) (DE: "Schwindelgefühl bei geschlossenen Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Vertigo (DE: "Gleichgewichtsstörungen") *

Mark only one oval.

	1	2	3	4
None					Severe

Stomach awareness (DE: "Magenbeschwerden") *

Mark only one oval.

	1	2	3	4
None					Severe

Burping (DE: "Aufstoßen") *

Mark only one oval.

	1	2	3	4
None					Severe

Experiment Procedure

In the experiment you will be asked to perform a task in a virtual environment while wearing a head-mounted display as well as headphones. You will see and hear pairs of virtual actors performing an act of speech. You will then be prompted to decide, for each pair, which one has the stronger "social presence" (this term is defined on an introductory slide during the experiment). Each trial lasts about 12 to 15 seconds. The experiment will be conducted in blocks of 12 trials (about 2.5 minutes each) and will end once all 12 blocks have been completed. The experiment usually takes about 30 minutes. You may take short breaks between blocks, but please try to hold your concentration throughout each block, as the trials within a block happen consecutively. Thank you! (Please click "continue".)

You are now ready to start the experiment. Please contact the experimenter.

If you have completed the experiment, please click "continue".

Simulator Sickness Questionnaire (Post)

General discomfort (DE: "Unwohlsein") *

Mark only one oval.

	1	2	3	4
None					Severe

Fatigue (DE: "Ermüdung") *

Mark only one oval.

	1	2	3	4
None					Severe

Headache (DE: "Kopfschmerzen") *

Mark only one oval.

	1	2	3	4
None					Severe

Eyestrain (DE: "Ermüdung der Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Difficulty focusing (DE: "Schwierigkeiten mit der Sehschärfe") *

Mark only one oval.

	1	2	3	4
None					Severe

Increased salivation (DE: "Erhöhte Speichelbildung") *

Mark only one oval.

	1	2	3	4
None					Severe

Sweating (DE: "Schwitzen") *

Mark only one oval.

	1	2	3	4
None					Severe

Nausea (DE: "Übelkeit") *

Mark only one oval.

	1	2	3	4
None					Severe

Difficulty concentrating (DE: "Konzentrationsschwierigkeiten") *

Mark only one oval.

	1	2	3	4
None					Severe

Fullness of head (DE: "Druckgefühl im Kopfbereich") *

Mark only one oval.

	1	2	3	4
None					Severe

Blurred vision (DE: "verschwommene Sicht") *

Mark only one oval.

	1	2	3	4
None					Severe

Dizzy (eyes open) (DE: "Schwindelgefühl bei geöffneten Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Dizzy (eyes closed) (DE: "Schwindelgefühl bei geschlossenen Augen") *

Mark only one oval.

	1	2	3	4
None					Severe

Vertigo (DE: "Gleichgewichtsstörungen") *

Mark only one oval.

	1	2	3	4
None					Severe

Stomach awareness (DE: "Magenbeschwerden") *

Mark only one oval.

	1	2	3	4
None					Severe

Burping (DE: "Aufstoßen") *

Mark only one oval.

	1	2	3	4
None					Severe

Post Questionnaire

Did you feel immersed in the virtual world? *

Mark only one oval.

	1	2	3	4	5
no						yes

Were you distracted from the virtual world by real-world ambient noise? *

Mark only one oval.

	1	2	3	4	5
no						yes

Have you been able to see parts of the real laboratory during the experiment? *

Mark only one oval.

	1	2	3	4	5
no						yes

Do you think the experiment task was too difficult? *

Mark only one oval.

	1	2	3	4	5
no						yes

Do you think the experiment was too long? *

Mark only one oval.

	1	2	3	4	5
no						yes

How would you subjectively describe your level of attention during the experiment? *

Mark only one oval.

	1	2	3	4	5
very low						very high

Which strategy did you use (e.g., concentrating on certain signals, making a "decision from the gut", etc.)? *

 

 

 

 

 

Any observations regarding the difficulty of the task that you made during the experiment and would like to share?

 

 

 

 

 

Additional comments:

 

 

 

 

 

Slater-Usoh-Steed Questionnaire (SUS)

Please rate your sense of being in the virtual environment, on a scale of 1 to 7, where 7 represents your normal experience of being in a place. *

I had a sense of “being there“...

Mark only one oval.

	1	2	3	4	5	6	7
not at all								very much

To what extent were there times during the experience when the virtual environment was the reality for you? *

There were times when the virtual environment was the reality for me...

Mark only one oval.

	1	2	3	4	5	6	7
not at all								almost all the time

When you think back to the experience, do you think of the virtual environment more as images that you saw or more as somewhere that you visited? *

The virtual environment seems to me to be more like...

Mark only one oval.

	1	2	3	4	5	6	7
images that I saw								somewhere that I visited

During the time of the experience, which was the strongest on the whole, your sense of being in the virtual environment or of being elsewhere? *

I had a stronger sense of...

Mark only one oval.

	1	2	3	4	5	6	7
being elsewhere								being in the virtual environment

Consider your memory of being in the virtual environment. How similar in terms of the structure of the memory is this to the structure of the memory of other places you have been today? By ‘structure of the memory’ consider things like the extent to which you have a visual memory of the virtual environment, whether that memory is in colour, the extent to which the memory seems vivid or realistic, its size, location in your imagination, the extent to which it is panoramic in your imagination, and other such structural elements. *

I think of the virtual environment as a place in a way similar to other places that I have been today...

Mark only one oval.

	1	2	3	4	5	6	7
not at all								very much so

During the time of your experience, did you often think to yourself that you were actually in the virtual environment? *

During the experiment I often thought that I was really standing in the virtual environment...

Mark only one oval.

	1	2	3	4	5	6	7
not very often								very much so

B. Data: Questionnaire

C. Data: Experiment