Content
● Recapitulation Course 4
○ Navigation and Controllers
○ Cybersickness
● Hardware
○ Output (Visual, Auditory, Haptic, Vestibular, and Olfactory Channels)
○ Input (Tracking Systems, Input Devices)
● Conclusions
● Bibliography
2
R - Navigation
R - Controls
4
R - Cybersickness
Interface to VR - Output
A key component of a VR experience is how the user
perceives the environment. Their physical perception of the virtual world is based entirely on what the computer displays. We use the term display
broadly to mean a method of presenting information to any of the senses The human perceptual system has at least five senses providing information to the brain. Three of these senses-visual, aural, and haptic-are commonly presented with synthetic stimuli in a VR experience
VR systems fool the senses by output of computer-generated stimuli rather than natural stimuli to one or more of these senses
The inclusion of additional senses almost always improves immersiveness 6
Arrangements for Sensory Displays
There are three basic arrangements for all sensory displays: stationary, headbased, and hand-based.
1. Stationary displays (like rear projection screens and audio speakers) are fixed in place
2. Head-based displays (HBDs) are worn on or in some way attached to the user's head and move in
conjunction with the head
3. Hand-based displays (such as palm-top style and glove devices) move in conjunction with the user's hand
Visual Displays
1. Stationary displays
● Fishtank VR
● Projection VR 2. Head-based displays
● Occlusive HMDs
● Nonocclusive HMDs 3. Hand-based displays
● Palm VR
8
Visual Depth Cues
Humans perceive information regarding the relative distance of objects in a host of ways; these indicators of distance are called depth cues:
1. Monoscopic image depth cues
Interposition, Shading, Size, Linear perspective, Surface texture gradient, Height in the visual field, Atmospheric effects, Brightness
2. Stereoscopic image depth cue (stereopsis) 3. Motion depth cues
4. Physiological depth cues
Visual Depth Cues - Monoscopic image
Can be seen in a single static view of a scene (photographs and paintings)
● Interposition - An occluding object is closer
● Shading - Shape and shadows
● Size - The larger object is closer
● Linear Perspective - Parallel lines converge at a single point; Higher the object is (vertically), the further it is
● Surface Texture Gradient - More detail for closer objects
● Atmospheric effects - Further away objects are blurrier and dimmer
10
Visual Depth Cues - Stereoscopic Image (Stereopsis)
Stereopsis is derived from the parallax between the different images received by the retina in each eye (binocular disparity). The stereoscopic image depth cue depends on parallax, which is the apparent displacement of objects viewed from different locations
Stereopsis is particularly effective for objects within about 5 m. It is especially useful when manipulating objects within arms' reach
12
Visual Depth Cues - Motion
Come from the parallax created by the changing
relative position between the head and the object being observed (one or both may be in motion)
Depth information is discerned from the fact that objects that are nearer to the eye will be perceived to move more quickly across the retina than more distant objects. Basically, the change in view can come about in two ways: the viewer moves or the object moves.
14
16
Visual Depth Cues - Physiological
Are generated by the eye's muscle
movements to bring an object into clear view
Accommodation is the focusing adjustment made by the eye to change the shape of its lens https://www.youtube.com/watch?v=fXd3L4izfQI Convergence is the movement of the eyes to bring an object into the same location on the retina of each eye
18
Visual Displays - Monitor Based or Fishtank
The simplest form of VR visual display utilizes a standard computer monitor
The fishtank paradigm is classified as stationary display VR (it is unlikely that the monitor itself would move during use) Interface Issues of Fishtank VR: keyboard, standard mouse, trackball, or 6-DOF pressure stick (e.g., the
Spacetec Spaceball)
It is less immersive than most other VR visual displays
Visual Displays - Projection-based VR
The screen may be much larger than the typical fishtank VR display
Most projection VR systems are rear-projected Projectors need alignment with a computer
(which provide multiple graphical outputs) or multiple computers synchronized
The two common methods of creating
stereoscopic imagery on a projected VR display use either shutter glasses or polarized glasses
20
Cave Automatic Virtual Environment (CAVE)
CAVE is an empty room with at least four display walls — three walls surrounding you + the floor
CAVE - Projectors and Displays
The projectors behind the walls are there to project high-resolution
images. So, in order to perceive 3D stereo vision the user would wear a pair of shutter glasses. This is because at any instant they only see through one of their glasses.
Here, the displays are synced to the shutter glasses so they will produce images that are for the correct eye at the correct moment and this
switches at least 60 times per second to produce the illusion of 3D stereo vision, similar to HMDs
https://www.youtube.com/watch?v=Gb9ayYGM-4c
22
Visual Displays - Flat Panel Displays
Properties of Visual Displays
Visual Presentation Properties
● Color, Spatial resolution, Contrast, Brightness, Number of display channels, Focal distance, Opacity, Masking, Field of view, Field of regard, Head position information, Graphics latency tolerance, Temporal resolution (frame rate)
Logistic Properties
● User mobility, Interface with tracking methods, Environment requirements, Associability with other sense displays,
Portability, Throughput, Encumbrance, Safety, Cost
24
Head-based displays VR
HBDs are not stationary (they move in conjunction with the user's head)
- head-mounted displays (HMDs)
- counterweighted displays on mechanical linkages (e.g., the BOOM)
- small screens designed to display a virtual image several feet away (e.g., the Private Eye)
- experimental retinal displays (which use lasers to present the image directly onto the retina of your eye) - slightly movable, nickelodeon-type displays that the
Head-based displays VR (2)
The resolution can vary from the relatively few pixels
offered in early systems (160 x 120 color pixels per eye) to very high resolution (over 3840 x 2160 pixels per eye)
HBDs allow for stereoscopic image depth cues (use dual visual outputs - one for each eye)
Any lag in the tracking and image-generation systems causes a noticeable problem for the viewer (simulator sickness)
26
Oculus Rift
Oculus Rift package which consists of 1 Head Mounted Display (HMD), 2 hand controllers and 2 sensors
Oculus Rift - HMDs
HMDs have two displays — one in front of each eye:
1. Lens 2. Display
3. Tracking Technology
28
Oculus Rift - Virtual Screens
The two images in front of
each eye are slightly different and this successfully gives you the experience of 3D stereo vision
These devices also have head-tracking motion
sensors to provide the user with dynamic control of their viewpoint
CAVEs and HMDs
Both CAVEs and HMDs are designed for single users. You can have VR applications with multiple users but each one will need their own display device, that is either a CAVE or HMD.
The CAVEs have a big screen with high resolution. In comparison, the HMD is currently much lower in their resolutions. The CAVEs take a large and dedicated space while the HMDs are more flexible in their space
requirements
30
Handheld VR
A handheld VR display consists of a screen small enough to be held by the user NaviCam project - camera is used to recognize features in the environment A handheld VR display has a screen, some form of tracking, and a method of transferring imagery to the screen
Benefits of Stationary Displays (Fishtank and Projection)
● Higher resolution (at the same price in comparison with HMDs)
● Wider field of view
● Longer user endurance (i.e., can stay immersed for longer periods)
● Higher tolerance for display latency
● Greater user mobility (fewer cables)
● Less encumbering
● Lower safety risk
● Better for group viewing
● Better throughput
32
Benefits of Head-based Displays (Occlusive and Nonocclusive)
● Lower cost (for lower resolution models)
● Complete field of regard
● Greater portability
● Can be used for augmenting reality
● Can occlude the real world
● Less physical space required (compared with multiscreened stationary displays)
● Less concern for room lighting and other environmental factors
Benefits of Hand-based Displays
● Greater user mobility
● Greater portability
● Can be combined with stationary VR displays
34
VR - Aural Displays
Headphones are analogous to head-mounted visual displays
Headphones may be constructed to isolate the participant from sounds in the natural world or to allow real-world sounds to overlap with virtual sounds
Speakers allow multiple participants to hear the sounds
Aural Localization Cues
Localization is the psychoacoustic phenomenon in which a listener can determine the direction and distance from which a sound emanates Spatialization describes the act of creating the illusion that a sound is emanating from a specific 3D location
Convolvotron gives the virtual reality application developer a method of simulating these phenomena to create the illusion of directional
sound
The ventriloquism effect exploits the psychoacoustic phenomenon that sound is likely to be coming from where it looks like it should be
coming from 36
Aural Primary Cues
Three primary “cues” are used by the human brain to establish the spatial position of sound sources in our environment:
● interaural time differences (ITD) - the difference in the arrival time of a sound between the two ears
● interaural intensity differences (IID) - the difference in the intensity of a sound between the two ears
● spectral cues
Aural Other Cues
Elevation Cues - Pinna spectral cues - are changes in the frequency profile of sounds resulting from the size and shape of your head and shoulders
Attenuation - can aid in sound localization because high frequencies in air are dampened faster than low frequencies
Reverberation levels - allowing source location to be represented relatively faithfully during the early portion of a sound
Doppler shift - for sound sources that are moving toward or away from a listener
Head movement - improve the accuracy of sound localization 38
Properties of Aural Displays
Aural Presentation Properties
● Number of display channels, Sound stage, Localization, Masking, Amplification
Logistic Properties
● Noise pollution, User mobility, Interface with tracking methods, Environment requirements, Associability with other sense displays,
Portability, Throughput, Encumbrance, Safety, Cost
Head-based Aural Displays- Headphones
Head-based aural displays (headphones) move with the participant's head, are for one person only, and provide an isolated environment
Can seal off the real world using closed-ear headphones, or allow real-world sounds to be heard along with synthetic sounds with open-ear (hear-through) headphones
40
Stationary Aural Displays-Speakers
Speakers are the stationary aural display system Generally correspond more closely with
projection visual displays, but it is possible to use speakers with head-based visual displays It is also possible to combine the two types of aural display systems: bass sounds could be displayed via a subwoofer speaker, while the rest of the virtual world sounds are displayed via headphone
Benefits of Stationary Displays (Speakers)
● Works well with stationary visual displays
● Does not require sound processing to create a world-referenced sound stage (i.e., one that remains stable to the virtual world)
● Greater user mobility
● Little encumbrance
● Multiuser access means faster throughput
42
Benefits of Head-based Displays (Headphones)
● Works well with head-coupled visual displays
● Easier to implement spatialized 3D sound fields
● Masks real-world noise
● Greater portability
● Private
Haptic Displays
Haptic, from the Greek, relates to physical contact or touch
The nature of the haptic interface dictates that it is a form of both input and output: as output it is physical stimuli displayed by the computer, and
because of its physical connection to the
participant, it is also an input device to the computer Haptic perception involves the combined
sensations of kinesthesia and taction
44
Haptic Displays - Sensations
Taction is the sense of touch that comes from sensitive nerve sensors at the surface of the skin
Kinesthesia (or proprioception) is the perception of movement or strain from within the muscles,
tendons, and joints of the body
Most haptic displays are in some way hand-based.
A few belong to the foot-coupled display category
Haptic Displays - Applicability
Use of haptic display in VR is on the rise in applications that involve training or evaluation of manual tasks, such as medical operations or
serviceability testing of mechanical equipment
An example of the latter is a virtual wrench constrained-by the haptic display system-to only those movements possible in the real world
46
Haptic Displays - Categories
1. End-effector displays - provide a means to simulate grasping and probing objects
2. Tactile displays - touching, grasping, feeling surface textures, or sensing the temperature of an object
3. Robotically operated shape displays - use robots to present physical objects to the user's fingertips or to a fingertip surrogate
4. 3D Hardcopy is the automated creation of physical models based on computer models, which provides a haptic and visual representation of an object
Properties of Haptic Displays
Haptic Presentation Properties
● Kinesthetic cues, Tactile cues, Grounding, Number of display channels, Degrees of freedom, Form, Fidelity, Spatial resolution, Temporal
resolution, Latency tolerance, Size Logistic Properties
● User mobility, Interface with tracking methods, Environment
requirements, Associability with other sense displays, Portability, Throughput, Encumbrance, Safety, Cost
48
Tactile Haptic Displays
Tactile displays focus on the skin's ability to interpret stimuli
The categories of skin (dermal) stimuli are pressure, temperature, electricity, and pain
Available actuators for tactile displays, including inflatable bladders, vibrators, pin arrangements,
temperature-regulating devices, and specialized pressure devices, such as a chest thumper (a low-frequency
speaker worn on the chest that provides thumps and rumble effects)
Vestibular Display
The vestibular perceptual sense maintains balance.
The human organ that provides this perception is located in the inner ear
It helps humans sense equilibrium, acceleration, and orientation with respect to gravity
There is a strong relationship between the vestibular and visual systems. Inconsistency between cues such as the horizon line and balance can lead to cybersickness
Vestibular display is accomplished by physically moving
the user 50
Vestibular Display - Applicability
Motion base systems are common in large flight simulator systems used by the military and commercial transport pilots
Also, in in entertainment venues, such as flight and driving arcade systems and group rides
Olfactory Display
Heilig experimented with smell (in Sensorama), exposing participants on a simulated motorcycle ride to several aromas, including food smells as one "drove" past a restaurant and exhaust fumes from larger vehicles
Individual identifiable smells that can be perceived by a participant are called odoronts
During operations, surgeons use their sense of smell to detect specific substances, such as necrotic tissue within the body
https://www.youtube.com/watch?v=LCYe4eul3TA
https://www.youtube.com/watch?v=3GnQE9cCf84 52
Other Senses
Display to senses such as taste and magnetoreception has been subject to less research
Taste display has the obvious obstacle of health concerns, along with a limited number of useful application areas that would benefit from its development Magnetoreception is the ability to perceive the Earth's magnetic field. Some experiments have also suggested that humans have a limited ability of
magnetoreception
https://www.youtube.com/watch?v=uQdjr3y9aBU
Content
● Recapitulation Course 4
○ Navigation and Controllers
○ Cybersickness
● Hardware
○ Output (Visual, Auditory, Haptic, Vestibular, and Olfactory Channels)
○ Input (Tracking Systems, Input Devices)
● Conclusions
● Bibliography
54
Interface to the Virtual World - Input
User monitoring - includes the continuous tracking of both user movements and
user-initiated actions (pressing a button or issuing a voice command)
World monitoring - supplies real-world input and information about change in the virtual world, brought about through time or user manipulation
User Monitoring - Methods
Active methods - spoken commands, physical controls (wands, joysticks, steering wheels,
dashboards, and keyboards, props, platforms), gestures (hand, head, feet, eye)
Passive methods - tell the computer how and where the participant is moving and where they are looking (includes tracking of the body and position
tracking)
Mind-Controlled VR
https://www.youtube.com/watch?v=7gC4P_FKOhA
56
User Monitoring - Position Tracking
1. Electromagnetic 2. Mechanical
3. Optical
4. Videometric 5. Ultrasonic 6. Inertial
7. Neural (muscular)
User Monitoring - Body Tracking
Body Posture and Gestures - system
determines the current position of the user or some part of the user's body
Posture - the static position of a body part or
group of parts, such as an extended index finger or a clenched fist
Gesture - a specific user movement that occurs over time (make a grasping motion (clenching the fist) to indicate your desire to grab an object)
58
User Monitoring - What body parts are Tracking?
Body parts - the head, the hand and fingers, the eyes, the torso, the feet, other body parts (here are body functions, such as temperature, perspiration, heart rate, respiration rate,
emotional state, and brain waves)
The use of physical objects other than body parts to estimate the position of the
participant. These physical objects are usually props and platforms
User Monitoring - Improving Tracking
Predictive analysis is a computational process that can be used to effectively increase
precision while reducing latency
Calibrating the system for operation within a specific environment can help reduce errors Combining tracking methods can sometimes provide favorable results by taking advantage of the good qualities of each method to
overcome the limitations of another
60
Physical Input Devices - Basic
Devices range from simple handheld objects to large cockpit-style platforms in which the user can sit
Physical controls are the individual buttons (0,1), switches (0,1,2,..), and valuators (0-1) (sliders and dials)
Physical controls can be generic (can be used in multiple applications) or can be specific interfaces (for example for musical or puppetry performance)
Multiple devices can be consolidated into a single input device (joystick, keyboard, piano keyboard, wind
Physical Input Devices - Props
A user interface prop is a physical object used to represent some object in a virtual world
Props allow for more flexible and more intuitive interactions with virtual worlds
The goal in the use of props is to create an interface that the user manipulates in a natural way
For example, the use of a real golf putter as a prop
provides the texture of the grip and the proper mass and center of gravity that give the realistic momentum
necessary to manipulate a virtual putter 62
Physical Input Devices - Platforms
Platforms are larger, less mobile physical
structures used as an interface to a virtual world The platform of a VR experience is the place where the participant sits or stands during the experience
Many of these platforms also can provide sense-of-balance (vestibular) feedback by mounting them on a motion base (an active floor or cockpit that is moved by hydraulics)
Physical Input Devices - Platforms Types
The most common types being
● ring platforms
● Kiosks
● ambulatory platforms (treadmill, bicycle, wheelchair)
● vehicle platforms (cockpit)
● stationary VR displays (large-screen room, drafting board)
64
Speech Recognition (Audio Input)
Speech recognition provides an excellent opportunity for natural communication with computer systems
Speech recognition systems generally map audio sounds to text strings, which then are matched with a set of
preprogrammed possible responses
There are three methods of activating selective listening:
1. push to talk (using a button or a microphone on/off switch) 2. name to talk (user says an activation word followed by an
instruction)
World Monitoring
Information can be gathered and brought into the
experience from sources not directly related to the participant (a virtual world that evolves over time, with or without participants)
A virtual world server may contain the position of other active participants and/or the current state of the virtual world
Real-world input is sometimes used to create portions of the virtual world in real time (weather-monitoring
station or satellite imagery contained in a simulation) 66
Persistent Virtual Worlds
Persistent Virtual Worlds - is one that exists and evolves whether or not it is currently being experienced via VR or another medium
User manipulations in a persistent world can remain intact until another user (or agent) comes along and changes them (allows for asynchronous communication) Multiuser dimensions/dungeons (MUDs) are persistent text-based virtual worlds that individual users can modify and leave for future participants to find
Bringing the Real World into the Virtual World
Real-world data can be gathered by video cameras and other measuring devices and can be used in:
● Analyzing and exploring acquired scientific data (a weather system or a traffic monitoring system)
● Preventing users from colliding with objects in real world (when the user freely walks about in a room)
● Preparing for a real-world dangerous task (for rescue and mining operations)
● Planning for modifying or using real-world space (real-estate)
● Educational experiences (history, geography)
68
Transducers
Transducers are used to retrieve real-world data Transducers include such devices as microphones, weather stations, LIDAR (light detection and ranging), video cameras, and electromagnetic position sensors Transducers can be used in VR systems to help create a richer virtual world
Input in Unity
70
Conclusions
The way participants interact with a virtual reality system greatly influences their experience with the virtual world
The modes of interaction affect how easy the system is to use, how
mentally immersed the participant feels, and the range of possible user actions
Achieving mental immersion is not as simple as physical immersion, but it can be greatly aided by display of the virtual world to multiple senses
A significant portion of the input process consists of monitoring the user's actions
Questions
How can we interact with a VR system? (Input/Output)
How can we synchronize virtual worlds between multiple users?
72
Bibliography
● Sherman, W.R., Craig, A. B. (2003) Understanding Virtual Reality. Elsevier, Science.
● Olwa, A. (2009) Unobtrusive Augmentation of Physical Environments :
Interaction Techniques, Spatial Displays and Ubiquitous Sensing. PhD Thesis https://www.researchgate.net/publication/277995946_Unobtrusive_Augmenta tion_of_Physical_Environments_Interaction_Techniques_Spatial_Displays_and_
Ubiquitous_Sensing
●
Links
● Spinning Up in VR — Part 3: Virtual Reality Hardware
https://medium.com/@mnrmja007/spinning-up-in-vr-part-3-virtual-reality-hardware-e91825465f4d
● Seeing 3D from 2D Images. How to make a 2D image appear as 3D! https://slideplayer.com/slide/3425866/
● AR / VR Science Note 004: Sound Cues and 3D Localization
http://practicalar.com/2016/11/30/ar-vr-science-note-004-sound-cues-and-3d-localization/
● Mind-Controlled VR https://www.technologyreview.com/2017/08/09/68005/mind-controlled-vr-game-really-works/
● Unity Manual - Input https://docs.unity3d.com/Manual/Input.html
●
74
