Lecture 10 - Multisensory Perception

Question 1

Multisensory perception

Answer 1

• sensory dominance = conflict between two senses and one modality dominates over other.
• may be some overlap as perception is about forming estimates of the physical properties of objects
• a single property we are perceiving is often conveyed across many senses = intersensory redundancy (there is redundancy in info communicated across different senses)
• we make perceptual estimates of object positions using info. what if we receive conflicting info?

Question 2

visual dominance

Answer 2

• rock and victor (1964) presented slightly incongruous info to sight and touch. subjects feel object from underneath without seeing own hand and visual impression of the size and shape of object is distorted visually. 1 trial. asked to either choose which object best matches size of object, choose which object best matches size without seeing object, draw object size estimate
  > subjects impressions dominated by visual input - when you feel and see an object at same time their impression is smaller than object acc is when viewing through reducing lense
  > 1 in 5 reported awareness of the conflict
  > conc vision dominates touch and even shapes it
• some evidence of vision affecting hearing is ventriloquists.
• Jackson (1953) - tested if our estimate of an object’s location is dominated by vision or hearing. 2 types of stimulus combos: ecologically meaningless (beep & light) or meaningful (steam and kettle). subjects sat in array at each position and presentation of occurrence manipulated to affect where steam and sound came from. asked where position of sound was from
  > at 30 degree dif get strong visual capture of auditory localisation. only 2.6% correct auditory but for visual 97% correct
  > as you inc spatial dif between info locations get less capture by the visual info and better auditory info = auditory dominance when there is small dif between auditory and visual stimulus

Question 3

mcgurk effect

Answer 3

• effect of vision on speech
• resulting perception is a novel composite resulting from fusion e.g. hearing b and seeing g results in perceiving d
• may be explained by difs in voice onset timing?

Question 4

sensory dominance

Answer 4

• posner et al (1976) - sensory dominance is due to adaptive attentional mechanism
• vision is less capable of exogenous orienting over other modalities (harder to startle someone with visual stimuli)
• humans therefore preferentially attend to vision more
• attention may have some role e.g. in mcgurk effect. other effects are automatic even if you know there is a conflict between senses
• need to identify scenarios in which vision is dominated by other senses

Question 5

auditory dominance

Answer 5

• hearing provides temporal info. vision is based on chemical transduction which takes longer than vibrational transductions in ear
• temporal judgements done in brainstem and auditory perception reaches brain quicker
• 2 scenarios:
  1. resolving ambiguous visual events - auditory cue to distinguish between visual events can get auditory capture of vision
  2. resolving temporal differences - sekular et al (1997) - bouncing balls illusion. when you use auditory cue can influence perception to seem like they bounce off rather than pass through
  > can also influence timing. get inc likelihood of perceiving bouncing off if sound onset is before. sound onset within time window of visual coincidence inc likelihood of perceiving bouncing.

Question 6

is vision for space and hearing for time?

Answer 6

• hearing afford greater accuracy at resolving temp difs and rhythmic patterns of perception.
• early exp biased towards spatial perception and locatlisation and visual dominance better suited here
• Shams et al (2000) - show auditory dominance by considering temporal properties of a stimulus rather than spatial
• judgement of black flashes with beeps:
  > perceive two flashes due to two beeps but multiple flashes are not interpreted as single flash when one beep heard.
  > shows auditory capture and dominance for temporal features of stimulus but reversing this does not interpret the same.
  > presenting fewer beeps than flashes does not affect no. flashes you perceptually experience = auditory effect only occurs if no. auditory events is bigger than no. visual events

Question 7

theory of sensory dominance: discontinuity hypothesis

Answer 7

• Shimoio and Shams (2001)
• the modality that carries a signal that is more discontinuous (and more salient - more structure stimuli with peaks and valleys) becomes influential or modulating modality. sometimes vision dominates over auditory sometimes auditory dominates over visual.
• one direction
• has some effect but cannot explain all cases of visual dominance or other forms of multisensory integration

Question 8

theory of sensory dominance: modality appropriateness hypothesis

Answer 8

• Welch and Warren 1980. for some tasks some modality is more appropriate than the other
• when two sensory modalities provide discrepant info about some characteristic of an event, the resolution of the discrepancy will favour the modality that is the more precise in registering that event
  > spatial tasks - vision
  > temporal tasks - hearing
• similar theories have considered modality precision - whatever modality allows you to detect fine detections in stimuli is going to be most useful for that task

Question 9

multisensory neurons

Answer 9

• Stein - there is no animal in which there is a complete segregation of sensory processing
• stein and meredith studied effects of visual-auditory interactions on neurons and behaviour
  > focused on orienting response in cats. cats sensitive to predators. deciding where to orient is more a combined effect of combining inputs becoming greater than each sense individually
• studied supeiror collicuus (midbrain recieves info from retina, involved in spatial orienting). predominantly responds to visual stimuli while inferior colliculus is auditory based. found:
  > superficial layers - visual neurons. stimulating = orienting behaviours
  > deep layers - diverse preferences of neurons as some respond to visual, auditory or both.
• tried to identify what laws are that govern an indiv layer/neuron reacts to stimuli
  1. law 1 - super-additivity
  > neural response combined visual and auditory stim is greater than sum of unisensory responses
  > presentation together causes massive inc in firing rate but depends on some conditions
  2. law 2 - inverse effectiveness
  > degree of the additive response is inveresely related to the strength of unisensory responses
  > weak unisensory cues are more effective as multisensory cues. weaker when presented alone means the greater the super-additive effect
  > with optimal unisensory cues multisensory responses can be sub-additive
  > additive response depends on spatial and temporal congruity
  3. law 3 - spatiotemporal coincidence
  > response enhancement when V and A stimuli placed within respective RFs. response depression if outside.
  > only get these multisensory effects when two events occur in same region of space
  > get strong responses when events occur within critical temporal incidence period

Question 10

stein et al 1989

Answer 10

• cats orient to V and A stimuli to get food. can vary V and A occurring.
• unisensory stimuli are near threshold (important for near inverse effectiveness) gives best effect
• performance when both cues together is better than from either cue alone = multisensory effect
• this satisfies super-additivity principle: actual response greater when you use raw sum when accounting for probability
• exp 2 trained to oriento to V and manipulated V and A to be spatially disparate. shows behaviour suppression suggesting effect of spatial coincidence
• some evidence for superadditivity in humans. Risberg & Kubjer (1978) - visual and auditory speech perception shows large super additivity in perception.
• Calvert et al (2000) - found super and sub additive responses to visual/auditory speech perception in superior temporal sulcus (super when VA together). VA shows superadditive effect when congruent, subadditive effect when incongruent.

Question 11

maximum likelihood

Answer 11

• Ernst and banks (2002) provided framework for conditions in which brain combines info.
• implies humans integrate visual and other cues in a statistically optimal way.
  > each cue is weighted based on it’s reliability
  > does not require high level appropriateness explanation as we have ways to quantify unisensory estimates
• tested this using the maximum likelihood estimation rule.
• when there is discrepancy we expect vision to dominate. but what if vision is degraed to be less reliable using visual noise
  > as you inc noise, the function becomes less steep. this means inc noise the judgement of object height is biased towards haptic height. as you degrade V info get a haptic capture
  > in accordance with MLE, sensitivity is greater/better with both cues combined than either alone
• therefore:
  > brain does not simply select most appropriate modality
  > it combines sensory info from different modalities into a single estimate to reduce variability