lecture 4 - multisensory integration Flashcards
(41 cards)
definitions
-multisensory
-cross-modal
-integration
Multisensory: More than one modality is used in perception
Cross-modal: Interactions between different
modalities → one sense affects perceptions provided by a different sense
Integration: Merging information from different modalities into a unified percept
how is perception multisensory
Perception is multisensory in natural interactions with the environment
- Information from different senses can either be complementary or redundant/ overlapping
multisensory information ________ reliability of the percept and provides a more complete representation of the world
Multisensory information increases
reliability of the percept and provides a more complete representation of the world
(+ increases resistance to interference)
different modalities can provide convergent information about…
-what does this mean for the CNS
Different modalities can provide convergent information about the same external event/properties
- CNS has to disentangle cases where stimulation of different senses is unrelated and where it is related
simple heuristics for integration
(3) (how do we know information from 2 senses belong together)
1) Temporal correlation:
Stimulation of different modalities occurs at, (roughly) the same time.
2) Spatial congruency:
Stimuli in the different senses come from approximately the same location
3) Inverse effectiveness:
Reduced benefit of multisensory integration the stronger the unimodal signal of a cross-modal
cue is → Multisensory response is stronger when one stimulus by itself is quite weak
what is inverse effectiveness
Degree to which a multisensory response exceeds the response of the most effective modality specific
stimulus component declines as the effectiveness of the modality-specific stimulus component increases
inverse effectiveness on a neural level
-what is the point of this
example of one neuron and its reaction to a weak stimulus
Multi-modal neurons in Superior Colliculus (SC) (relevant for rapid orienting of attention):
-when neuron reacts to JUST visual input there is a weak respons, same as when it reacts to auditory input and has a weak response
- Spikes produced by combination of visual and auditory event (5) is larger than the individual
neural spikes in response to visual (1) and auditory stimuli (2) - Superadditivity of spike counts:
Multisensory response is greater than the sum of uni-sensory responses - Sum usually only larger for weak inputs (near threshold) → aids detection of weak stimuli
→ speeds up behavioural responses
inverse effectiveness
-photos of the dog , 1,2,3
1) dog far away from window, weak auditory stimuli, and periphery
2)
Superadditivity: Both cues are weak – response exceeds the sum of the separate inputs
- Additivity: As cues become stronger unisensory responses become stronger → integrated response is not different from the sum of the responses to each component
- Subadditivity: Combined input is smaller than the sum of the two uni-sensory inputs (but still exceeds the largest single input response)
neural mechanisms
subcortical areas : the superior colliculus
-where is it located
-how many multinsensory neurons
-do neurons overlap
Located in the mid-brain –
important for orienting
behaviour and fast motor
reactions
- High(est) proportion of
multisensory neurons
(extensively studied) - Neurons show overlapping
spatial maps for visual, auditory
and somatosensory modalities
neural mechanisms
-cortical areas
-how many multisensory neurons and where
- what did the studies of this area focus on
- Multisensory neurons are
found in most areas – often in combination with unimodal neurons
-Even in areas previously considered modality specific (e.g., neurons in visual cortex respond to tactile cues, and neurons in primary auditory cortex are activated by
visual lip movements
Studies in primates primarily focussed on posterior parietal cortex (converging information from visual, vestibular, tactile
and auditory system
cross modal integration
-how is input from two senses combined perceptually?
Different modalities are combined to yield the best estimate of the external properties
The modality that provides more reliable information is given more weight (greater reduction in uncertainty)
cross modal integration examples
→ e.g., vision strongly influences auditory localisation
(ventriloquist effect) – vision is spatially more accurate
→ audition can dominate vision in temporal properties,
e.g., auditory flutter drives perception of visual flicker (the two flash illusion
→ Modality appropriateness hypothesis
what is semantic congruency
Semantic congruency (consistent
meaning of two stimuli) Semantic congruency refers to the degree of similarity or compatibility between two pieces of information, such as words or stimuli, based on their meaning or semantic content.
semantic congruency _______ multinsensory stimulus integration and…
Semantic congruency (consistent
meaning of two stimuli) strengthens
multisensory stimulus integration and corresponding behavioural
performance
semantic congruency and its affect on speed of participants responses
Semantic congruency of visual and
auditory stimuli affects the speed of
participants responses
→ faster target detections when
visual stimulus is accompanied by a
semantically congruent sound
two visuals of- teleophone and frog
two sounds of teleophone and frog
-did a combination- press a button whenever you see or hear the telephone
-if you only see the teleophone (visual stimulus) you have a reaction time of jus under 5 ms, and if you only hear it there is similar reaction time
-incogruent : see the teleophone and hear the frogor vice versa (slow reaction time)
-
crossmodal illusions
-what is the parchment-skin Illusion
-complementary information improves the reliability of our perception
-sound modifies tactile sensations
-enhanced high frequency feedback makes the skin feel drier
-temporal coincidence required
incongruent information can result in unexpected precepts due to _______ _______
sensory interactions eg audio-visual illusion of the MCGurk effect)
-you listen to auditory stimulus ‘ba’, have a visual stimulys/actor pronouncing ‘ga’
-usually the perception is a fusion response (da / tha)
explain the parchment skin illusion in depth
exp
-relationship between auditory information and skin perception
-sound modifies tactile sensations
deprivation of one modality can modify the development and integration of remaining modalities
-enhanced high frequency feedback makes the skin feel drier
-temporal coincidence required
parchment skin illusion in blind people
Less susceptible to the illusion – ability to ignore irrelevant auditory input in the tactile task
parchment skin illusion ,
vision, audition and touch
study on blind people
-what does this show
-study
-control
-10 early onset blind participants
-had to rate their tactile perception on scale -5 to +5
-if high frequencies accentuated (makes it feel rougher), works well in sighted participants
Robust illusion in sighted humans
* 7 of the early blind participants were not/ minimally susceptible to the illusion
* Only 3 of the early blind participants showed small effects in the expected direction
- Multisensory perception might not be innate but is – at least to some extent - based on experiences during early
development → following visual deprivation extensive cross-modal changes occur (re-organisation of perceptual system)
Cross modal plasticity in the Cerebral cortex
study by hamilton et al 2000
-what does this case suggest
Case of blind woman (born blind) who lost ability to read Braille (code) following bilateral occipital lesions (usually processing vision) following stroke in primary visual cortex (processes vision area) but didn’t loose any other tactile sensitivity , just specific to reading braille
- Occipital cortex involved in decoding spatial and tactile information for Braille reading -
- Suggests that there may be a critical period of susceptibility for the recruitment of the occipital cortex for haptic information processing (in congenitally blind)
Cross-modal Plasticity in the Cerebral Cortex
TMS – Study
Cohen et al., (1997),
TMS briefly disrupts the electric activation patterns of the neurons in the cortical area it is applied to. short bursts of magnetic stimulation applied to certain areas of the brain
- Blind Braille readers and sighted participants (blindfolded) who had to identify embossed Roman letters
- Occipital stimulation (visual area) disrupted Braille reading in blind participants but not tactile discrimination in sighted participants (but disrupted their visual performance)
- Visual cortex recruited for somatosensory processing in early blind
Cross-modal Plasticity after Sensory Deprivation mechanism
-1 Reorganisation in areas associated with the deprived modality
if you don’t have vision, the occipital cortex seems to reorganise process
Primary sensory areas are able to process information from remaining modalities
- Sensory inputs shape the functional architecture of the brain
- Reorganisation likely to be limited to early-onset (sensitive period- when brain is still elastic)
- Caution:
Difficulty to clearly distinct between
primary brain areas and neighbouring areas (usually multi-modal) – small spatial resolution of TMS, PET and MRI it might not be clear where the reorganisation in primary areas occurs
Cross-modal Plasticity after Sensory Deprivation mechanism
2. Reorganisation of multi-modal
areas in the cortex:
Behavioural compensation for missing modality is mediated by enhanced
recruitment of multi-modal areas
- Reorganisation in multi-modal areas also for late-onset deprivations
- Example: Enhanced recruitment of
posterior STS (area of multi-modal
integration) in deaf individuals when attending to moving visual displays
