Multisensory Processes: A Balancing Act across the Lifespan

Multisensory processes are the product of a dynamic reweighting of physical stimulus characteristics and learned associations. This reweighting occurs across multiple timescales, ranging from long term (i.e., developmental and lifespan) to short-term (i.e., during the learning and encoding of multisensory relations). We propose a novel theoretical framework that combines traditional principles associated with stimulus characteristics (i.e., space, time, effectiveness) with a new principle of dynamic reweighting of stimulus characteristics and learned associations across different timescales. The novel theoretical framework emphasizes the plastic and dynamic nature of multisensory processing across the lifespan and thus accounts for improvements in multisensory perception and behavior under ‘normal’ circumstances and offers plausible remediation strategies for treating patients in whom sensory or multisensory function is compromised. Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales. Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales. Our world comprises a vast array of information that is coded in different sensory modalities. This presents a fundamental challenge for cognitive, perceptual, and motor systems because the multitude of multisensory inputs needs to be seamlessly integrated and appropriately segregated to form coherent perceptual representations and drive adaptive behaviours. Fortunately, the nervous system possesses specialized architectures and processing mechanisms that enable the combination and integration of multisensory information. These mechanisms and brain networks solve fundamental issues associated with sensory ‘binding’ (see Glossary) and consequently provide marked behavioral and perceptual benefits. Take as an example our ability to comprehend a single speaker in a noisy room. Seeing the speaker's mouth can increase the intelligibility of the auditory signal by 15 dB [1Sumby W.H. Pollack I. Visual contribution to speech intelligibility in noise.J. Acoust. Soc. Am. 1954; 26: 212-215Crossref Scopus (1879) Google Scholar]. This ability to extract speech information from visual cues begins in infancy when infants start babbling and need to learn native speech forms, when they have to disambiguate unfamiliar speech, and when they are learning two languages [2Lewkowicz D.J. Hansen-Tift A.M. Infants deploy selective attention to the mouth of a talking face when learning speech.Proc. Natl Acad. Sci. U.S.A. 2012; 109: 1431-1436Crossref PubMed Scopus (314) Google Scholar, 3Pons F. et al.Bilingualism modulates infants’ selective attention to the mouth of a talking face.Psychol. Sci. 2015; 26: 490-498Crossref PubMed Scopus (123) Google Scholar]. Extending this further, deaf individuals can learn to watch mouth movements to understand speech by lip reading, which results in extensive cross-modal plasticity within regions of the cerebral cortex [4Bavelier D. Neville H.J. Cross-modal plasticity: where and how?.Nat. Rev. Neurosci. 2002; 3: 443-452Crossref PubMed Scopus (717) Google Scholar]. Other examples of how multisensory interactions facilitate behavior and perception abound and include faster and more accurate detection, and (see The of Multisensory Scholar, The of the Scholar, The of Multisensory for The and for multisensory and integration have in and multisensory functions the of regions as as and regions this by that sensory systems have the to one processing in et multisensory function of the visual PubMed Scopus Google Scholar, Sci. PubMed Scopus Google Scholar, Multisensory in the 2015; PubMed Scopus Google The of multisensory are to be in that they a of that flexibility on sensory function et of PubMed Scopus Google Scholar]. and of multisensory the relative of versus in the product of a multisensory of we these as the physical characteristics of the stimuli and the the learned associations that are the different sensory modalities. these of stimulus characteristics and learned associations, we to highlight the dynamic and that are across a of ranging from the to is that physical stimulus characteristics a in multisensory interactions the and perceptual The of these characteristics include the and temporal of the stimuli to one as as relative in a these stimulus characteristics are to be to the of a multisensory We that these stimulus characteristics function in rather that they with one et principles of multisensory Neurosci. 2012; PubMed Scopus Google Scholar, et of multisensory and on multisensory PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar]. the of stimuli relative in a In addition, we how these and interactions are by and have to the nature of of multisensory when stimulus are et of PubMed Scopus Google Scholar, D. et attention and multisensory Sci. PubMed Scopus Google Scholar, et and multisensory Neurosci. 2015; PubMed Scopus Google Scholar, et and in multisensory PubMed Scopus Google Scholar, et to multisensory PubMed Scopus Google Scholar]. this that a of interactions the physical characteristics the of the stimuli and the learned associations that are short-term and the product of a multisensory the that is a of for the importance of the separate of and on multisensory on on of Multisensory of inputs from multiple sensory modalities long for regions of cortex as as for in The of the This traditional by of a of et of integration in Neurosci. 2002; PubMed Google Scholar, et from visual and to auditory cortex in the PubMed Scopus Google Scholar, Multisensory in visual in PubMed Scopus Google Scholar, et auditory and visual PubMed Scopus Google Scholar, et and a in Neurosci. PubMed Scopus Google in et multisensory function of the visual PubMed Scopus Google Scholar, Sci. PubMed Scopus Google The and behavioral and perceptual of these cross-modal are a of are the of multisensory of ranging from single in to of The of Multisensory Scholar, The of Multisensory Scholar]. is that multisensory results in or different from based on the of the fundamental processes to the of multisensory the physical of the stimuli to be and temporal stimuli in or both and behavioral and temporal stimuli in these the of these the and are et principles of multisensory Neurosci. 2012; PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar, et integration of cues by PubMed Scopus Google stimuli results in the These of principles and cues from a and highly stimuli need The of the of multisensory across and in different brain the developmental is one in which inputs from the sensory systems in that and as inputs from the different that are or by inputs from multiple (i.e., multisensory and increase in over et is for the of multisensory Neurosci. PubMed Scopus Google Scholar, et of multisensory Neurosci. 26: PubMed Scopus Google Scholar, of multisensory and multisensory integration in Neurosci. PubMed Google Scholar]. the of be in the developmental from behavioral and with the from that interactions based on the physical of multisensory inputs for the of multisensory interactions et and multisensory a PubMed Scopus Google Scholar, et from and Sci. 2015; PubMed Scopus Google Scholar]. (i.e., for the nature of multisensory processes. is that stimulus characteristics dynamically and in with a of processes and task et of PubMed Scopus Google Scholar, D. et attention and multisensory Sci. PubMed Scopus Google Scholar, et and multisensory Neurosci. 2015; PubMed Scopus Google Scholar, et and in multisensory PubMed Scopus Google Scholar]. The of inputs from multiple sensory modalities long for regions of cortex as as for in The of the This traditional by of a of et of integration in Neurosci. 2002; PubMed Google Scholar, et from visual and to auditory cortex in the PubMed Scopus Google Scholar, Multisensory in visual in PubMed Scopus Google Scholar, et auditory and visual PubMed Scopus Google Scholar, et and a in Neurosci. PubMed Scopus Google in et multisensory function of the visual PubMed Scopus Google Scholar, Sci. PubMed Scopus Google The and behavioral and perceptual of these cross-modal are a of are the of multisensory of ranging from single in to of The of Multisensory Scholar, The of Multisensory Scholar]. is that multisensory results in or different from based on the of the fundamental processes to the of multisensory the physical of the stimuli to be and temporal stimuli in or both and behavioral and temporal stimuli in these the of these the and are et principles of multisensory Neurosci. 2012; PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar, et of and on multisensory PubMed Scopus Google Scholar, et integration of cues by PubMed Scopus Google stimuli results in the These of principles and cues from a and highly stimuli need The of the Scholar]. The of multisensory across and in different brain the developmental is one in which inputs from the sensory systems in that and as inputs from the different that are or by inputs from multiple (i.e., multisensory and increase in over et is for the of multisensory Neurosci. PubMed Scopus Google Scholar, et of multisensory Neurosci. 26: PubMed Scopus Google Scholar, of multisensory and multisensory integration in Neurosci. PubMed Google Scholar]. the of be in the developmental from behavioral and with the from that interactions based on the physical of multisensory inputs for the of multisensory interactions et and multisensory a PubMed Scopus Google Scholar, et from and Sci. 2015; PubMed Scopus Google Scholar]. (i.e., for the nature of multisensory processes. is that stimulus characteristics dynamically and in with a of processes and task et of PubMed Scopus Google Scholar, D. et attention and multisensory Sci. PubMed Scopus Google Scholar, et and multisensory Neurosci. 2015; PubMed Scopus Google Scholar, et and in multisensory PubMed Scopus Google Scholar]. In this review we of the relative of physical stimulus characteristics and learned associations to multisensory processing We on two different temporal as a to highlight the dynamic that is these The temporal multisensory processing across the that is a developmental from an on stimulus characteristics to a on learned associations (see D.J. The of multisensory systems perceptual Sci. PubMed Scopus Google for a review of from of The temporal is and to the processing of information to behavioral and perceptual that are by We that multisensory processes within this are highly and the of learned associations and task in multisensory as in the of a sensory individuals can be to or to an sensory to functions of the we that multisensory processes can be by task and under circumstances in et of in the visual and auditory and 2015; PubMed Scopus Google the of multisensory function begins D.J. and multisensory perceptual PubMed Scopus Google of in and have on the D.J. The developmental and of multisensory from single to In The of Multisensory Scholar, et Scopus Google Scholar, and plasticity of multisensory Neurosci. Google Scholar, et and plasticity of and information Am. Acad. PubMed Scopus Google Scholar]. this that multisensory processing and the brain that over the of and that they on to be a developmental reweighting as that physical characteristics are more and more and learned are principle that from is that multisensory systems developmental and in infants have the physical stimulus characteristics of the sensory world and how stimulus as and to that are and thus associated with a the and of a a developmental stimulus characteristics as and temporal are the of learned perceptual can stimulus they provide the on which representations can be and the in the of more and processing The on these stimulus characteristics within the developmental of the brain regions for regions for more perceptual and different brain regions and different and with different to physical characteristics versus learned associations is in our of the temporal The that multisensory processes are on an to physical stimulus and that multisensory systems and to stimulus characteristics is by infants multisensory based on cues as D.J. Cross-modal in Scopus Google and temporal D.J. et perception and Scopus Google Scholar, and perceptual learning in PubMed Scopus Google and to the multisensory of and This of processing is by an of perceptual that infants to multisensory based on (i.e., infants and and and speech and and and speech The of a more multisensory perceptual system is that infants to a of auditory and visual of information in the the of more multisensory infants and as they with native they multisensory inputs and native F. et of speech perception in Natl Acad. Sci. U.S.A. PubMed Scopus Google Scholar, D.J. The of perception in Natl Acad. Sci. U.S.A. PubMed Scopus Google This is as multisensory perceptual D.J. The of multisensory systems perceptual Sci. PubMed Scopus Google Scholar, D.J. and multisensory perceptual PubMed Scopus Google and the of and selective infants’ to and that have (i.e., that are that are of during this infants’ ability to more multisensory associations as a of with the This developmental multisensory perceptual and the of more and specialized multisensory perceptual and is by and the of infants the ability to the of auditory and visual and speech information D.J. perception of speech PubMed Scopus Google Scholar, D.J. of temporal in PubMed Scopus Google Scholar, et perceptual of an in Sci. Scopus Google as as the of and speech and and D.J. The of multisensory systems perceptual Sci. PubMed Scopus Google Scholar, D.J. of temporal in PubMed Scopus Google Scholar, The perception of speech in PubMed Scopus Google Scholar, information in and is in Scopus Google Scholar]. infants to the multisensory speech as the et in PubMed Scopus Google which the of auditory and visual speech of infants to et of multisensory in and 2015; PubMed Scopus Google and perception of of PubMed Scopus Google as multisensory perceptual by they start to to the of speech in a mouth [2Lewkowicz D.J. Hansen-Tift A.M. Infants deploy selective attention to the mouth of a talking face when learning speech.Proc. Natl Acad. Sci. U.S.A. 2012; 109: 1431-1436Crossref PubMed Scopus (314) Google and by they to the multisensory of speech and the multisensory of native speech D.J. et of the multisensory of speech in and the of 2015; PubMed Scopus Google Scholar, D.J. F. of in PubMed Scopus Google This developmental is the for and of et to and in PubMed Scopus Google of information et in PubMed Google these illustrate a developmental in which the ability to more multisensory characteristics is on an ability to characteristics In the we that the of multisensory integration and in on physical stimulus In this we that as and regions and (i.e., to on an of the for this developmental can be in the in the relative weighting of these in to the temporal of auditory and visual are of multisensory temporal function and one of the more of these is based on the of a the of within which is highly that stimuli from two different sensory modalities be a single perceptual D.J. of temporal in PubMed Scopus Google Scholar, Multisensory in a Neurosci. PubMed Scopus Google Scholar, Multisensory temporal task and stimulus PubMed Scopus Google Scholar, et the and temporal stimulus that multisensory integration in 2012; PubMed Scopus Google Scholar]. a a of adaptive because to visual and auditory from different This occurs the that visual and auditory different as they to the sensory and the that across the in and perceptual functions of multisensory interactions in auditory PubMed Scopus Google on the developmental of the that in the multisensory temporal Sci. 2012; PubMed Scopus Google Scholar]. that the for different stimuli and in infants D.J. perception of speech PubMed Scopus Google Scholar, D.J. of temporal in PubMed Scopus Google et temporal for are PubMed Scopus Google and Multisensory temporal task and stimulus PubMed Scopus Google a on physical stimulus characteristics on learned associations. for the is that this is different of the for the stimuli (i.e., the for and as and et temporal for are PubMed Scopus Google Scholar]. This is to the one the developmental by physical stimulus In this the be that the integration of stimuli and the integration of more these developmental that the importance of can which are learned and can an in in the of multisensory processes and the that The the in multisensory systems as as the reliance on to multisensory interactions as one the of these is to the over which they of is to multisensory plasticity in plasticity can provide an the physical characteristics and learned associations and the over which this dynamism this the we in this that on multisensory temporal because is to more multisensory because on in the perceptual that on a task the of the relative of an auditory and visual stimulus results in a in the ability to et the temporal of multisensory Neurosci. PubMed Scopus Google Scholar, et of multisensory perceptual Neurosci. 2012; PubMed Scopus Google Scholar]. when with on a (i.e., they a or multisensory temporal by the to by These of the the of for more than the of to the stimuli to during et the temporal of multisensory Neurosci. PubMed Scopus Google Scholar]. on a task with and to the for to the in The of the multisensory temporal and in developmental PubMed Scopus Google on a that is in the physical characteristics of the stimuli (i.e., temporal the of learned associations (i.e., the based on in based on the of an on to the stimuli that are to the in multisensory temporal are in as in the of to a or of [4Bavelier D. Neville H.J. Cross-modal plasticity: where and how?.Nat. Rev. Neurosci. 2002; 3: 443-452Crossref PubMed Scopus (717) Google Scholar, F. of multisensory from and Neurosci. Google Scholar, et the brain to understand the a Rev. PubMed Scopus Google Scholar]. results highlight that different perceptual and can be by physical characteristics versus learned associations. These plasticity provide an the in multisensory processes and over the of a or learned associations can processes that are in physical stimulus characteristics (i.e., temporal an can these multisensory functions on an that they that we to of have to on this by on fundamental processes as learning and of on multisensory to a task that within a single et of visual and multisensory by PubMed Scopus Google Scholar, et brain multisensory to PubMed Scopus Google Scholar, The of multisensory in PubMed Scopus Google Scholar, et of based on multisensory 2012; PubMed Scopus Google Scholar, et PubMed Scopus Google Scholar, et multisensory auditory and visual 2015; PubMed Scopus Google Scholar]. This the of an stimulus in a multisensory a with the that the ability to the stimulus when in a single of or is for in a multisensory (i.e., the and of the versus in a is or the multisensory the with the of an or an stimulus in the These that the encoding of information in a multisensory during the of a single of that to information processing and information is to of multisensory the multisensory nature of these task (see The of multisensory 26: PubMed Scopus Google for a review of This for a (i.e., on the of to of a learned multisensory (i.e., the versus the physical stimulus characteristics (i.e., the of on sensory processing and and brain have that these multisensory during the of stimulus processing and within visual and auditory how information and can sensory processing et of visual and multisensory by PubMed Scopus Google Scholar, et brain multisensory to PubMed Scopus Google Scholar, The of multisensory in PubMed Scopus Google Scholar, et of based on multisensory 2012; PubMed Scopus Google Scholar, et auditory sensory processing of is by visual PubMed Scopus Google Scholar, et of auditory in the of Neurosci. 2015; PubMed Scopus Google Scholar]. have in the of This that the to which an auditory and visual information can on the the of an brain to multisensory stimuli on they a or during provide multisensory brain one in and based within a single et PubMed Scopus Google Scholar]. In to for our of multisensory processing and the within which multisensory encoding and these challenge which have that is when encoding and et Scholar]. an associated with multisensory they are to the processing of of this is to these across the lifespan to and is the of the we are multisensory processes to more remediation that can be in This of for patients with Multisensory in Neurosci. 2012; PubMed Scopus Google or sensory et brain as a task for visual 2012; PubMed Scopus Google (see and in to new and as these in et brain as a task for visual 2012; PubMed Scopus Google Scholar]. The how the and plasticity of multisensory processes across the lifespan the physical characteristics (i.e., of stimuli in the how a dynamic reweighting these stimulus characteristics and learned associations shapes multisensory processing as and how multisensory encoding can of to learning and The is how the can be learned associations as one multisensory in the can information processing be by learned associations and the task to the characteristics of the sensory this is the the can be as to to the physical characteristics of sensory information. the principle be that task and in et of in the visual and auditory and 2015; PubMed Scopus Google that the brain is or as as multisensory for information from the different this et multisensory function of the visual PubMed Scopus Google Scholar, Sci. PubMed Scopus Google in can be in individuals by (i.e., sensory the in this are to in to visual et in the visual Neurosci. PubMed Scopus Google Scholar, et by sensory the Neurosci. PubMed Scopus Google Scholar, sensory the of Rev. Neurosci. PubMed Scopus Google Scholar, et visual of visual PubMed Scopus Google Scholar]. need to be with of in visual of individuals et of visual cortex in the 2015; PubMed Scopus Google Scholar, et visual is the visual and and Neurosci. 2015; PubMed Scopus Google Scholar, et in the of and visual Neurosci. 2015; PubMed Scopus Google as as visual of visual et with sensory the visual form in the 2012; PubMed Scopus Google Scholar, Visual cortex in by PubMed Scopus Google Scholar]. the of the principles and processing the in the of sensory with in In the one that the the the to be by sensory sensory cross-modal PubMed Scopus Google is to based on the versus of the sensory inputs et of PubMed Scopus Google Scholar, multisensory processing in auditory PubMed Scopus Google The by the to which sensory regions are specialized that the brain to one of stimuli of that for as a task is the cortex of that the cortex an in the of This from that both and individuals the cortex in to these individuals learn to extract shapes from as when a or when to et in the visual Neurosci. PubMed Scopus Google Scholar, et during of in and Neurosci. Google Scholar]. novel cross-modal one that information is and within the Multisensory in a novel PubMed Scopus Google Scholar]. individuals the cortex when is as when to the and an auditory task or an auditory task et in the visual Neurosci. PubMed Scopus Google Scholar]. the in the of this in the this with the in the cortex et of in visual and to 2015; PubMed Scopus Google Scholar]. Other examples of processes include the of the form during or et with sensory the visual form in the 2012; PubMed Scopus Google and the of the during the of the of as Visual cortex in by PubMed Scopus Google Scholar]. examples of and have in in the of brain in the of visual or when the in How the task of brain regions is and from the and across These drive plastic in to in the sensory or sensory that brain regions are as sensory for the inputs that they they are as task based on the that they or the of the inputs information that they of sensory this can be short in the and brain and stability in sensory processing sensory 2015; PubMed Scopus Google Scholar]. these illustrate how the weighting of stimulus characteristics versus learned associations can be to a where function (i.e., task or can drive multisensory function of the stimulus characteristics in et of 2015; PubMed Scopus Google Multisensory processes are and for the and of accurate perceptual and is multisensory and integration the and the behavioral and perceptual of these processes in is and in is how the physical characteristics of the stimuli and the learned associations in a to the multisensory product (see Our review that multisensory processing in and that is plastic and in in that multisensory networks to the of the sensory world to the nature of and behavioral task and we that multisensory processes can learning and under ‘normal’ circumstances for remediation in of sensory highly plastic and dynamic and and the of multisensory How multisensory processes across the during the are the multisensory integration the and are these across brain are the mechanisms that the of information across the a or that this across are the on multisensory are the of is the to or individuals in the of facilitate or with sensory as in the of or accounts for the and task of a brain How this and how is How can the brain the task of sensory and of with a is a brain to a new task or based on the sensory multisensory in are the in of in the of the are in individuals for whom the the et temporal for are PubMed Scopus Google Scholar, The of the multisensory temporal and in developmental PubMed Scopus Google Scholar]. with multisensory as on the The of the multisensory temporal and in developmental PubMed Scopus Google thus be to from individuals with to multisensory stimuli of multisensory learning in et of visual and multisensory by PubMed Scopus Google are the brain for the in multisensory How multisensory interactions and how are processes with in multisensory in the product of a multisensory How and and the of multisensory How multisensory processes across the during the are the multisensory integration the and are these across brain are the mechanisms that the of information across the a or that this across are the on multisensory are the of is the to or individuals in the of facilitate or with sensory as in the of or accounts for the and task of a brain How this and how is How can the brain the task of sensory and of with a is a brain to a new task or based on the sensory multisensory in are the in of in the of the are in individuals for whom the the et temporal for are PubMed Scopus Google Scholar, The of the multisensory temporal and in developmental PubMed Scopus Google Scholar]. with multisensory as on the The of the multisensory temporal and in developmental PubMed Scopus Google thus be to from individuals with to multisensory stimuli of multisensory learning in et of visual and multisensory by PubMed Scopus Google Scholar]. are the brain for the in multisensory How multisensory interactions and how are processes with in multisensory in the product of a multisensory on an of the from the as as the of in The of and the from the of and is a and is by an as as the for from the of and the and the processes different information is coded as from the (i.e., In multisensory different information to stimulus as and occurs in the processes information from one can or drive in brain regions associated with sensory example is the of sensory when a an auditory and visual speech a novel example is when an auditory and visual in the perception of This in we with the visual from the and mouth to more processes in to inputs from multiple an developmental to and with the native to of perceptual and to perceptual of an and poorly perceptual this to the of for native multisensory inputs and a in to that information one when visual is or for a the of within which is a that stimuli be integrated or as by behavioral or perceptual is a for multisensory temporal The is on the temporal principle of multisensory the in and behavior are when the stimuli are in temporal