Date Published: February 20, 2019
Publisher: Public Library of Science
Author(s): I-Hsuan Chen, Qingqing Zhao, Yunfei Long, Qin Lu, Chu-Ren Huang, Zhiqiang Cai.
Modality exclusivity norms have been developed in different languages for research on the relationship between perceptual and conceptual systems. This paper sets up the first modality exclusivity norms for Chinese, a Sino-Tibetan language with semantics as its orthographically relevant level. The norms are collected through two studies based on Chinese sensory words. The experimental designs take into consideration the morpho-lexical and orthographic structures of Chinese. Study 1 provides a set of norms for Mandarin Chinese single-morpheme words in mean ratings of the extent to which a word is experienced through the five sense modalities. The degrees of modality exclusivity are also provided. The collected norms are further analyzed to examine how sub-lexical orthographic representations of sense modalities in Chinese characters affect speakers’ interpretation of the sensory words. In particular, we found higher modality exclusivity rating for the sense modality explicitly represented by a semantic radical component, as well as higher auditory dominant modality rating for characters with transparent phonetic symbol components. Study 2 presents the mean ratings and modality exclusivity of coordinate disyllabic compounds involving multiple sense modalities. These studies open new perspectives in the study of modality exclusivity. First, links between modality exclusivity and writing systems have been established which has strengthened previous accounts of the influence of orthography in the processing of visual information in reading. Second, a new set of modality exclusivity norms of compounds is proposed to show the competition of influence on modality exclusivity from different linguistic factors and potentially allow such norms to be linked to studies on synesthesia and semantic transparency.
The modality exclusivity effect is based on the observation that words referring to strong sensory meanings link to the perceptual regions of the brain [1, 2, 3, 4, 5]. Studies have shown that conceptualization corresponds to the neural systems in charge of perception and motor control [2, 6, 7]. In particular, modality exclusively mapping has been found in many studies [8, 9]; for example, studies have found that one word maps to one modality [7, 8, 10, 11]. The mapping asserts that there is a strong association between a specific word and a specific modality. However, empirical evidence has shown that a property referred to by a word may not only be conceptualized in one specific modality; instead, a property can be perceived in unimodal, bimodal, or multimodal experiences [12, 13, 14, 15, 16]. For example, English sweet generally refers to gustatory experience, while it can also be used to describe visual experiences as in sweet smile. This linguistic phenomenon is often referred to as linguistic synesthesia and generated important literature in linguistics focusing on the potentially universal directionality constraints among different senses [17, 18, 19]. The modality exclusivity norms have been developed in this context for different languages such as English and Dutch for relevant research in this field [14, 15, 16]. For instance, the modality exclusivity norms supported Strik Lievers and Winter’s  new study from the linguistic perspective showing that sense modalities provide cognitive motivation for grammatical categories of sensory lexicon. Since previous studies primarily focus on Indo-European languages, modality exclusivity norms for a non-Indo-European language such as Mandarin Chinese will provide an important dataset to show similarities across languages and also language-dependent variation in modality exclusivity, especially in light of recent studies showing cross-lingual variations of synesthetic mapping directionality based on Mandarin Chinese and Korean [19, 21].
In Mandarin, a disyllabic sensory word can be composed of two monosyllabic sensory words from the same sensory experiences or from different sensory experiences. These two-character sensory words are often used to describe the sensory experiences related to the etymology of its components. For example, the disyllabic word 濕潤 shi1run4 ‘moist’ is composed of two characters with similar meanings: ‘wet’ and ‘moist’ respectively. The two characters are both from the haptic modality, and the compound still describes haptic experiences. These disyllabic words can also have two component characters from different sense modalities. For example, 痛苦 tong4ku3 ‘painful’ is composed of two characters from haptic experiences (‘pain’) and gustatory experiences (‘bitterness’) respectively. However, the meaning of this disyllabic word is basically haptic. The term then was extended to the mental interpretation of suffering.
This rating task shows that the semantics encoded in the components of the two-character words may influence the participants’ evaluation. For the majority of the compounds, their dominant modality is still related to the etymology of the component characters. In the visual, haptic, and gustatory dominant modalities, the dominant modality is the same as one of the two components for the majority of the compounds. Notably, the compounds which are rated to be dominant in the visual modality can contain no visual component. Within this group, the compounds contain one haptic component, which strongly suggests either synesthetic or associative relations between the visual and haptic modalities.
This current study is designed to reflect characteristics of the morphology and orthography of Mandarin Chinese, as well as the role they may play in reading strategy. The rating tasks on the Mandarin modality exclusivity are done in both single-character and two-character compound words in this paper in order to examine how Mandarin native speakers evaluate sensory words with different morphological complexity. The exclusivity ratings of mono-syllabic words written with one character allows clarity in the identification of original and basic sense modality as well as the relationship between that modality and all highly rated modalities. The experiment on disyllabic two-character compounds provides the context for us to observe more complex interactions among multiple sense modalities. Our results show that the sense modality property of a word is generally multimodal for both mono- and di-syllabic words, which is in line with the modality ratings of other languages [14, 15, 16]. In both studies, the visual dominance is salient for both monosyllabic words and disyllabic compounds. In particular, compounds with the visual dominant modality can generally describe sensory experiences not explicitly represented by either component of the compounds.
The paper provides the first sets of modality exclusivity ratings of a language with explicit representations of sense modality in the writing system. Regardless of different types of component composition, Chinese characters typically have a semantic radical that contains explicit information related to the experiences from different modalities. Leveraging this encoded modality information in our analysis, the collected modality exclusivity norms show that Chinese orthography plays an important role in the perception of the modality of a word. The most interesting discovery is that the awareness of direct phonological encoding in a character seems to strengthen the exclusivity rating of the auditory modality. We also set up a set of exclusivity norms for compounds involving components encoding different modalities. We found that the dominant modality of such compounds could come from the modality of one or neither of the components. However, it is conceivable that other variables such as semantic transparency of compound and component words  may also play a role. In addition, expansion and elaboration of compound modality exclusivity norms in Chinese and other languages may lead to new directions of research.