Research Article: Modern health worries: Deriving two measurement invariant short scales for cross-cultural research with Ant Colony Optimization

Date Published: February 7, 2019

Publisher: Public Library of Science

Author(s): Gabriel Olaru, Oliver Wilhelm, Steven Nordin, Michael Witthöft, Ferenc Köteles, Rodrigo Ferrer.


Worries about possible harmful effects of new technologies (modern health worries) have intensely been investigated in the last decade. However, the comparability of translated self-report measures across countries is often problematic. This study aimed to overcome this problem by developing psychometrically sound brief versions of the widely used 25-item Modern Health Worries Scale (MHWS) suitable for multi-country use. Based on data of overall 5,176 individuals from four European countries (England, Germany, Hungary, Sweden), Ant Colony Optimization was used to identify the indicators that optimize model fit and measurement invariance across countries. Two scales were developed. A short (12-item) version of the MHWS that represents the four-factor structure of the original version and an ultra-short (4-item) scale that only measures the general construct. Both scales show that overall levels of health worries were highest in England and Hungary, but that the main reason for concern (e.g. electromagnetic radiation or food related fears) differs considerably between these countries. This study also shows that even if measurement invariance of translated self-report instruments across countries is problematic, it can be optimized by using adequate item selection procedures. Differences of modern health worries across countries and recommendations for cross-cultural research are discussed.

Partial Text

Worries about potentially adverse health effects of scientific and industrial progress appear to be longstanding phenomenon. Prominent examples are health complaints related to railroad accidents known as the ‘railway spine syndrome’ in the early 19th century as well as the more recent phenomenon of ‘electric allergy’ [1]. With the start of the third millennium, these “modern health worries” (e.g. worries about adverse health effects of genetically modified food) have become a focus of psychometric research in health psychology and behavioural medicine. However, cross-cultural research on modern health worries is lacking. For an unbiased comparison of across countries, measurement invariance needs to be established before mean-level differences can be analysed [2]. This paper summarizes the current state of knowledge concerning modern health worries (MHWs), as assessed by the Modern Health Worries scale (MHWS) [3]. We subsequently present how measurement invariance of the scale can be improved using Ant Colony Optimization [4–6]. The derived short scales are used to investigate country specific differences in MHWs. Recommendations and advantages of this procedure compared to classical approaches for dealing with a lack of measurement invariance (e.g. partial measurement invariance) are subsequently discussed.

We first tested the different factor models on the full 21-item scale under configural measurement invariance across the four countries (see online repository at for fit indices and full loading structure of all models). The correlated factor model fitted the data worst (CFI = .974; RMSEA = .106). The correlations between the factors were similarly high as in previous studies (r = .55 – .89; on average r = .73). Even though the higher-order model is more parsimonious, it fitted the data slightly better, but still yielded insufficient model fit (CFI = .975; RMSEA = .103). The second-order factor loadings were on average .87, which supports the presence of a second-order factor. The bi-factor model of MHWs resulted in better fit (CFI = .984; RMSEA = .087), but suffered from a large number of negative or low loadings on the Toxic Intervention factor and a lack of robustness across countries. Given the large proportion of low factor loadings and difficult interpretability of the factors, we decided against using this model [46]. This problem was also apparent in the bi-factor ESEM model, which otherwise fitted the data well (CFI = .994; RMSEA = .064). However, as noted before, the scale scores can only be meaningfully interpreted when model fit, factor loadings and the theoretical foundation of the model are sound. In the case of the bi-factor model, the interpretability of the factors and corresponding scale scores was problematic due to the unclear factor pattern. While the ESEM correlated factor model yielded better model fit (CFI = .989; RMSEA = .081) than the CFA counterpart, a large number of cross-loadings were significant (151 out of a total of 252). Only 16 out of 63 possible cross-loadings per country were robust across countries (i.e., always or never significant). Note that despite being reduced by the inclusion of cross-loadings, the factor correlations were still reasonably high (average r = .61). Despite the better model fit of the ESEM and bi-factor models, we decided retain the theoretical model of MHWs and remove problematic items (instead of adding additional parameters to reduce misfit). We thus applied the higher-order factor model, which was the most parsimonious model and yielded the strongest factor loadings. However, the unsatisfactory RMSEA levels under configural measurement invariance indicated that the full scale does not adequately represent the theoretical structure of MHWs. As such, higher levels of measurement invariance across countries were not tested. Using ACO, we improved the absolute model fit of both short scales beyond the fit of the original scale (see Table 4). To account for possible effects of item reduction on the fit indices (i.e. improving model fit due to reduced model complexity), we compared the ACO models to item selection by chance. We randomly selected 1,000 12-item models and computed the 1st and 99th percentile of CFI, RMSEA and ΔCFI to examine the distribution of model fit (configural measurement invariance: CFI = .975-.994; RMSEA = .078-.132; scalar measurement invariance: CFI = .969-.990; RMSEA = .080-.117). As can be seen, ACO optimized absolute model fit for both measurement invariance levels beyond the 99th CFI or 1st RMSEA percentile of the model fit distribution.

ACO was successful at improving model fit and measurement invariance of the modern health worries scale (MHWS) on a cross-cultural sample, while also maintaining the factor structure of the measurement. We developed two short forms that provide a valid and measurement invariant measurement of the construct across four European countries, namely Hungary, Sweden, Germany and England. We then compared the levels of the various facets and the general factor of MHWs across these countries.

ACO has shown to be an adequate tool for improving measurement invariance in a cross-cultural setting. The short (12-item) version of the MHWS maintains the original four-factor structure of the original scale, while also yielding good model fit and measurement invariance across the four countries assessed in this study. The ultra-short (4-item) scale is appropriate for the measurement of the general construct, but is unable to detect meaningful differences across countries at the factor level. In addition, the reduced item number will negatively affect measurement precision. We hence recommend maintaining the original factor structure when shortening measurement inventories. This can be done easily using ACO, as it will select item sets of a fixed size for a pre-defined model instead of removing items sequentially. In this study, the English and Hungarian sample showed higher levels of MHWs than the German and Swedish sample. While general levels of MHWs were similar between the Hungarian and English sample, Hungarians seemed to be more concerned about Radiation than the other countries. Participants from England were more worried about Toxic interventions. Concerns about Environmental pollution were high in both these countries. General levels of MHWs were similarly low for German and Swedish participants.




Leave a Reply

Your email address will not be published.