Research Article: Using Item Response Theory to evaluate the psychometric characteristics of questions in a Brazilian programme and the performance of dental teams in primary care

Date Published: May 31, 2019

Publisher: Public Library of Science

Author(s): Suellen R. Mendes, Renata C. Martins, Juliana V. M. Mambrini, Antônio Thomaz G. Matta-Machado, Grazielle C. M. Mattos, Jennifer E. Gallagher, Mauro H. N. G. Abreu, Thiago Saads Carvalho.

http://doi.org/10.1371/journal.pone.0217249

Abstract

First, to assess the psychometric properties of key questions included in a public sector evaluation of primary dental care in Brazil; and second, to evaluate the performance of dental teams in relation to these items.

Secondary analysis of a national primary care dataset monitoring quality and access to dental care. Data were collected through face-to-face interviews with representatives of dental teams participating in the ‘National Programme for Improving Access and Quality of Primary Care’. Twenty-three mandatory questions about the dentists’ reported delivery of dental procedures were included in the analysis. Item Response Theory (IRT) modelling was applied to measure the psychometric properties of the instrument—level of difficulty and discrimination parameter of each item—and then to estimate dental team performance scores based on these parameters. Based on IRT, possible scores ranged from -4 to +4.

Three of the 23 mandatory items were removed due to poor internal consistency, resulting in a scale of 20 items for assessing dental team performance. The results showed variation in procedures delivered by the dental teams; whilst more than a half of the procedures were executed by at least 80% of the dental teams, those relating to dentures (partial/total) and frenectomy (lingual/labial) were performed by less than 30%. Amongst the 20 items included in the model, those related to partial/total dentures and oral cancer follow-up presented higher levels of difficulty and were less frequently provided. The items relating to the treatment of deciduous teeth and access to the dental pulp of permanent teeth had the highest discrimination parameters and, consequently, greater weight in the performance’s score estimation; therefore, dental teams that did not perform these items had the lowest performance scores. In the present study, dental team performance scores ranged from -3.66 to +1.87 with a mean/median of -0.06/+0.01.

The findings suggest that whilst the items within the instrument demonstrated some potential to discriminate between poor and very poor teams, they were ineffective in discriminating between poor and good teams. Whilst Brazilian dental teams perform many mandatory procedures, variation in the nature of their delivery of care requires further investigation to enhance service provision to the population.

Partial Text

Brazil is a large country that covers approximately 47% of South American landmass, with a population estimate of 209 million inhabitants across five geographical regions (north, northeast, centre-west, southeast and south) and marked socioeconomic disparities [1]. Since 1988, the Brazilian population has been provided with access to all levels of health care through the Brazilian National Health System (known as Sistema Único de Saúde and abbreviated to ‘SUS’), which is free at the point of delivery. It is therefore one of the biggest health care systems in the world, serving approximately 60% of the Brazilian population; none-the-less, whilst it is open to all, however, some people choose to seek private care [2,3].

The correlation of the 23 items from the original scale presented with a low Cronbach’s alpha coefficient, and three items were therefore removed. The total Cronbach’s alpha of the 20 remaining items was 0.66, which is considered acceptable internal consistency [24]. It is important to note that the dental procedures which were unreliable in the model were also less commonly provided: ‘anatomical and functional impression for dentures/prostheses’ (8.2%), ‘removal of cysts’ (21.4%), and ‘removal of impacted teeth’ (30.1%).

The items included in the estimation of dental team performance are mandatory in the Brazilian PHC dentistry, although the dental teams are not forced to perform items which are beyond their competence. Whilst most of the dental procedures were reportedly executed by most dental teams; it has been possible to observe through the two PMAQ-AB cycles (2011/2012 and 2013/2014) that the additional procedures included after NOHP was introduced had the lowest individual performance frequencies, and so present higher levels of difficulty. Most notably they related to the provision of partial/total dentures and oral cancer monitoring. Furthermore, it is of great concern that certain dental teams did not even meet the requirements to participate in the evaluation and they clearly need urgent investigation and support to ensure that they are fit for purpose of PHC.

The findings suggest that items within the SUS evaluation instrument demonstrated some potential to discriminate between poor teams from the very poor ones and ineffective in discriminating the teams considered poor from the good ones. Whilst Brazilian dental teams perform many mandatory procedures, variation in the delivery of care requires further investigation to enhance service provision to the population.

 

Source:

http://doi.org/10.1371/journal.pone.0217249