All published articles of this journal are available on ScienceDirect.
Psychometric Attributes of the DISC Predictive Scales
Abstract
Introduction:
This study was designed to evaluate the psychometric attributes and screening efficiency of a Spanish version of the Children Predictive Scales (DPS) against the Spanish Diagnostic Interview Schedule for Children (DISC-IV).
Method:
This pilot cross-sectional study included 61 children aged 9 to 14 years in a mainstream school. The following psychometric attributes were analyzed: acceptability, scale assumptions, internal consistency, and precision, as well the predictive validity (AUC).
Results:
The scale did not show ceiling or floor effects (6.4%, 1.3%, respectively). The internal consistency was high (α=0.92), and the standard error of measurement was adequate (SEM=1.54). The overall DPS AUC was 0.72 against DISC IV corresponding diagnosis.
Conclusion:
The Spanish version of the DPS-4.32 seems to be a reliable and precise tool for screening mental health disorders in a school-age population.
INTRODUCTION
Over the past 10 years, and particularly after the publication of the Global Burden of Disease [1], significantly more attention has been paid to mental disorders of childhood and adolescence. Recent data have shown that between 3% and 18% of children suffer from psychiatric disorders, and these disorders can cause significant functional impairment [2]. In addition, psychiatric disorders can be identified in 25 - 75% of patients with juvenile neurological disorders such as epilepsy, tic disorders or mental retardation [3-6]. Therefore, in several settings (e.g., clinical assessments, epidemiological studies of neurological disorders in children), it is important to screen for psychiatric disorders. While short screening measures are sometimes preferable, these do not necessarily produce diagnostic categories based on Diagnostic and Statistical Manual of Mental Disorder (DSM) criteria [7]. The DISC Predictive Scales (DPS) are a series of diagnosis- specific short-forms of the Diagnostic Interview Schedule with Children (DISC), and include only the DISC items that are most predictive of DSM-IV DISC diagnosis [8]. The full DPS can be used to accurately screen for specific DSM disorders, and they also have the potential to considerably speed up structured diagnostic interviewing [9]. Although the screening efficiency of the DPS has been widely examined, showing high sensitivity, specificity and negative predictive values [10-12], only a few clinimetric attributes such as test-retest reliability and internal consistency have been examined in English versions [10,11]. Therefore, this study was firstly aimed to analyze the remaining psychometric attributes of the DPS (validity, precision and some aspects of acceptability) and to test the screening efficiency of a Spanish version of the DPS against the Spanish DISC-IV, in a school-based sample of children and adolescents in Spain.
METHODS
Design
This was a pilot, cross-sectional study carried out in the Neurology Department of the General Yague Hospital, Burgos, Spain. This project was part of a larger epidemiological protocol for the study of the prevalence of tic disorders and associated comorbidities in a school-based sample of children, designed to validate this scale for the field study. The study protocol was in accordance with the Helsinki declaration. It was approved by the ethical review board of the Burgos Hospital Complex Ethics Committee and the local school authorities, which act as ethical committees in each school. The anonymity of each child was rigorously preserved, in line with national privacy legislation. Prior to starting, this study was discussed at teacher meetings and a letter of invitation and consent form was sent to the children’s home for consideration. Written consent from a parent/guardian was requested. If the parent declined participation in the study, no data were collected on that child.
Subjects
For this pilot study, a selected school was chosen, and two different developmental stages were included to ascertain a convenience sample of both children and adolescents. All subjects were either in the middle year of elementary school (aged 9-10 years) or the middle year of a single local mainstream school (teenagers aged 13-14 years) in Burgos (Spain). Subjects with prior learning difficulty, pervasive developmental disorders or any other psychiatric diagnosis were not excluded for the study.
Measures
The following measures were administered: 1) The computerized Spanish Generic DISC-IV [13], a respondent-based structured interview with parallel parent and youth (9-17 years) versions. This scale includes 358 stem questions and nearly 1,300 contingent questions. Using the previous 12-months as a time frame, the DISC-IV may be used to generate more than 30 DSM-IV childhood diagnoses. For this study, the following DISC-IV modules were administered: anxiety disorders, mood disorders, disruptive behaviour disorders, and alcohol/substance abuse; 2) An updated computerized Spanish version of the original DPS, provided by the developing authors (CP Lucas), which contains 18 subscales (agoraphobia, social phobia, specific phobia, generalized anxiety disorder, obsessive-compulsive disorder [OCD], major depressive disorder, attention-deficit/ hyperactivity disorder [ADHD], oppositional defiant disorder, and conduct disorder, alcohol, marijuana and other substances abuse). The DPS is made up of a brief diagnosis-specific self-report inventory to identify youths endorsing symptoms and who are highly likely to meet diagnostic criteria [9]. The DPS also measures impairment with seven questions that inquire about parent and teacher reactions to, and limitations resulting from, youths’ feelings and behaviour. The total DPS score is therefore the sum of DPS symptomatology and DPS impairment. We used the cut-off scores for an adolescent population provided by the original developers [9,14] (see Table 2). A DPS total score ≥ 9 indicated the presence of probable mental disorders when at least social phobia, general anxiety, depression alcohol, marijuana and other substance have been included in the screen; 3) A Health-Related Quality of Life Scale (HRQoL): Peds-Qol 4.0 (Spanish version of the Paediatric Quality of Life Inventory) Generic Core Scales [15], as a self-report HRQoL measure for children and adolescents. In this scale, items are reverse scored and linearly transformed to a 0-100 scale, so that higher scores indicate better HRQoL.
Procedure
Written informed consent was obtained from parents. Appropriate verbal instructions were given to all subjects. For subjects aged 9-10 years, computer-assisted face to-face interviews for the DISC-IV and DPS were performed, and the Peds-QoL scales were administered by the trained investigators. For this group, the DISC-IV and DPS were completed by the parents and the Peds-QoL scales by the children. For subjects aged 13-14 years, all the former scales were completed by the teenagers, including the DISC-IV and DPS. The DPS and DISC-IV scales were administered in the same order to all youth.
Data Analysis
Data were summarized as means ± standard deviation (SD) or counts (percents). In terms of data quality, any value for fully computable data that exceeded 95% was considered acceptable. The following psychometric attributes were analyzed: acceptability, scale assumptions, internal consistency, and precision. Scale assumptions were explored in relation to average scores in the included sample. Acceptability was explored in relation to score ranges, floor and ceiling effects. A ceiling effect is defined as an undesirable measurement outcome occurring when the dependent measure puts an artificially low ceiling on how high a participant may score, and on the contrary a floor effect, when the dependent measure artificially restricts how low scores can be. In this regards we took a of 15% for ceiling and floor effects as satisfactory, and skewness of score distributions [accepted limits: limits: -1 to +1] [16,17]. Internal consistency was determined on the basis of item-total correlation (corrected for overlap, criterion value ≥ 0.40), and Cronbach's alpha coefficient (α value, with a value of ≥0.70 being deemed acceptable) [18]. An exploratory factor analysis (principal components method, orthogonal varimax rotation) was performed to ascertain the structural characteristics of the scale.
Precision was assessed using the standard error of measurement (SEM = SD × √[1 - reliability coefficient (α value)]). The upper level of the 95% confidence interval (CI) is represented by 1.96 × SEM. An arbitrary maximum SEM value of half a standard deviation (meaning a reliability index ≥ 0.75) was adopted. The results obtained were interpreted following the published recommendations for health measurement tools [19]. Validity was assessed using the convergent validity and known-groups validity. For convergent validity, we hypothesized a low-moderate association among DPS subscales, and between DPS and Peds QoL scores (Spearman rank correlation coefficient values, rs ≥ 0.20). DPS scores were compared among males and females, and known-groups validity were tested using the Mann-Whitney test to determine the scale's capacity (DPS total score ≥ 9) to distinguish among patients grouped by severity based on impairment of HRQoL (PedsQoL scores).
The sensitivity, specificity, positive predictive value (PPV) and the negative predictive value (NPV) of the DPS against DISC-IV corresponding diagnosis were calculated. The area under the ROC curve (AUC) of the DPS-and the predictive value according to distribution of their scores against DISC-IV corresponding diagnosis in the included patients group were used as measurements of predictive validity. Analyses were confined to those disorders that were measured on both the DISC and the DPS. The AUC provided an estimate of the diagnostic accuracy of the DPS against DISC-IV corresponding diagnosis. Higher AUC reflect a greater probability that a child with a disorder on the DISC IV will score above the cut-off score of the designed DPS subscale. AUC values greater than 0.5 reflect above chance-level accuracy, and an AUC of 1.00 indicates perfect accuracy [20]. Dichotomous analyses, relating DPS and DISC disorder used chi-square analysis. Analyses then used logistic regression to predict DISC diagnostic disorders from their DPS counterparts. We used a level of significance of p<0.05. All the statistical analyses were two-tailed and performed using the Statistical Package for Social Sciences (SPSS), version 17.
RESULTS
The total sample consisted of 61 Caucasian subjects (33 [54%] males and 28 [46%] females) out of 94 eligible subjects. There was a total 33 dropouts (36%, 20 [60%] males and 11 [40%] females) from the original eligible sample due to refusal or contact failure. The non-enrolled group was similar to the enrolled group in terms of gender (p=0.50) and age (all subjects were obtained from the same age groups). The enrolled subjects were 100% computable for the DPS, 93% for the DISC-IV, and 100% for the Peds QoL. The estimated mean time for completing the DPS was approximately 20 minutes. Clinical and demographic characteristics are shown in Table 1.
Table 2 shows the mean DPS scores; the mean DPS total score was 5.5 (range = 0 - 36). No significant differences in DPS scores were seen in males versus females [6.4 (6.7) vs. 4.4 (3.7), p=0.72]. The DPS showed neither a ceiling effect (6.4 %) nor a floor effect (1.3 %) for the total scores. A skewness value of the total DPS-4.32 score was 2.97, higher than the standard values, indicating non-normal distribution of the scores. The Internal Consistency was high for the scale as a whole (α=0.92), and relatively low-moderate for the different subscales, ranging from 0.45 to 0.84, The item-corrected total correlation was weak-moderate, ranging from rs=0.005 to 0.70, as well as the total scores of the different DPS subscales with each other, ranging from rs=0.05-0.34.
With regards to the scale precision, the SEM was adequate 1.54 (95% CI upper limit of 4.64). The factor analysis identified 23 factors that explained 41% of the variance in the DPS scores. Factor I grouped 47 items out of 98 (51%), factor II grouped 14 items (15%), factor III grouped 12 items (13%), factor IV grouped 10 items (4%), and the remaining 15 items (16%) by others. Instead for cluster analysis, the factorial analysis identified 4 factors that explained 42% of the variance. Factor I grouped agoraphobia, panic attack, generalized anxiety disorder, oppositional defiant disorder, conduct disorder, substance abuse and impairment subscales; factor II grouped specific phobia, ADHD and depression subscales; factor III grouped OCD, mania and eating disorder subscales; and factor IV grouped social phobia and anxiety separation. With regards to convergent validity, the DPS showed moderate correlation with the Peds QoL total scores, with rs scores ranging from -0.23 to -0.38. As far as known-groups validity was concerned, the adolescents aged 13 years with a DPS total score ≥ 9 compared to those with a DPS total score < 9 had worse HRQoL with a mean Peds QoL score of 61.5 (7.1) vs. 72.9 (7.2), p=0.003]. Instead, a trend toward significance in HRQoL impairment was found in children aged 9 years (DPS total score ≥ 9 vs. < 9, with a mean Peds QoL score of 65.4 (6.8) vs. 72.9 (9.5), p=0.08 (Table 3).
Fifty six (92%) subjects screened positively for at least one DPS symptom, and co-occurrence of any DPS symptom was high (41 subjects [71%]). However, the overall prevalence of any possible psychiatric disorders (based on the DPS with a total cut-off score ≥9) was 5.3% (10 subjects), and 6.3% (12 subjects) with the DISC-IV. Based on the DISC-IV, the most frequent disorder was specific phobia (6 subjects [10%], followed by OCD (3 subjects [5%]), ADHD (2 subjects [3%]), and anxiety separation, conduct disorder, alcohol abuse in 1 subject [2%], respectively. Likewise, among those patients who fulfilled criteria for a possible psychiatric diagnosis based on the DPS cut-off scores (see Table 2), the most frequent possible diagnosis was specific phobia (5 subjects [8%]), followed by ADHD (4 subjects [6%]), and the remaining in less than 5% of the subjects. Overall, the DPS showed a low sensitivity, especially in OCD and specific phobia (41% for any DPS diagnosis, and 33%, for OCD and specific phobia, respectively), but with excellent specificity and NPV. The overall DPS AUC was 0.72 (95% CI: 0.57-0.87) against DISC IV corresponding diagnosis (Fig. 1, Table 4).
DISCUSSION
The DPS was developed as a mental health screen for specific disorders in young populations. The present study, which follows the principals and methods of the Classical Test Theory [18,19], shows the DPS applicability to the Spanish population, demonstrating their cross-cultural applicability. For the DPS, the parameters that measured the quality and acceptability of the data were basically satisfactory, and were situated within the standard values except for skewness. Although the internal consistency, measured by the Cronbach α, was excellent for the whole scale, it was moderate for the different subscales α values. A previous study has shown higher α values when a cluster diagnosis was used instead of single subscales assessing individual
diagnosis [11]. A possible explanation is that from a statistical point of view, α values are related to the number of the items included, and therefore subscales with fewer items may have lower α values. Likewise the item-total correlation was low-moderate, and the correlation among the different subscales were also weak-moderate. This could be explained by the fact that the different subscales were designed to assess unrelated symptoms (e.g., ADHD is not closely related specific phobia). In this study, the DPS has an adequate SEM value, indicating that overall, the DPS shows a suitable inter-relationship between the scale items at a point in time, a lack of random error and a possible high precision of the scale.
The exploratory factor analysis indicated a 23 factor structure, which explained only 41% of the variance in the DPS scores. The majority of the items of the DPS were grouped into 4 factors, as well as the diagnostic cluster analysis performed with the subscales. Although the present study identified data reduction analysis generally comparable to those of prior investigations based on DISC disorders [9,21], some differences in terms of the included items, understandable in the context of measurement and sample variations, are apparent, and require further investigations.
In our study, we found a lower prevalence of psychiatric diagnosis in the young population compared to other studies using the DPS [22]. Since our primary objective was the validation of the screening procedure, this prevalence estimate should be interpreted with caution, particularly as the sample included in this study was small. Low prevalence base rates of the disorders in the sample, allowed us to positively diagnose only 6 disorders (OCD, ADHD, specific phobia, anxiety separation, conduct disorder and alcohol dependence) based on the DISC-IV. Because, based on the literature, prevalence estimates for OCD and anxiety paediatric disorders are relatively high (anywhere from 1% to 4% for OCD, and about 6% for phobia) [23,24], in our study DPS seems to be most valuable as a screening tool for certain psychiatric diseases such as phobia, and ADHD but not sensitive enough to screen for OCD and anxiety. With regards to the predictive validity, the overall AUC of the DPS and its different subscales against the DISC-IV was 0.72, similar to a previous study where the DPS was used as a screening measure for incarcerated youths [8], and a clinical and community sample [10]. However, methodological issues such as the use of individual psychiatric diagnosis versus cluster diagnosis, and the different characteristics of the samples make comparisons and extrapolation to the overall child community or clinical population difficult.
In terms of cross-cultural validated screening psychiatric questionnaires, one of the most widely used is the Strengths and Difficulties Questionnaire that has shown excellent psychometric attributes both in community and clinical samples [25,26]. However, future studies are needed to address its diagnostic algorithms and its relationship with DSM diagnosis in more detail. Another widely used screening questionnaire is the Child Behaviour Checklist DSM oriented scale [5]. A prior study has shown similar screening efficiency compared to the DISC-IV [27]. However, taking into consideration the shorter length of the DPS, and the possibility that it may be administered as a computerized self-report scale, the DPS can offer several advantages for epidemiological research.
DPS scores were able to differentiate groups of patients based on HRQoL impairment. Therefore, in agreement with other studies, we have noticed a decrease in HRQoL among children and adolescents with mental disorders [28]. Further studies are needed with larger samples to predict HRQoL by using psychiatric screening assessments in community samples.
Our study has several limitations, such as the relatively small sample of patients and likely selection bias, the high dropout rate, the lack of a clinical sample, the possibility of recall bias, and the lack of knowledge regarding longitudinal psychometric properties. Likewise, a number of studies suggest that child and parent reports of symptoms can show poor agreement depending on the child's age and the type of disorder assessed [13], and therefore for the older group we only used self-administered interviews, different results could be obtained if the DPS and DISC-IV data were obtained by interviewers. In one interesting article by Steenhuis et al., parents-self administered interview of the ADHD section of the DISC-IV (Internet version) was compared to the interviewer administration of the same section [29]. According to this preliminary study, highly comparable diagnostic outcomes between self-administration through Internet of a single DISC-IV module and interviewer administration were found. Further research studies are required to confirm the reliability of self-administration and age-frame of the DISC-IV and DPS. On the other hand, the use of DPS cut-off scores obtained from the US adolescent population can also be considered as controversial, and require a larger study to confirm their utility in the Spanish young population. Nevertheless, we highlight, in agreement with other studies, the difficulty of performing epidemiological surveys in school samples with high dropout rate [30]. However, taking into account that some of the former data have been provided in previous publications [8,10], we believe that this study has the advantage of illustrating the DPS its psychometric attributes better and confirming the cross-cultural validity and feasibility as a screening tool. We have validated the DPS against the DISC IV, rather than against a clinical evaluation. Although it was established decades ago for adults, structured interviews are much more reliable than clinical judgment and have high validity vis-à-vis clinicians [31]. In this regard, the present finding of DPS/DISC IV concordance should be expanded by comparisons between DPS and clinical interviews.
In conclusion, the DPS seems to be an acceptable, consistent, cross-culturally valid, precise, and potentially responsive scale. It seems to be promising for mental health screening instrument for DSM-IV diagnosis, especially for certain diagnosis such as phobia and ADHD in a school community sample, but require further studies using larger samples. However, there is no doubt that the DPS has to be used with caution, and must be accompanied by an appropriate clinical assessment (other structured interviews or assessments by clinical experts) in order to provide a formal diagnosis of psychiatric illness.
FINANCIAL DISCLOSURE
SACYL. Biomedicine project GRS 157-A.
Health Research Grant PI 070846.
European General Development Co-funding.
ACKNOWLEDGEMENTS
We would like to thank Christopher P Lucas for his support.