Moreover, noted that even studies controlling for Barnum effects may be compromised by a halo rating response bias whereby differences in clinicians ratings between real and bogus reports are artifactually inflated by one obviously invalid element of the bogus narrative. To eliminate or reduce this confound, D. K. Snyder et al. argued for interweaving real and bogus interpretive elements within CBTI narratives when controlling for Barnum effects in customer satisfaction studies. Other than the study regarding the CBTI for the Marital Satisfaction Inventory, no other study could be found using this methodology.
Viewed as expert systems, CBTIs comprise automated procedures ¦ in which the knowledge, problem-solving strategies, and evaluative judgment of one or more experts are built into a computer program, enabling the program to assist in emulating the process that the expert follows in attacking a problem. Regarded from this perspective, the appropriate model for evaluating the validity of CBTIs is one that examines how well the computer-based narrative addresses relevant assessment questions in a manner similar to that of an expert clinician.
Provided a thorough description of procedures critical to the development and initial validation of computerized systems that model expert opinion, Among these are (a) developing an initial question list, (b) developing a preliminary statement library, (c) revision of the statement library by an expert panel, (d) assigning score ranges to statements, and (e) examining whether the resulting CBTI narratives provide the same (or better) conclusions as human experts could derive from the same data, Acknowledged that expert-opinion modeling comprises only an initial validation strategy for CBTIs.
Similarly, asserted that the expert-simulation strategy ¦ essentially maintains the status quo and produces improvements in assessment accuracy only to the degree that the emulated expert (or experts) improves. Consequently, CBTIs based on expert systems should ultimately be evaluated on the basis of their accuracy in predicting external criteria. (West 2005) External Criterion Studies External criterion studies evaluate the validity of CBTI systems by examining the congruence of statements or ratings derived from the computer-based narrative with independent observations or ratings of the test respondent.
And emphasized the importance of external criterion studies in CBTI validation but also acknowledged their paucity and methodological shortcomings. In the prototypic validation study adopting this approach, independent criterion ratings for respondents are provided by clinicians with extensive knowledge of the client but without information regarding test scores, or by expert raters based on exhaustive review of clinical records (although this latter approach has limitations of its own. Independent ratings of the test respondent are also derived from the computer-based narrative.
Various approaches can be used to analyze congruence including correlations, joint-contingency analysis, or Q-sort techniques. As with customer satisfaction studies, ratings can also be derived from bogus reports or from profiles of the stereotypic respondent as a control for evaluating the incremental validity of ratings obtained from bona fide narratives. Alternatively, ratings can be derived from multiple CBTI systems developed for the same instrument. Although for few instruments other than the MMPI does this option exist?
Scholar appropriately noted problems with conceptualizing clinicians ratings as the standard by which to evaluate CBTI systems accuracy. Clinician ratings may themselves lack intra- and interrater reliability and may have limited validity resulting from (a) incomplete or inaccurate understanding of the test respondent, (b) failure to recognize base rates, (c) restricted expertise in domains to be rated, (d) lack of familiarity with the CBTI system, or (e) other biases in clinical judgment (see comments on improving clinical judgment in the following section).
Emphasized a potential fatal flaw in external criterion studies in which ratings of the respondent on non test criteria are contaminated by raters have been influenced (either directly or indirectly) by the individuals test scores. Sources of such criterion contamination are both numerous and subtle: With the exception of rigidly excluding initial test results from any treatment considerations, they are nearly impossible to eliminate.
For example, if a patient is assigned to a particular treatment regimen based in part on initial test findings, and a clinicians ratings of that patient are influenced by knowledge of base-rate phenomena characterizing individuals receiving that treatment, then clinical ratings of the patient will reflect an arti-factual congruence with ratings derived from the computer-based narrative, Offered useful recommendations for obtaining criterion measures that may circumvent or reduce criterion contamination.
Scholars delineated additional considerations potentially compromising external criterion studies. Similar to those for customer satisfaction and expert-emulation studies, these include failure to (a) obtain sufficiently large, representative samples of raters and test respondents, (b) use appropriate rating scales, (c) evaluate discriminate or incremental validity, (d) assess independent narrative elements produced by different decision rules, and (e) examine potential moderators of CBTI validity (eg, assessment context and respondent characteristics). (Singh 1998)