Youth Risk/Need Assessment: An Overview of Issues and Practices

4. Research Findings

4. Research Findings

4.1 Methodological limitations

There are several different types of risk/need assessment tools currently being used in Canadian jurisdictions (see Appendices 1 and 2). Only a few tools (LSI-R, YLS/CMI, Level of Service-Ontario Revised [LSI-OR]) are the subject of published peer-reviewed research, which reports on reliability and validity (mainly predictive validity). We cannot comment extensively on those tools (Offender Risk Assessment and Management System [ORAMS], Youth Management Assessment [YMA], Youth Community Risk/Need Assessment [YCRNA], YLS/CMI Screening Version) for which published empirical research is not available. Most of the research purportedly validating these tools has been conducted by government or contract researchers and is not readily available nor are datasets for testing these tools accessible.

The following identifies the most common methodological issues raised in the interviews and outlined in academic literature on risk/need assessment tools for young offenders. This review focuses exclusively on LSI-based tools, and primarily on the YLS/CMI and Youthful Offender - Level of Service Inventory (YO-LSI). The Risk/Need Assessment - Case Management Review (RNA) and the Offender Risk Assessment and Management System - Primary Risk Assessment (ORAMS-PRA) for young offenders are adaptations of the YLS/CMI and YO-LSI respectively. Eight of the 12 provinces and territories participating in this study have adopted some version of an LSI-based tool.

A dominant theme to emerge from the interviews and academic literature is the need for further research studies examining the validity and reliability of risk/need instruments specifically with young offender populations. Youth workers, policy contacts, and researchers alike identified the need for more extensive research on these tools. For the most part, these instruments have been adopted without proper validation and reliability studies. The implication is that jurisdictions are placing confidence in and making decisions about youth on the basis of tools that have not been validated for use in their regions.

Surprisingly little research has been conducted on risk/need instruments with young offender populations. Many of the tools were initially developed for and tested on adult offenders and then subsequently adopted and/or adapted for use with young offenders with little or no testing. At the time of this study, no research with a sufficient young offender sample was conducted on the LSI-OR or the ORAMS.[12] The YLS/CMI is one tool where studies have reported significant constructive validity (i.e., how well the tool classifies risk) and predictive validity (i.e., whether the tool predicts risk accurately) with a young offender population aged 12 to 15 (Costigan 1999; Costigan and Rawana 1999; Jung and Rawana 1999; Hoge and Andrews 1996; Jung 1996; Lindsey 2000; Poluchowicz, Jung, and Rawana 2000; Rowe 2002; Schmidt, Hoge, and Robertson 2002). But, here too, the studies are limited. As one of the authors of the YLS/CMI, Robert D. Hoge (2002), acknowledges, "normative and psychometric data for the YLS/CMI are preliminary at the moment" (p. 390). In a study comparing three risk instruments, including the YLS/CMI, Hoge reported that "these measures have not been fully developed as actuarial measures, and it is not possible at present to specify the levels of decision accuracy they might produce. However, the available results are promising" (p. 392-3). Given time limitations and difficulty in locating many of these studies (many of the findings are reported only in conference presentations, unpublished theses, or in internal ministry reports that are not readily available), an in-depth analysis of this research was not possible.

We conducted an international literature search, which revealed few published peer reviewed academic studies of youth risk/need assessments tools. Many of the existing reports and studies are produced by governments or contract researchers - these were not accessible within the timeframe of the study. Reports or studies relevant to the standardized tools used in Canadian youth jurisdictions (i.e., YLS/CMI and LSI-R) were obtained and are reviewed below.

It should be stated that a potential problem with LSI-based research is that most of the research has been conducted by those working within the justice system or by the authors of the tools and/or their immediate students. This does raise the potential for research to be used to validate such instruments. It would be preferable for future studies to be conducted by those without a vested interest in the promotion of these tools.

On the basis of the interviews conducted, one of the apparent limitations of LSI-based tools concerns the specific sample used to develop and test LSI assessments. All studies on the YLS/CMI, YO-LSI, and RNA were conducted in jurisdictions within Ontario, specifically in South-Central and Northern Ontario. When using these tools outside these jurisdictions an assumption is made that the specific Ontario samples are representative of youth across the country. While this may be the case, if it is not, the validity of the tool may be compromised. In other words, the tool may prove to have differential validity in different jurisdictions. Few regions have evaluated, or are in the process of evaluating these tools for specific use in their jurisdictions. While this is partially due to lack of adequate resources, it also reflects the degree to which regions have placed confidence in an instrument without proper validation.

A similar issue arises in relation to the number of risk levels (low, moderate, high, very high) and the cut-off scores (e.g., a total risk score less than 9 or between 9 and 22 is considered low and moderate respectively). These were all established on the basis of results with young offenders in Ontario. The tools do allow cut-off levels to be adjusted and, indeed, the authors of the tool recommend that each region develop their own normative data to determine whether the cut-off levels or scores in the guide need adjusting for specific use in their jurisdiction. Apart from Ontario, only one province, Manitoba, has attempted to "norm" the tool (i.e., develop regional cut-off scores). Potential regional differences could reduce the predictive validity of the tool if the cut-off scores are not normed for each region adopting a risk/need tool. Moreover, the validity of the risk categories, specifically the differentiation between high-and very high-risk levels requires further exploration. In general, few young offenders fall within these categories and studies have not been able to conclusively establish the validity of these categorical distinctions. In fact, some regions have chosen to collapse the two categories. The implication of these practices on the overall validity of the tool needs to be explored.

The psychometric validity of the items used in some of the risk/need scales requires careful attention. This refers to whether the individual items in each scale (as opposed to the instrument as a whole) are statistically valid and reliable predictors of risk/need (i.e., can they differentiate between recidivists and non-recidivists). Studies have not examined the extent to which some of the items may potentially target disadvantages experienced by minority groups, or reflect moral and social values that are rejected by certain segments of a youth population. These concerns arise specifically in relation to the YO-LSI included in the ORAMS-PRA for young offenders. The YO-LSI consists of 82 items, some of which include receipt of social assistance, psychiatric history of parents and siblings, sexual experience (e.g., promiscuity, use of birth control), has children, lives in high crime area, has tattoos, and intellectual disorder. Some of these items may be characteristic of recidivists, but they may not impact on criminogenic risk/need.

It is important to understand how the items in the tools impact on the final risk score. The actual correlation between each item and recidivism is usually quite small, such that the predictive accuracy of any single item is relatively weak (Ministry of the Solicitor General and Correctional Services [MSGCS] 1995). However, the addition of a number of risk/need items, however loosely correlated to recidivism, should provide greater overall predictive accuracy. Hence, the specific number of items can vary considerably from scale to scale since eliminating one item should not significantly impact on the overall risk score. One of the stated strengths of the LSI approach is precisely its flexibility and the fact that it is amenable to further development and refinement. However, each of the items (as opposed to the tool itself) should be tested to ascertain whether they are theoretically relevant and statistically significant predictors of recidivism. According to the developers of the tool, only the YLS/CMI and the LSI-OR are theoretically based and directly concerned with the measurement of criminogenic risk/need factors (Andrews and Bonta 1998; Andrews, Robinson and Hoge 1984). Additional items may be important for treatment considerations but are not adequate predictors of recidivism and, consequently, should not be used to establish the overall risk score. Identifying the central predictors of criminogenic risk/need is central to risk/need assessment tools and has a direct bearing on the type and quality of interventions.

This concern was raised in a number of interviews, particularly with those using the YO-LSI to assess Aboriginal youth. Some of the items, specifically those relating to parental relations, socio-economic status, accommodations, sexuality, having children at a young age, and leisure activities, among others, may not be sensitive to the cultural distinctiveness of Aboriginal groups. In light of such concerns, some of the participants indicated that they do not record information on sensitive issues. This exclusion of selected items (and, potentially, their inclusion as well) may result in differential interpretation of risk/need for Aboriginal youth and, consequently, impact on the effectiveness of treatment.

Items in the YLS/CMI may also be poor predictors of recidivism for certain age cohorts. The YLS/CMI was developed and tested on Phase I young offenders (aged 12 to 15 at the time of the offence). Preliminary findings suggest that perhaps items relating to family circumstances and parenting may be less relevant in determining criminal activity for older youth, specifically those between the ages of 16 and 17 (Hoge, Andrews and Leschied 1996).

The following summarizes some of the key methodological concerns emanating from this section.

  1. Adequate reliability and psychometric and predictive validity have not been demonstrated on young offender populations across the country. Some research in Ontario does exist but it is preliminary in nature and insufficient to validate the use of these tools on a wide range of young offenders. The psychometric validity of the items in the tools has not been sufficiently researched. Only one study, a Masters thesis, was found to test the psychometric validity of the tools with a young offender population (Jung 1996). This raises concerns over whether some items may potentially target disadvantages experienced by minority groups or reflect moral and social values that are rejected by certain segments of a youth population.
  2. Tools developed in Ontario are being used in other jurisdictions across the country without proper validation studies. One needs to question the use of convenience samples to establish validity that are then used to generalize to populations that may differ substantially. Further issues pertaining to validity include the absence of research examining external or construct validity (Zimmerman et al. 2001).
  3. Although the tools may, in the future, demonstrate adequate predictive validity, they DO NOT measure dangerousness, nor can they identify the severity of an offence. A common concern identified by respondents is the fear that a high-risk score may be interpreted by some as an indication that a youth poses a greater risk to society, the institution, or to oneself. When used at the pre-sentence stage, the danger exists that the courts may assume that a high-risk youth poses a greater danger to society and sentence accordingly. Risk scores merely identify who is more likelyto re-offend. They do not differentiate between type of recidivism, that is, they cannot differentiate between who will breach a probation order (e.g., not appear for a meeting) versus who will commit an assault. In fact, the tools may prove to be better predictors of low-level criminal behaviour like failure to comply with a programme. The ability of the tools to accurately predict recidivism for all types of re-offences has yet to be considered.
  4. The tools CAN NOT identify who with certainty will actually re-offend. They merely predict who is more likelyto re-offend.
  5. Some jurisdictions are using the tools at the pre-sentencing stage, yet none of the tools were designed for use with young offenders at this point. The only tools tested on a young offender population, the YLS/CMI, were developed as a case management tool.
  6. No studies have examined the validity or reliability of the tools for case management purposes. No research has explored to what degree, or how valid the tools are in informing and determining case management decisions.
  7. Different concerns about these tools emerge when they are administered in a non-research setting. Practitioners administering these tools ought to be well trained on how to administer the tools. Presently, many tools include vague criteria such as "could make better use of time," "Non-rewarding parental relations," "inconsistent parenting," "peer interactions," "supportive of crime," "poor social skills," "underachievement," "inadequate supervision," "problems with teachers," "no personal interests," "inadequate guilty feeling." It is difficult to determine how consistently these criteria are interpreted. These criteria can involve substantial amount of speculation and morally laden subjective assessments. The assessor's choice of informants and interpretations of the authenticity of their claims is not transparent. Audits are not consistently performed and there are wide variations on the type and quality of training assessors receive.

4.2 Accountability and defensibility

Practitioners consistently claimed that the advent of risk assessment tools has resulted in more defensible and accountable practices of assessment. This development was seen as positive and progressive. Practitioners maintained that decisions made using structured risk assessment tools were more defensible than gut feelings,which were seen as the basis of discretionary judgments. One Crown Attorney noted that recommendations for pre-trial detention and sentencing were more persuasive and defensible if you could show that the recommendations were based on a systematic review and analysis of the areas of risk and need shown by research to be related to recidivism. Further, it was believed that not only were these tools useful in presenting evidence to the court but that they also enhance the defensibility of pre-trial and sentencing decisions to the general public. Many suggested the tools made the reasoning behind the case plan more transparent because these tools used objective criteria that was empirically based and that were uniformly applied. In short, it is seen as justifying "what is done to the youth".

Interestingly, there was variation in opinions about the usefulness of risk and need assessments. Some respondents felt that risk and need assessments were most useful for new youth workers and those with less experience who had not yet developed a strong intuitive sense. Some suggested risk assessments were a "matter of common sense," but that common sense is not persuasive in court or in terms of the public perceptions of just and rational decision-making. Clearly, one of the appeals of risk assessments is that they ensure that a decision is defensible. As one respondent noted, "They back the [Probation Officer] up if something goes wrong - you can demonstrate that you used a standard approach that is empirically based".

4.3 Subjective/moral criteria

While considerably more structured than clinical judgment, these tools do not, in practice, substitute for professional discretion, nor are they objectiveor apolitical,as some postulate. Professional judgment is a key component of risk assessment (Andrews and Bonta 1998). User manuals for both the YLS/CMI and the LSI-R encourage the exercise of professional discretion and acknowledge that the completion of these assessment forms[13] requires "subjective judgments on the part of the professional who completes them" (Hoge and Andrews 2002:5). While scales are useful in structuring decision-making and ensuring that practitioners are looking at all relevant risk and need factors, managerial decisions about particular offenders are still heavily influenced by personal judgment, particularly in terms of how practitioners define and rate intangible needs. Current and past research and interviews with practitioners show considerable variability in how assessment criteria are interpreted (Hannah-Moffat and Shaw 2001).

Risk/need assessments provide decision makers with risk/need scores and categorical ratings from high to low depending on the tool used. One of the often disregarded restrictions of risk/need assessments is their reliance on a multitude of subjective/moral judgments about the youth's character and his/her lifestyle. Several risk tools (e.g., YSL/CMI, ORAMS - PRA, LSI-R, YCRNA) currently used in Canadian jurisdictions to assess youth include criteria that require practitioners to evaluate various aspects of a youth's life and to make a series of moral determinations about his/her hygiene, leisure habits, friends, parents, study habits, and consumptive and sexual behaviours. For instance, the ORAMS (section #7 Primary Risk Assessment),[14] which is used to prepare court reports or for sentenced youth upon admission to the correctional system, scores the following items:

Has drunk alcohol, leisure time spent mostly in unconstructive activities (e.g., hanging around arcades, partying, doing crime), sexually active, has sex with more than one partner (i.e., is promiscuous), doesn't take precautions for birth control and safe sex, has a child or children, past victimizations, has tattoos/self-inflicted scars or burns, was uncooperative/belligerent during the assessment.

Comparable difficulties exist with other risk/need tools outlined in this report. The inclusion of these criteria raises two concerns. First is the relevance of these criteria to the management of the youth (i.e., psychometric properties discussed above). Clearly, some of the above-listed items will guide practitioners in developing intervention plans for individual youth but other criteria are of questionable empirical and legal relevance. Some have suggested that the criteria in some tools, "dangerously convolutes factual with interpretive information, and is therefore potentially very misleading if a person who completes it is not properly trained" (Cole and Angus 2003:17).[15] The use of such extralegal criteria to formulate recommendations about sentencing and to establish who is suitable for diversion is undoubtedly problematic and may in fact discriminate against large numbers of youth. Such judgments embedded in risk/need assessment tools often escape careful scrutiny by practitioners, in particular, judges and defence counsel of the youth.

Second, even if these criteria were established as empirically or legally relevant, there is a lack of transparency in how the practitioners determine the presence or absence and relevance of particular items contained in assessments. Some risk/need assessment tools (e.g., YLS/CMI) provide space for comments that contextualise ratings. Considerable variability remains in how practitioners record this information and in how they use and interpret the information they collected from the youth and collateral contacts. There are some concerns about gendered and racialized interpretations of risk and need criteria and the importance of the age of the youth (and developmental stage) in making determinations about the appropriatenessof particular behaviours. A few respondents felt that these tools inappropriately label youth and that these labels are punitive and stick with the youth their entire time in the system.

  • [12] Lina Girard's Ph.D. Dissertation on the LSI-OR did include information on young offenders, but the sample size (n=31) was too small to substantiate the validity of the tool.
  • [13] Tools like the YLS/CMI contain separate sections (section IV) for practitioners to document their own assessment. Assessors are required to check one of four boxes (low, moderate, high, very high), indicating their professional assessment of the offender's risk. If this assessment differs from that which the tool ascribes, the assessor is required to document the reasons for the discrepancy.
  • [14] The ORAMS is used in NWT and Manitoba.
  • [15] These authors further indicate that the courts have had difficulty defining and applying the term psychopathy because the term is fraught with descriptive and conceptual difficulties (Cole and Angus 2003:17).
Date modified: