By: Genie Jackson, PhD, Research Team Lead at equivant Pretrial
The use of validated risk assessments to achieve optimal, measurable reductions in pretrial failure rates is key. Validation helps to ensure the instruments your agency utilizes are objective and standardized based on pretrial failure. equivant Pretrial recommends that agencies validate their assessments no less frequently than once every three years to ensure that the instrument is identifying risk accurately in your agency’s population and to identify any disparate effects or biases in the tool based on gender, race, or ethnicity.
Below is the validation process used by the equivant Pretrial Research team:
- Information Gathering: We first meet with the agency to determine analysis type, stakeholders, and timeline. Measurement outcome data may vary, based on local definition, and will most likely need to be extracted from multiple sources.
- Power Analysis: Next, we determine the sample size required to detect differences in failure rates across risk levels based on expected failure rates. This applies not just to the entire sample, but to the subgroups to be studied, such as gender-defined subgroups..
- Data Cleaning: After obtaining the data from the various sources, the information is merged and arranged into timelines for each individual in the sample, demonstrating date of screening, failure events, disposition date, and other relevant events.
- Assessment Result Analysis: Then, we look at the assessment results for the individuals in each group. The basic question is: did the assessed risk correlate with their behavior regarding court appearances and arrest for new criminal offenses? Other considerations include release conditions, release rates, release timeframes, all of which may be aggregated by gender, race, or ethnicity, ZIP code, and offense type.
- Industry-standard Percentages Comparison: As a rule of thumb, a risk instrument that performs well should have high risk individuals failing at a rate at least ten percentage points higher than the base rate. Sometimes this is not possible, as in situations where the base rate is very low, but there should still be evidence that the group classified as high risk has a higher failure rate than the overall failure rate, and that the group classified as low risk has a lower failure rate than the overall failure rate. If this is not the case, it is time to dig into the why. Would changing the risk level cutoffs in the scale improve the situation? Is the problem with one of the items or the way it is being administered? Answering these questions and taking steps to address the identified problem can resolve the issue in most cases.
- Formal Report and Recommendations: Finally, we take what we have learned from analyzing the data and prepare a formal report, including recommendations. These recommendations may include cut point adjustments, training tips, or improved ToolTip wording. After the agency has reviewed the final report, we make a formal presentation highlighting the main results of the study outcomes.
Many states and jurisdictions mandate externally validated pretrial assessments. In addition, criminal justice agencies are driven by the mission of increased appearance rates and pretrial success by providing due process while assuring community safety. To discuss more about your state’s legislative requirements for the use and validation of pretrial risk assessments, click here.
