Potential Issues with Criminal Justice Data

Summary: I wrote this document based on my experience validating a risk assessment instrument. These were some of the issues, rewritten for generalizability, that we encountered. Accessible, quality data is often a project bottleneck, and I’ve found these helpful to consider before working on a project with criminal justice data.

Comments: I wrote these notes in the context of evaluating a risk assessment instrument, so “risk factors” are independent variables and “outcomes” are dependent variables.

Potential Issues

• Data System Diversity – Risk factor information can require data from several systems, potentially including individual demographic data, pretrial systems data, criminal history data, arrest data, jail booking data, and court disposition data.
• Agency Diversity – An individual jurisdiction can also have multiple courts, jails, and other agencies that maintain the same type of data but stored in different systems.
• Data Encoding Diversity – These data are often encoded in different ways across different systems, requiring reconciliation.
• Insufficient Data Definition – Jurisdictions or agencies might not define or capture specific data required for risk factors or outcomes if it’s not required for existing systems.
  • e.g. Felony/Misdemeanor offense information is commonly captured, but violent/non-violent offense information is often not. An accurate lookup of whether an offense is violent might not exist.
• Definitional Diversity – Definitions of criminal history factors and severity may differ across jurisdictions.
• Intra-Agency System Changes – Agencies adopt new systems for the same data over time.
  • This requires linking records for an individual across time.
  • Legacy systems may not have every piece of requisite data for an accurate criminal history.
  • Jurisdictions have different practices for archiving historical data from legacy data systems. For example, full criminal history information may be unavailable.
• Individual Data Veracity – The consistency of the data for any one individual varies across all of these systems.
  • Reconciling various demographic descriptors (such as Hispanicity) across these systems has implications for determining predictive bias.
• Documentation Quality – Documentation about the data in different systems is often incomplete, out-of-date, or incorrect.
• Proprietary System Inconsistencies – Proprietary systems can store data in different formats and have different options for exporting data.
• Absence of New Data Storage – Scores from risk assessment instruments, as well as decisions made using them, aren’t stored in existing systems. This requires historical analysis (i.e. revalidation) based off risk factors where a real-time analysis with calculated assessment scores would be more precise.