FMCSA proposes substantial revisions to its Safety Measurement System (SMS)
Updated: Feb 17, 2023
The Federal Motor Carrier Safety Administration (FMCSA) is proposing fairly substantial changes to its Safety Measurement System (SMS) also known as CSA, which the agency uses to prioritize motor carriers for enforcement. According to a notice published in the Federal Register on February 15, 2023, "FMCSA continues its commitment to continuously improving SMS to identify motor carriers that present the highest crash risk through a transparent and effective system." We'll break it all down in this article.
Background
The FMCSA first implemented SMS in 2010, as part of the broader Compliance, Safety, Accountability (CSA) program. As we've discussed in other articles, the agency and its state partners rely on SMS to identify high risk motor carriers and prioritize them for enforcement. Through SMS, carriers are scored in seven categories known as BASICs, based on violations discovered during roadside inspections and in compliance reviews. Carriers receive percentile scores in each of the BASICs, indicating how they are performing relative to their peers. This have come to be known as CSA scores.
In 2017, the U.S. Congress demanded FMCSA commission an independent study of SMS in light of heavy criticism of the system over the years. The agency engaged the National Academy of Sciences (NAS) to conduct that study, which it completed that year. NAS ultimately concluded that SMS, in its current form, "is structured in a reasonable way and its method of identifying motor carriers for alert status is defensible.” However, it went on to make a handful or recommendations to the FMCSA, including that it consider adopting a better statistical model as the backbone of the system. The recommended model, used in many other industries, is known as the Item Response Theory or IRT.
FMCSA declines to adopt an IRT model
In proposing changes to its SMS, the FMCSA is explicitly declining to adopt an IRT model as an alternative to the existing SMS methodology. According to its notice, the Agency conducted a "full review" of an IRT model and how to move forward with it as a prioritization tool. It established a committee of experts to design and test the model using inspection data from its MCMIS database. Ultimately, the agency concluded the IRT model had many "limitations and practical challenges." "As a result, FMCSA has concluded that IRT modeling does not perform well for the Agency’s use in identifying motor carriers for safety interventions, and therefore, does not improve overall safety.”
FMCSA identified five main concerns with an IRT model:
IRT is heavily biased towards identifying smaller carriers that have few inspections with violations and limited on-road exposure to crash risk. When the safety event groups and data sufficiency standards used in SMS were applied to the IRT model, IRT produced similar results to SMS.
IRT does not use Vehicle Miles Traveled (VMT) or Power Units (PUs) to adjust for differences in on-road exposure in the Unsafe Driving BASIC. As a result, IRT identified carriers with much lower crash rates in that BASIC compared to SMS.
IRT modeling is not readily understandable by most stakeholders or the public. IRT’s inherent complexity makes it challenging for the industry and public to replicate and interpret results. While SMS results can be reproduced and explained using simple math, IRT requires an advanced understanding of statistical modeling and analysis.
A motor carrier cannot independently compute its IRT results. IRT results can be computed only for the entire carrier population. A carrier would not be able to identify how specific violations or areas of regulatory noncompliance impacted its prioritization status or how it could improve its status.
IRT’s runtime is incompatible with FMCSA’s operational needs. The IRT model takes four weeks to run as compared to two days for SMS. The long runtime would make it difficult to make even minor changes to the system.
So, rather than proceed with a complete overhaul of SMS, the agency will continue to improve the existing system.
Proposed revisions to SMS
The agency is proposing several changes to the existing SMS methodology. These can be broken down as follows:
Reorganized and updated safety categories, including new segmentation, to replace current BASICs
Consolidated violations
Simplified violation severity weights
Proportionate percentiles instead of safety event groups
Improved Intervention Thresholds
A greater focus on recent violations
An updated Utilization Factor.
The agency has included a detailed overview of these changes in a document titled “Foundational Document” in the docket with its most recent notice. It has also published a preview site, which allows carriers to see how the changes will look once they take effect and to preview how their own existing CSA scores might change once that happens.
Reorganization of BASICs into "Safety Categories"
In terms of "reorganizing" the current BASICs, the FMCSA is proposing to combine the existing drug/alcohol BASIC into the Unsafe Driving category and then splitting out the Vehicle Maintenance BASIC into two separate categories that will be called Vehicle Maintenance and Vehicle Maintenance: Driver Observed. BASICs will now be called "safety categories."
According to the notice, because drug/alcohol violations discovered during roadside inspections are few and far between, it makes sense to simply include the violations in the Unsafe Driving category as opposed to splitting them out on their own. Similarly, all "driving while out-of-service" violations (aka "jumping" OOS orders), which are currently spread across the various BASICs to which the underlying OOS violation relates, will now be included under the Unsafe Driving safety category.
The agency explains that the Vehicle Maintenance category is the currently the largest BASIC. Breaking it into two separate categories would, according to the agency, help carriers and law enforcement better understand the nature of their vehicle maintenance issues and target them for correction. More specifically, the new Driver Observed category would encompass maintenance violations that could have been identified by the driver in a pre- or post-trip inspection. All other maintenance violations (i.e., those normally discovered by a mechanic during an annual inspection) would fall into the general Vehicle Maintenance category.
Consolidating & Reorganizing Violations
The agency is also proposing to consolidate the existing universe of violations that filter into the SMS system. According to the notice, SMS currently accounts for 959 separate violations. The revised system will group similar violations together with one another, resulting in 116 violation "groups." For example, the existing 73 separate hours-of-service violations will be combined into a total of 9 violation groups.
A complete list of all SMS violations and the new categories into which they will fall is available here.
While all 959 violations can and will still be cited by law enforcement during roadside inspections, SMS will group similar violations for purposes of prioritizing carriers for enforcement. Put differently, "If a motor carrier receives more than one of the violations in a violation group during a single inspection, the new methodology would treat that set of violations as a single violation when calculating the carrier’s measure in that safety category."
The agency rationalizes this change on its preview site: "There are often multiple ways to cite a carrier for the same underlying safety issue. FMCSA’s analysis indicated that, for prioritization purposes, determining whether a safety issue is identified is more important than determining how many ways it was documented. Grouping a motor carrier’s violations before analyzing their data would ensure that motor carriers are treated fairly by holding carriers with similar safety issues to the same standards, regardless of how those issues were documented. This would prevent the inconsistencies in safety category measures that occur when multiple violations are cited for the same underlying safety issue during one inspection."
Simplified Violation Severity Weights
Another significant revision to SMS will be a simplification of violation severity weights. Under the current system, violations are assigned a severity weight of 1-10. Many have criticized these weights over the years as overly subjective. To address this, the agency is proposing to assign each violation group a weight of either 1 or 2. Out of service violations and violations in the Unsafe Driving category that are so-called "disqualifying" will receive a weight of 2, and all other violations in the system will have a weight of 1.
According to the agency, "After conducting analysis on multiple approaches, FMCSA determined that assigning customized weights to all violations was not as important as noting that the violation occurred. This simplified approach identifies carriers with higher crash rates for prioritization and makes it clearer why a specific violation is weighted more heavily than others."
The agency's preview site includes the following example of how its violation grouping and severity weights will work.
Proportionate Percentiles
The agency is also proposing revisions to the way SMS handles safety event groups to try and remove sudden "jumps" in percentile scores based on changes to a carrier's safety event group. Under the new system, SMS would use safety event groups only to calculate benchmark median values of each grouping. Carriers' proportionate percentiles would then be calculated from a weighted average of percentiles based on those benchmarks. And once a benchmark is established, a carrier's percentile would be based solely on the carrier's own performance rather than the performance of its peers.
The agency describes it this way: This methodology has several benefits compared to the safety event group approach used in SMS:
Customized to a carrier’s exact number of events.
Ensures stable results for carriers by only allowing for gradual percentile changes from month to month when dropping or adding events.
Allows an individual carrier’s change in measure to have greater influence on their percentile. There will no longer be substantial percentile increases without a corresponding measure increase.
Changes to Intervention Thresholds
Lastly, FMCSA is proposing minor changes to some of its existing intervention thresholds, the cutoffs at which carriers are prioritized for enforcement. Thresholds for the Unsafe Driving, Crash Indicator, and Hours of Service safety categories will remain the same. In addition, the existing thresholds for the Vehicle Maintenance category will extent to the new Vehicle Maintenance: Driver Observed category. The Driver Fitness thresholds will increase from:
80% to 90% for general carriers.
65% to 75% for passenger carriers.
75% to 85% for HM carriers.
And the the HM Compliance thresholds will increase from 80% to 90% for all carrier types.
Additional Changes
The agency is also proposing a few more minor revisions to SMS, including the following:
A greater focus on more recent violations. Under the current system, carriers' scores are impacted by all violations incurred over the past 24 months. While the system will continue to consider all such violations, carriers who have incurred no new violations in a particular safety category in the past 12 months will no longer have a percentile score in that category.
Carrier segmentation. Currently, SMS segments carriers into those that primarily operate straight vehicles and those that primarily operate combination vehicles when calculating percentiles for the Unsafe Driving and Crash Indicator BASICs. The new methodology would retain this segmentation, and would also segment the following safety categories: Segment HM Compliance by Cargo Tank and Non-Cargo Tank carriers; Segment Driver Fitness by Straight and Combination carriers.
Updated Utilization Factor. Currently, SMS uses a "utilization factor" to help ensure that measures in the Unsafe Driving and Crash Indicator safety categories account for carriers’ different levels of exposure to inspections and crashes. In SMS, the Utilization Factor is applied to carriers that drive up to 200,000 VMT per average Power Unit (PU)—in other words, a carrier’s VMT per average number of vehicles they have on the road. The new methodology would extend the Utilization Factor to carriers that drive up to 250,000 VMT per average PU.
Conclusion
While the FMCSA's proposed revisions to SMS are not nearly as substantial as they would have been had the agency adopted an IRT model, they still have the potential of significantly impacting the way motor carriers are prioritized for enforcement. The agency has set up a preview of the new system, which carriers can access at this link: https://csa.fmcsa.dot.gov/prioritizationpreview/. According to its notice, the agency will hold a series of question and answer sessions about these changes over the coming months. The agency is also seeking public comments through May 16, 2023, which can be filed in Docket No. FMCSA-2022-0066.
If you have questions about how these changes may impact you, please feel free to contact us. We will continue to monitor the status of these changes and and further information from FMCSA on this topic. Be sure to subscribe to our newsletter for the latest updates on this and other regulatory topics.
About Trucksafe Consulting, LLC: Trucksafe Consulting is a full-service DOT regulatory compliance consulting and training service. We help carriers develop, implement, and improve their safety programs, through personalized services, industry-leading training, and a library of educational content. Trucksafe also hosts a monthly live show on its various social media channels called Trucksafe LIVE! to discuss hot-button issues impacting highway transportation. Trucksafe is owned and operated by Brandon Wiseman and Jerad Childress, transportation attorneys who have assisted some of the nation’s leading fleets to develop and maintain cutting-edge safety programs. You can learn more about Trucksafe online at www.trucksafe.com and by following Trucksafe on LinkedIn, Facebook, Twitter, and YouTube.