NHTSA "CODES" Study
Crash Outcome Data Evaluation System
Analysis by the AMA
From: Sean Maher - American Motorcyclist Association
Date: January 3rd, 1996
The Crash Outcome Data Evaluation System (CODES) Study: Comments and Observations from the American Motorcyclist Association.
The National Highway Traffic Safety Administration (NHTSA) recently released a Report to Congress, The Benefits of Safety Belts and Motorcycle Helmets: Based on Data from The Crash Outcome Data Evaluation System (CODES). NHTSA was provided with five million dollars to fulfill the requirement for this report contained in Section 1031 (b) of the Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA):
"(1) In General. -- The Secretary shall conduct a study or studies to determine the benefits of safety belt use and motorcycle helmet use for individuals involved in motor vehicle crashes and motorcycle crashes, collecting and analyzing data from regional trauma systems regarding differences in the following: The severity of injuries; acute, rehabilitative and long-term medical costs, including the sources of reimbursement and the extent to which these sources cover actual costs; government, employer, and other costs; and mortality and morbidity outcomes. The study shall cover a representative period after January 1, 1990."
This study extends beyond fatalities to include an analysis of non-fatal injuries and their related costs for treatment. In addition to reporting the average inpatient charges associated with the treatment of injuries, NHTSA provides 'effectiveness' measures as an overall assessment of how the use of a safety belt or helmet can reduce injuries or prevent death.
Voluntary helmet use advocates will undoubtedly be confronted with elements of this report as they preserve or work to restore voluntary helmet use within their respective states. It is important to be familiar with the full study, rather than the parcels of information that the press is repeating. Doing so reveals that NHTSA's best attempt to build a case for mandatory helmet laws, has fallen well short of the mark.
The following comments are offered for your consideration as you develop strategies to respond to CODES:
Perhaps one of the most important facts to remember when evaluating CODES is that it is not nationally representative. NHTSA states on page 14 of the CODES Report to Congress, "The reader should note that the overall results presented in this report reflect only the 7 CODES states and the case selection criteria for the safety belt and motorcycle helmet analyses. They are not intended to be nationally representative."
New York contributes nearly forty percent of the motorcycle rider data. It is likely that this over-representation increases the average inpatient charge estimates. Only six of the seven CODES states are used for the helmet analyses: Hawaii, Maine, Missouri, New York, Pennsylvania and Wisconsin. Utah participated in the safety belt study, however their data is excluded from the helmet study.
Of the states contributing data to the helmet study, New York's data represents nearly forty percent of the study sample. This likely results in considerable bias in the reported economic results. NHTSA concedes on page 14 of the CODES Report to Congress, "Because these methods of computing averages are strongly influenced by number of cases, data from larger states may disproportionately influence the overall weighted average."
To illustrate the limited applicability of this report and the effect that one state can have on the results consider the following:
The draft Report to Congress did not include data from Pennsylvania.
The draft Report to Congress indicates motorcyclists were less dependent (19 percent) upon public sources for medical costs than motor vehicle operators (21 percent).
The final Report to Congress includes Pennsylvania's data.
The final Report to Congress indicates motorcyclists are more dependent (23 percent) upon public sources for medical costs than motor vehicle operators (16 percent).
The addition of one state causes a significant shift in the outcome. This underscores the wide margin for error inherent in this report, reinforces the fact that states are unique and that transportation issues should be evaluated within that context.
The 'effectiveness' ratings for helmet use are inflated. NHTSA uses two methods to calculate 'effectiveness'; one for seat belts another for helmet use. In order to increase the sample size for the helmet portion, NHTSA did not control for contributing risk factors, as they do with the seat belt portion. This leads to inflated 'effectiveness' ratings for the helmet study.
NHTSA states on page 15 of the draft Report to Congress, "Estimates of 'effectiveness' without these factors incorporated were generated as part of the CODES analysis. These estimates were higher in most of the states, an indication of the influence of these risk factors on crash outcomes."
In our comments to the draft Report to Congress, the AMA notified NHTSA that their methodology inflates 'effectiveness' ratings for helmet use. NHTSA's response was to change the final Report to Congress on page 16 to read, "Estimates of 'effectiveness' without these factors incorporated were generated as part of the CODES analysis. These estimates of 'effectiveness' with and without risk factors were not significantly different in most of the states."
No change of methodology was used between the draft and final reports. However, NHTSA estimates changed from "higher in most of the states" to "not significantly different in most of the states." We can only assume this is an intentional misrepresentation of the facts.
NHTSA acknowledges the minimal effectiveness of helmets in preventing any type of injury, indicating that helmets prevent head injury, not all injuries. In a separate analysis that addresses only head injuries and more specifically, brain injuries, NHTSA found that helmets are effective in preventing brain injury. However, the same analysis indicates that the average inpatient charges for unhelmeted riders were less than those of helmeted riders.
NHTSA found that the average inpatient charge for a helmeted motorcyclist receiving a brain injury was $26,985, compared to $26,805 for an unhelmeted motorcyclist (see chart). NHTSA also found that the average inpatient charge for a helmeted motorcyclist not receiving a brain injury was $12,736, compared to $11,730 for an unhelmeted motorcyclist.
This same analysis found helmets to be 67 percent effective in preventing brain injuries. Although this estimate is impressive on its own, brain injuries represent the top of a pyramid in terms of motorcycle injury distribution. NHTSA has acknowledged the minimal effectiveness of helmets in preventing most injuries.
The Technical Report supplement to CODES shows inconsistencies for helmet effectiveness, which were not included in the Report to Congress. In the Technical Report, NHTSA calculates odds ratios in addition to 'effectiveness' ratings, as an alternative measure for effectiveness. Utah's odds ratios indicate a non-protective effect for helmets in preventing death. Pennsylvania's odds ratios are termed "non-significant" in the prevention of injury, as are Hawaii's. The odds ratios are not presented in the final Report to Congress.
NHTSA attributes these inconsistencies to the small sample size, different patterns of helmet use and the type of injury. However, the same sample, the same patterns of helmet use and the same types of injury are present in the estimates of 'effectiveness,' which are presented in the final Report to Congress.
In addition, NHTSA excludes Utah's data from the helmet portion of the study. NHTSA's rationale for this decision is because Utah did not include a code for not wearing a helmet on their traffic report. The code Utah uses is 'helmet use unknown.'
The decision to exclude Utah's data was made after Utah had been awarded a grant, based on its capabilities to participate in this study, and after Utah collected its data. Had the 'unknown' factor been a legitimate concern, it stands to reason that Utah would not have been selected as a CODES participant state.
The AMA recognizes helmets as an effective piece of safety equipment and encourages their use. However, the determination to use a helmet should be that of adults, free from governmental mandates. The results of the CODES study indicate that helmet use has no impact on the cost of an injury after it has occurred. In addition, this study shows that helmets are minimally effective in preventing any type of injury after an accident has occurred.
NHTSA was provided with five million taxpayer dollars to develop a compelling case for mandatory helmet use laws. Having generated a report with no national applicability, using questionable methodology for the helmet study and with mixed results for helmet use, NHTSA has fallen well short of the goal.
In their limited applicability, the draft and final Reports to Congress show that motorcyclists are no more dependent upon public sources for medical costs than motor vehicle operators. These reports show that helmets are minimally effective in preventing any type of injury from occurring, underscoring the importance of accident prevention. Furthermore, one analysis showed the inpatient charges for unhelmeted motorcyclists to be slightly more than helmeted motorcyclists. While another analysis, more specific to head and brain injury, showed the inpatient charges for unhelmeted motorcyclists to be less than helmeted motorcyclists.
It remains clear that the most effective way to mitigate the cost of motorcycle injuries is through awareness and education aimed toward accident prevention, a strategy widely recognized and accepted by the motorcycling community. The NHTSA, however, continues to embrace a failed strategy of imposing mandatory helmet use laws upon all motorcyclists, a strategy that has been repeatedly rejected by the motorcycling community.
For further information:
Copies of the CODES report to Congress on the Benefits of Safety Belts and Motorcycle Helmets and the CODES Technical Report may be obtained from the National Center for Statistics and Analysis, National Highway Traffic Safety Administrations, 400 7th Street, SW, Room 6125, Washington, DC 20590 or send a FAX request to 202-366-7078.
Copies of the draft CODES report and Technical Report can be obtained from Sean Maher at the American Motorcyclist Association, 33 Collegeview Rd., Westerville, OH 43081. Please send a 9x12, SASE w/$2.00 of postage.
If you have questions or comments, please call Sean Maher at the American Motorcyclist Association, (614) 891-2425.