We used the five dimensions of the RE-AIM model to evaluate a smoke alarm (SA) installation program called Operation Installation (OI), which was implemented in 36 high risk census tracts in Dallas, TX, from 2001-2011. More than 20,000 SAs were installed in 8,134 houses through OI. The RE-AIM model showed that the program had a modest reach into the target population (26.5%), and a relatively high effectiveness (63%) at preventing deaths and injuries in program houses. Adoption and implementation remained high throughout the time period. Individual level maintenance of SAs was high initially (91.8%), but rapidly declined and was only 20% after 10 years.
Application of the RE-AIM model to evaluate this long-term SA installation program highlighted areas that warranted improvement, especially for the dimensions of Reach and individual-level Maintenance. The model may be useful for evaluating the impact of other injury prevention programs.
Academic Editor: Prachi Arora, University of Wisconsin - Madison
Checked for plagiarism: Yes
Review by: Single-blind
Copyright © 2016 Shelli Stephens-Stidham, et al.
The authors have declared that no competing interests exist.
Deaths from residential fires and burns remain an important health problem, accounting for 2,520 fire deaths in the U.S. in 2011.1 A smoke alarm (SA) increases the chance of surviving a house fire by 2- to 3-fold.2, 3 While reported SA prevalence is as high as 97% in the U.S. as a whole, houses at highest risk of death and injury from fires continue to have the lowest SA use.4, 5, 6, 7
Based on local data for Dallas, TX,2 we targeted census tracts with the highest per capita rates of house fire-related deaths and injuries and those with lowest median income, for SA installations through a program entitled Operation Installation (OI)2.OI is a joint program of the Injury Prevention Center of Greater Dallas (IPC) and the Dallas Fire Rescue Department (DFRD), and involves door-to-door canvassing and SA installation. A 10-year outcome evaluation of OI from 2001-2011 showed that the adjusted case rate in program houses was 63% lower than non-program houses in the same census tracts.8
However, factors beyond effectiveness can affect program impact.9, 10, 11, 12, 13 The Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework developed by Glasgow, et al provides a comprehensive approach to evaluating an intervention’s overall public health impact.10, 12, 14 In this model, Reach addresses the percentage and representativeness of participants, Effectiveness addresses the impact of the intervention on the targeted outcome, Adoption addresses the proportion and representativeness of settings that deliver the program, Implementation addresses the extent to which the program is delivered as intended, and Maintenance addresses both the individual and organizational (setting level) program delivery over time.10 Of these RE-AIM dimensions, Reach and Effectiveness are individual-level dimensions, Adoption and Implementation are Organizational/setting-level dimensions, and Maintenance has both individual-level and organizational/setting-level dimensions.10
We used the RE-AIM components to measure the impact of OI. Although the RE-AIM model has been applied to a limited number of injury prevention programs, including a Tai Chi program to prevent older adult falls13and sports injury programs,15, 16 to our knowledge, it has never been applied to a long-term SA installation program.
Operation Installationbegan in 1999 in target census tracts in Dallas that had high rates of house fire-related deaths and injuries. We began systematic data collection for the program in April 2001, and included data through April 2011. Details of the program can be found in a previous paper.8
The five RE-AIM dimensions include Reach, Effectiveness, Adoption, Implementation, and Maintenance, both at the individual level and at the setting level.10, 12 For each RE-AIM dimension, we calculated cumulative results at 2 years and 10 years after initiation.
Reach was calculated by dividing the total number of persons residing in program homes by the total population of the OI census tracts, as previously described.8 Program homes were defined as homes that had received a SA through OI between 2001 and 2011. Non-program homes were all other houses in the same census tracts that did not receive a SA through OI during the same time period.
The representativeness of the OI census tracts was determined by comparing demographic characteristics of the program homes with non-program homes.8
Effectiveness was determined by comparing rates of house fire-related deaths and injuries for program and non-program populations. Details of the effectiveness calculations can be found in a previous paper.8
Adoption was defined as the number of fire departments recruited to participate in the SA distribution.
Implementation was defined as the extent to which the Dallas Fire-Rescue and IPC implemented OI, including the average number of SA’s installed per house. For cost calculations, detailed information regarding time and costs (including materials, salaries, and volunteer hours) were collected during the time period when funding from the Centers for Disease Control and Prevention (CDC) was available – October 2006 through September 2012. We assigned a volunteer hour cost to be $21.79 per hour, which is a standard volunteer hour cost assigned by a Texas government agency. Based on all costs incurred during that time period (2006-2012), we extrapolated the cost of the project for the study period (2001-2011) in 2013 US dollars, the cost per house visited, and cost per SA installed.
Maintenance at the individual level was measured by re-visiting a random sample of homes that had received SAs through OI 2, 4, 6, 8 and 10 years previously, and determining the proportion of SA’s that were still functional. Details of this study have been published by McCoy, et al.17Maintenance at the setting level was determined by the number of canvassing sessions and number of census tracts canvassed during the 10-year period from 2001 through 2011.
Statistical comparison of rates in program and non-program homes for the calculation of effectiveness was done through conditional maximum likelihood estimates of the Rate Ratio, and expressed as Rate Ratio (RR) with corresponding 95% confidence interval (CI) and P-value.
In the 36 census tracts where OI was conducted, 20,127 SAs were installed in 8,134 homes (28,570 persons reside in those homes). Those 36 census tracts contained 32,480 homes (107,705 persons reside in those homes). The cumulative population reach after two years was 5.7% (6125/107,705) of the total population in the 36 census tracts, and for the 10-year time period the cumulative population reach was 26.5% (28,570/107,705).
In terms of representativeness, individuals in program homes were somewhat more likely to own their home (71.1% vs. 63.7%), have a Hispanic head of household (43.5% vs. 35.5%), and have a head of household over 64 years of age than non-program houses (31.3% vs. 21%). Each of these differences was statistically significant (p < 0.05 by chi square).
For the first 2 years after SA installation, the rate of deaths and injuries in program houses was 83% lower than non-program houses (1.9 vs. 11.5 deaths and injuries per 100,000 population, respectively; RR, 0.16; 95% CI, 0.01 to 0.89; p< 0.05). For the entire 10-year period, the rate was 63% lower in program houses compared to non-program houses (3.5 vs. 9.5 deaths and injuries per 100,000 population, respectively; RR, 0.32; 95% CI, 0.10 to 0.84, p<.02).8
Only one fire department (DFRD) was recruited to participate in the program. The intervention was implemented as designed throughout the 10-year period. The average number of SAs installed per house increased from 1.4 during the first two years, to 2.5 for the entire ten years.
The total project cost in 2013 US dollars was $1.14 million; the cost per program home was $169.41, and the cost per SA installed was $56.71.
For the first two years, the program canvassed an average of 5.0 census tracts per year in 8.5 sessions per year. For the entire 10-year period, the program canvassed and average of 3.6 census tracts in 6.2 sessions per year. At the individual level, 91.8% of houses had at least one working SA 2 years following the original installation compared to only 19.9% after 10 years.17The results of the various RE-AIM parameters are summarized in the Table 1.Table 1. RE-AIM Parameters at 2 years and 10 years, Operation Installation, Dallas, TX 2001-2011
|Participation rate: Population in program homes/total population||6,125/107,705||28,570/107,705|
|Death & Injury Rate for Program vs. Non-Program homes*||1.9* vs. 11.5*||3.5* vs. 9.5*|
|Participation rate at Fire Department||1 of 1||1 of 1|
|Total no. SA’s installed||2,456 SA’s installed||20,127 SA’s installed|
|Average no. SA’s installed per house||1.4 SA’s per house||2.5 SA’s per house|
|Costs of O.I.:||N/A||Total Cost: $1.141 million|
|Cost per home:………….$169.41|
|Cost per SA installed: $ 56.71|
|-Ave. no. census tracts canvassed/year;||5.0 census tracts per year;||3.6 census tracts per year;|
|-Ave. no. of sessions/ year||8.5 sessions per year||6.2 sessions per year|
|- houses with SA’s working||91.8 % of SA working @ 2 years post installation||19.9% of SA working @ 10 years post installation|
The RE-AIM model has been used to evaluate dimensions of community-based public health programs to determine whether programs are worth sustained investment.9, 10, 11, 12, 13, 18 To our knowledge, it has not been applied to a long-term SA installation program.
Our analysis of OI used the RE-AIM dimensions and found several strengths in the program, including its effectiveness, and its adoption and implementation by the DFRD throughout the 10-year period. The DFRD has now operationalized OI to the point that the program is part of its regular operations. In addition, the cost of the program per program home and per smoke alarm installed, was somewhat lower than previously described.19 This improved efficiency may be related to the large number of SAs installed through OI. Such economies of scale may be important to the cost effectiveness of such programs.19
Our analysis also identified areas for improvement for OI. Although its effectiveness for preventing house fire-related deaths and injuries was high (83% for the first 2 years), it clearly declined after 6 years post installation. This decline in effectiveness may have been due to a decrease in SA function over time. Only approximately 20% of houses still had at least one working SA 10 years following installation; this is somewhat lower than the 33% found by Jackson, et al from a multi-site evaluation.20 Based on the RE-AIM evaluation, DFRD is now installing tamper-resistant smoke alarms and re-canvassing previous project neighborhoods and homes to replace smoke alarms that have been removed or no longer function.
In addition, the project reached only a quarter of the target population. To increase the reach, postcards are being mailed to homes in the project neighborhoods to alert residents when DFRD will be canvassing the neighborhoods and installing smoke alarms. Yard signs have also been used to notify residents in advance of scheduled smoke alarm canvassing sessions.
There are several limitations to this study. Because only one fire department was involved, it is difficult to generalize about the adoption component of the RE-AIM evaluation; other fire departments may not be able to integrate a program such as OI into its ongoing functions. In addition, implementation of OI was most active during the years that DFRD was receiving grant funding from CDC; it is more challenging to implement OI without ongoing support. In addition, we had no household data from non-participating households; we utilized census data to calculate demographic data for this group, which may not be at the same level of accuracy as the data collected from participating households.8
As suggested by Hanson, it is useful to assess the broader population impact of a public health program, using dimensions such as implementation and maintenance, in a real world setting.21 The five RE-AIM dimensions provided insight into areas to focus our future activities for improvement of OI. The RE-AIM model may be a beneficial tool in evaluating other injury prevention programs as well.
SSS, MM, CR and GRI contributed to the concept and design of the project. MM, CR, JJB and EC contributed to the acquisition of data. SSS, MM, JJB and GRI contributed to the analyses and interpretation of data. SSS, MM and GRI had access to all data and are responsible for the overall content. All authors contributed to drafting and revising the article and approved the final version.
We would like to thank the following people for their assistance in the project: Kevin Sipes (retired) and Debra Carlin (retired) with the City of Dallas Fire-Rescue Department for their oversight of the project, and members of the Inspection and Life Safety Education Division for the coordination and collection of data; Merissa Yellman for assistance in cost calculations; and David Beveridge, Injury Prevention Center Administrative Assistant, for assistance in formatting various parts of this article.