Executive Summary
An advanced-technology Integrated Safety and Security Enforcement System (ISSES), now deployed at three commercial vehicle inspection sites along interstate highways in Kentucky, was evaluated from the point of view of system performance, potential effects on inspection selection efficiency (choosing the highest-risk trucks from the stream of commerce), user acceptance, and costs.
Objectives. The overall purpose of the Kentucky Commercial Vehicle Safety Applications (CVSA) Evaluation was to provide an independent assessment of the performance, usability, safety benefits, and wider applicability of an advanced system for screening trucks at the roadside. The system, known as ISSES, has been deployed along selected interstate highway routes by the Kentucky Transportation Cabinet, under a grant from the Federal Motor Carrier Safety Administration (FMCSA). ISSES is made up of a radiation monitor, a thermal inspection device, a laser scanner/vehicle detector, a license plate recognition system, and a USDOT number reader. The system has potential applications for both highway safety and nuclear detection/homeland security. The present evaluation is focused on the first deployment, at Laurel County, near London, Kentucky, on I-75 northbound. This Technical Report describes the evaluation objectives, the data collection and data analysis methods, and the evaluation results. Goals of the evaluation were to estimate whether the ISSES will make highways safer and more secure, and to determine how the ISSES makes the commercial vehicle inspection process more efficient and effective. The main aspects of this evaluation were (a) system performance, (b) inspection selection efficiency, and (c) user acceptance/system costs.
Data Collection. For the system performance study, available data were collected from all of the relevant ISSES subsystems, vendor information, state inspections conducted on commercial vehicles during a 2-week field observation, and past inspections at the London site and other comparable sites in Kentucky. Several visits were made to the London site for observations. For the inspection efficiency study, data were collected on the decisions made by inspectors at the London site, compared with data available from the ISSES during the same period, and with data potentially available from external state and national databases. For the user acceptance and cost study, data were collected in a series of interviews and researcher observations at the London site.
Data Analysis. For the system performance study, data were analyzed by comparing actual performance data versus manufacturer and system integrator specifications and user documentation. For the inspection efficiency study, the current methods for selecting trucks to inspect were documented, and the safety characteristics of the general commercial vehicle population at the site were determined. Statistical models were used to project effects of ISSES on the efficiency of selecting the highest-risk trucks for inspection. These results were used to model the potential reductions in truck crashes, based on an improved ability to take these highest-risk trucks out of service (OOS) until they are repaired. The views and attitudes of the inspectors and others working with ISSES at the London site were summarized, to document perceived system usability and ideas for future deployments, and to depict the one-time and recurring dollar costs to deploy and operate the ISSES in the field.
Schedule. Planning for the evaluation began in June 2005, and the full-scale independent evaluation began in October 2006. Field observations and most interviews took place in June 2007.
Results. Overall, despite the fact that commercial vehicle law enforcement staffing levels in Kentucky did not allow for full-time, dedicated use of the ISSES by inspectors at the time of the evaluation, the subsystems that were under evaluation in this task were found to be performing effectively in a stand-alone mode. During the field observation, the radiation monitor generated a relatively large number of gamma alarms for naturally occurring substances, causing inspectors generally to ignore most of the radiation alarms. Also, despite having been installed for two years, the roadside system was not yet integrated with in-state or national databases of historical safety information on carriers or vehicles, so the ISSES was not able to provide instant, "actionable" historical information that the inspectors could apply in their decision-making. Such integration has the potential, if implemented, to afford significant benefits in vehicle screening and safety enforcement, according to the modeling performed in connection with the inspection efficiency study. The ISSES software and components now deployed—though operational—are considered to be in a development mode as of late 2007.
The vendor informed the evaluation team that the company attempted to use commercial, off-the-shelf technologies for the ISSES whenever possible. While this approach provides advantages with respect to reducing first costs and allowing the state to begin using subsystems like the thermal inspection camera and radiation monitor immediately, it also increases the cost and difficulty of integrating disparate commercial systems.
Kentucky's current inspection selection methods were compared with potential applications of ISSES technology across a set of scenarios, used to model improvements in commercial vehicle safety, as measured by changes in the rates of OOS orders issued to commercial vehicles. Applying various combinations of inspection selection strategies and available or envisioned technologies for real-time vehicle identification and safety information exchange at the roadside, in a hypothetical statewide deployment supporting about 44,000 vehicle inspections and 86,000 driver inspections in a year, the ISSES was estimated to contribute to incremental reductions of between 63 and 629 commercial vehicle-related crashes per year, reductions of between 16 and 163 personal injuries, and reductions of up to 7 fatalities.
The overall user acceptance of the system was difficult to measure for two main reasons. First, the KVE inspectors interviewed in 2007 for this evaluation reported having had very little training on the system, which had been in place for two years prior to the evaluation. The vendor reported having held a training session when the system was installed in 2005. Offsite training and onsite exercises for response to radiation alarms were also provided. Secondly, the workload and day-to-day priorities of the inspectors generally cause them to rely more on visual evidence, professional experience, and judgment than on advanced-technology screening devices such as ISSES when they make inspection selection decisions. Thus the ISSES was in place and operating at the inspection station, but was not being used to any effective extent during the period of the evaluation.
The deployment took place in a larger enforcement context that has up to now emphasized and rewarded inspectors for the numbers of inspections they complete per time period, not necessarily for achieving high rates of OOS orders. Thus the purpose of the ISSES (to help inspectors focus on the trucks with the worst safety records, and in effect drive upward the rate of OOS orders) is not directly aligned with the traditional, quota-driven goals of the inspectors in Kentucky. Both approaches, i.e., the productivity of completing a large number of inspections and the efficiency or effectiveness of identifying a high proportion of safety violations per time period, are valid goals of commercial vehicle inspection. This institutional disconnect in Kentucky, however, affected the degree to which the inspectors perceived the ISSES as helping them achieve their personal and organizational job goals. The inspectors who were interviewed tended to regard the ISSES as a technology that could potentially help in their inspection duties, but they had not had the breadth and depth of experience in using it to permit a detailed characterization of the degree of user acceptance.
The system at Laurel County cost $350,000 to install initially. The Kentucky Transportation Center (KTC) at the University of Kentucky, which was involved in ISSES contracting between the state and the vendor, reported that funds from Oak Ridge National Laboratory were also used in the original Laurel County installation and deployment, and that subsequent systems installed in other Kentucky counties have cost the state approximately $500,000 each to procure and install. The state has also paid approximately $109,000 in follow-up costs paid to the system vendor between November 2006 and August 2007 for ongoing maintenance, off-site monitoring of system status, and on-site troubleshooting, such as system calibration, testing, adjustment, technical support, training, and repair at all of the ISSES sites. Not all of these follow-up costs have been directly related to actual ISSES maintenance and repair, however. Some fraction of the costs were for administrative activities, software programming support, and communications protocol development for the nuclear detection subsystem.
Conclusions. The KVE inspectors at Laurel County were not using the ISSES to any great extent during the period of the field study. According to interviews with inspectors and with staff from the KTC, the ISSES hardware was functioning satisfactorily, but the state's current enforcement staffing levels—and an organizational emphasis on the quantity of inspections completed, as opposed to the rate of OOS orders issued—prevent inspectors from having the time or incentives to make effective use of the information being displayed by the ISSES.
Although they were not yet integrated with any state or national data sources, the portions of the ISSES under evaluation in this study appeared to be performing as designed. The system has the potential to reduce commercial vehicle related crashes, injuries, and fatalities substantially if deployed more widely. Such wider deployment would also depend on the ISSES being connected with current and historical sources of safety and inspection data, which would enable inspectors to focus their efforts on the highest-risk carriers, vehicles, and drivers, while allowing the safer, more compliant vehicles to continue traveling past the weigh station. Overall, to the extent that they had been exposed to the ISSES, the users at the Laurel site were positive toward it and appeared to recognize its potential, but within their current organizational environment, they regarded it as more of a developmental test or research device than as a tool that they wanted to use immediately in their day-to-day commercial vehicle inspection and law enforcement duties.