Table of Contents

3. Evaluation Approach

3.1 Evaluation Overview

The primary intent of the FOT was to determine the extent to which existing vulnerabilities in the transportation of hazardous materials can be reduced, thereby reducing the potential for a catastrophic event with resulting loss of life and property. The benefit-cost analysis was designed to measure this benefit and determine which component technologies or integrated systems offer the best mix of improved security balanced against reasonable costs for deployment and operations.

Deploying these technologies and systems will require an investment by the trucking industry, which is an industry with very low returns – a profit margin of 2 percent is not uncommon. An important aspect of this evaluation was to determine whether or not the deployment of these technologies and systems would generate a positive return on investment for industry. The potential return from deployment was quantified as a measurable improvement in operating efficiencies or improvement in overall operations. The objective assessment of this "bottom line" impact is critical for determining policy options for structuring a deployment program: market-based, where operating efficiencies drive deployment; or a mandate in the interests of national security; or a package of incentives designed to encourage and facilitate deployment.

3.2 Assessment Frameworks

The main evaluation impact categories examined by the Evaluation Team are safety, security, and operational efficiency. As detailed in Figure 3-1, these impact categories feed the benefit-cost analysis according to macroeconomic/societal (macro) public sector benefit-cost results (stemming from security and safety benefits) and microeconomic/industry (micro) private sector benefit-cost results (derived from operational efficiency improvements and enhanced safety). The macro/societal and micro/industry benefit-cost measurements analysis was conducted to determine the following:

• Are the industry operational efficiency benefits significant enough to drive widespread industry deployment of test technology systems?
• If not, are the macro benefits large enough to warrant government action to facilitate wide-scale national deployment?

The evaluation assessments were conducted within the scope of the FOT and extended the FOT findings to the larger universe of truck-based HAZMAT shipments (for the four primary load types) through rigorous analytical frameworks. These frameworks utilized primary and secondary industry survey data; detailed motor carrier census records; market analysis of technology products and services that are commercially available; and the opinions provided by two august groups of leading national experts in HAZMAT shipping, public safety, security and risk assessment – an Expert Steering Committee and a 26-member Delphi Panel.

The assessments determined what measurable benefits exist, and established and implemented analytical frameworks to monetize potential benefits and to weigh these against any costs that would have to be incurred to realize the benefits. Detailed discussion of methodology are presented in Volume III, Section 3: HAZMAT FOT Security Benefits Assessment and Section 4: Benefit-Cost Analysis and Industry Deployment Potential, and are summarized in the following sections of this synthesis document.

Figure 3-1. Evaluation Framework

It should be noted that the test technologies were designed to enable real-time communications and information exchange among drivers, dispatchers and other authorized parties; to track assets; to secure vehicles, loads, and shipping documentation; and to enable driver or automated exception alerts in response to crises or deviations in operational characteristics outside of set parameters. The technologies themselves and their usage are not specifically designed to provide explicit or traditional safety benefits (i.e., reducing the frequency and severity of crashes).

The test technologies are not designed to warn drivers of obstacles in proximity to their vehicles, lane departure, imminent vehicle rollover conditions, or conditions signaling driver fatigue. The exceptions include the beneficial impacts of frequent driver/dispatcher communications that allow a dispatcher to assess the driver's condition; position tracking to determine possible speeding, or capabilities that provide responders to HAZMAT incidents timely notification of the incident; and location and the type and quantity of HAZMAT involved to enhance the rapidity and appropriateness of response. Potential safety benefits in terms of crashes avoided and enhanced emergency response are proffered, but are mostly qualitative in nature. Quantitatively, the evaluation focused on the remaining two key assessment areas: Security and Operational Efficiency.

The estimation of benefits and costs, payback periods, and industry deployment potential was based on a stepwise analytical framework. Benefits were derived as operational improvements and reductions in potential impacts of terrorist activities involving truck-based HAZMAT shipments. The framework for these assessments (as illustrated in Figures 3-2 and 3-3) is based on the following inputs:

• Technology performance and participant perceptions, as defined by the FOT, established technology functionality, efficacy, user acceptance, and operational improvements. These data also provided inputs to the Delphi process described in Volume III, Section 3.
• HAZMAT carrier demographics (total number of trucks and fleet size distributions) for each load type that was included in the FOT. These data, defined through queries of the FMCSA's Motor Carrier Management Information System (MCMIS), established the number of trucks that would represent potential full deployment of the test technologies. The results of the Deployment Team's motor carrier survey. Returned by 153 motor carriers, the respondent demographics represented a broad diversity of fleet sizes, range of operations, routing variability, general operational characteristics and levels of fleet management technologies currently used and those to be employed in the near-term. These results were validated using other industry technology deployment studies and applied to the demographics of HAZMAT carriers reported in the FMCSA Motor Carrier Management Information System (MCMIS) database to estimate levels of current technology market penetration and total market potential.[3]
• A Technology Compendium, which defined current and near-future levels of motor carrier technology adoption, and pricing and functionality of commercially available technology products with similar capabilities as the technologies testing during the FOT. The latter provided a range of potential industry costs.

Other important evaluation goals included: assessing technical performance of the test suites; user acceptance and perceptions of the pros and cons of using the technologies; issues of deployment costs; privacy and the potential for governmental intervention; and defining the market potential for the deployment of the technologies.

The data were collected through direct observations; reviews of technology transaction records; interviews and survey questionnaires of motor carrier and state agency participants and non-participants; and consultation of secondary data sources. These assessments are the basis for the overall evaluation findings.

The processes involved in the Efficiency and Security Assessments are detailed in Volume III, Sections 2 and 3, respectively, and are summarized in the sections that follow.

Figure 3-2. Operational Efficiency Benefit-Cost Analysis Process Flows

Figure 3-3. Security Benefit-Cost Analysis Process Flows

previous | next