3. Evaluation Strategy Overview

This section presents an overview of the evaluation strategy developed for the ETC/E-Screening Interoperability Pilot Project to reflect the overall goals and objectives established for the Pilot Project by the project participants. The following five evaluation goals were identified:

Assess the impact of interoperability on motor carrier mobility.
Assess the impact of electronic screening on motor carrier safety.
Identify industry and government efficiency gains from ETC/E-screening.
Assess the impact of electronic screening on the environment, in particular, reduction in diesel emissions.
Assess overall customer satisfaction, both government and industry.

Supporting objectives for the evaluation goals were developed to provide quantitative and qualitative analysis of the system impacts, to document institutional challenges, and to identify lessons learned. The impacts and challenges identified were carefully explored and documented to help provide guidance for other regions considering similar integration projects. The evaluation findings will be used by other agencies to assess the appropriateness of ITS integration as a potential solution to their local problems.

These objectives were derived from the objectives of the project participants. Two types of data were collected: quantitative data on operational performance (e.g., travel time, out-of-service rates) and qualitative data (e.g., customer satisfaction ratings).

The ETC/E-Screening Interoperability Pilot Project evaluation structure is based on standard evaluation practices originally developed by USDOT. For each evaluation goal, hypotheses were formulated to identify anticipated impacts to the system. One or more measures of effectiveness (MOEs) are associated with each hypothesis to assess the accuracy of the hypothesis. Required data and data sources are identified for each MOE. The goals, hypotheses, MOEs, and data sources identified for each study for the evaluation of the ETC/E-Screening Interoperability Pilot Project are summarized in Table 3-1.

**Table 3-1. Evaluation Goals, Hypotheses, and MOEs**
Goal	Hypothesis	MOE	Data Sources or Requirements
Improve mobility at weigh stations and toll collection facilities.	ETC and E-screening will improve the mobility of transponder-equipped commercial vehicles at weigh stations and toll collection facilities.	Travel time through facilities. Travel time variability through facilities. Number of commercial vehicles passing through weigh stations per day.	Field measurement of travel times and travel time variability through facilities. Field counts of commercial vehicles. Weigh station records.
Improve safety.	Carriers with transponders will maintain compliance with safety standards. Enforcement personnel will be better able to identify non-compliant or unsafe carriers. Crash rates involving commercial vehicles will be reduced at both weigh stations and toll facilities. Station closings due to ramp backups onto the mainline when station is operating at capacity based on traffic volume will be reduced.	Number of compliant carriers with transponder inspected per day. Out-of-service rates for transponder-equipped and non-transponder-equipped vehicles. Crash rates. Number of times stations must close per day due to ramp backups when stations are operating at capacity based on traffic volume, and duration of closures.	Enforcement records/ out-of-service reports. Weigh station records.
Improve efficiency of motor carrier operations for government and industry.	Data sharing will improve inter-agency coordination, thereby improving efficiency of motor carrier operations. Enforcement agencies will establish standardized criteria for bypass, inspection selection, and other enforcement activities to improve identification of non-compliant carriers.	Costs associated with reduced fuel consumption and travel time. Costs associated with enforcement activities (number of enforcement officials, hours of operation). Number of inspections on one trip. Agency procedures and policies.	Calculations of yearly fuel and travel time savings for industry. Enforcement agency records. Agency documents on enforcement policies and procedures.
Reduce fuel consumption and emissions at toll facilities.	The use of one transponder for both ETC and E-screening will help promote industry acceptance and use. Drivers will perceive a time savings association with use of the technology. Enforcement officials will benefit from the carriers’ use of the technology.	Industry acceptance/ endorsement of technology. Incentives offered by state agencies to encourage use of transponders. Drivers’ perceived time savings. Enforcement officials’ assessment of technology and perception of benefits.	Surveys/interviews/Focus groups with motor carriers, drivers, and enforcement officials.

In addition to assessing the five goals established for the evaluation, the Evaluation Team also documented the following institutional and technical challenges:

How participating agencies identified and resolved technical and policy issues related to the use of a single transponder for both ETC and E-screening.
How industry concerns on such issues as access to and use of ETC and E-screening event data were addressed and the process established for discussing these issues with industry.
How E-screening enrollment and bypass criteria were established and agreed to.
Who was responsible for reviewing carrier applications for E-screening and how this was accomplished.
How transponder administrator services were addressed.
How participating motor carriers were recruited, and what plans are being developed for larger scale recruitment.
How E-screening in Maryland and Connecticut will be used as the baseline for expanded deployment of electronic screening capabilities at weigh and inspection stations located elsewhere in each State.
How the Maryland Electronic Screening Program will operate as a "program neutral" electronic screening program, and what this means for achieving national E-screening interoperability.

Since institutional and technical challenges are identified and addressed throughout the project, documenting institutional and technical challenges was not established as a stand-alone evaluation goal. As particular tests supporting the other objectives were conducted, questions related to or information required for the institutional challenges component were incorporated, as appropriate. To this end, data collection activities for institutional and technical challenges test was designed to complement and be incorporated with the other evaluation activities and detailed tests.

Detailed test plans were developed for each of the evaluation components proposed for the evaluation of the ETC/E-Screening Interoperability Pilot Project.²⁹ Each test plan defined the objective, approach, and work steps for each evaluation component, as summarized below:

Mobility Test Plan - Investigate mobility improvements in terms of travel time through facilities and the volume of trucks processed per day at the weigh stations.
Safety Test Plan - Document the improvements in safety enforcement and carrier compliance with safety standards.
Operational Efficiency Test Plan - Examine the impacts of interagency coordination on motor carrier operational efficiency.
Environmental Test Plan - Examine the environmental impacts from reduced waiting times at toll and weigh station facilities.
Customer Satisfaction Test Plan - Investigate the level of improved customer satisfaction resulting from more convenient payment of tolls and screening process improvements.
Institutional and Technical Challenges/Lesson Learned Test Plan - Identify the lessons learned from both institutional and technical challenges.

In preparing the test plans for specific evaluation goals, numerous test activities were identified that could be consolidated to ensure efficient use of evaluation resources. For example, one test combined the planned timing of delays at weigh stations and toll facilities. This particular test activity generated data used to complete the mobility, operational efficiency, and environmental tests, and also impacted the customer satisfaction and institutional challenges tests. Prior to conducting field work, all test and design data collection activities were identified to ensure appropriate collection of all necessary data.

The toll facilities and weigh stations used to conduct field tests were selected based on discussions and cooperative efforts with project participants in each respective State. Field measurements at toll facilities required one site visit per facility, which was based on the extensive use of toll tag transponders by the trucking industry. These measurements facilitated the collection of both "Before" (without transponder) and "After" (with transponder) data. Initially, two site visits were planned to collect the Before and After data at weigh stations. This plan was later modified and reduced to one visit given since only the Perryville Weigh Station was operational for E-screening during the evaluation period. There was no need to collect After project data as a result.

As initially proposed, the evaluation assessed the use of the Mark IV Fusion transponder at the Peace Bridge International Border Crossing near Buffalo, New York. Initially, the National Customs Automation Program (NCAP) transponder deployment, with integration to the International Trade Data System (ITDS), was carried out by U.S. Customs and Border Patrol and offered the opportunity to assess an additional interoperable application as an evaluation component. As part of the evaluation, the transponder/ITDS implementation effort was to be assessed as follows:

A "case study" assessment of the potential use of a universal transponder as part of this test, consistent with the "universal transponder" deployment effort described in the SOW for the I-95/Northeastern U.S. region.
A "case study" assessment of the implementation of the ITDS as part of this border crossing system, including interviews with all major key players to determine the perceived benefits and institutional challenges of deploying these systems.

However, the final test at the Peace Bridge did not use the Mark IV Fusion transponder, and the opportunity to develop the case study did not materialize.

The Peace Bridge "universal transponder" and ITDS deployment was based on "smart card" technology, not transponder-based technology. Since the intended objective of the proposed case study was to measure the expanded interoperability of transponder technology to include border crossing and customs functions, this was not feasible.

During a presentation to FHWA and FMCSA on the preliminary results of data collection, the Evaluation Team was asked if any data collection had been performed to measure the impact of ETC on congestion, mobility, and environmental impacts in urban areas. The Evaluation Team replied that this was not the case, and was then asked to consider expanding the data collection efforts to include urban areas.

Based on this request and the fact that the Peace Bridge case study did not materialize, the SOW for this evaluation was amended to exclude the case study, and instead, to substitute the following items:

This modification to the Statement of Work requires the following additional tasks:

The collection of data at two additional sites - the Tappan Zee Bridge and the George Washington Bridge. The intent of this additional collection data is to:

Estimate the mobility, efficiency and environmental impacts resulting from the use of electronic toll collection in an urban setting.
Expand the customer satisfaction assessment to include an assessment of the benefits perceived by end users from using ETC in an urban setting.
Expand the environmental impact analysis to model the potential emissions reductions resulting from the use of ETC and electronic screening.
The period of performance for these two tasks shall be concurrent with the overall period of performance for the current evaluation.

The deliverables for this modification shall be included in the overall report prepared for the evaluation. The deliverables shall include:

An assessment of the mobility and efficiency benefits of the use of ETC in an urban setting.
A statistical assessment of potential emissions reductions resulting from the use of ETC and E-screening technologies. ³⁰

²⁹ETC/E-Screening Interoperability Pilot Project Detailed Test Plans, July 2002.
³⁰Electronic Toll Collection/Electronic Screening Interoperability Pilot Project, Statement of Work Modification to DTFH61-96-C-00098, SA9834M2, published July 2004.

Previous | Next