2. Evaluation Approach

A kick-off meeting was conducted in Monterey, California at MST headquarters on November 1, 2006, to begin preliminary discussions and lay the foundation for the overall evaluation process as discussed later in this section. The following steps were proposed for conducting the evaluation:

Develop Evaluation Plan;
Develop Test Plan;
Collect data;
Conduct data analysis;
Conduct on-site interviews at MST; and
Prepare and submit final evaluation report.

The last four steps (i.e., data collection through the final report) were proposed to be conducted separately for Phases II and III of the evaluation. The final report for Phase III (this document) also includes findings from Phase II.

2.1 Evaluation Plan

2.1.1 Evaluation Phases

USDOT-sponsored ITS evaluations are traditionally divided into phases. During Phase II, the Evaluation Team collects data before the technologies are deployed and summarizes the "before" data in a Phase II Report. Phase III occurs once the technologies have been deployed (after the approval of Phase II results), when the Evaluation Team collects the "after" data and presents the findings of the "before and after" analyses in a Phase III Report. In the case of the MST ITS Augmentation Project, many of the technologies were deployed during Phase II, presenting the Evaluation Team with an opportunity to document the findings of the "before and after" analyses earlier, in the Phase II report. However, there were project components that were deployed during Phase II, namely real-time information and Google Transit trip planner. To allow the timely publication of evaluation results for the technologies already deployed, the Evaluation Team proposed that the scope of both phases be conducted in the following manner:

Phase II: The "before and after" analyses were performed for those technologies that were fully deployed and operational. For those technologies that did not have a definite deployment frame, only the "before" analysis was performed. The Phase II Report was prepared with results from these analyses.
Phase III: This phase consisted of evaluating the technologies that were deployed by the end of 2008. The Evaluation Team collected the "after" data and customer satisfaction surveys during 2009.

For this Phase III report, the Evaluation Team performed the following activities:

Collected data for the "after" analysis following the implementation of the new technologies;
Collected customer satisfaction survey data at the Monterey Transit Plaza and Salinas Transit Center;
Performed data analysis and tested the hypotheses associated with Phase III technologies;
Developed the Draft Evaluation Report for Phase III and provided the report to FTA/FHWA for review and comment; and
Incorporated FTA/FHWA comments, revised, and finalized the Final Evaluation Report for Phase III and submitted to FTA/FHWA for final review and approval.

2.1.2 Evaluation Hypotheses

The evaluation hypotheses and measures of effectiveness (MOEs) were developed in the initial phase of this evaluation project. The hypotheses that were submitted and approved in the Evaluation Plan were identified in two distinct categories: "key hypotheses" and "secondary hypotheses."⁷

Both key and secondary hypotheses were tested in the in Phase II report based on test procedures identified in the Test Plan. Section 6 describes results obtained from test of these hypotheses.

The key hypotheses for this evaluation are:

The project will result in a reduction in operations and planning costs and improved service planning;
The project will result in improved on-time performance of MST operation;
The project will result in an increase in the reliability of services;
The project will enhance system productivity; and
The project will result in an improvement in maintenance scheduling and planning.

The secondary hypotheses for this evaluation are:

The project will result in improved customer satisfaction;
The project will result in an increase in ridership;
The project will result in an improvement in driver/ passenger security;
The project will result in a reduction in the travel times of specific routes where TSP is deployed;
The project will help reduce response time for incidents and emergency management;
The project will result in a reduction in vehicle hours;
The project will reduce the number of customer complaints;
The project will result in improved facility security;
The project will establish a comprehensive reporting system; and
The project will result in reduced cases of false financial claims.

Please refer to Appendix A for a detailed list of measures of effectiveness (MOEs) for all hypotheses.

The evaluation of these hypotheses can be categorized based on their contribution to individual operational departments at MST. Hence, the evaluation (including both Phase II and Phase III activities) was conducted by studying each of the following topics:

Transit Planning and Operations;
Maintenance and Incident Management;
Safety and Security;
MST Reporting Capabilities;
Finance; and
Customer Service.

The preceding categories were defined by considering the complexities of the evaluation based on the MST goals for technology implementation, the nature of data analyses for this evaluation, and the types of data and data collection methods needed for analyses.

The hypotheses related to the real-time Information, trip planner and customer satisfaction were tested in Phase III of the evaluation. The hypotheses related to the TSP and smart card payment could not be investigated because the deployments were not completed during the evaluation period.

2.2 Test Plan

2.2.1 Data Collection

Phase II of the evaluation project required collecting data for both quantitative and qualitative analyses. Archived data from the ACS was collected for the time period from April 15 to May 31 for each year from 2003 through 2007 to test the hypothesis related to on-time performance. In addition to the ACS data, information such as operational statistics (e.g., vehicle revenue-miles and passenger-miles), cost and revenue data, information on performance measures, accident or incident statistics, and financial claim/recovery data was collected to test other hypotheses. Further, the Evaluation Team conducted several interviews during a 2-day on-site visit at MST in August, 2008. These interviews were conducted with representatives from planning, operations, finance, maintenance, customer service, management, safety and security, and information technology. Please refer to Appendix D for a detailed list of questions asked in these interviews.

In Phase III, the quantitative analysis was conducted by collecting a larger amount of automatic vehicle location (AVL) data over a more specific timeframe (i.e., starting in 2005) as compared to Phase II dataset. As part of the Phase III data collection, the Evaluation Team exported raw AVL data directly from the database of the CAD/AVL system during the site visit in June 2009. This data was further processed and imported into a SQL Server database and Microsoft Excel in order to perform data analyses. Further, a qualitative analysis was performed on data collected through intercept surveys. The customer survey was primarily focused on evaluating real-time information and MST's service reliability. However, the results of the survey complement the findings from the quantitative analysis of AVL data, which was conducted to determine if there were any on-time performance trends.

2.2.2 Analysis

A quantitative analysis was performed on ACS data to calculate on-time performance of MST vehicles for selected time periods in both Phase II and Phase III. Since on-time performance measurement standards were changed by MST during the selected time periods (see Section 3.1.1.1.1), the Evaluation Team measured on-time performance based on new MOEs. The on-time performance was measured based on the positive or negative deviation (in number of minutes) of the actual arrival time from the scheduled arrival time. A detailed approach for this analysis is discussed later in this report (see Section 3.1.1.1.1 for Phase II approach and 3.1.1.1.2 for Phase III approach). Also, the annual trends for other statistics (e.g., ridership, revenue, productivity and accidents) were calculated.

In addition to performing quantitative analysis, a qualitative assessment of the impact of technologies on daily activities of all MST departments was performed in Phase II. The qualitative analysis was conducted based on information obtained from MST staff during the interviews conducted in August 2008.

Finally, data obtained from customer intercept surveys were analyzed in Phase III to measure customer perceptions of the impact of technology on MST Service.

Section 3 summarizes findings obtained from the data analyses, on-site interviews (conducted in Phase II), and intercept surveys (Phase III) to describe the impact of various technologies on MST.

⁷ Secondary hypotheses are those that address goals that are relatively less significant in improving daily operations at MST in comparison to key hypotheses.

Previous | Next