EXECUTIVE SUMMARY
The purpose of this document is to present the findings from Phase II and Phase III of the evaluation of the Intelligent Transportation Systems (ITS) Augmentation Project that was implemented at Monterey-Salinas Transit (MST) in Monterey, California. This project, implemented using Federal Fiscal Year 2003 ITS Integration Program Funds, originally included automated passenger counting (APC) technology, digital video surveillance, on-board automated vehicle annunciation (AVA), smart-card-based fare payment, transit signal priority (TSP), web-based trip planning, and real-time information systems. All of these systems were implemented as of June 2009 except for smart card fare payment and TSP. Deployment of these two technologies has been delayed for institutional and technical reasons.
Monterey-Salinas Transit (MST) is one of the three major transit providers in the Monterey Bay area and serves Monterey County and Southern Santa Cruz County. MST provides service to the 352,000 residents of Monterey County, including the RIDES curb-to-curb paratransit transportation service for residents of the MST service area who need specialized transportation. MST serves the Monterey Peninsula and Salinas Valley areas with 36 fixed routes and provides service between these two urban areas of Monterey County via Highway 68 and Highway 1. MST also operates trolleys that serve the area along the Monterey and Pacific Grove waterfronts.
Background on the ITS Augmentation Project
MST received Federal earmark funding in 2003 to procure and implement technologies to meet its daily organizational needs. These technologies were implemented under a project named the ITS Augmentation Project. The ITS Augmentation Project was primarily intended to assist MST in improving customer information and convenience, on-time performance, operational efficiency and management, and safety and security. Additionally, the project was expected to enhance reporting and implement automated passenger counting.
MST's goals for the ITS implementation as they relate to Federal Transit Administration (FTA)/USDOT goals were identified as follows:
| FTA/USDOT Goals | MST Issues Addressed by ITS Technologies |
|---|---|
| Increased Productivity |
|
| Improved Mobility |
|
| Improved Efficiency |
|
| Improved Safety |
|
The following technologies were selected by MST for implementation as part of the ITS Augmentation Project:
- On-board technology for automatic passenger counting (APC);
- Automatic vehicle Announcements (AVA);
- Upgraded transit management and planning software;
- Digital cameras on buses and in transit centers;
- Security images to local public safety agencies;
- Archived Data User Service (ADUS);
- Real-time information system (RTIS);
- Transit Signal Priority (TSP);
- Translink program (San Francisco Bay Area smart card fare payment system); and
- Upgraded MST website to provide trip planning and bus location information.
Some of the above technologies (e.g., passenger counting and AVA) were integrated with the existing Advanced Communication System (ACS) system, which was procured under a separate grant in 2002. The ACS is a global positioning system (GPS)-based computer aided dispatch and automatic vehicle location (CAD/AVL) system.
Most of the technologies identified in the ITS Augmentation project were implemented as of June 2009. MST has decided not to use the Translink program due to current institutional issues among participating agencies and the vendor. The agency plans to procure smart card fare payment technology in partnership with the Santa Cruz Metropolitan Transit District.
MST decided not to participate in the TakeTransit trip planner offered by the Metropolitan Transportation Commission (MTC) in the San Francisco Bay Area even though that was identified in the original scope of this project. Instead, MST launched the Google Transit trip planner in November 2008.
MST has not yet implemented TSP because all of the project stakeholders have yet to reach consensus regarding the benefits of this technology. The dynamic message signs (DMS) providing real-time information have been installed at two locations. Please refer to Section 1.2 of this report for further details on the deployment status of each technology.
MST has procured other technologies since the implementation of the ITS Augmentation Project to reduce manual process and improve efficiency, including implementing a financial accounting and management software (FAMIS)1 in 2006, a new payroll system in 2008 and maintenance management system (MMS) in 2006. MST has also installed on-board internet access on vehicles serving the two commuter routes.
Background on the MST ITS Evaluation
The USDOT's ITS Joint Program Office (JPO), housed within the Research and Advanced Technology Administration (RITA) established a National ITS Evaluation Program to determine the impacts of ITS deployments across the country. The objective of these evaluations is to document findings that can be used by a wide variety of external audiences such as planners, engineers, and managers. The results of these evaluations assist in the planning and implementing of future ITS projects with help of lessons learned from systems already implemented.
The ITS/JPO selected the MST ITS Augmentation Project for an evaluation to be conducted by an independent Evaluation Team comprised of SAIC and TranSystems (referred to as the Evaluation Team in this report). The two-fold purpose of this evaluation is to evaluate the use of archived ITS data for improving service planning and operations and to evaluate the costs and benefits of implementing the MST ITS Augmentation Project. The ACS was evaluated along with the technologies that are part of the Augmentation Project since these technologies were integrated with the ACS. Further, the ACS was evaluated because it contributes significantly to achieving the ITS Augmentation Project goals (e.g., on-time performance). Also, the Evaluation Team analyzed the impact of other technologies that were not procured with the Federal earmark funding (e.g., FAMIS and MMS) but have made significant contributions to improvements at MST.
The evaluation was conducted in three phases. Phase I of the evaluation was an initial assessment of the implementation of the Augmentation Project by the Evaluation Team. This Phase was completed in October 2005 when the Evaluation Team submitted its Phase 1 findings to the ITS/JPO. These findings assisted the ITS/JPO in deciding to proceed with a full evaluation of MST's Augmentation Project. Phase I results are not included in this Evaluation Report.
Phase II, completed in December 2008, focused on a "before and after" analysis of the technologies that had been fully deployed as of August 2008. Additionally, the Phase II report provided information on technologies that were planned to be deployed in the future.
In Phase III, the impacts of real-time information and the Google Transit trip planner were evaluated by analyzing results of the customer intercept surveys that were conducted. Since TSP and smart card payment were not deployed at the time of Phase III evaluation, an analysis was not performed for these technologies. Further, all inconclusive hypotheses from Phase II were revisited in Phase III.
The evaluation approach adopted for Phases II and III is discussed in Section 2 of this report.
Conclusions of the Evaluation
This report describes the findings from both Phase II and Phase III of the evaluation. The evaluation was conducted by testing the hypotheses identified in the Evaluation Plan submitted to USDOT in January 2007. Data collection and analysis were conducted according to the Test Plan submitted to USDOT in June 2007. The evaluation was based on six key hypotheses and nine secondary hypotheses. These hypotheses were tested by analyzing quantitative data obtained from MST and qualitative data obtained by interviewing MST staff (staff interviews were conducted in Phase II), and by conducting intercept surveys of MST customers at two major transfer centers (surveys were conducted in Phase III). The Evaluation Team would like to thank MST staff members since they have made significant contributions to this evaluation by providing data and anecdotal information, and by assisting with the planning for the customer survey.
The evaluation of ITS deployment at MST has resulted in the identification of key factors about MST's experience related to the procurement, implementation, management, and utilization of ITS technologies. Also, the evaluation identified the impacts of the technology on various departments at MST. Further, the customer satisfaction survey conducted in Phase III of the evaluation helped measure customer perception of the impacts of the deployment of technology at MST.
The following paragraphs provide a summary of the evaluation findings with respect to both key and secondary hypotheses. The results of testing the hypotheses revealed that they were either supported or inconclusive. For example, a few of these hypotheses (e.g., related to the improvement of on-time performance, and increase in ridership) were not supported by the data. The contribution of related technologies was not obvious due to the involvement of external factors (e.g., service change and operational improvements). Further, given that it takes considerable time for technologies such as those deployed at MST to stabilize, to become integral to agency operations and management, and to be accepted by staff, all hypotheses were re-examined in Phase III. However, as discussed below, in many cases, the conclusions were not markedly different from those that were determined in Phase II. Also, some hypotheses did not require retesting in Phase III since quantitative or anecdotal evidence obtained in Phase II had already provided sound conclusions.
The key hypotheses for this evaluation are:
- Hypothesis: The project will result in a reduction in operations
and planning costs and improved service planning. In Phase II, the
Evaluation Team found increases in annual revenue and annual revenue per
passenger-mile from the time of the technology implementation. However,
it was not obvious that the improvements were due to technology. Also, quantitative
estimates of benefits perceived by MST departments were not available for
most technologies. MST provided some basic estimates of savings from the
deployed technologies, such as the HASTUS scheduling software, and fuel
management systems highlighted in Section 3.7.5 of this report.
Also, as part of the staff interviews conducted in Phase II, MST provided anecdotal evidence of benefits perceived from ITS implementations (as of August 2008), which provides the basis for the fact that technology contributed to service planning and operations improvements. MST reported improvements in service planning due to the accuracy and reliability of the archived ACS data used in recent comprehensive operational analysis (COA) studies. Also, MST has been able to reduce the cost of data collection by reducing the manual effort required by COA studies (e.g., recruitment of temporary staff). Among other benefits, MST utilizes archived data from the ACS for analysis with the help of other tools such as ArcView and Microsoft Excel and Access for planning needs (e.g., using passenger count data for determining stop and shelter needs and appropriate locations).
This hypothesis was not revisited in Phase III since anecdotal evidence obtained in Phase II led to the conclusion that new technologies played a significant role in improving MST service. MST staff considers these improvements to be a result of changes made in the service based on recommendations resulting from ACS data analyses.
- Hypothesis: The project will result in improved on-time performance
of MST operation. The intent of this hypothesis was to determine if
there were improvements in schedule adherence due to the availability of
real-time vehicle information for dispatchers and supervisors. Also, the
Team wanted to evaluate the impact of MST's ability to adjust schedules
by utilizing the archived ACS data. However, this hypothesis could not
be supported with the results obtained from the quantitative data analysis.
The results were inconclusive in Phase II because MST had made several changes
in planning and operations during the time period selected for analysis.
Therefore, this hypothesis was re-examined in Phase III of the evaluation.
In Phase III, the Team analyzed a larger amount of AVL data (daily schedule adherence data from March 2005 through June 2009) but, similar to Phase II, the analysis results did not suggest any consistent trends for schedule adherence (i. e., consistent improvement after AVL deployment). It should be noted that, in most cases, vehicles were found to be on-time per MST standards (less than 5 minutes late). Also, the variability across the evaluation timeframe was well within two minutes. However, as concluded in Phase II, it cannot be confirmed that the use of the ACS system alone had an impact on MST's on-time performance. Various changes in the system implemented by MST during the evaluation timeframe could have contributed to the trends as well (Figure ES1).
Image details
Figure ES1. Timeline of Events Related to ACS Deployment
Even though the hypothesis cannot be supported with the results from the quantitative analysis, MST staff input obtained during Phase II interviews provided an understanding of the positive impacts of ACS. MST staff believes (see their detailed input in Section 3.1.2.2.3) that on-time performance has improved since the technology implementation and technology has contributed directly or indirectly to this improvement (e.g., by providing data for COA analysis and subsequent service restructuring).
- Hypothesis: The project will result in an increase in the reliability
of services. In Phase II, the Evaluation Team concluded that the reliability
of MST service should be measured by performing a qualitative assessment
of customers' perception of on-time performance. Based on the customer survey,
it can be concluded that at least 70 percent of the riders who were surveyed
in Monterey and Salinas are "satisfied" or "very satisfied" with the on-time
performance of MST service. Also, a similar percentage of riders are "satisfied"
or "very satisfied" with the ease of making transfers. These statistics
indicate that a significant number of MST riders find MST service reliable.
- Hypothesis: The project will enhance system productivity. This
hypothesis is supported by several statistics that serve as indicators of
productivity improvements (e.g., revenue per passenger mile and passenger
miles). These statistics were calculated in Phase II of the evaluation;
however, these statistics are inconclusive since it is not clear from the
productivity indicator data whether the improvements are due to technology
implementation or other changes in the organization.
This hypothesis was revisited by examining productivity data for fiscal years 2008 and 2009. A trend indicating some improvement in productivity was based on several of the statistics (e.g., revenue and revenue-hours), but in most cases, the statistics (e.g., boarding/revenue-hour) showed inconsistencies when compared to the trends observed in Phase II. These inconsistencies were mostly due to a decrease in MST ridership in fiscal years 2008 and 2009. Thus, the results of the quantitative data analyses were inconclusive.
However, as reported in Phase II, based on staff interviews, MST staff believes that the technology has assisted them in increasing their productivity by carrying more passengers during the same service hours with improved scheduling. MST also pointed out during staff interviews that a productivity increase may not be an absolute indicator of service improvements since a decrease in productivity sometimes benefits the organization by contributing to on-time service. For example, reducing the number of passengers on overcrowded buses can reduce dwell times at stops and subsequently improve schedule adherence of those buses.
- Hypothesis: The project will result in an improvement in maintenance
scheduling and planning. This hypothesis is supported by the information
provided by the maintenance department during on-site interviews conducted
as part of Phase II of the evaluation. MST staff believes that the MMS has
enabled them to track daily maintenance activities such as inventory control,
maintenance-workflow management, and fuel management. Other systems such
as the ACS and video surveillance system assist MST by enabling the agency
to review on-board system performance logs and by to monitor the quality
of maintenance work (through reviews of recorded videos), respectively.
As reported in Phase II, the Evaluation Team also wanted to evaluate the capabilities and impact of the remote diagnostics system implemented as part of the ACS. However, MST discontinued the remote diagnostics feature after initial use since the diagnostics were completely unreliable. MST was receiving an overwhelming number of false alarm messages, which led maintenance supervisors to ignore the remote diagnostics.
We did not revisit this hypothesis in Phase III since anecdotal evidence obtained from Phase II interviews were sufficient to conclude that there are positive impacts of the deployment of MMS and ACS on managing maintenance activities at MST.
The secondary hypotheses for this evaluation are:
- Hypothesis: The project will result in improved customer satisfaction.
Customer intercept surveys were conducted at the Monterey Transit Plaza
and Salinas Transit Center locations in mid-September 2009. The overall
satisfaction with MST was found to be very high with riders at both locations
— they provided ratings that averaged a score of 4 out of a possible 5.
Also, this overall average was reflected in riders' perceptions of MST's
on-time performance, the ease of making transfers, the frequency of service,
the hours of service, and the number of routes served. In fact, almost three-fourths
of the survey respondents said that they were "satisfied" or "very satisfied"
with each of the system attributes discussed above.
Further, riders were very positive in their views of the AVA system with 7 out of 10 (from both locations) reporting that they "agree" or "completely agree" that the announcements are loud enough, help them find their stop, and give them enough time to get ready before the stop. Salinas riders were positive in their views of the electronic signs. Again, almost three-fourths of surveyed riders reported that they "agreed" or "completely agreed" that the signs were useful, accurate, easy to understand, and conveyed the correct bus status information.
However, although a majority of riders appear to be satisfied with many aspects of the MST bus service, it appears that many of the riders are not yet making use of the benefits of (or have access to) the most recent MST technological advances such as real-time travel information and automated trip planning.
Thus, it can be concluded, based on the results of the intercept surveys, that the technology implementation has resulted in improved customer satisfaction.
- Hypothesis: The project will result in an increase in ridership.
The data provided by MST shows an increasing trend in ridership since 2003.
However, this information does not support the hypothesis as it is not clear
if the ridership increases have been due to just technology implementations.
This hypothesis was revisited during Phase III by analyzing customer satisfaction obtained from intercept surveys. As discussed above, a large number of MST riders are satisfied with the service.
- Hypothesis: The project will result in an improvement in driver and
passenger security. The Evaluation Team obtained several anecdotal
references that support this hypothesis through MST staff interviews conducted
in Phase II. The general perception at MST is that security systems have
helped them create a safer environment for MST riders and coach operators.2
MST has posted placards on-board vehicles that inform riders that they are
under video surveillance.
The local police consider MST buses as "mobile surveillance units." MST's ability to provide video evidence of criminal activities that involve MST buses with the help of on-board cameras has helped improve MST's relationship with the local police.
The on-board security cameras assist MST primarily in capturing evidence of any criminal activity. Additionally, these cameras have continually assisted MST in reducing the number of insurance claims submitted by passengers (e.g., related to slip and falls). Also, the video evidence assists MST in protecting drivers from being victims of false customer complaints.
Since the Phase II results discussed above provided evidence regarding the impact of surveillance systems on driver and passenger security, this hypothesis was not revisited in Phase III.
- Hypothesis: The project will result in a reduction in the travel
times of specific routes where TSP is deployed. This hypothesis was
not tested in Phase III since MST has not yet implemented TSP.
- Hypothesis: The project will help reduce response time for incidents
and emergency management. The hypothesis can be supported by information
provided by operations and maintenance staff from interviews conducted during
Phase II. However, the Team did not receive any quantitative estimates of
improvements in response time.
It was reported in Phase II that the availability of the ACS assists MST staff to track vehicle locations in real-time and enables them to send a supervisor to the accident site immediately. Also, MST drivers can select a specific text message from the list of canned messages on mobile data terminals (MDTs) and send that to the dispatcher to notify operations that there has been an incident enabling drivers to avoid making a voice call, if necessary. Text messaging capability has helped MST reduce the voice radio traffic by 60 percent. Also, starting fall 2008, MST supervisor will be able to connect remotely to the ACS to obtain any additional information that is needed while responding to an incident.
The ACS enables MST to provide and monitor evacuation services in the event of natural disasters such as the wildfires that happened during summer 2008. For example, during the recent wildfire event in Big Sur, MST was able to develop and manage task forces using MST vehicles through the use of the ACS.
Also, the number of incidents has been reduced in recent years subsequently contributing to reduced insurance premiums.
Since anecdotal and quantitative evidence obtained in Phase II provided sufficient information to conclude that there are improvements in incident response time, this hypothesis was not re-examined in Phase III.
- Hypothesis: The project will result in a reduction in vehicle hours.
The intent of this hypothesis was to test whether the technology has assisted
MST in reducing the number of revenue hours since 2003. Since annual revenue-hour
statistics do not show a consistent increasing or decreasing trend, this
hypothesis could not be supported. The number of revenue-hours decreased
between 2003 and 2005, but an increasing trend can be seen since 2005. This
inconsistency could be due to operational changes (e.g., addition of more
trips to a route) implemented by MST throughout the evaluation timeframe.
This hypothesis was not re-examined in Phase III since the trend of revenue vehicle-hours alone does not provide a reasonable indication of the impacts of technology on overall MST operations. Trends in revenue vehicle-hours need to be examined in conjunction with the number of passengers carried by those vehicles.
A reduction in deadhead-miles or deadhead-hours may be a reasonable indicator of improvements in transit operations, but the Team did not have access to the data required to measure the non-revenue miles or hours experienced by MST vehicles.
- Hypothesis: The project will reduce the number of customer complaints.
This hypothesis was not tested completely in Phase II since MST did not
have a record of the number of customer complaints for the "before" and
"after" cases. However, it was indicated by MST during the Phase II interviews
that the reduction in the number of complaints should not be an absolute
indicator of improved customer service. They have noticed that the number
of complaints have increased since MST developed an efficient process to
track and respond to a customer complaint. It is evident that customers
like to provide more comments and feedback only when they are assured of
receiving a response.
This hypothesis was not revisited in Phase III since no additional data could be obtained to make additional conclusions.
- Hypothesis: The project will result in improved facility security.
This hypothesis is supported by the facts and anecdotal references obtained
during on-site interviews as part of Phase II of the evaluation. The physical
facilities are equipped with cameras and the closed circuit television (CCTV)
technology that enable the real-time video monitoring of facilities by the
safety and security group. MST staff believes that the video monitoring
capability has assisted MST in reducing vandalism activities and creating
a more secure environment for MST riders waiting at transit centers.
Also, MST has implemented controlled access to its facilities with. The access is restricted to MST staff with a valid proximity card-based identification. This implementation is assisting MST in securing its physical facilities (headquarters and the transit centers) by restricting entrance to only authorized employees.
- Hypothesis: The project will establish a comprehensive reporting
system. This hypothesis cannot be supported with the available information
as the reporting process could not be evaluated "before" and "after" the
technology.
However, MST staff believes that the current reporting needs to be improved. The standard reports provided by various deployed systems (e.g., ACS, MMS, FAMIS) do not necessarily provide the information needed by MST employees. MST had hired an outside consultant to conduct a needs assessment for reporting; however, it was reported by MST during the Phase III onsite visit that the project could not be completed. Thus, this hypothesis is not supported with currently available information.
- Hypothesis: The project will result in reduced cases of false financial
claims. Through the staff interviews conducted in Phase II, MST provided
several anecdotal references (see Section 3.3.2.2) that serve as evidence
of financial savings due to the implementation and use of technologies,
primarily the video surveillance system. The video playback component of
the ACS also assists MST in responding to customer complaints related to
late arrivals or departures.
The on-board cameras have helped MST save money in various false complaints and accidental damage claims from passengers. MST reported that it recovered $70,000 during fiscal year 2007. However, before the installation of the video surveillance system, cost recovery was only in the order of $800 - $1800. Also, MST had to pay $3 million in settlements due to lack of sufficient evidence, which could have been mitigated with the help of an additional exterior camera on the bus.
Since the findings from Phase II were sufficient to support this hypothesis, it was not re-examined in Phase III.
Figure ES2 highlights the overall benefits found from this evaluation. Even though the Evaluation Team was not able to derive conclusions regarding the direct impact of technology for specific expected changes (e.g., increased ridership, improved on-time performance), anecdotal information obtained from MST staff has provided significant evidence to show that, so far, technology has made significant improvements in operations and planning. Also, survey findings reveal a high satisfaction among MST riders that can be partially attributed to changes in the system based on recent studies (which used ACS data for analyses), or new customer-centric technology implementations (e.g., AVA or real-time information signs). Generally, technologies have played a significant role in improving the efficiency of all departments as reported by the MST management. Improved efficiency has helped MST achieve cost savings as well. It is expected that even more benefits will be realized as these technologies are increasingly relied upon to perform specific operational and management functions.
| Summary of Overall Benefits Improved decision-making based on facts/information available from ACS and other systems. Organizational Improvements:
Current benefits have generated management support for future technology deployments (e.g., MST Board of Directors adopted technology as a priority). Return on Investment:
|
Technologies have primarily helped MST operations by enabling the agency to track vehicles in real-time and respond to incidents and emergency situations quickly. Also, HASTUS and the ACS, along with other tools, have helped MST improve planning which has subsequently helped the agency to run better operations (e.g., improved on-time performance resulting from route changes and schedule adjustments). The impact of the video surveillance system is significant as well because it has created a safer rider environment and has enabled MST to defend itself against lawsuit claims and reduce insurance-related costs. The maintenance department has experienced benefits through the MMS as it assists MST in improving the workflow process and quality control.
Figure ES3 highlights the overall lessons learned identified during this evaluation. The technology implementations provided an opportunity for MST to learn several lessons that will help them in future procurements. As MST plans to replace some of their systems (e.g., the ACS) with upgraded and better technologies, they believe that the prior deployment experience gives them enough confidence to procure from and negotiate with vendors, and manage the implementation of those technologies.
| Summary of Lessons Learned
|
1 FAMIS is a term used by MST to refer to its financial and accounting system. This is not an official product name for the system.
2 "Coach Operators" is the term used by MST for bus drivers.