6. Conclusions
The evaluation of the ITS deployment at MST has resulted in the identification of key factors about MST's experience related to the procurement, implementation, management, and utilization of ITS technologies. Also, the evaluation identified the impacts of the technology on various departments at MST. Further, the customer satisfaction survey conducted in Phase III of the evaluation helped measure the customer perception of the impacts of the technology deployment at MST.
The following paragraphs provide a summary of the evaluation findings with respect to both key and secondary hypotheses. The results of testing the hypotheses revealed that they were either supported or inconclusive. For example, a few of these hypotheses (e.g., related to the improvement of on-time performance, and increase in ridership) were not supported by the data. The contribution of related technologies was not obvious due to involvement of external factors (e.g., service change, and operational improvements). Further, given that it takes considerable time for technologies, such as those deployed at MST, to stabilize, to become integral to agency operations and management, and to be accepted by staff, all hypotheses were re-examined in Phase III. However, as discussed below, in many cases, the conclusions were not markedly different from those that were determined in Phase II. Also, some hypotheses did not require retesting in Phase III since quantitative or anecdotal evidence obtained in Phase II had already provided sound conclusions.
The key hypotheses for this evaluation are as follows:
- Hypothesis: The project will result in a reduction in operations
and planning costs and improved service planning. In Phase II, the
Evaluation Team found increases in annual revenue and annual revenue per
passenger-mile from the time of the technology implementation. However,
it was not obvious that the improvements have been due to technology. Also,
quantitative estimates of the benefits to MST departments were not available
for most technologies. MST did provide some basic estimates of savings from
technologies, such as from the deployment of the scheduling software and
fuel management systems highlighted in Section
3.7.5 of this report.
Also, as part of the staff interviews conducted in Phase II, MST provided anecdotal evidence of the benefits perceived to result from ITS implementations (as of August 2008), which provides the basis for the assertion that technology contributed to service planning and operations improvements. MST reported improvements in service planning due to the accuracy and reliability of the archived ACS data used in recent comprehensive operational analysis (COA) studies. Also, MST has been able to reduce the cost of data collection by reducing the manual effort required by COA studies (e.g., recruitment of temporary staff). Among other benefits, MST uses archived data from the ACS for analysis with the help of other tools such as ArcView, Microsoft Excel, and Microsoft Access for planning needs (e.g., using passenger count data for determining stop and shelter needs and identifying appropriate shelter locations).
This hypothesis was not revisited in Phase III since anecdotal evidence obtained in Phase II led to the conclusion that new technologies played a significant role in improving MST service. MST staff considers these improvements to be a result of changes made in the service based on recommendations resulting from ACS data analyses.
- Hypothesis: The project will result in improved on-time performance
of MST operation. The intent of this hypothesis was to determine if
there were improvements in schedule adherence due to the availability of
real-time vehicle information for dispatchers and supervisors. Also, the
Team wanted to evaluate the impact of MST's ability to adjust schedules
by utilizing the archived ACS data. However, this hypothesis could
not be supported with the results obtained from the quantitative data analysis.
The results were inconclusive in Phase II because MST had made several changes
in planning and operations during the time period selected for analysis.
Therefore, this hypothesis was re-examined in Phase III of the evaluation.
In Phase III, the Team analyzed a larger amount of AVL data (daily schedule adherence data from March 2005 through June 2009) but, similar to Phase II, the analysis results did not suggest any consistent trends for schedule adherence (e.g., consistent improvement after AVL deployment). It should be noted that, in most cases, vehicles were found to be on-time per MST standards (less than 5 minutes late). Also, the variability across the evaluation timeframe was well within 2 minutes. However, as concluded in Phase II, it cannot be confirmed that the use of the ACS system alone had an impact on MST's on-time performance. Various changes in the system implemented by MST during the evaluation timeframe (see Figure 9 for a series of changes) could have contributed to the trends as well.
Even though the hypothesis cannot be supported with the results from the quantitative analysis, MST staff input obtained during Phase II interviews provided an understanding of the positive impacts of ACS. MST staff believes (see their detailed input in Section 3.1.2.2.3) that on-time performance has improved since the technology implementation and technology have contributed directly or indirectly to this improvement (e.g., by providing data for COA analysis and subsequent service restructuring).
- Hypothesis: The project will result in an increase in the reliability
of services. In Phase II, we concluded that the reliability of MST
service should be measured by performing a qualitative assessment of customers'
perception of on-time performance. Based on the customer survey, it can
be concluded that at least 70 percent of the riders who were surveyed in
Monterey and Salinas are "satisfied" or "very satisfied" with the on-time
performance of MST service. Also, a similar percentage of riders are "satisfied"
or "very satisfied" with the ease of making transfers. These statistics
indicate that a significant number of MST riders find MST service reliable.
- Hypothesis: The project will enhance system productivity. This
hypothesis is supported by several statistics that serve as indicators of
productivity improvements (e.g., revenue per passenger-miles and passenger-miles).
These statistics were calculated in Phase II of the evaluation. However,
these statistics are inconclusive since it is not clear from the productivity
indicator data whether the improvements are due to technology implementation
or other changes in the organization.
This hypothesis was revisited by examining productivity data for fiscal years 2008 and 2009. A trend indicating some improvement in productivity was based on several of the statistics (e.g., revenue and revenue-hours), but in most cases, the statistics (e.g., boarding/revenue-hour) showed inconsistencies when compared to the trends observed in Phase II. These inconsistencies were mostly due to a decrease in MST ridership in fiscal years 2008 and 2009. Thus, the results of the quantitative data analyses were inconclusive.
However, as reported in Phase II, based on staff interviews, MST staff believes that the technology has assisted in increasing MST's productivity by allowing the agency to carry more passengers during the same service hours with improved scheduling. MST also pointed out during staff interviews that a productivity increase may not be an absolute indicator of service improvements since a decrease in productivity sometimes benefits the organization by helping it provide on-time service. For example, reducing the number of passengers on overcrowded buses can reduce dwell times at stops, and subsequently improve the schedule adherence of those buses.
- Hypothesis: The project will result in an improvement in maintenance
scheduling and planning. This hypothesis is supported by the information
provided by the maintenance department during on-site interviews conducted
as part of Phase II of the evaluation. MST staff believes that the MMS has
enabled the agency to track daily maintenance activities such as inventory
control, maintenance-workflow management, and fuel management. Other systems
such as the ACS and the video surveillance system assist MST by enabling
staff to review on-board system performance logs and to monitor the quality
of maintenance work (through reviews of recorded videos), respectively.
As reported in Phase II, the Team also wanted to evaluate the capabilities and impact of the remote diagnostics system implemented as part of the ACS. However, MST discontinued the remote diagnostics feature after initial use since the diagnostics were completely unreliable. MST was receiving an overwhelming number of false alarm messages which led them to ignore the remote diagnostics.
We did not revisit this hypothesis in Phase III since anecdotal evidence obtained from Phase II interviews were sufficient to conclude that there are positive impacts of the deployment of MMS and ACS on managing maintenance activities at MST.
The secondary hypotheses for this evaluation include the following:
- Hypothesis: The project will result in improved customer satisfaction:
Customer intercept surveys were conducted at the Monterey Transit Plaza
and Salinas Transit Center locations in mid-September 2009. The overall
satisfaction with MST was found to be very high with riders at both locations
- they provided ratings that averaged a score of 4 out of a possible 5.
Also, this overall average was reflected in riders' perceptions of MST's
on-time performance, the ease of making transfers, the frequency of service,
the hours of service, and the number of routes served. In fact, almost three-fourths
of the survey respondents said that they were "satisfied" or "very satisfied"
with each of the system attributes discussed above.
Further, riders were very positive in their views of the AVA system, with 7 out of 10 (from both locations) reporting that they "agree" or "completely agree" that the announcements are loud enough, help them find their stop, and give them enough time to get ready before the stop. Salinas riders were positive in their views of the electronic signs. Again, almost three-fourths of surveyed riders reported that they "agreed" or "completely agreed" that the signs were useful, accurate, easy to understand, and conveyed the correct bus status information.
However, although a majority of riders appear to be satisfied with many aspects of the MST bus service, it appears that many of the riders are not yet making use of the benefits of (or have access to) the most recent MST technological advances such as real-time travel information and automated trip planning.
Thus, it can be concluded, based on the results of the intercept surveys, that the technology implementation has resulted in improved customer satisfaction.
- Hypothesis: The project will result in an increase in ridership.
The data provided by MST shows an increasing trend in ridership since 2003.
However, this information does not support the hypothesis as it is not clear
if the ridership increases have been due to just technology implementations.
This hypothesis was revisited during Phase III by analyzing customer satisfaction obtained from intercept surveys. As discussed above, a large number of MST riders are satisfied with the service.
- Hypothesis: The project will result in an improvement in driver and
passenger security. The Evaluation Team obtained several anecdotal
references that support this hypothesis through MST staff interviews conducted
in Phase II. The general perception at MST is that security systems have
helped them create a safer environment for MST riders and coach operators.
MST has posted placards on-board vehicles that inform riders that they are
under video surveillance.
The local police consider MST buses as "mobile surveillance units." MST's ability to provide video evidence of criminal activities that involve MST buses with the help of on-board cameras has helped them improve their relationship with the local police.
The on-board security cameras assist MST in primarily capturing evidence of any criminal activity. Additionally, these cameras have continually assisted MST in reducing the number of insurance claims submitted by passengers (e.g., related to slip and falls). Also, the video evidence assists MST in protecting its drivers from being victims of false customer complaints.
Since the Phase II results discussed above provided evidence regarding the impact of surveillance systems on driver and passenger security, this hypothesis was not revisited in Phase III.
- Hypothesis: The project will result in a reduction in the travel
times of specific routes where TSP is deployed. This hypothesis was
not tested in Phase III since MST has not yet implemented TSP.
- Hypothesis: The project will help reduce response time for incidents
and emergency management. The hypothesis can be supported by information
provided by operations and maintenance staff from interviews conducted during
Phase II. However, the Team did not receive any quantitative estimates of
improvements in response time.
It was reported in Phase II that the availability of the ACS assists MST staff to track vehicle locations in real-time and enables them to send a supervisor to the accident site immediately. Also, MST drivers can select a specific text message from the list of pre-loaded messages on mobile data terminals (MDTs) and send it to the dispatcher to notify operations that there has been an incident; in this way, the driver can avoid making a voice call, if necessary. Text messaging capability has helped MST reduce the voice radio traffic by 60 percent. Also, starting fall 2008, MST supervisor will be able to connect remotely to the ACS to obtain any additional information that is needed while responding to an incident.
The ACS enables MST to provide and monitor evacuation services in the event of natural disasters, such as the wildfires that struck during summer 2008. During the recent wildfire event in Big Sur, MST was able to develop and manage task forces using MST vehicles through the use of the ACS.
Also, the number of incidents has been reduced in recent years, subsequently contributing to reduced insurance premiums.
Since anecdotal and quantitative evidence obtained in Phase II provided sufficient information to conclude that there are improvements in incident response time, this hypothesis was not re-examined in Phase III.
- Hypothesis: The project will result in a reduction in vehicle hours.
The intent of this hypothesis was to test whether the technology has assisted
MST in reducing the number of revenue hours since 2003. Since annual revenue-hour
statistics do not show a consistent increasing or decreasing trend, this
hypothesis could not be supported. The number of revenue-hours decreased
between 2003 and 2005, but an increasing trend can be seen since 2005. This
inconsistency could be due to operational changes (e.g., addition of more
trips to a route) implemented by MST throughout the evaluation timeframe.
This hypothesis was not re-examined in Phase III since the trend of revenue vehicle-hours alone does not provide a reasonable indication of the impacts of technology on overall MST operations. Trends in revenue vehicle-hours need to be examined in conjunction with the number of passengers carried by those vehicles.
A reduction in deadhead-miles or deadhead-hours may be a reasonable indicator of improvements in transit operations, but the Evaluation Team did not have access to the data required to measure the non-revenue miles or hours experienced by MST vehicles.
- Hypothesis: The project will reduce the number of customer complaints.
This hypothesis was not tested completely in Phase II since MST did not
have a record of the number of customer complaints for the "before" and
"after" cases. However, it was indicated by MST during the Phase II interviews
that the reduction in the number of complaints should not be an absolute
indicator of improved customer service. MST staff has noticed that the number
of complaints have increased since MST developed an efficient process to
track and respond to a customer complaint. It is evident that customers
like to provide more comments and feedback only when they are assured of
receiving a response.
This hypothesis was not revisited in Phase III since no additional data could be obtained to make additional conclusions.
- Hypothesis: The project will result in improved facility security.
This hypothesis is supported by the facts and anecdotal references obtained
during on-site interviews as part of Phase II of the evaluation. The physical
facilities are equipped with cameras and the closed circuit television (CCTV)
technology that enables the real-time video monitoring of facilities by
the safety and security group. MST staff believes that the video monitoring
capability has assisted the agency in reducing vandalism activities and
creating a more secure environment for MST riders waiting at transit centers.
Also, MST has implemented controlled access to its facilities. The access is restricted to MST staff with a valid proximity card-based identification. This implementation is assisting MST in securing its physical facilities (headquarters and the transit centers) by restricting entrance to only authorized employees.
- Hypothesis: The project will establish a comprehensive reporting
system. This hypothesis cannot be supported with the available information
as the reporting process could not be evaluated "before" and "after" the
technology.
However, MST staff believes that current reporting needs to be improved. The standard reports provided by various deployed systems (e.g., ACS, MMS, FAMIS) do not necessarily provide the information needed by MST employees. MST had hired an outside consultant to conduct a needs assessment for reporting. However, it was reported by MST during the Phase III onsite visit that the project could not be completed. Thus, this hypothesis is not supported with currently available information.
- Hypothesis: The project will result in reduced cases of false financial
claims. Through the staff interviews conducted in Phase II, MST provided
several anecdotal references (see Section
3.3.2.2) that serve as evidence of financial savings due to the implementation
and use of technologies, primarily the video surveillance system. The video
playback component of the ACS also assists MST in responding to customer
complaints related to late arrivals or departures.
The on-board cameras have helped MST save money on various false complaints and accidental damage claims from passengers. MST reported that it recovered $70,000 during fiscal year 2007. However, before the installation of the video surveillance system, cost recovery was only in the order of $800- $1800. Also, MST had to pay $3 million in settlements due to lack of sufficient evidence, which could have been mitigated with the help of an additional exterior camera on the bus.
Since the findings from Phase II were sufficient to support this hypothesis, it was not re-examined in Phase III.
Figure 77 highlights the overall benefits found during this evaluation. Even though the Evaluation Team was not able to derive conclusions regarding the direct impact of technology for specific expected changes (e.g., increased ridership, improved on-time performance), anecdotal information obtained from MST staff has provided significant evidence to show that, so far, the technology has made significant improvements in operations and planning. Also, survey findings reveal a high satisfaction among MST riders which can be partially attributed to changes in the system based on recent studies (which used ACS data for analyses), or new customer-centric technology implementations (e.g., AVA or real-time information signs). Generally, technologies have played a significant role in improving the efficiency of all departments as reported by the MST management. Improved efficiency has helped MST achieve cost savings as well. It is expected that even more benefits will be realized as these technologies are increasingly relied upon to perform specific operational and management functions.
| Summary of Overall Benefits Improved decision-making based on facts/information available from ACS and other systems. Organizational Improvements:
Current benefits have generated management support for future technology deployments (e.g., MST Board of Directors adopted technology as a priority). Return on Investment:
|
The ITS technologies implemented have primarily assisted MST operations by enabling real-time vehicle tracking and quick response to incidents and emergency situations. Also, HASTUS and the ACS, along with other tools, have helped MST improve its planning, which has subsequently helped the agency to run better operations (e.g., improved on-time performance resulting from route changes and schedule adjustments). The impact of the video surveillance system is also significant because it has created a safer rider environment and has enabled MST to defend itself against lawsuit claims and to reduce insurance-related costs. The maintenance department has experienced benefits through the MMS as it assists MST in improving the workflow process and quality control.
Figure 78 highlights the overall lessons learned identified during this evaluation. The technology implementations provided an opportunity for MST to learn several lessons that will help the agency in future procurements. As MST plans to replace some of its systems (e.g., the ACS) with upgraded and better technologies, agency officials believe that the prior deployment experience gives it enough confidence to procure from and negotiate with vendors and manage the implementation of new technologies.
| Summary of Lessons Learned
|