ITS - Intelligent Transportation Systems Report ITS Home Page
Table of Contents

Executive Summary

Under the guidance of the Volpe Center, the California Department of Transportation (Caltrans) was one of two organizations selected to conduct a test bed in support of the United States Department of Transportation's (USDOT) SafeTrip-21 Initiative. The Caltrans' test bed is located in the San Francisco Bay Area and is referred to as the California Connected Traveler (CACT) Test Bed. The second test bed is along the I-95 Corridor. Under the direction and funding of the RITA ITS Joint Program Office, SAIC was selected to conduct an independent national evaluation of the technologies being deployed as part of the two test beds. This document presents the findings of the national evaluation of one of the three applications that comprise CACT Test Bed, the Networked Traveler-Transit/Smart Parking (NT-T/SP) application.

PROJECT BACKGROUND

The NT-T/SP test was developed by the California Partners for Advanced Transit and Highways (PATH) and the University of California (UC) at Berkeley. PATH is administered by the UC Berkeley Institute of Transportation Studies, with a mission to develop solutions to the problems of California's surface transportation systems through cutting edge research.

EVALUATION APPROACH

The evaluation approach was driven by a series of objectives that align with the USDOT's goals for the SafeTrip-21 initiative. Each objective was supported by corresponding hypotheses and measures of effectiveness, which in turn were used to identify specific data sources for the key activities for the evaluation. These data sources provide a detailed bank of knowledge relevant to the application, and a comprehensive look at lessons learned and the success of the NT-T/SP test. To achieve the evaluation objectives, the evaluation team implemented the following key activities:

SUMMARY OF FINDINGS

The NT-T/SP test involved the deployment of real-time transit information and trip planner applications that serve travelers along the US-101 corridor in San Francisco, California. With the support of several regional partners, PATH developed a set of web-based and smart phone-based applications branded as "PATH2Go" including the PATH2Go Trip Planner website, the PATH2Go Web Based Traveler Information website, and the PATH2Go Smart Phone Application. While the original scope of the test was to only provide parking and transit information for trips on the US-101 corridor, it was later expanded to provide real-time information to travelers across most transit agencies throughout the San Francisco Bay Area . This "one-stop shop" for traveler information was designed to help travelers make better pre-trip planning decisions in terms of mode selection and to serve as a tool for planning transit trips from any origin to any destination considering all transit options available. The applications also supported one another by allowing users to send personalized trip information from one of the website applications to the smart phone application so users could take the information with them.

The NT-T/SP test is unique in several ways. First, the test integrated a broad range of real-time multi-modal transportation information, including transit and parking information. While this is not unusual for traffic information, or for transit information provided by individual transit agencies, the NT-T/SP test integrated real time traffic, transit, and parking information across multiple agencies. Second, the test incorporated a technique referred to as "geo-fencing" to prevent the use of the smart phone application while driving. PATH developed a geo-fencing technique which attempted to determine which mode the traveler was using, in order to allow transit users to continue to receive updates while on the move but prevent drivers from using the information while driving.

The NT-T/SP test successfully provided real time transit information to internet and smart phone users for the duration of the evaluation, with close to no service disruption.

For smart phone users, the geo-fencing technique developed by PATH appears to have been effective in preventing use of the application while driving. The 5 mph threshold set by the PATH project team appeared to block the application for all smart phones when the GPS signal was available. However, if a smart phone is unable to obtain a GPS signal for significant portions of time, the application may be open for use. Although this was primarily due to limitations in the smart phone capabilities, this situation as well as others where satellite connections are limited does identify a disadvantage to designing geo-fencing functionality that relies heavily on smart phone GPS data to prevent distracted driving. On arterials and local roads where speeds are more variable due to greater occurrences of red lights at intersections, congestion, or stop and go traffic; the smart phone application is constantly being blocked and unblocked as driving speeds fluctuate above and below 5 mph. The ability to access the information on the application at low speeds or while stopped does present the opportunity for distraction to drivers regardless of whether or not users recognize the 5 mph design threshold.

The geo-fencing design can distinguish between users driving along a transit route versus users taking transit. While occasionally a user may be able to access the smart phone application while driving along a transit route, the time and distance constraints as well as the route matching and trip history requirements implemented into the geo-fencing design were mostly successful at preventing drivers from mimicking transit trips to gain access to the applications. In practice, the likelihood of users going to these lengths to gain access to the smart phone application is probably low. It is highly unlikely that a normal user would a) know enough about the server logic to know geo-fencing exceptions, b) go to such lengths to access the application while driving, or c) stumble across this scenario during normal travel behavior.

The evaluation team did observe several instances where the smart phone application was blocked while truly riding transit. Although unrelated to distracted driving, implementing a geo-fencing design into the smart phone application that primarily provides transit information may detract from the user experience of actual transit riders.

The PATH2Go applications experienced steady growth in registered users throughout the evaluation period while daily use of the PATH2Go Smart Phone Application and the PATH2Go website applications fluctuated throughout the evaluation period. The initial targeted marketing efforts of advertising the applications on the MTC 511 website, distributing a press release, and handing out flyers at transit stops/stations were effective in attracting registered users and increasing awareness of the website applications. The most significant increase in usage of the website applications came as the result of a Twitter post on a popular account followed by transit riders that use Caltrain. The website traffic generated by the Twitter post increased the total number of absolute unique visitors by 104 percent and the total number of visits by 66 percent in the span of five days.

Over the course of the evaluation period, which began with the launch on July 29, 2010 and ended November 15, 2010, the PATH2Go applications attracted over 900 registered users, 67 percent of which downloaded and used the smart phone application at least once and may have also used the website applications. 34 percent of the smart phone application users downloaded the app and only opened it once without returning; leaving 66 percent that used it more than once. According to a study produced by an iPhone application analytics firm called Pinch Media, it is actually common for smart phone applications to be abandoned after one use.1 33 percent of users registered on the project website, but only used the website applications. By the end of the evaluation period, the PATH2Go website applications attracted a total of 916 absolute unique visitors that accounted for 1,664 total visits to the website, and an average of 1.82 visits per user.

Usage analysis for each of the PATH2Go applications provided insight into the success of efforts put forth to market the application as well as insight into typical user behavior while accessing the applications. The usage analysis suggests that newer, more progressive forms of marketing like using social media websites such as Twitter can be significantly more effective in increasing awareness of real-time traveler information like the PATH2Go applications. Although still effective, more traditional forms of marketing like preparing a press release do not seem to generate the same level of exposure as quickly as a targeted social media effort without being covered by a major media source. Although the fluctuating website usage was greatly increased using Twitter, the impact was short-lived as usage quickly returned to its rolling pattern of approximately five to thirty website visits per day only a few days after the Caltrain tweet. While social media may have a greater ability than traditional marketing to attract a large number of visitors to a website quickly, the usefulness of or need for the information available on a website is generally what drives return visits.

With over 55 percent of users having only visited the website one time, regular or return users of the website were not as common. User frequency can often be an indicator of user acceptance and need for a website, but a number of factors may explain why less than half of website users returned for another visit. Possible explanations for this trend include:

With the exception of possible insight from the user survey, there is no definitive way to determine which of these explanations is responsible for the greater number of one-time users versus return users. Regardless, a low visit frequency may indicate low user acceptance or usefulness. Alternatively, it may simply be an indicator of how visitors use the information on the website relevant to the various possibilities listed above.

A web-based user survey was distributed to all registered users of the PATH2Go applications resulting in 121 survey responses. Respondents provided valuable insight into user perceptions and perceived benefits of the information provided by the applications. Most had heard about them from a web-based source (e.g., a web search, the MTC website, a link from another transportation site, or from an electronic message). A high proportion reported they heard it from a friend or colleague, which further indicates the importance of informing the public with "word of mouth" methods, especially as it comes from trusted sources.

Usage of the tools showed relatively low usage patterns, especially for the web-based applications. At least one-half of respondents reported never having used the Trip Planner or Traveler Information site and approximately one-fourth had not used either site in the week before completing the survey. Use of the Smart Phone application was slightly higher with only approximately one-half of respondents reporting they had never used it or had not used it in the past week. One-half of the respondents reported they use the applications to plan their transit trips and these are for regular trips (e.g., commuting to work or school).

While-half of the users reported having received the "Warning: Application Disabled While Driving" message, two-thirds of those who got the warning reported that it occurred relatively infrequently – less than 25 percent of the time. However, when it was received it was reported as annoying by 70 percent of users. This observation was borne out by respondent comments that focused on trying to use the application while riding on transit vehicles or as passengers in automobiles and "being blocked."

When considering the attributes and value of all three PATH2Go applications, users were generally pleased with them though there were areas where the applications could be improved such as retrieving the information. For instance, approximately one-third reported that it was difficult to find what they were looking for and that the information was not well presented on the applications. However, at least 50 percent of respondents reported the applications provided them with the information they were looking for and stated that the information on the applications is valuable. Almost 40 percent reported the information is well organized.

There was also strong agreement that the ability to access multiple transit services and having reliable arrival and departure information was important. Finally, most respondents reported that having the transit information available to them made them more confident about using transit, though not as many respondents went as far to say that the application led them to choosing an alternative mode to their usual transportation method.

The PATH2Go tools will remains operational through 2011, with new Caltrans funding.

CONCLUSIONS

Overall, the NT-T/SP test was a success, as witnessed by the large number of registered users and website visits, and by the extent to which registered users provided positive feedback (including infrequent users and those who wished to provide constructive suggestions on how to enhance the initial, beta, system.)

The NT-T/SP test resulted in significant insight into the understanding of distributing real time transit information. The test demonstrated the ability to integrate transit, traffic, and parking information across multiple agencies in real time. The test highlighted the potential for distributing personalized information via the internet and smart phones, and to do so without causing driver distraction.

There were many technical challenges associated with the test, one example being the geo-fencing technique. While geo-fencing has existing for some time, the requirements for this test were different in that it had to prevent use of the smart phone application in one mode (cars) while allowing it in others (train and bus.) This was achieved by PATH, although arguably to the detriment of the overall user experience, essentially because users found it to be annoying and, at times, inappropriate.

With few exceptions, the geo-fencing technique developed by PATH was effective at blocking the use of the smart phone application in cars, and by extension was therefore able to minimize distracted driving. However, the technique was not foolproof, in part because it depends upon smart phones being able to access a GPS signal. Without this signal, the geo-fencing technique will be unable to calculate whether the user is moving faster than the 5 mph threshold. Users with a detailed knowledge of the design of the geo-fencing technique may be able to mimic a transit vehicle while travelling in a car, although this is considered a remote possibility. What is more likely is for a user to be blocked from using the smart phone application while riding transit, as the user may not have planned a trip in accordance with the requirements of the geo-fencing design. Under these circumstances the application will assume any such riders are actually in a car, and consequently block access to the application.

The NT-T/SP test adopted a design that implemented the geo-fencing technique that used a server-based method, rather than using a client-based approach on the smart phone devices. With any application design for smart phones, developers can decide whether to host the code and source information for certain application functionality on the server-side of the application or the client-side of the application. In other words, the decision-making can either take place on servers hosted by the developers or on the smart phone itself. The geo-fencing design was integrated into the PATH2Go Smart Phone Application using server-side logic, which allowed for a thin client-side design. Implementing a server side geo-fencing design prevented the design team from having to address differences in the operating systems of Windows Mobile, iPhone, and Android smart phones that could have an effect on geo-fencing performance. Client-based functionality would have required wrapping up all of the code and sources into the application download, which would have been demanding on the smart phone in terms of application size, processor speed, and battery life depending on complexity.

Overall, the PATH2Go tools experienced steady growth in the number of registered users - a possible indication of user acceptance. However it was clear that multiple approaches are necessary to raise awareness of the tools, rather than relying on a single approach. Equally, it was clear that repeated measures are necessary to retain momentum in growth. It is likely that each marketing technique affects each user in a different way; therefore, many different efforts are usually required to reach out to a wide variety of users.

Well over half of respondents "strongly agreed" or "agreed" that the information was valuable and this is contrasted with only 14 percent who "strongly disagreed" or "disagreed." In fact, when using the applications, respondents felt that having information for multiple transit services was very useful. Almost two-thirds "strongly agreed" or "agreed" that supplying information for multiple services (e.g., Caltrain, BART, SF Muni) was very helpful. This was especially true for those trips that were non-routine and may have involved multiple services or services they normally did not use.

Additionally, there was relatively strong agreement from respondents that the real-time departure and arrival information supplied on the application was valid. While one-fourth reported they did not have enough experience to rate this, 40 percent reported they "strongly agreed' or "agreed" that the schedule information was reliable. Only 12 percent "strongly disagreed" or "disagreed" that the information was reliable.



1 Gonsalves, Antone. "Apple App Store Downloads Often Abandoned." InformationWeek. InformationWeek Business Technology Network. 20 Feb. 2009. Web. 6 Dec 2010. [ Return to note 1. ]

Previous | Next