PART III: SUMMARY AND CONCLUSIONS
6. Summary and Conclusions
6.1 SUMMARY
The NT-T/SP test involved the deployment of real-time transit information and trip planner applications that serve travelers along the US-101 corridor in San Francisco, California. With the support of several regional partners, PATH developed a set of web-based and smart phone-based applications branded as "PATH2Go." The primary purpose of the test was to provide information to travelers in real-time across all transit agencies that serve the US-101 corridor. This "one-stop shop" for traveler information was designed to help travelers make better pre-trip planning decisions in terms of mode selection and to serve as a tool for planning transit trips from any origin to any destination considering all transit options available.
The NT-T/SP test is unique in several ways. First, the test integrated a broad range of real-time multi-modal transportation information, including transit and parking information. While this is not unusual for traffic information, or for transit information provided by individual transit agencies, the NT-T/SP test integrated real time traffic, transit, and parking information across multiple agencies. Second, the test incorporated a technique referred to as "geo-fencing" to prevent the use of the smart phone application while driving. PATH developed a geo-fencing technique which attempted to determine which mode the traveler was using, in order to allow transit users to continue to receive updates while on the move but prevent drivers from using the information while driving.
The NT-T/SP test successfully provided real time transit information to internet and smart phone users for the duration of the evaluation, with close to no service disruption.
For smart phone users, the geo-fencing technique developed by PATH appears to have been effective in preventing use of the application while driving. The 5 mph threshold set by the PATH project team appeared to block the application for all smart phones when the GPS signal was available. However, if a smart phone is unable to obtain a GPS signal for significant portions of time, the application may be open for use. Although this was primarily due to limitations in the smart phone capabilities, this situation as well as others where satellite connections are limited does identify a disadvantage to designing geo-fencing functionality that relies heavily on smart phone GPS data to prevent distracted driving. On arterials and local roads where speeds are more variable due to greater occurrences of red lights at intersections, congestion, or stop and go traffic; the smart phone application is constantly being blocked and unblocked as driving speeds fluctuate above and below 5 mph. The ability to access the information on the application at low speeds or while stopped does present the opportunity for distraction to drivers regardless of whether or not users recognize the 5 mph design threshold.
The geo-fencing design can distinguish between users driving along a transit route versus users taking transit. While occasionally a user may be able to access the smart phone application while driving along a transit route, the time and distance constraints as well as the route matching and trip history requirements implemented into the geo-fencing design were mostly successful at preventing drivers from mimicking transit trips to gain access to the applications. In practice, the likelihood of users going to these lengths to gain access to the smart phone application is probably low. It is highly unlikely that a normal user would a) know enough about the server logic to know geo-fencing exceptions, b) go to such lengths to access the application while driving, or c) stumble across this scenario during normal travel behavior.
The evaluation team did observe several instances where the smart phone application was blocked while truly riding transit. Although unrelated to distracted driving, implementing a geo-fencing design into the smart phone application that primarily provides transit information may detract from the user experience of actual transit riders.
The PATH2Go applications experienced steady growth in registered users throughout the evaluation period while daily use of the PATH2Go Smart Phone Application and the PATH2Go website applications fluctuated throughout the evaluation period. The initial targeted marketing efforts of advertising the applications on the MTC 511 website, distributing a press release, and handing out flyers at transit stops/stations were effective in attracting registered users and increasing awareness of the website applications. The most significant increase in usage of the website applications came as the result of a Twitter post on a popular account followed by transit riders that use Caltrain. The website traffic generated by the Twitter post increased the total number of absolute unique visitors by 104 percent and the total number of visits by 66 percent in the span of five days.
Over the course of the evaluation period, which began with the launch on July 29, 2010 and ended November 15, 2010, the PATH2Go applications attracted over 900 registered users, 67 percent of which downloaded and used the smart phone application at least once and may have also used the website applications. 34 percent of the smart phone application users downloaded the app and only opened it once without returning; leaving 66 percent that used it more than once. 33 percent of users registered on the project website, but only used the website applications. By the end of the evaluation period, the PATH2Go website applications attracted a total of 916 absolute unique visitors that accounted for 1,664 total visits to the website, and an average of 1.82 visits per user.
The usage analysis suggests that newer, more progressive forms of marketing like using social media websites such as Twitter can be significantly more effective in increasing awareness of real-time traveler information like the PATH2Go applications. Although still effective, more traditional forms of marketing like preparing a press release do not seem to generate the same level of exposure as quickly as a targeted social media effort without being covered by a major media source. Although the fluctuating website usage was greatly increased using Twitter, the impact was short-lived as usage quickly returned to its rolling pattern of approximately five to thirty website visits per day only a few days after the Caltrain tweet. While social media may have a greater ability than traditional marketing to attract a large number of visitors to a website quickly, the usefulness of or need for the information available on a website is generally what drives return visits.
With over 55 percent of users having only visited the website one time, regular or return users of the website were not as common. User frequency can often be an indicator of user acceptance and need for a website, but a number of factors may explain why less than half of website users returned for another visit. Possible explanations for this trend include:
- Length of Evaluation Period.
- User Travel Frequency using Transit.
- Usefulness of Real-time Information.
- Perceived Value of Website.
- Website Functionality.
With the exception of possible insight from the user survey, there is no definitive way to determine which of these explanations is responsible for the greater number of one-time users versus return users. Regardless, a low visit frequency does not necessarily indicate low user acceptance or usefulness, but may more so be an indicator of how visitors use the information on the website.
Registered users provided valuable feedback on the PATH2Go tools. Most had heard about them from a web-based source (e.g., a web search, the MTC website, a link from another transportation site, or from an electronic message). A high proportion reported they heard it from a friend or colleague, which further indicates the importance of informing the public with "word of mouth" methods, especially as it comes from trusted sources.
Usage of the tools showed relatively low usage patterns, especially for the web-based applications. At least one-half of respondents reported never having used the Trip Planner or Traveler Information site and approximately one-fourth had not used either site in the week before completing the survey. Use of the Smart Phone application was slightly higher with only approximately one-half of respondents reporting they had never used it or had not used it in the past week. One-half of the respondents reported they use the applications to plan their transit trips and these are for regular trips (e.g., commuting to work or school).
While-half of the users reported having received the "Warning: Application Disabled While Driving" message, two-thirds of those who got the warning reported that it occurred relatively infrequently - less than 25 percent of the time. However, when it was received it was reported as annoying by 70 percent of users. This observation was borne out by respondent comments that focused on trying to use the application while riding on transit vehicles or as passengers in automobiles and "being blocked."
When considering the attributes and value of all three applications, users were generally pleased with them though, there were areas where the applications could be improved such as retrieving the information. For instance, approximately one-third reported the applications were not trouble free; it was not easy to find the information they were looking for; and the information was not well presented. However, one-half of respondents reported the applications provided them with the information they were looking for and slightly more than half reported that the information on the applications is valuable. Almost 40 percent also reported the information is well organized.
There was also strong agreement that the ability to access multiple transit services and having reliable arrival and departure information was important. Finally, most respondents reported that having the transit information available to them made them more confident about using transit, though there was not overwhelming evidence that having this information would lead to them choosing an alternative mode to their usual transportation method.
The PATH2Go tools will remains operational through 2011, with new Caltrans funding.
6.2 CONCLUSIONS
The conclusions that follow are grouped according to the evaluation objectives for the NT-T/SP test.
6.2.1 Observe the consumer response to the NT-T/SP application
Overall, the NT-T/SP test was a success, as witnessed by the large number of registered users and website visits, and by the extent to which registered users provided positive feedback (including infrequent users and those who wished to provide constructive suggestions on how to enhance the initial, beta, system.)
6.2.2 Understand the technical and institutional issues associated with distributing multi-modal information to smart phone users
The NT-T/SP test resulted in significant insight into the understanding of distributing real time transit information. The test demonstrated the ability to integrate transit, traffic, and parking information across multiple agencies in real time. The test highlighted the potential for distributing personalized information via the internet and smart phones, and to do so without causing driver distraction.
There were many technical challenges associated with the test, most notably the geo-fencing technique. While geo-fencing has existing for some time, the requirements for this test were different in that it had to prevent use of the smart phone application in one mode (cars) while allowing it in others (train and bus.) This was achieved by PATH, although arguably to the detriment of the overall user experience, essentially because users found it to be annoying and, at times, inappropriate.
6.2.3 Test the ability of geo-fencing as a method to prevent distracted driving
With few exceptions, the geo-fencing technique developed by PATH was effective at blocking the use of the smart phone application in cars, and by extension was therefore able to minimize distracted driving. However, the technique was not foolproof, in part because it depends upon smart phones being able to access a GPS signal. Without this signal, the geo-fencing technique will be unable to calculate whether the user is moving faster than the 5 mph threshold. Users with a detailed knowledge of the design of the geo-fencing technique may be able to mimic a transit vehicle while travelling in a car, although this is considered a remote possibility. What is more likely is for a user to be blocked from using the smart phone application while riding transit, as the user may not have planned a trip in accordance with the requirements of the geo-fencing design. Under these circumstances the application will assume any such riders are actually in a car, and consequently block access to the application.
6.2.4 Understand the development process and institutional issues associated with implementing a server-based geo-fencing method versus a client-based method on mobile devices
The NT-T/SP test adopted a design that implemented the geo-fencing technique that used a server-based method, rather than using a client-based approach on the smart phone devices. With any application design for smart phones, developers can decide whether to host the code and source information for certain application functionality on the server-side of the application or the client-side of the application. In other words, the decision-making can either take place on servers hosted by the developers or on the smart phone itself. The geo-fencing design was integrated into the PATH2Go Smart Phone Application using server-side logic, which allowed for a thin client-side design. Implementing a server side geo-fencing design prevented the design team from having to address differences in the operating systems of Windows Mobile, iPhone, and Android smart phones that could have an effect on geo-fencing performance. Client-based functionality would have required wrapping up all of the code and sources into the application download, which would have been demanding on the smart phone in terms of application size, processor speed, and battery life depending on complexity.
6.2.5 Measure usage of the NT-T/SP application
Overall, the PATH2Go tools experienced steady growth in the number of registered users - a possible indication of user acceptance. However it was clear that multiple approaches are necessary to raise awareness of the tools, rather than relying on a single approach. Equally, it was clear that repeated measures are necessary to retain momentum. It is likely that each marketing technique affects each user in a different way; therefore, many different efforts are usually required to reach out to a wide variety of users.
6.2.6 Analyze the perceived accuracy and usefulness of mode shift alerts and en-route transit information
Well over half of respondents "strongly agreed" or "agreed" that the information was valuable and this is contrasted with only 14 percent who "strongly disagreed" or "disagreed." In fact, when using the applications, respondents felt that having information for multiple transit services was very useful. Almost two-thirds "strongly agreed" or "agreed" that supplying information for multiple services (e.g., Caltrain, BART, SF Muni) was very helpful. This was especially true for those trips that were non-routine and may have involved multiple services or services they normally did not use.
Additionally, there was relatively strong agreement from respondents that the real-time departure and arrival information supplied on the application was valid. While one-fourth reported they did not have enough experience to rate this, 40 percent reported they "strongly agreed' or "agreed" that the schedule information was reliable. Only 12 percent "strongly disagreed" or "disagreed" that the information was reliable.