2.  EVALUATION APPROACH AND OBJECTIVES

2.1 Introduction

The evaluation of the Central Puget Sound Regional Fare Coordination project responds to the objectives of the ITS JPO as well as the partners of the RFC Project.  When reviewing earmarked projects to select candidates for evaluation, the USDOT seeks to identify those projects that meet a minimum set of criteria that include:

• Potential to fill selected data gaps;
• Likelihood that the project will be completed in a reasonable timeframe, thereby yielding timely evaluation results;
• Clearly defined integration of project components;
• Adequacy of funding to achieve project objectives; and
• Relevance of the project to other specific FHWA ITS program needs.

The USDOT selected this particular RFC Project for evaluation because it represents one of the nation’s most visible and leading efforts to establish a multi-agency electronic fare card system, and because of the high level of national interest in establishing other programs of this type.

A case study evaluation was considered the most useful approach to apply in this instance.  Case study evaluation design involves documentation of the institutional processes established and followed by the project participants, as opposed to a more traditional before-after evaluation design that compares data collected prior to and following the implementation of a project in order to assess its impacts.  Because the time horizon for full implementation of the Central Puget Sound RFC Project is long, and evaluation results useful to others are needed as soon as possible, a case study approach was felt to be most appropriate and beneficial.[7]

The evaluation examines the regional governance issues associated with the development of the project.  It also seeks to document institutional processes that extend beyond governance issues to include a more complete set of issues faced in such areas as finance, contracts, fare card technology integration, and relationships among the agency partners and with their customers and the vendor.  This approach seeks to provide a detailed assessment of the key issues faced across many aspects of the RFC Project and a discussion of how they were addressed and resolved.  Overall, the evaluation seeks a balance between analysis breadth and depth by keeping the focus on key issues, partner motivations and perspectives on these issues, decision processes used in addressing the issues, and outcomes and benefits derived from these processes.  In this way, the evaluation intends to highlight those aspects of the RFC Project planning and implementation processes that will be of most value to others.  More detailed descriptive and technical data on specific aspects of the RFC Project are available through the project partners but will only be covered here to the extent necessary to set the context for an evaluation of the issues.

2.2 Objectives of the RFC Evaluation

The scope of this evaluation has evolved since the development of the initial Scope of Work (SOW) in May 2002.  The original SOW included two evaluation components, as follows:

• An assessment of the processes by which the Project Partners identify, address and resolve a variety of institutional and policy issues that are expected to be confronted when seeking to implement a complex new regional fare card program among seven transportation agencies, each with a history of different management and operating systems, needs, interests, and constituencies.
• An evaluation of selected other data that are already being collected by the partner agencies or data that otherwise are readily available and relevant to the goals of the IPAS program.  In this sense the case study format will be expanded to include an examination of these additional data, as long as it doesn’t involve additional original (primary) data collection requirements by the contractor.  Examples might include available cost and revenue data, driver performance logs, bus boarding statistics, or data on the diffusion of the new fare media.  The specifics will depend on learning more about what data will be readily available that are directly relevant to the evaluation objectives.
• As assumption underlying the initial SOW was that the evaluation period would extend through the beta test and initial deployment of the RFC Project components.  An evaluation strategy document was issued in June 2003 that discussed needed changes in the original evaluation objectives noted above, based on discussions with the Project Partners.  It became clear that the full implementation of the RFC Project was going to be extended beyond the original projected timeframe and that it would not be feasible or useful to extend the evaluation period to coincide with the beta testing.  Therefore, the kinds of data outlined in the second bullet objective above would not be available to the evaluation.

The ITS JPO directed the project evaluation team in November 2004 to include in the work scope a new focus on preparing a set of lessons learned reports for posting on the USDOT web site.  Thus, the objectives of this evaluation have evolved over time, consistent with the realities of the RFC Project implementation schedule and the needs of the ITS JPO.  The current objectives, as agreed upon in discussions between the ITS JPO, the RFC administrator and the evaluation team, include:

1. Document the institutional and organizational history, and a timeline of the important milestones of the RFC Project.
2. Analyze the processes by which the project partners identify, address and resolve issues associated with planning for and implementing the RFC Project.
3. Identify the ways in which the RFC Project integrates the partner agencies.
4. Seek to distinguish findings that are applicable to other agencies and settings, and those that are unique to the Central Puget Sound regional context.
5. Assemble historical and projected cost data and coordinate with the JPO cost database to contribute to a current understanding of the cost implications of the RFC Project.
6. Prepare a set of Lessons Learned for on-line posting based on the findings from the evaluation.

In order to meet these objectives, the evaluation team has reviewed a comprehensive set of project documentation and prior relevant research studies.  The team has interviewed numerous staff and management in each of the seven partner agencies, the vendor representative, regional FHWA and FTA staff, and members of the Regional Team that is responsible for the administration of the project and the contract with the vendor.  In addition, the team has reviewed reports and evaluations of several other national fare card projects, such as TransLink^â in the San Francisco Bay Area[8] and the ORANGES Field Operational Test (FOT) program in Orlando, Florida.[9]

As of this writing, project development is still underway.  It is expected that a “beta” test of the system, involving all back office operations and equipping a subset of the partner agencies’ fleets, will take place in the third quarter of 2006, followed by full system installation in the first three quarters of 2007, and full revenue service operations in the fourth quarter of 2007.

2.3 Prior Research

The research team reviewed literature documenting the history and evolution of automated fare and smart card technology in the U.S. and around the world.  The literature is focused on transit applications of automated fare payment systems and smart card evaluations and deployments.

Smart card technology has been evaluated and deployed in several locations across the U.S. and around the world during the past 10-15 years.  In recent years, smart card evaluations have generally taken place as part of a beta test or demonstration phase of a comprehensive regional fare card system.  In the U.S., examples of these systems can be found in Chicago, Orlando, the San Francisco Bay Area, Washington, D.C., Phoenix, Los Angeles, Ventura County in Southern California, and Seattle.  International examples include applications in London, Paris, and Hong Kong.

Prior to deployment of smart card technologies, the literature focused on “proof of concept” evaluations and limited deployment demonstrations.  In 1994, the Partnership for Advanced Transit and Highway (PATH) conducted an evaluation of a Los Angles smart card Field Operational Test (FOT) for the California Department of Transportation (Caltrans) and the Federal Transit Administration (FTA).[10]  The FOT deployed automated fare collection (AFC) technologies on three transit properties.  The FOT deployed two smart card technologies with different capabilities.  The first was a contactless radio frequency (RF) smart card that only needed to pass within a few inches of a passenger interface unit (PIU) in order to complete the transaction.  The second technology involved a smart card requiring insertion in a PIU to complete the transaction.

The results of the FOT indicated that the technology worked well in Los Angeles.  Contactless RF smart cards proved to be extremely reliable and convenient.  However, the research team recommended that a lockout period[11] be imposed on the use of the RF cards in order to prevent unintentional multiple transactions.  The smart cards requiring contact demonstrated a higher degree of reliability relative to magnetic stripe cards but did not perform well compared to the RF cards.  The smart cards requiring insertion into PIUs were inconvenient and exhibited a technology failure rate above levels deemed appropriate for transit applications.

The Los Angeles FOT also demonstrated that system performance improved significantly over time.  The lockout period requirement was an example of a system enhancement recognized and addressed during the FOT.

In another early test of the smart card technology, the Régie Autonome des Transports Parisiens (RATP) network in Paris, considered among the most innovative in the world and one of the first transit operators to adopt an AFC system in 1973, examined two technologies designed to replace its out-of-date system of magnetic stripe cards.  The RATP conducted its own evaluation in 1996.[12]  The smart card system was necessary because the magnetic stripe cards were unreliable, with up to 10 percent of all users experiencing card demagnetization.  The smart card system allowed RATP to offer a range of products and created marketing and customer service opportunities through its electronic purse (multiple accounts on the same card) function.  RATP concluded that contactless smart cards were the most effective fare medium, representing the next generation of fare card technology.

The UK conducted its own evaluation of the contactless smart card in the 1990s, with pilot tests conducted on mass transit systems across the country.[13]  In one test, more than 180 buses were equipped with on-board fare transaction processors, and approximately 20,000 smart cards were issued to customers.  On average, nearly 8,000 daily transactions were recorded during the demonstration.

The results of a pilot test conducted in the Harrow area, located northwest of London demonstrated that:

• customer satisfaction associated with the smart card was very high,
• the technology proved to be extremely reliable,
• operators found the technology easy to operate, and
• the amount of management information generated by the new system far exceeded that historically acquired through traditional surveys.

Based on the outcome of the pilot test, the technology evaluators found that most of the key objectives of an AFC system could be obtained through deployment of a contactless smart card.  The study went on to predict a future development path of the smart card technology, which included wider exploitation of the card, enhanced incentive schemes, standardization of the technology and expanded use of e-purse capabilities.

Chicago’s AFC system was the focus of a 2000 study performed by Foote and Stuart.[14]  This study examined the relation between the AFC system and shifts in fare media usage.  The study reported that in its second year of full operation, 1.2 million passengers were using AFC on Chicago Transit Authority (CTA) bus and rail systems.  Between 1996 and 1999 there was a significant shift in the use of fare media:

• token usage was phased out entirely (down from 42 percent of total revenue in 1996),
• cash payments declined from 52.1 percent to 35.1 percent of total farebox revenue,
• prepaid pass usage grew to 18.5 percent of total farebox revenue, and
• fare cards grew to 46.4 percent of total farebox revenue.

The study’s findings also suggest that implementing the AFC system contributed to a larger than anticipated passenger growth rate of 4.3 percent in 1999.

Smart card technologies applied to broader transportation applications have gained favor in recent years.  One such system, ORANGES, was developed in Orlando, Florida.  ORANGES was deployed with multi-agency collaboration as a U.S. DOT ITS Field Operational Test (FOT).  It included a central payment and clearinghouse system with smart cards used in multiple transportation settings – transit, toll and parking.  The U.S. DOT published in 2004 a Phase I risk assessment report as part of a national evaluation. [15] The report documented both qualitative and quantitative goals for the ORANGES system.  Quantitative goals included those targeting revenue enhancement, increased market penetration, cost savings, time savings and expanded use of prepaid cards.  Qualitative goals centered on customer satisfaction, ease of use and information generated by the system.  The ORANGES FOT was concluded in 2004.

The initial FOT included deployment on transit (two LYNX bus lines), toll (Holland East toll plaza) and parking (Central Boulevard, Library and Market Street garages in Orlando) systems.  During the test, partner agencies planned to maintain 800-1200 cards.  The FOT is ongoing; however, there have been several lessons documented from the experience to date.  The primary lesson to agencies interested in using smart card technology is that it is important not to underestimate the complex nature of interoperability and integration issues.  The design phase of the ORANGES system took much longer than originally expected.  Delays during the design phase were attributed to the lack of documented requirements for the system prior to the selection of the vendor.  The U.S. DOT report also cited issues relating to the recruitment of passengers willing to participate in the demonstration as a complicating factor.  Many of the enrolled participants used their cards on a limited basis or not at all.  The research team recommended putting extra effort into the recruitment and screening of pilot test participants.  More specific recommendations include: performing education/outreach with the participating customers, examining usage patterns of the recruits and conducting follow-up in order to isolate and eliminate participants who are unlikely to make use of the technology.  Other lessons included the importance of being conservative when ordering smart cards (order more than enough to cover the pilot test in case existing models are discontinued) and to recognize tradeoffs between low cost equipment offered by vendors interested in the high profile nature of the project and performance.

Transit operators in Ventura County, California took part in a pilot test of an AFC system between January 1996 and October 1999.  The demonstration project was coordinated by the Ventura County Transportation Commission (VCTC).  Following deployment, FHWA and FTA presented a case study as an examination of a project where the outcome did not meet expectations.[16]  Problems that were experienced during the demonstration included numerous operational shortcomings, inconsistent data and infrequent reports.  South Coast Area Transit’s system never achieved full operability during the pilot test due to reliability problems.  Despite these shortcomings, the pilot test was considered a step forward in terms of enhanced multi-agency coordination.

The Smart Passport case study concluded that a successful multi-agency fare collection system required the following elements:

• A regional champion,
• Adequate staff resources,
• Regular and open communication,
• Vendor with a local presence,
• Pricing structure that compares favorably with the existing fare structure,
• Extensive staff training,
• System requirements established prior to selection of vendor,
• Data requirements established during planning phase,
• Defined report formats,
• Clearly defined clearinghouse and settlement responsibilities designated during the planning phase,
• Effective and comprehensive marketing strategy,
• Programs that offer users incentives, and
• Surveys that are conducted in order to determine customer satisfaction levels and to define strategies for enhancing customer satisfaction.

The research team also noted that, provided that the key elements listed above are achieved, agencies would do well to consider alternative business models and implementation strategies.  No single strategy will fit all cases.

A Charles River Associates (CRA) study published in 2002 evaluated the demonstration phase of the TransLink^® system.[17]  TransLink^® is a smart card transit fare collection system currently under development in the San Francisco Bay Area.  The CRA study examined the stated goals of the TransLink^®  system and evaluated data collected during the system’s demonstration phase in order to draw conclusions regarding the early system performance.

TransLink^® was installed in elements of systems operated by the six largest transit agencies located in the San Francisco Bay Area: AC Transit, BART, Caltrain, Golden Gate Transit, San Francisco Muni, and Valley Transportation Authority (VTA).  In the demonstration phase, TransLink^® was tested on every major transit mode in the Bay Area.

The regional transit coordinating council for the San Francisco Bay Area, known as the Partnership Transit Coordination Committee (PTCC), established key goals for the TransLink^â system.  These were focused on enhancements to passenger convenience, efficiency and security of the fare collection system, business opportunities that enhance revenues or reduce costs and the consolidation and coordination of transit services in the San Francisco Bay Area.

During the demonstration phase, the technology performed reasonably well.  In terms of technological issues, no overwhelming problems were detected.  Minor issues relating to poor performance among handheld card readers and some reports of out of service equipment, primarily on buses and streetcars, were noted.

In order to examine user acceptance, 4,137 passengers were initially recruited to participate in the demonstration, followed later by an additional 2,033 volunteers.  Users experienced a high degree of satisfaction with TransLink^® and showed interest in using the system on a permanent basis once it reaches full operability.  Most customers found TransLink^® easy to use and, when problems arose, their experience with the TransLink^® Service Bureau’s help and information line was viewed as highly positive.  Customers did, however, experience problems with the add value machines and the audio components of the TransLink^® system.

CRA’s examination of the TransLink^® demonstration phase indicated that significant additional staff training was needed.  During the demonstration phase, there were issues with out-of-service equipment, which users attributed as much to poor operation as to technological malfunction.

A recent publication sponsored by the Transit Cooperative Research Program (TCRP) signaled the future of transit fare collection through the development of recommended standards for AFC systems.[18]  This study examined current multi-agency AFC systems in order to draw conclusions from what did and what did not work well during the implementation of these systems.  The study did not recommend specific technologies, noting that successful regional AFC systems can be achieved through the application of a variety of technologies.  Instead, the study focused on institutional, operational and financial issues.

Based on the examination of nine regional operating systems located around the world, the TCRP study proposed a multi-phased approach to the development of regional fare collection systems.  The study found that multi-jurisdictional systems developed in stages were most successful.  That is, most jurisdictions appeared to require initial steps towards regionalization (e.g., Puget Pass) before realizing a comprehensive regional fare card system such as this RFC Project.  Recommended standards included those relating to:

• Formation of governance structures responsible for setting broad program policies for the system, managing programs, operation functions and implementing the project,
• The importance of coordination in the setting of policies related to fares, media formats, customer service activities, operating policies and cost-recovery methods,
• Design of financial standards and clearinghouse protocol, and
• Development of operating policies related to pricing, sales procedures, training and system operation and maintenance.

The literature suggests that AFC systems are following a development path with contactless smart cards as the preferred fare media.  To date, this has included the development of the technology, “proof of concept” testing, deployment of contactless smart card systems in multi-jurisdictional settings, use of the card to implement complex fare policies and application of smart cards in broader transportation settings.  The literature suggests that next steps along this development path could be the adoption of standards for implementing regional AFC systems and wider use of the electronic-purse capabilities of the card as a marketing tool for public transit agencies interested in forming partnerships with private partners.  The evolution of electronic fare collection technologies is likely to accelerate in the near future as experience from the many deployments taking place across the U.S. generates information that will smooth the path for agencies undertaking future deployments.

Previous  |  Table of Contents  |  Next

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS