3. Evaluation Methodology

The main objective of this evaluation was to measure or confirm the expected outcomes of the CRRAFT system. To accomplish this objective, the evaluation methodology had four main phases:

1. Develop hypotheses
2. Determine approach to assess hypotheses
3. Collect Data
4. Analyze Data

This evaluation approach was first presented in the Evaluation Plan and Test Plan document, which discussed the first two phases and provided guidelines to conduct Phases 3 and 4. The following sections describe the efforts undertaken during Phases 1 through 3. The data analysis (Phase 4) is described at length in the next chapter - Evaluation Results.

3.1 Hypotheses Development

The first step of the evaluation was to take the expected outcomes presented in Table 2-2 and develop them into hypotheses. The hypotheses are statements that describe the expected outcomes in a more detailed and measurable manner. The expected outcomes and the set of 12 hypotheses developed are listed in Table 3-1. Hypotheses one, two, three and five are considered "key hypotheses."

3.2 Determining Approach to Assess Hypotheses

The assessment of the hypotheses was done through Measures of Effectiveness (MOEs) assigned to each hypothesis (i.e., change in overall time saved preparing reports and invoices, time saved scheduling trips). For most of the hypotheses, the MOEs were assessed before and after the implementation of CRRAFT to understand the impact of the system. Three approaches were designed to collect and analyze the data necessary to assess the MOEs:

1. Analysis of quantitative measures
2. Analysis of surveys and interviews with transit providers
3. Analysis of interviews with funding agencies and ATRI

3.2.1 Analysis of Quantitative Measures

This is an analysis of standard operating performance metrics typically used by transit providers, and changes in those measures before and after CRRAFT. It also included measurable aspects of the invoicing and reporting process before and after CRRAFT. The data for this analysis came primarily from system performance figures archived by either the transportation providers or the PTPB, and invoices and invoice submission logs archived by the PTPB.

Because CRRAFT was already in operation throughout New Mexico, before and after comparisons of these measures depended on the availability of archived data related to these performance metrics. The Evaluation Team noted that a lack of knowledge on the availability of archived data limited the extent to which the plans for the evaluation could be finalized. In order to remove this ambiguity, a preliminary assessment was conducted. The primary purpose of this assessment was to identify the types of archived data that were available to support this evaluation and to determine if sufficient quantitative data of usable quality were available for a before/after analysis of the CRRAFT system. Table 3-2 summarizes the results of this assessment. Note that not all of the hypotheses shown in Table 3-1 are included in this summary since the Preliminary Assessment focused only on hypotheses that required the evaluation of quantitative data elements.

3.2.2 Analysis of Surveys and Interviews with Transit Providers Staff

To offset the limited amount of archived quantitative measures available, the national evaluator proposed to conduct an analysis of the processes used before and after CRRAFT in order to estimate the impact of CRRAFT. For example, CRRAFT provides a uniform and consistent set of reports to PTPB, which should decrease the amount of manual processing required to handle those reports. Both quantitative and qualitative subjective data were collected for this portion of the study, largely through interviews and surveys with operating agencies.

An attitudinal survey was used to obtain user opinions on the impact of CRRAFT on their operations. In order to facilitate analysis, the survey consisted primarily of questions whose responses are numbers in a linear scale (e.g., "What impact has CRRAFT had on the time spent preparing reports for PTPB?" with responses like "1 It takes a lot longer with CRRAFT; 2 It takes a little bit longer with CRRAFT; 3 The time required is about the same with and without CRRAFT; 4 It takes a little bit less time with CRRAFT; or 5 It takes a lot less time with CRRAFT").

3.2.3 Interviews with Funding Agencies and ATRI

Interviews were conducted with staff of the PTPB, NMDOT, and the ATRI to review and discuss lessons learned and best practices with respect to the implementation, operations and maintenance of the CRRAFT system that were not previously identified in the CRRAFT Case Study . Interviews with ATRI staff responsible for developing and maintaining CRRAFT provided costs of developing CRRAFT, plans for future enhancements to the system, and the expected costs for these enhancements. Because one of the benefits of CRRAFT is the creation of a uniform processing system for the 27 participating agencies, which facilitates future enhancements, this type of information is important in understanding the full benefits of CRRAFT.

3.2.4 Summary

The hypotheses and these analysis approaches for assessing them formed the basis for the evaluation. A summary of the evaluation approach including hypotheses, MOEs, data sources and evaluation method is presented in Table 3-3.

3.3 Data Collection

In this phase of the evaluation, and based on the approaches previously determined, the Evaluation Team collected data from a number of sources, including relevant documents and archived data from the CRRAFT system. Also included in the data collection efforts were surveys and interviews with transit provider and funding agency staff. The following subsections describe the data collection efforts.

3.3.1 Data and Documents

The Evaluation Plan identified several electronic and physical documents that were needed to assist with the assessment of the hypotheses. These documents are also summarized in Table 3 3 above under the column called Data Sources. These documents and data were collected as follows.

The PTPB files include a folder for each transit provider receiving Section 5311 and/or Section 3037 (JARC) funding. Separate files are kept for each fiscal year. Each of these files includes, among other things, a log of the invoice/payment process. These invoice submission logs for FY02 and FY04 were consulted by the Evaluation Team. One of these logs is presented as an example in Appendix A.

These invoice submission logs provided, among others, the date of initial invoice submission, any problems or errors with the invoice, the date of the final invoice submission had there been errors on the initial invoice, the date the invoice was entered into the PTPB database, and the date the invoice was given third level approval at PTPB (accepted for payment). These data provided the team with the ability to analyze the submission lag (days between the date at which invoices are actually submitted and the submission deadline) before (FY02) and after (FY04) the implementation of CRRAFT. With these data, the team was also able to determine how often errors were found in the invoices and re-submissions were required before and after CRRAFT. The time required to resolve erroneous invoices was also determined. Finally, the data also allowed the Team to calculate the time required by PTPB to approve the invoices, before and after CRRAFT.

The Evaluation Plan anticipated the review of invoice re-submittal records kept by the transit providers. However, not all the providers kept these records and the Evaluation Team decided to conduct the analysis based solely on the records in the Submission Logs kept at the PTPB. Yet, the issue of invoice re-submission was included in the surveys to the Providers.

The Evaluation Team collected the monthly invoices submitted by the transit providers to the PTPB before (FY02) and after (FY04) CRRAFT. Section 5311 providers include number of passengers, number of revenue miles, hours of service provided, and operating cost on those invoices. Section 3037 providers include number of passengers and operating costs. For FY02 the monthly invoices were Excel-based invoices, for FY04 the invoices were produced by CRRAFT. An example of these invoices is shown in Appendix B.

The data in the invoices were utilized to calculate commonly used operational metrics (i.e. ridership, cost per revenue hour) to compare the performance of transit providers before (FY02) and after (FY04) CRRAFT.

Some of the data found in CRRAFT for FY04 were found to be out of range. Specifically, Carlsbad, Hobbs, Las Vegas, Los Lunas, Taos, and Zuni had suspicious figures for several entries of monthly vehicle miles, and Navajo Nation, Taos, and Zia Therapy had out of range figures for a few entries of monthly vehicle hours. For example, the monthly vehicle hours for Zia Therapy range between 501 and 591 in all months in FY04 except for November 2003, in which the recorded vehicle hours are 85,580. The agencies were asked about these extremely high figures of vehicle hours and vehicle miles, and they cited errors in the CRRAFT system as the cause for these outliers in the data. Navajo Nation and Zia Therapy only had one month out of twelve with erroneous vehicle hours data. Thus, the outlier number was thrown out and replaced by the average of the remaining eleven months. This procedure could not be repeated with the other agencies because the number of months with suspicious figures was higher. Hence, Taos was not included in analyses including vehicle hours and no analyses were done with the vehicle miles metrics.

An HSD-ISD evaluation was conducted in June 2002, which included an audit of nine transit providers. The Evaluation Team collected a copy of this report because the section on quarterly fiscal reporting includes documentation of discrepancies between what providers billed NMSHTD and the amount they were reimbursed. These data will help assess Hypothesis No. 7 about the extent of unauthorized trips.

During the preliminary assessment, it was mentioned that this report was produced by ATRI; however during the data collection effort, the Team found out that although ATRI did conduct an informal review of available commercial products that could be used to provide the types of services included in CRRAFT, an official report on the topic was never written or could not be located. The Evaluation Team determined during the early stages of the project that if this report was not available, then doing independent research to reproduce those results was not a cost-effective alternative. Hence, the assessment of the MOE associated with this report was abandoned.

A case study of CRRAFT that provides information on CRRAFT, its origins, its costs, and plans for its future was also collected and reviewed. The case study was mostly used to obtain background information as needed. This CRRAFT case study was one of the "best practices" analyzed by Multisystems (now part of TranSystems) for a report submitted in 2002.

3.3.2 Surveys of Transit Provider Staff

Surveys of transit provider staff were conducted by the Evaluation Team to obtain opinions on the impact of CRRAFT on transit operations. Derived from the expected outcomes and hypotheses, a survey was developed that explored user attitudes and perceptions of CRRAFT's impact on such day-to-day operations as preparing monthly invoices, scheduling demand responsive trips, and operational efficiency and cost. The survey consisted of four types of questions: multiple choice (where responses were selected from a list of six choices), fill-in questions on time estimation, ranking of CRRAFT features, and two open-ended (free text) questions. A copy of the survey questions are shown in Appendix C.

The survey was administered to CRRAFT users at 26 transit agencies (representing about 60 users). Prior to distributing the surveys, each transit agency supervisor was contacted via telephone. The telephone call provided an opportunity to present an overview of the evaluation and survey, obtain some background information from the transit agency, and enlist their participation. Afterwards, the surveys were distributed in late December 2004 and early January 2005 via E-mail, and were returned either by E-mail, fax, or U.S. mail. In some cases, follow-up phone calls were made to transit agencies to clarify responses or comments on completed surveys or to make a request for their survey(s). Forty surveys were returned representing 24 of the 26 transit providers (a return rate of about 92%). At some transit agencies, each CRRAFT user completed one survey. At others, multiple users completed a single survey. This difference was permitted because at some transit agencies CRRAFT responsibilities were distributed to multiple staff members (e.g., Person A responsible for the scheduling, Person B preparing the monthly invoices).

The survey data were manually entered into a database and descriptive statistics were computed using SPSS (Statistical Program for the Social Sciences) statistical software. In cases where multiple surveys were returned from the same transit agency, an average rating (or score) was computed for each question and used in the SPSS analysis to simplify the interpretation of the statistics. This data manipulation method reduced the bias that would result in over-weighting ratings from agencies submitting multiple surveys, especially when compared to surveys submitted from transit agencies that combined the responses from multiple users into one survey. The survey results are presented in Chapter 4 of this report.

3.3.3 Interviews with Funding Agencies, PTPB, ATRI and Transit Providers

The Evaluation Team interviewed the managers and staff at the NMHSD, NMDOT, ATRI, and several transit agencies. Two different interview guides were developed. The interview guide for funding agency managers/staff (see Appendix D) investigated what was working well with CRRAFT, what needed improvement, and what other functions would be useful additions to CRRAFT. The interview guide for Transit Agency Managers (see Appendix E) focused on gathering information about their experience with the CRRAFT application and the effect CRRAFT has had on their organization. Transit providers were split over time savings, efficiency, accuracy, operational impact, and overall opinion of CRRAFT.

NMDOT and ATRI interviews were held in face-to-face meetings that took place in early December 2004. NMHSD and most transit agency interviews were held over the telephone in December 2004 and January 2005. Most interviews were completed in 30 to 60 minutes.

Since the number of interviewees was relatively small, a statistical analysis of the responses was not conducted. The results of these interviews, which were primarily anecdotal in nature, were used to guide follow-up discussions and explore operational changes resulting from the use of CRRAFT.

In addition to the initial interviews, follow-up discussions were conducted with five transit agencies in April 2005 during the New Mexico Public Transport Association Conference in Albuquerque. These discussions allowed a more thorough investigation of issues that were raised when the data and survey results were analyzed. It also provided an opportunity for gaining a clearer picture of the performance and perceptions of CRRAFT.

Previous | Next

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS