3. Summary of Evaluation Activities

This section provides a summary of the originally planned and later modified evaluation activities conducted as part of the independent national evaluation of the Airport ITS integration and RIMS projects.

For the originally planned evaluation, a structured approach was used in developing the plan for evaluating the GLITS ITS Integration project. The original evaluation activities presented here are based on the review of evaluation documents (GLITS Evaluation Plan and the Airport ITS Integration Detailed Test Plan); project documents (Earmark funds application, MDOT-sponsored research reports, maps, cost estimates, etc.); site visits and meetings; and follow-up discussions with the GLITS management team and stakeholders. Project goals and objectives were used as the basis for developing evaluation objectives, study areas, and the approach for investigating each key hypothesis.

The GLITS project evaluation originally consisted of a study of system impacts. The system impact study was envisioned to measure or confirm the expected outcomes of the system in terms of the safety, mobility, and customer satisfaction impacts. However, because of unforeseen events, changes in project leadership, and various other factors which resulted in deployment delays for both the Airport ITS integration and RIMS projects, significant changes to the evaluation occurred.

The emphasis of the evaluation was modified from a system impact study to a case study investigation of key events and the development of lessons learned based on the stakeholders' experiences. The intent of the new evaluation was to focus on the series of events that contributed to the delays and develop a set of lessons that would be useful for other agencies. In summary, this evaluation is intended to result in a:

Case Study report of key events influencing the Airport ITS integration and RIMS project deployments.
Posting of the report on the USDOT electronic document library (<http://www.its.dot.gov/library.htm>).
Set of lessons learned based on GLITS stakeholders' experiences.
Posting of lessons on the USDOT ITS Lessons Learned Knowledge Resource (<http://www.itslessons.its.dot.gov/>).

The following sections provide a description of the original evaluation objectives, approach, and study areas and the modified evaluation objectives.

3.1 Original Evaluation Objectives

Initially, the overall evaluation objectives were to provide a quantitative and qualitative analysis of the system impacts of the ITS integrations and to identify lessons learned from the Airport ITS Integration and RIMS projects. The impacts and institutional challenges were to be carefully explored and documented to help provide guidance for other jurisdictions considering similar deployment and integration projects. In addition, the findings of the evaluation were intended to be used by other agencies assessing the appropriateness of ITS integration as a potential solution to similar problems.

3.1.1 Original Evaluation Approach

The evaluation approach proposed for the original evaluation was based on project performance goals based on the expectations of the project stakeholders and the USDOT's evaluation objectives. For each project, project stakeholders' goals were established through dialogue in the national ITS evaluation nomination process. USDOT evaluation goals were developed to meet the system impact study requirements identified in the evaluation Statement of Work (SOW).

For each goal, in coordination with the project stakeholders, the Evaluation Team developed a key hypothesis to form a basis for testing and measuring the impact of the proposed ITS solutions Each key hypothesis resulted in developing associated measures of effectiveness (MOE) to provide the basis for comparison of "before" and "after" cases. The MOEs provided suggestions for data requirements, which were used in discussion with project stakeholders to identify specific data sources to support the evaluation.

Table 1 and Table 2 present an overview of the original evaluation approaches, including the goals, key hypotheses, MOEs, data sources, and planned analyses associated with the two components of the GLITS project: the Airport ITS Integration and the RIMS projects, respectively.

**Table 1. Airport ITS Integration Project Evaluation Approach Overview**
Goal	Key Hypothesis	MOE	Data Sources or Requirements	Data Analysis Method
Increase traveler mobility in the area.	Project assumes real-time traffic data will be made available to the traveling public, thereby contributing to increased mobility.	Travel time studies. Roadway volume counts. Vehicle average operating speeds. Roadway and intersection level of service.	Wayne County DPS (Roads) and MDOT agency reports. Wayne County DPS (Roads) and MDOT agency reports. Wayne County DPS (Roads) and MDOT agency reports. Archived data from Wayne County DPS (Roads) and MDOT reports.	Quantitative analysis of archived and collected data. Quantitative analysis of archived and collected data. Quantitative analysis of archived and collected data. Quantitative analysis of archived and collected data.
Improve the level of safety.	Improve safety by informing the traveling public well in advance of incidents, potential hazards, and security issues.	Reduction in speed variation across high crash frequency freeway sections Crash frequency.	Wayne County DPS (Roads) and MDOT agency reports. Wayne County DPS (Roads) and MDOT agency reports.	Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis. Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis.
Improve customer satisfaction.	Project assumes that the real-time information made available through ITS technologies to the traveling public on airport conditions (traffic, parking availability, curbside access, and safety) will contribute to increased customer satisfaction.	Customer perceptions.	Surveys/interviews.	Qualitative analysis of survey/interview responses.

**Table 2. RIMS Project Evaluation Approach Overview**
Goal	Key Hypothesis	MOE	Data Sources or Requirements	Data Analysis Method
Improve the satisfaction of county residents.	Customer satisfaction will increase as the DPS makes more real-time information available.	Customer perceptions.	Surveys/interviews.	Quantitative analysis of survey/interview responses.
Increase traveler mobility in the region.	The provision of real-time traffic and roadway condition information will contribute to improved mobility by reducing delays and increasing travel time reliability.	Customer perceptions. DPS maintenance staff perceptions.	Survey/interviews. Wayne County DPS (Roads) and MDOT agency reports.	Quantitative analysis of survey/interview responses. Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis.
Improve winter maintenance activities and construction delays.	Using GIS, the RIMS will improve DPS operations by increasing awareness of potential problems; efficiently routing public complaints; providing enhanced responsiveness to in-road maintenance; and reducing design time.	DPS staff perceptions. Maintenance schedules. Records of lane closures. Public comments.	Survey/interviews. Wayne County DPS (Roads) and MDOT agency reports. Wayne County DPS (Roads) and MDOT agency reports. Wayne County DPS (Roads) and MDOT agency reports.	Qualitative analysis of survey/interview responses. Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis. Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis. Quantitative analysis of archived data; development of forecast based on relationships revealed during data analysis.
Improve safety, especially at work zones.	The RIMS will provide real-time information on traffic and road conditions, ultimately improving safety by informing the traveling public well in advance of incidents and potential hazards (including weather).	Crash frequency.	Wayne County DPS (Roads) and MDOT agency reports.	Quantitative analysis of archived and collected data; development of forecast based on relationships revealed during data analysis.

3.1.2 Data Collected and Analyzed

For the RIMS project evaluation, no evaluation data were collected. However, pre-deployment (baseline) data were collected at the DTW for the Airport ITS Integration project evaluation. For the traveler mobility analyses, traffic count data were manually collected for non-peak and peak traffic days at three locations on October 22-23, 2003 and November 26, 2003. The baseline data showed that the majority of inbound and outbound vehicles used the I-94 (northern) access road rather than the I-275 (southern) access road. See appendix A for the analysis relating to traffic flow data for vehicles entering and exiting the DTW.

Air passenger counts for a 15-month period from October 2002 through December 2003 were collected to describe the number of scheduled passengers, air taxi (commuter) passengers, and charter passengers. In addition to total counts, the data were also disaggregated by direction of travel (arrival or departure) and flight destination (domestic or international). See appendix B for the analysis relating to air passenger counts.

Parking lot transactions for a 2-year period from February 2002 through February 2004 from the Smith/Berry Terminal and McNamara Terminal were also collected and analyzed in support of this evaluation. These parking data, collected by location and parking facility were aggregated into monthly totals and daily averages. See appendix C for the analysis relating to airport parking lot transactions.

For the Safety and Security analyses the DTW Wayne County Airport Authority, Division of Public Safety provided crash data from State of Michigan Traffic Crash Reports. The crash data included all vehicle crashes occurring on the airport roadways and parking areas from February 2002 through January 2004 and were characterized by: Date, Time Period, Location, Road Condition, Type of Crash, and Accident Severity. See appendix D for the analysis relating to airport crash data.

3.1.3 Summary

The evaluation originally was planned to be an investigation of system impacts and the identification of lessons learned based on stakeholder experiences regarding the Airport ITS Integration and RIMS projects. An evaluation approach was developed for each project based on the anticipated project's performance goals. Pre-deployment data were collected for the Airport ITS Integration project but not the RIMS project. Appendices A through D show the preliminary analyses of the airport traffic flow, air passenger counts, airport parking lot transactions, and airport crash data.

Although pre-deployment data were collected at the airport, deployment delays and various other factors (discussed in section 4) resulted in the originally planned evaluation being modified. The modified evaluation focuses on the events that contributed to the delays and the development of a set of lessons learned. The following sections describe the modified evaluation.

3.2 Modified Evaluation Objectives

A new SOW was developed to modify the GLITS evaluation objectives and activities to conduct the evaluation. The modified evaluation consists of a case study of the Airport ITS integration and RIMS projects, and the development of Lessons Learned. As such, the new evaluation activities included:

Interviews of key personnel at MDOT, Wayne County, and the Wayne County Airport Authority to identify the events, issues, and factors that have influenced the GLITS deployment.
Collecting supportive information.
Documenting Lessons Learned according to ITS JPO-prescribed format.
Delivering Draft and Final Case Study Report.
Presenting findings to ITS JPO.

3.2.1 Conduct Interviews

Interviews with the key personnel at MDOT, Wayne County, and the Wayne County Airport Authority were conducted to identify the events, issues, and factors that have influenced the GLITS deployment. The interviews with key personnel were used to identify, describe, and capture the various factors that have influenced the GLITS deployment. In addition, the interviews were used as inputs for the development of lessons learned.

3.2.2 Collect Supporting Information

Supporting information was collected to investigate, identify, and examine information describing the events and factors that have influenced the GLITS deployment. The type of supporting information examined was:

Project planning documents or architecture diagrams.
Maps/photos of airport roadway and construction activities.
Meeting minutes.
News articles providing background information or insight into key events.

The information was used to develop a deeper understanding of the key events and factors and was included or cited as appropriate.

3.2.3 Develop Lessons Learned

A series of lessons learned were developed from the management, planning, design and development, procurement, leadership, legal, and human resources experiences of the MDOT, Wayne County, and Wayne County Airport Authority agencies. The lessons learned were written in a format that is compatible with those residing on the ITS Lessons Learned Knowledge Resource Website.

The ITS Lessons Learned Web Page on the ITS JPO Website¹⁴ was used as a reference for additional guidance in documenting and formatting the lessons learned. Potential focus areas for lessons learned will include those identified in Table 3.

Table 3. Lessons Learned Categories and Subcategories
Lesson Category	Subcategory
Management & Operations	Operations Maintenance System Data & Storage Evaluation & Performance Measurement M&O Tools & Models
Policy & Planning	Policy Planning Architecture Programming (TIP / SIP) Planning Tools & Models
Design & Deployment	Project Management Requirements & Design Standards & Interoperability Implementation Quality Assurance & Testing Design Tools & Models
Leadership & Partnership	Leaders & Champions Partnerships & Agreements Awareness & Outreach Media Coordination Organizational Management & Structure
Funding	Federal State Regional & Local Private Innovative Financing
Technical Integration	Functional Jurisdictional Legacy Systems
Procurement	Work Allocation Method of Award Contract Form Contract Type Terms & Conditions
Legal Issues	Intellectual Property Liability Privacy Labor Rules & Regulations Disputes & Claims
Human Resources	Personnel Management Recruiting Retention & Turnover Training

3.2.4 Develop Case Study Report

The Case Study Report documents the events, factors, impacts, and lessons learned that have affected the deployment of the GLITS project.

3.2.5 Presentation of Findings and Conclusions

A briefing to ITS JPO will present a summary of the findings and conclusions. The findings portion will include a summary of key events, factors, and issues that have resulted in deployment challenges and delays. The conclusions portion will include a summary of the lessons learned and an update of the deployment status and institutional/management changes that have occurred to address the issues.

¹⁴ ITS Lessons Learned Website resource, last accessed on October 23, 2006 at: <http://www.itslessons.its.dot.gov/>.

Previous | Next