< Real-Time Data Coverage | Table of Contents | Uses of Real-Time Data >

4 Data Quality

Data quality is the suitability of the data for its intended purpose. It is most commonly assessed by how accurate the data is (i.e., how close the reported data is to the “ground truth”), but it includes other metrics such as timeliness and availability. Clearly, traveler information must be of sufficient quality that users trust the information presented to them enough for it to influence their travel decisions. This section discusses how quality is measured; tools used to measure data quality; and industry perspectives on real-time traveler information data quality across the traffic, transit, parking and freight modes.

4.1 Data Quality Metrics and Parameters

4.1.1 Data Quality Metrics

A white paper entitled, “Defining and Measuring Traffic Data Quality,” which was a product of an FHWA Traffic Data Quality Workshop in 2002, proposed a battery of data quality metrics including accuracy, completeness, validity, timeliness, coverage, and accessibility. In addition to identifying metrics, this paper suggests that data quality is only meaningful relative to the intended purpose of the data. While the white paper is specific to the traffic mode, these same metrics apply to other modes as well. In this treatment of the subject for real-time applications, we discuss quality in terms of accuracy, timeliness, and reliability. Other metrics such as validity, completeness, and accessibility relate more to archived data. Each of these measures can be used in a specification, and real-time data should be evaluated against them, with the possible exception of timeliness, which is difficult to measure. We will discuss data validation more in a subsequent section.

4.1.1.1 Accuracy

Accuracy is how close the reported data is to “ground truth,” or actual conditions. Accuracy can be measured in many different ways depending on the type of data being considered. The simplest case is regarding a discrete event. One measure of accuracy is the percentage of time events are reported when they actually occur. An inaccurate message may be a missed event (false negative) or a message that persists beyond the life of the actual event (false positive). For non-discrete events, such as traffic congestion data, accuracy may be calculated in different ways including mean error[1], mean absolute error[2], root mean squared error, mean absolute percent error, mean squared error, etc. Ideally, mean error—or bias—can be adjusted for, as it represents a consistent over or underreporting in the data. However, the source of the bias must be considered. For some data sources, errors tend to arise when congestion occurs and it is not captured in the data due to an over-reliance on historical data or a scarcity of real-time data points.

There is no universal consensus on which accuracy measure should be used for different applications. It also matters whether speed or travel time is considered. Speed is typically preferred because it controls for varying segment lengths. Percent error is consistent across possible ranges of speed. When using mean speed error or mean absolute speed error, it is important to qualify what range of speed is being considered, and for this reason, it is typically reported in “bins,” such as below 30 mph, 30 mph to 50 mph, or greater than 50 mph. An error of 10 mph is much greater if the ground truth speed is 10 mph than if it is 60 mph.

For the I-95 Corridor Coalition INRIX evaluation, data quality was measured on two criteria—average absolute speed error and the speed error bias. The mean absolute speed error was required to be within 10 mph within each of the following ranges of observed speed—below 30 mph, 30 mph to 45 mph, 45 mph to 60 mph, and greater than 60 mph. Data quality requirements were in place whenever the volume was greater than 500 vehicles per hour.

AVL data for bus tracking is evaluated on the basis of its positional accuracy. From a technical standpoint, it is usually very precise, often accurate to within 30 feet of actual vehicle location when properly maintained. In terms of providing accurate information to the public, while the public realizes that information cannot be perfect, information should be 95-percent to 98-percent accurate for riders to trust the information, meaning that information is within an acceptable error the vast majority of the time.

4.1.1.2 Timeliness

Timeliness, also referred to as latency or lag, is the time between when actual conditions occur and when those conditions are reflected in the real-time information source. It may have several components depending on the type of data:

Timeliness can be difficult to measure. Acceptable values for real-time information are typically 10 minutes or less. Therefore, measuring timeliness requires a good estimate of when the actual event occurred (e.g., an incident or a traffic slowdown) and what is the comparable point in the real-time information stream.

4.1.1.3 Reliability

Reliability is sometimes also considered “availability,” the amount of data available compared to the amount intended to be available. For instance, if some or all of a real-time data stream is down for some period of time, it is less than perfectly reliable. It is typically represented in percentage terms.

Poor reliability can have many causes, especially considering there is a sequence of events that must occur between data collection and dissemination. For example, the Los Angeles County Metropolitan Transit Authority’s communications network provides 99-percent radio frequency availability, but that does not mean that every piece of information needed for accurate real-time information is available to the system. Vehicle maintenance issues, operator log-on issues, scheduling errors, and system-related issues can all contribute to the overall reliability of the system. Severe weather can also be detrimental to system performance. Poor reliability can undermine the public’s confidence in a real-time information system. However, there is no national standard to determine when data should no longer be displayed due to system unreliability, with each agency determining its own threshold and refresh rates.

4.1.2 Measuring Data Quality

Historically, the quality of traveler information data is not often formally measured and published, especially for the public sector. Rather, most internal assessments have focused on customer satisfaction and feedback. Past research has attempted to define reasonable standards for data quality for different applications, but actual reports on whether these standards are met are rare. The next section will provide more detail on actual levels of data quality observed in recent evaluations.

With the growth in private firms providing traffic data to the public sector, measuring data quality is becoming more important. New innovative data collection techniques rely on combining traffic data from multiple sources to arrive at real-time estimates. Because many of these methods are new and unproven in all conditions, public agencies cannot rely on their familiarity with known technologies (e.g., point sensors) to understand the quality of this data. Therefore, public sector agencies must validate the data they are buying against the levels specified in the contract documents.

Some agencies typically measure the quality of data that is used to communicate traveler information to the public by:

In measuring the accuracy of traffic data, it is important to note that the baseline of “ground truth” against which the data is to be measured must also be measured. This introduces an amount of error in the “ground truth,” which must be considered when determining the accuracy of the data provided. The I-95 evaluation used Bluetooth tracking devices to measure segment travel times for ground truth. Because of the variability between individual vehicles and the potential for error in the Bluetooth tracking technology itself, it was decided that the error in the INRIX data be calculated as the distance from the 95-percent confidence band (Standard Error of the Mean or “SEM Band” in the table) of the Bluetooth data rather than the mean (Summary Report for I-95 Corridor Coalition Vehicle Probe Project: Validation of INRIX Data July-September 2008, January 2009).

Another issue revealed by the I-95 evaluation is a need to filter out outlier data points. Even though the data may meet quality standards in aggregate, there were some occurrences of random spikes in the data that could cause potential issues for real-time applications such as posting travel times on DMS (I-95 Corridor Coalition Vehicle Probe Project: Validation of INRIX Data July-September 2008, page 9).

Recently, Chicago RTA has planned to conduct field testing to develop accuracy specifications. The agency is most concerned with developing an improved projection of the average customer’s experience with RTA’s real-time applications, including how they perceive information available to them while at a bus stop.

Current Quality Perspectives

How Accurate Does Real-Time Traveler Information Need to Be?

High-quality data may come at a substantial cost to a real-time information system. Information providers must choose a level of data quality that is in line with the needs of their users.

As per customer demands, transit agencies are working to improve real-time information systems to reflect real-time conditions. Real-time transit information quality is inherently variable, but data must be highly accurate in order to be useful. GPS AVL generally allows for extremely accurate data in real time, usually to within 30 feet of actual vehicle location and seldom more than 100 feet. Most agencies strive for data that is accurate at least 95 percent of the time, often striving for numbers as high as 98 percent.

Many parking vendors and operators assert that actual sensors perform at a very high level of accuracy, often upwards of 95 percent to 99 percent. However, given the large number of vehicles entering and exiting a parking facility, even a small number of errors can accumulate to a large total error. For example, if 1,000 vehicles pass a sensor with 99-percent accuracy and only 990 are scanned, the system would calculate that there are 10 more spaces in the facility than are actually available. Over a period of 10 days, similar readings could increase this error to 100 extra spaces, undermining the system’s usefulness. Thus, many facilities recognize the need to do periodic manual resets, often on a daily or weekly basis. Counting spaces manually and testing observed accuracy will also assist operators in developing appropriate baselines and buffers. Portland Airport noted that except for situations involving severe weather, their sensors were accurate enough to provide useful information. However, the operator’s systems still required daily recounts, usually discovering that their system is generally off by about 2 percent to 20 percent per day depending on volume and weather. For entry/exit systems, resetting counters will always be necessary in the foreseeable future, but the goal is to decrease the frequency with which resets must be performed. Many facilities are striving for a goal of every 2 weeks by installing more accurate sensors and using existing sensors more efficiently.

Due to the variable size of commercial vehicles, vendors have had much more difficulty developing sensors that were accurate enough to be used for commercial vehicle smart parking deployments. FMCSA is working with two vendors to develop and test more accurate sensors, but both vendors have yet to develop a model that is accurate enough for a pilot implementation.

For traffic data, various thresholds for data quality have been proposed in different contexts. The I-95 Corridor Coalition, for example, required that the data it procured be within 10 mph within various ranges of speed. At 30 mph, the top of the lowest speed range, that equates to a 33-percent error. At 60 mph, the bottom of the highest speed range, that equates to a 17-percent error. A 2004 FHWA Report entitled, Traffic Data Quality Measurement, proposed targets of 10-percent to 15-percent root mean squared error, 95-percent reliability (termed “completeness”) and “close to real-time” for timeliness, for traveler information applications. The RTSMIP proposed rule includes the real-time information data quality targets listed in

4.2 Current Quality Perspectives

4.2.1 How Accurate Does Real-Time Traveler Information Need to Be?

High-quality data may come at a substantial cost to a real-time information system. Information providers must choose a level of data quality that is in line with the needs of their users.

As per customer demands, transit agencies are working to improve real-time information systems to reflect real-time conditions. Real-time transit information quality is inherently variable, but data must be highly accurate in order to be useful. GPS AVL generally allows for extremely accurate data in real time, usually to within 30 feet of actual vehicle location and seldom more than 100 feet. Most agencies strive for data that is accurate at least 95 percent of the time, often striving for numbers as high as 98 percent.

Many parking vendors and operators assert that actual sensors perform at a very high level of accuracy, often upwards of 95 percent to 99 percent. However, given the large number of vehicles entering and exiting a parking facility, even a small number of errors can accumulate to a large total error. For example, if 1,000 vehicles pass a sensor with 99-percent accuracy and only 990 are scanned, the system would calculate that there are 10 more spaces in the facility than are actually available. Over a period of 10 days, similar readings could increase this error to 100 extra spaces, undermining the system’s usefulness. Thus, many facilities recognize the need to do periodic manual resets, often on a daily or weekly basis. Counting spaces manually and testing observed accuracy will also assist operators in developing appropriate baselines and buffers. Portland Airport noted that except for situations involving severe weather, their sensors were accurate enough to provide useful information. However, the operator’s systems still required daily recounts, usually discovering that their system is generally off by about 2 percent to 20 percent per day depending on volume and weather. For entry/exit systems, resetting counters will always be necessary in the foreseeable future, but the goal is to decrease the frequency with which resets must be performed. Many facilities are striving for a goal of every 2 weeks by installing more accurate sensors and using existing sensors more efficiently.

Due to the variable size of commercial vehicles, vendors have had much more difficulty developing sensors that were accurate enough to be used for commercial vehicle smart parking deployments. FMCSA is working with two vendors to develop and test more accurate sensors, but both vendors have yet to develop a model that is accurate enough for a pilot implementation.

For traffic data, various thresholds for data quality have been proposed in different contexts. The I-95 Corridor Coalition, for example, required that the data it procured be within 10 mph within various ranges of speed. At 30 mph, the top of the lowest speed range, that equates to a 33-percent error. At 60 mph, the bottom of the highest speed range, that equates to a 17-percent error. A 2004 FHWA Report entitled, Traffic Data Quality Measurement, proposed targets of 10-percent to 15-percent root mean squared error, 95-percent reliability (termed “completeness”) and “close to real-time” for timeliness, for traveler information applications. The RTSMIP proposed rule includes the real-time information data quality targets listed in Table 4.1.

Table 4.1: RTSMIP Real-Time Information Data Quality Targets

 

Category of Information
Timeliness for Delivery
 Availability
(in percent)		 Accuracy
(in percent)
Metropolitan Areas
(in minutes)		 Non-Metropolitan Areas
(in minutes)
Construction Activities:
Implementing or removing lane closures 10 20 90 85
Roadway or lane blocking traffic incident information 10 20 90 85
Roadway weather observation updates 20 20 90 85
Travel time along highway segments 10 NA 90 85

Source: Real-Time System Management Information Program Notice of Proposed Rule Making

Agencies recognize the importance of providing travelers with high-quality data, including data that is perceived as accurate. For real-time traveler information to be effective, travelers must trust the information being provided to them. While the public does not expect information to be perfect, highly accurate data is necessary. It is also imperative to consider that the level of accuracy required will vary depending on who is requesting the information.

Public sector systems vary substantially in the timeliness of the information they are able to provide. 511 systems often require information from highway patrol CAD systems, for which there is often a delay in the timeliness of information, which can substantially impact the flow of information. While not in real time, automatic permitting applications rely on providing timely information. Any road updates including new construction or detours must be integrated into systems to ensure that dynamic routing algorithms are accurate.

All this being said, there is no one standard for accuracy upon which all can agree, even disregarding the different applications for the data collected for real-time traveler information. Furthermore, there is a difference between the accuracy that is desired and the accuracy that is tolerated. Ultimately, however, users are the key measure of quality, both for agency traveler information systems as well as for private-sector data collection and dissemination systems. And, different users will have different quality expectations and thresholds.

4.2.2 How Do Agencies View the Accuracy of Traveler Information?

While expressing a desire to improve accuracy, most transit agencies felt they were already able to provide a level of service such that customers were satisfied.

In terms of the traveling public’s perception of data quality, some agencies indicated that the public generally understands that there is an inherent lag time between what their traveler information is based on (e.g., travelers who have just completed their trip) and the actual door-to-door time they experience. As a result, the public is generally forgiving of some inaccuracies in the information they’re given. The level of accuracy that is required for the public to make educated decisions is different from region to region, but it may not change between an urban road and a rural road in the same region. Some areas validate data with travel time runs to view it from the user experience, but this gets difficult in a larger urban setting covering more roadways. If the level of accuracy required from the data is set at a high standard, if that standard is not met, the information provided to the traveler is typically still adequate to make an informed decision.

Some agencies noted that states are placing increased emphasis on data quality, although they recognize that more could be done to improve data quality. One important reason for this emphasis is the growth and use of data archives. Agencies are seeing value in archiving their real-time information and actively trying to make it part of their culture and business processes. However, archived data generally needs to be of a higher quality than real-time data to be useful for all of its possible applications, such as transportation planning and performance monitoring. Some agencies suggested that operations staff typically have a lower threshold for data quality for real-time decision making.

Just because the agency is receiving information does not mean that it should be disseminated to the public. Transit data accuracy levels in excess of 95 percent were reported as necessary to provide real-time data to the public. Many agencies implement controls within their system to ensure that the system stops displaying information when quality falls below acceptable accuracy levels. CTA uses a predictive algorithm to determine location that will stop displaying information if they do not receive AVL data from a vehicle for five polling cycles. In such cases, not showing any information is better than showing inaccurate information.

Some traffic incident reporting systems include automatic alarms or time-out functions to be able to alert staff of information that might be outdated, which helps them to monitor the quality of information being sent out through traveler information systems. Some agencies have established manual processes for reviewing and verifying information before it gets released to the public (such as an incident or closure). Although this adds to the resources and human intervention needed to operate these systems, agencies indicated it made for a better quality product.

4.2.3 What Is the Prevailing Quality of Traveler Information?

The prevailing quality of traveler information varies greatly by location, source, and type. This section presents some findings that provide insight into the prevailing quality of traveler information for the various modes in this study.

4.2.3.1 Observed Accuracy

Perhaps one of the most exhaustive evaluations of private sector traffic data quality has been the I-95 Corridor Coalition INRIX data program. This evaluation of the probe data provided by INRIX to the I-95 Corridor Coalition presented the industry with a number of lessons learned for probe data procurements. In addition, it revealed a great deal of information regarding the data quality that can be expected from private-sector providers using nationwide probe data, although this evaluation was specific to one firm (January 2009).

Generally, the INRIX data was within 5 to 12 mph of the mean of the “ground truth” data, which was collected using Bluetooth readers. While this was the best possible method of collecting ground truth for the number of segments in the study, it was an estimate itself and introduced its own uncertainty. As a result, the INRIX data was validated against the SEM band, which was equivalent to the 95-percent confidence interval of the Bluetooth data. Using this as a comparison, the INRIX data fell within 2 to 10 mph of the SEM band. Lower levels of accuracy were measured at low speeds, while traffic traveling closer to free-flow speeds was more accurate. It must be noted that this was one evaluation, albeit over many miles in four states, of one provider of real-time traffic information. It is not possible to make broad generalizations of the data quality of other providers of similar data in other locations. I-95 in the study area carries a great deal of traffic, including truck traffic. Other locations with lower volumes—especially truck volumes, which are more likely to be data probes—may not see the same results, even for the same data provider.

While probe data has been shown to be viable for freeways, arterial travel times from private-sector probe-based systems are not considered accurate enough for traveler information. Many factors are at play including not enough data points and high variability introduced by signals and driveway turning movements.

In addition to the I-95 Corridor evaluation, several cell-phone probe evaluations have been completed in the last several years. Cell-phone triangulation-based models, which were common before GPS was prevalent, have faced challenges. Companies typically enter into partnerships with one cell phone provider and are then dependent on data points that are anonymously tracked to derive segment speeds from data points. This creates a dependence on “hand-off data,” rather than making a direct determination on vehicle location. Performance (and data quality) appears to degrade for complex networks, which could be attributed to the hand-off strategy versus a more precise location strategy.

Additional evaluations of private-sector data in recent years have revealed the following (note that the market continues to change rapidly, and new products and technologies will warrant continued evaluation of vendor offerings):
Cell phone data is viable in free-flow conditions, but is not accurate in congested conditions.
Probe-based data is generally sufficient for congestion maps (red, yellow, green) but less suitable for travel times or operations purposes.

Another evaluation of two private-sector providers found the overall mean error of the two firms’ data to be 19.4 percent and 22.9 percent, respectively. However, errors for both were significantly higher during congested conditions than uncongested conditions, suggesting that historical data was relied upon heavily to fill in gaps in real-time data. The result is several congested periods were inaccurately reported as free flow.

Of the few real-time freight information systems today, a small number have developed robust measures of data quality, including tracking observed accuracy. Public-sector staff often have a more general understanding of the accuracy of their systems, but are unaware of more in-depth measures of data quality. Often data quality is judged simply as feedback from users, such as comparing estimated border wait times to observed ground truth. Many of these border wait-time estimates are considered relatively accurate the majority of the time, even though specific metrics do not exist. Certain areas are particularly lacking in high-quality data, including areas where situational awareness is limited, such as more rural areas where there is no TMC.

The accuracy of in-cab communications systems varies based on the type of information being provided. The trucking industry considers its systems highly accurate in their ability to track vehicle location—often within 5 minutes of real time. GPS as part of a communication system is usually accurate to within 30 feet of a vehicle’s actual location. However, weather and routing information is often not as accurate as required. For example, weather information often cannot be disseminated to drivers to forecast conditions for their expected locations in several hours or how the conditions will specifically affect commercial vehicles. The accuracy of routing is often limited in areas with current construction or in metropolitan areas with complicated or changing roads. The dynamic routing provided by the system is accurate enough to be useful, although it still requires improvement. Data quality is expected to improve as more companies deploy these systems and share information, particularly larger carriers.

Table 4.2 and Table 4.3 show observed data quality metrics in the transit and freight sectors.

Table 4.2: Observed Data Quality Metrics from Sample Transit Agencies

Data Quality		 Denver RTD, Colorado King County Metro, Washington TriMet, Portland, Oregon
Desired Observed Desired Observed Desired Observed
Accuracy 98% ~70 - 75% 95% 95% 99% Unknown
Vehicle Location Update Rate (polling rate) 30 seconds 2 minutes 60 seconds 2 - 3 minutes 30 seconds 1.5 - 2 minutes
Timeliness (refresh of information for traveler displays) 15 seconds 15 seconds 15 seconds 15 seconds 20 seconds 20 seconds
Reliability 99% of time 90% of time 99% of time 95% of time 99% of time 99% of time
Table 4.3: Typical GPS Telematics Data Quality
Accuracy Within ~30 feet
Timeliness Polling ~every 5 minutes
Reliability ~98 percent

4.2.3.2 Observed Timeliness

Evaluations of private-sector data in recent years have revealed that latencies of 5 to 10 minutes are typical of probe-based systems (note that the market continues to change rapidly, and new products and technologies will warrant continued evaluation of vendor offerings).

For transit agencies, observed polling rates ranged from as infrequent as every 5 minutes to as frequent as every 30 seconds. More frequent polling allows for improved accuracy. TriMet currently polls vehicles every 90 seconds but is working to increase the polling rate to every 30 seconds to improve customer information and dispatch accuracy. CTA buses report every 2 minutes, with count-downs on their electronic displays updated every 90 seconds to minimize errors. CTA has also implemented a system to cease the display of information when necessary if bad information is being provided. The CTA predictive algorithm will stop displaying information if an AVL unit fails to provide information for 5 polling cycles. Travelers usually prefer frequent updates so that they can match transit schedules to their own in real time.

Real-time parking information systems vary in the timeliness of the information being provided to parking operators and customers. Deployed sensors relay information to a centralized server nearly instantly, including space-by-space counters, which poll for vehicle presence every few seconds. However, there is a larger disparity in terms of how often information is relayed to the public. More advanced systems update Websites and DMS every 1 to 5 minutes. However, simpler systems may not update automatically, requiring an operator to update manually, which is only done every half-hour or less. Although these less frequent updates often are adequate for most patrons, information must be updated more frequently for a system to truly be in real time and fully optimize the use of its sensors. Ideally, real-time parking information updates should be provided at least every 15 minutes to stay relevant for users.

Border agents on the northern border strive to update information for the public every 5 minutes, although this is not always possible. However, border crossing information does not always need to be updated so frequently. For example, border agents on the southern border only collect border wait time information for their own purposes and only require updates every 30 minutes. Receiving information any more frequently would not be helpful to agents as they are unable to update their staffing any faster. In this way, it is important to consider the needs of individual user groups when assessing the required timeliness of real-time information.

Another example is the Otay Mesa Border Crossing in Texas, where CBP only wants refresh rates in 30-minute intervals. Faster refreshes would not be useful because CBP cannot change its staffing any faster. However, truckers might want new information in more frequent intervals, requiring a different communications medium with different refresh rates. Real-time systems must establish data quality through contractual arrangements tied to incentives. Technologies must be verifiable and match system needs.

4.2.3.3 Observed Reliability

Reliability measures the robustness of the system and the performance of its real-time information components over time. Most transit agencies reported that reliability was extremely high, often in excess of 99 percent. AVL systems have high reliability, especially when only simple location information is being detected. Requiring a robust suite of information technology equipment on vehicles can create complex systems that can require a substantial amount of onboard, data, communications, and network maintenance to remain reliable. Under optimal conditions, smart parking systems are highly reliable, with many systems capable of exceeding 99-percent data completeness and reliability. However, extreme conditions can reduce sensor reliability substantially, including adverse weather conditions such as snow or heavy rain or customer overflows at facility gates where cars cannot be distinguished from each other.

4.3 Trends

While data quality has improved in recent years, additional improvements are necessary. Data quality can be determined in a variety of ways. For example, a transit agency that uses WiFi or cellular for its communications network can provide coverage for 99-percent of a geographic area, but that does not equate to all information being 99-percent accurate, available, and reliable due to vehicle maintenance issues, operator log-on issues, scheduling errors, and system-related issues that all contribute to the overall quality of the data being reported by the AVL system. Agencies must have pro-active onboard equipment maintenance programs, active dispatcher management of operator log-on and service restoration updates, and accurate route and schedule development processes in place to overcome these issues.

The growth in private sector data is a key component of trends in real-time traffic data quality. Public sector data, which is primarily based on inductive loops, radar, acoustic, or magnetometer, has been consistent in quality for some time. The key variable with infrastructure-based data is how well the sensors are maintained. However, where sensors are deployed and well maintained, traffic data is generally good according to public sector representatives who participated in this study. The data quality of the private sector is very different, however. Instead of having good quality in localized areas and no data elsewhere, probe data sources can boast wide area coverage, but there is a continuum of quality over that coverage that varies by factors related to from where the data points are coming. The few evaluations cited above provide a glimpse into that data quality, but as new sources of data become available, it may change quickly. It is likely that the probe data quality will only improve over time and because it can scale to a very broad coverage area, it may one day take over as the predominant source of real-time traveler information. Note that there will always be applications that rely on volumes and other data that only point sensors can provide.

4.4 Gaps in Data Quality

4.4.1 Traffic Data Quality Gaps

Another significant area where gaps exist is data quality. This includes areas where data quality must improve as well as gaps in understanding of the levels of data quality that are required for different applications and how to measure it.

4.4.1.1 Factors that Limit Accuracy

It is only possible to have complete situational awareness at the moment the information arrives. Situations change, diminishing accuracy. Additional factors such as construction on local roads may also affect accuracy. Lack of communications may also limit situational awareness, including law enforcement agencies that do not coordinate information to provide robust situational awareness for the entire region. When providing information related to road closures and forecasting, information accuracy often depends on a human-made decision. For example, if flooding is expected to shut down a major interstate, an individual in a TMC or emergency operations center (EOC) must determine the proper information to provide to travelers, including whether the road is expected to be shut down and if so when it is expected to close and when it is expected to reopen. While traffic operations managers may desire completely accurate information, providing actionable forecasts often requires information that is less than perfectly accurate.

4.4.1.2 Data Quality Standards Gap

The quality of traveler information data is not often formally measured and published. Rather, most internal assessments focus on customer satisfaction and feedback. And, one person’s assessment of data quality is often different from another’s. Past research has attempted to define reasonable standards for data quality for different applications, but actual reports on whether these standards are met are rare. Implementation of the RTIP established under SAFETEA-LU Section 1201 may help to set national benchmarks for data quality for real-time information as it defines targets for timeliness, percent availability, and percent accuracy. In addition, the RTIP proposes to establish a standard data format to exchange traffic and travel conditions on major highways among state and local government systems and the traveling public. This being said, adoption of the standards referenced by the RTSMIP (IEEE 1512, TMDD, SAE J2354) are not yet widespread.

For public sector deployments, there is rarely an impetus to measure data quality unless it is part of a federal showcase where a formal evaluation is required. Even then, many of these evaluations focus on lessons learned rather than quantitative numbers.

4.4.1.3 Probe-based Data Validation Methods Gap

With the growth in private firms providing data to the public sector, measuring data quality is becoming more important. New innovative data collection techniques rely on combining traffic data from multiple sources in order to arrive at real-time estimates. Because many of these methods are new and unproven in all conditions, public agencies cannot rely on their familiarity with known technologies (e.g., point sensors) to understand the quality of probe-based data they procure from the private sector. Therefore, there is an inherent requirement that public sector agencies validate the data they are buying against the levels specified in the contract documents.

The most comprehensive evaluation of private sector data to date is underway by the I-95 Corridor Coalition. This has become a model to follow due to its rigor, although many other such evaluations preceded it. These prior evaluations of private sector data used a diversity of metrics including binary comparisons (e.g., speed ranges against a red-yellow-green map), absolute error, percent error, errors grouped by speed ranges, and data lag. Furthermore, the I-95 evaluation identified some unforeseen issues with validating data quality. First, in measuring the accuracy of traffic data, it is important to note that the baseline of “ground truth” against which the data is to be measured must also be measured. This introduces an amount of error in the ground truth estimate that must be considered when determining the accuracy of the data provided. The I-95 evaluation used Bluetooth tracking devices to measure segment travel times for ground truth. Second, there may be a need to filter out outlier data points. Even though the data may meet quality standards in aggregate, there may be periodic outlier data points that could present complications for certain real-time applications such as posting travel times on DMS. Finally, data latency or “lag” is an important measure of quality, but it is difficult to measure. Lag is defined as the difference in time between when a traffic event takes place (e.g., a slow-down caused by an incident or a bottleneck) and when that event is reflected in the data. It was determined as part of the I-95 evaluation that this was too difficult to measure, and it was discarded—at least temporarily—as an evaluation metric.

4.4.1.4 Probe-based Data Accuracy Gap

Evaluations of private sector data in recent years have revealed the following observations (note that the market continues to change rapidly and new products and technologies will warrant continued evaluation of vendor offerings):

4.4.1.5 Sensor Maintenance Gap

Public agencies face challenges in maintaining their existing sensor deployments. With limited funds, it is often difficult to fund maintenance even if capital dollars are available for new deployments. In California, according to the Caltrans PeMS, between 60 percent and 80 percent of Caltrans sensors are functioning at any given time. This is typical, if not good, for the industry. Further, there is often little incentive to maintain sensors to a high level of accuracy unless they support a high-profile application such as ramp metering. In practice, however, real-time traffic information is typically not important enough an application to drive sensor maintenance, which negatively impacts traffic data, signifying a gap in quality.

4.4.1.6 Travel Times Forecasting Gap

While the traveler information industry is striving toward full coverage of accurate real-time data, what travelers ultimately desire is a forecast of the traffic conditions that they will face, rather than what was measured in the immediate past. Even for travelers en route, conditions may change over the course of a long trip and current measurements may not accurately reflect that. For travelers accessing pre-trip information, there is a lag between when that information is accessed and when the trip is made. Multiple private sector data providers reported that they are working toward providing travel time predictions, although they are not currently offering them. One study participant compared the state of traffic information now to the way weather reports were many years ago. When weather prediction models were less reliable than they are today, weather reports focused on what the day’s weather was. As forecasting models have become more accurate, they now focus on the expected weather over the next several days. This evolution may take place similarly in the traveler information industry as the quality of the data improves and prediction models develop.

4.4.1.7 Real-Time Construction Information Gap

While construction information is valuable information for travelers such as recorded 511 construction messages, these systems often fall short of providing real-time information on the impacts of the construction. Construction information advisories will typically indicate the nature of the construction activity, the dates that it will be in effect, the times of day that lanes will be closed, and the types of delays one might expect, but they not typically inform drivers of current conditions. Furthermore, there are times when contractors will open or close a lane at times other that what was previously scheduled. While that may not be of consequence in rural areas, in urban areas, a contractor that is late getting out of the road in time for the morning rush, despite a contractual requirement to do so, may cause additional unplanned delays.

Within a DOT, there is likely someone who knows the status of a construction lane closure at any time, either as part of a permitting process or as part of project management. Ensuring real-time lane status information is available in a common database in real-time is feasible if the appropriate business processes are in place. While other types of events such as incidents or congestion must be discovered by a DOT, the DOT or other maintaining agency should be the originator of real-time construction information.

In addition, for 511 construction systems free-form text is entered as in the comments field, which is common for lane closure reporting systems. While it may be more readable for someone calling the 511 system or viewing the Web page, it is not usable by third parties who may wish to aggregate construction information into their travel time estimates, vehicle routing algorithms, or a common nationwide real-time incident data stream. As private-sector traffic information providers are trending toward nationwide coverage in line with their changing customer base, they require an ability to scale their data aggregation methods to the entire country. Common formats across jurisdictions facilitate higher-quality nationwide information from these providers, which benefits the industry.

4.4.1.8 Barriers or Challenges to Improved Data Quality

Industry practitioners and experts indicated the following barriers or challenges to improved quality of real-time traffic data:

4.4.2 Transit Data Quality Gaps

4.4.2.1 Data Standards Gap

Presently, many inconsistencies exist across the real-time information provided by transit operators. While developing integrated systems for regional systems, there is presently no national data standard for determining the definition of “a route,” as routes often converge and split with each other. There is also no national standard to quantify what is an acceptable accuracy for real-time information. To effectively share data and integrate real-time timetables, a common data structure must be developed so that each agency’s data structure can be mapped to a common data file. There is also a lack of standardized information in determining and sharing incident reports. Developing systems that use the same structure would allow for increased interoperability and improved real-time incident reporting to travelers.

4.4.2.2 Polling Rates Gap

Most agencies deployed systems over the last 10 years with an emphasis on command and control and currently only poll vehicles every several minutes, commonly every 3 to 5 minutes. Increasing polling frequency would greatly improve data quality to support accurate real-time information, although difficulties exist in achieving more frequent rates. Increasing the polling rate requires an increase in the amount of data transferred over wireless communications networks, which is limited by the amount of bandwidth available on local networks. Large fleets with communications networks at or near capacity will have difficulty increasing their polling rate, as the large number of AVL devices interfacing on the network may cause the network to run over-capacity. Likewise, expanding network capacity can be very costly to transit operators. One potential alternative for agencies is to use variable rate polling, which polls vehicles’ variable frequencies depending on proximity to the next stop, traffic conditions, and additional factors. System designs must consider the available location polling rate in order to avoid unintentionally misleading the public about the accuracy of information provided by the predictive algorithms.

4.4.2.3 Algorithms Gap

Many transit scheduling and real-time algorithms are unable to accommodate for the range of operational situations that occur in transit fixed-route operations. For example, “short-turning” a vehicle so that it does not complete its route as intended but instead turns around to focus on the highest traffic area of its route must be documented and shared with external systems in real-time if the predictive algorithm is to act upon this change in the schedule. Such operations can cause erroneous data, frustrating customers when the appropriate actions are not taken by the agency. This data accountability is a new concept for transit to address as they commit to publicly providing better real-time traveler information. However, data accuracy does limit the use of some real-time applications. For example, TriMet has developed a real-time bus mapper, but only uses it internally because accuracy limitations would likely result in the public misinterpreting the information.

4.4.3 Parking Data Quality Gaps

4.4.3.1 Dataset Size Gap

Although improved sensors are increasing data quality, improvements still need to be made to provide parking managers with the capability for better decision making. Current sensors and algorithms are unable to compensate for outliers and extreme circumstances in parking operations, such as special events, which cause erroneous information. One potential solution for this issue is to build larger datasets. This will allow parking managers to better understand driver and operator actions, create more robust algorithms, and improve forecasting and operations. Larger datasets will also allow parking managers a more holistic view of parking within a region, including how parking in one neighborhood affects another.

4.4.3.2 Standards Gap

Due to the fragmented nature of the parking industry, real-time parking information systems lack uniform standards. To aggregate parking information and pass it along to customers, there needs to be more standardized messages in terms of communicating parking availability and attributes. Many systems use similar green, yellow, and red indicators to display parking availability, but there is no standardization for what each of the colors mean from one facility to another.

Smart parking also requires more defined data standards. For example, facility attributes can have a substantial impact on customers’ parking decisions, including entry/exit points, security features, lighting conditions, and distances to notable landmarks. Similarly, no standardized system for how to measure each of these attributes has been implemented. Those standards that do presently exist as part of the national ITS architecture are outdated or do not promote increased information sharing. In addition, a national standard created organically through collaboration of the public and private sectors is needed to allow more thorough evaluations for comparing procurement methodologies for potential implementers, improved understanding of parking effects for policymakers, and more direct comparison of facility amenities for customers.

4.4.4 Freight Data Quality Gaps

4.4.4.1 Public-Sector-Related Gaps

While many public sector agencies report reasonably accurate systems, very few keep robust data quality metrics for freight information systems. It is the responsibility of the public sector to develop and promote useful freight information data standards to facilitate information flow between entities, including sharing non-proprietary information among public and private sector entities, as well as setting data quality standards.

4.4.4.2 Private-Sector-Related Gaps

Many commercial vehicles are without in-vehicle telematics or have outdated systems. The newest systems have greater ability to dynamically reroute based on changing conditions as well as track actual drivers’ hours expended and hours available. Since congestion is getting worse, it is of growing importance to provide high-quality information to improve decision making.

4.5 Closing the Gap and Roles for the US DOT

All modes of real-time information can benefit from the establishment of standards for collected data and performance measures to evaluate the accuracy and effectiveness of the use. Agencies should benefit from a national knowledge base concerning data quality, but maintain the ability to develop performance measures that address organizational goals. The US DOT should continue efforts to support research, support the development of white papers, promote partnerships like the 511 Coalition, and sponsor meetings and workshops for information dissemination and networking purposes. Detailed studies should be conducted to analyze and document the level of data quality that real-time information systems are able to provide and should provide. Polling rates for field devices should be evaluated to determine the optimal interval for information timeliness and accuracy.

The US DOT could improve real-time traveler information by increasing real-time data exchange and setting traveler information standards. To date, many standards are not as widely used as they should be and this hinders the ability to widely share and use information, which improves data quality. Making standards freely available would support adoption. In the larger IT industry, it is recognized that one must cater to developers if one desires them to write applications for a given platform. In an attempt to fill this void, unofficial versions of standards circulate that may have errors or may be outdated, undermining the use of the standards. Secondly, the US DOT could push for existing standards to be completed to eliminate ambiguities. This may result in less-than-perfect results, but the alternative is vague or ambiguous standards that are not in fact standard. Thirdly, a more open forum should be established for sharing lessons learned as well as a more open process for standards development. Finally, there needs to be clear test procedures or validation processes so that accurate implementations of the standards can be confirmed.

[1] Mean error is actually bias, i.e., the tendency to systematically over or underestimate.

[2] Mean absolute error is a more accurate representation of accuracy since high and low measurements do not cancel each other out.

Previous | Next