4.0 Implementation and Evaluation of Sacramento TRMC Weather Alert Notification System

The Caltrans District 3 Regional Transportation Management Center (RTMC) has been participating since 2007 in the FHWA weather information integration study to identify strategies to enhance TMC weather integration in support of operations, to implement selected strategies, and to participate in an evaluation of the results of their strategy implementation. The RTMC has utilized the FHWA TMC Weather Integration Self-Evaluation and Planning Guide and also prepared a weather integration plan. This section focuses on a strategy they implemented to integrate road-weather information into the RTMC operational advisory functions by implementing an automated weather alert notification system. This system is expected to provide timely traveler and road weather information to the public, particularly regarding fog, wind and frost conditions that can severely affect travel safety and mobility. This section briefly summarizes the implementation and evaluation of the system. A separate, stand-alone report⁶ provides more detailed information on those efforts.

4.1 Sacramento’s Weather Information Integration Plan

The Sacramento RTMC identified the following weather integration strategies using the Guide:

1. Frequency of road weather observations – identify the time intervals for collection of environmental sensor station data. This will reflect the level of detail for information that is required by the RTMC for decision making.
2. Extent of coverage – identify additional observations beyond the present environmental sensor station coverage that are required to provide the needed weather information for various road weather conditions to support RTMC decision making.
3. Weather information coordination – assign responsibility to an RTMC staff member for coordination of weather information/integration related activities, including training.
4. Alert notification – provide automatic alerts to RTMC personnel (and potentially others) when certain weather condition thresholds are exceeded.
5. Road weather data acquisition – identify the road weather data, including forecasted road weather conditions, needed to support RTMC decisions. Additionally, identify the level of technological sophistication needed to process and manage weather data.
6. Use of external weather information sources – identify the appropriate weather information sources, observed and predicted, not owned by the RTMC or other state agencies. Additionally, identify the integration techniques to incorporate these information sources into the RTMC decision-making processes.
7. Decision support – identify procedures and tools to integrate road weather information into the RTMC decision making processes.

There are overlaps among each of these strategies, and these interrelationships are an important consideration when planning implementation tasks. The Sacramento RTMC’s primary goal was to provide alert notification to their operators regarding existing weather conditions so that more effective public advisory notification actions could be taken in a more timely manner. The other strategies listed supported that goal. It was also decided to focus the weather integration strategies on operations associated with the Sacramento Valley.

4.2 Alert Notification System Implementation Objectives

In order to enhance their operational performance, particularly during severe winter weather events in the Sacramento Valley, the RTMC implemented a series of activities in 2009 to achieve the following five main objectives:

1. Improved coordination of operator response and decision making regarding the posting of traveler advisories during severe weather conditions;
2. Enhanced weather information coming into the RTMC;
3. Implementation of an automated alert notification system that would improve operational awareness of and response to severe weather;
4. Refined operational procedures to guide efficient and appropriate operator response to weather conditions; and,
5. Development and implementation of a training program to enhance awareness and strengthen the capability of operators to proactively use weather information, weather warnings and alerts in RTMC operations and in posting and removing advisory messages.

4.3 Alert System Implementation Tasks

The RTMC management committed to implementing the weather alert notification system through a series of tasks carried out during 2009, as follows:

1. Identify and assign an RTMC weather coordinator.
2. dentify weather information sources (observations and forecasts) and determine what is needed in addition to sources already available and access by the RTMC.
3. Properly calibrate and maintain Caltrans-owned road weather observation sensor sites.
4. Identify, procure and install additional weather observation systems.
5. Define alert mechanisms and thresholds, procedures, and timing.
6. mplement the weather alert notification system.
7. Define and implement a training program for operators and others.

Responsibilities were assigned, a schedule was established, and by December 2009 automated weather warnings and alerts were being delivered to operators 24/7 in the RTMC.

4.4 Evaluation Approach

Quantitative and qualitative data were collected to support evaluation measures in both the baseline (“before”) and post-deployment (“after”) periods. The quantitative data included weather condition readings from a set of RWIS sensors, records of warnings and alerts issued before and during weather events, information on messages posted on electronic signs in the valley, and operator entries in the Computer Aided Dispatch (CAD) TMC logs. The qualitative data were obtained through interviews with operators and managers of the RTMC, and they included observations and perceptions of changes and benefits being derived from the implementation of the weather alert notification system, and also institutional and organizational benefits received as a result of using this system. Data were collected throughout the baseline period of the implementation, and again after the strategies had been implemented and were operational for a period of time.

The evaluation focused on a few significant weather events during which the alert notification system was active. These events were analyzed in detail to understand how well the warnings and alerts tracked the actual weather conditions and how well the operators were able to use those warnings and alerts in supporting their decisions to post messages for the public. The evaluation also addressed the institutional process by which management sought to implement the system, make adjustments based on their real-time experiences with the system, and incorporate feedback from the operators.

The evaluation analysis compared outcomes between the “before” and “after” periods as a basis for identifying changes and benefits that could be attributed to the new integration strategies. Lessons learned were derived from the total implementation experience for the benefit of Caltrans and other TMCs that are considering enhancing their own level of weather integration.

4.5 Baseline Conditions and Challenges

The information collected during the baseline evaluation period identified needs that were closely aligned with the selection of weather integration strategies the RTMC intended to pursue and the tasks identified in their Implementation Plan. Table 20 below summarizes the primary baseline condition findings and corresponding implementation plan tasks that addressed these issues.

Table 20. Summary of Baseline Conditions and the Relationship to Implementation Plan Tasks

Baseline Conditions and Challenges	Implementation Tasks
General lack of focused and coordinated use of weather information.	Task 1: Identify and assign a RTMC weather coordinator.
RWIS sensors inaccurate and unreliable. No confidence in sensor data.	Task 3: Properly maintain/calibrate Caltrans RWIS data stations.
Lack of knowledge of all possible weather information sources that could be used to identify or confirm weather events.	Task 2: Identify weather information sources (observations and forecasts) available or accessible.
Lack of coverage of sensor data, mostly in areas that experience dense fog.	Task 4: Identify, procure, and install additional weather observation systems.
Too much dependency on field personnel to determine when a weather event begins and ends – their input is not always accurate or timely.	Task 5: Define alert mechanisms, thresholds, and procedures. Task 6: Implement weather alert system.
Lack of procedures or guidance regarding operations during inclement weather events.	Task 5: Define alert mechanisms, thresholds, and procedures.
Inconsistent response by the various RTMC operators during weather events.	Task 7: Define and implement training program.
Difficult to know when a weather event is beginning.	Task 6: Implement weather alert system.
Difficult to know when a weather event has ended.	Task 6: Implement weather alert system. (Note: future enhancements to the alert system are planned to address the issue of when events end.)
Significant delays between when an event begins and when signs are activated	All Tasks: Implementation of a weather alert system, operator training, and consistent use of operational procedures.

 

4.6 Alert Notification System Implementation

The RTMC management decided to build their automated alert notification system by activating a feature of the SCAN Web⁷ system (called SCAN Sentry) that issues email alerts when certain conditions (weather data thresholds) are met. This approach is limited to the six RWIS station locations in the Sacramento Valley and was selected as the easiest and least costly way to get an alert system up and running. The RTMC management saw this as a way to test and demonstrate how an alert system might help the operators, with the thought that it could be expanded in the future if it proved successful. The results of this evaluation will help the RTMC management decide if an expanded alert system is required to continue improving weather responsive operations.

The following is a summary of how the implemented alert system works:

1. Weather data from the six RWIS stations identified in Task 2 are posted every 10 minutes to the SCAN Web database. This includes all sensor data available at each location.
2. The SCAN Sentry system (software routine with access to the SCAN Web database) compares the weather measurements to the established warnings and alerts. The warnings and alerts identified above were incorporated in the SCAN Sentry system by RTMC management.
3. When the warning or alert conditions are met based on established thresholds, an email is sent to RTMC management and operators.
4. The SCAN Sentry system allows the manager to limit the number of alerts issued if the conditions persist. The maximum time (suspend time) between alerts is 3 hours. If a wind event continues to exceed (or repeatedly fluctuate through) the threshold for several hours, or even days, the operator will continue to receive alerts every 3 hours (but, not every 10 minutes).
5. The SCAN Sentry system does not have the capability of issuing an alert when the thresholds are no longer met, or not met for a given period of time, so operators are not alerted when weather conditions drop below thresholds. This is a limitation that the RTMC management would like to rectify with a new alert system in the future (second approach above).

The SCAN Sentry alert system was activated in the fall of 2009 and began issuing warnings and alerts based on the established weather data thresholds.

4.7 Alert Notification System Performance

Four significant fog and wind events that occurred between December 2009 and April 2010 were selected for a careful assessment, as case studies, of the performance of the weather alert notification system. These included:

1. A fog event that occurred December 17-18, 2009. While fog persisted throughout the valley over these two days, visibility conditions actually dropped below the indicated threshold in two distinct periods such that, for the purposes of assessing messaging performance, this can be interpreted as two separate fog events over this period of time.
2. A particularly severe and persistent wind event that occurred January 17-21, 2010, resulting in 176 alerts being issued.
3. A wind event that occurred March 12-13, 2010.
4. A wind event that occurred April 27-28, 2010.

Each event was evaluated with regard to 1) the timing and duration of the event, trends in the weather measure (sight distance for fog, average sustained speed and gust speed for wind, and temperature for frost) and when those measures crossed their pre-defined threshold, 2) the timing of issuance of automatically generated warnings and alerts that were received by RTMC operators via email, and 3) the timing of traveler advisory message activation and deactivation by the operators during the event. In assessing the RTMC operator responses to these weather events, several indicators were considered:

• Were the warnings and alerts issued appropriately and according to the designated thresholds?
• To what extent was the event covered by messaging to the public?
• Were the appropriate message signs activated based on receipt of alerts and readings from the various sensor sites?
• Were signs deactivated in an appropriate and reasonably timely way?
• Did the operators record information about the event and their decisions in the TMC log?

Follow-up interviews were conducted with four of the RTMC operators in June 2010 after they had experience with the weather alert notification system. The purpose of this interview was to learn how the alert system has been working from the operators’ perspective. Some of the same questions were addressed for this interview as had been covered in the baseline interviews conducted a year earlier. Operators in the follow-up interviews were asked to describe how they used the new alert system and whether it was helpful to them in improving the efficiency of their messaging decisions. The new procedures were discussed along with the factors operators consider when making advisory decisions. The analysis of some of the weather events from this evaluation were shared with the operators.

4.8 Evaluation Findings and Lessons Learned

The evaluation of the Sacramento RTMC weather alert notification system examined several adverse weather events in some detail in order to assess quantitatively how the alert system was performing and how the operators were able to use it in supporting their operational decisions regarding posting of advisory messages. The quantitative analysis focused primarily on the post-implementation data obtained from the RWIS sensors, the alert system records, and message sign records. Qualitative findings are based on interviews with selected operators before and after the implementation of the alert notification system.

The quantitative findings include the following:

• Timeliness of Alerts. Alerts should be issued to coincide with the start of an event, when conditions exceed the defined threshold. Across the four event periods analyzed, alerts were issued for the most part in a timely and accurate way; that is, 16 out of 18 times (for individual RWIS sensors) they were issued within +/- an average of 10 minutes of the time when the weather condition broke its defined threshold value at the beginning of the event. This is virtually right on time, given measurement error. Interpretation: The alerts were mostly well timed, indicating the alert notification system was working as planned.
• Timeliness of Message Activation. The RTMC aims to have messages posted on appropriate message signs for the duration of a weather event that exceeds the defined threshold. Fifteen individual sensor-reported weather events that exceeded threshold and/or lasted longer than 16 minutes should properly have had weather warning messages posted throughout those events; 13 of them did and two had no messages posted. For those event segments with some message coverage, coverage ranged between 27% and 100% of the duration of the event. Out of the 13 events with message coverage, 11 had coverage over 75% of the duration of the event. Interpretation: Messaging coverage was generally good but not complete. However, coverage improved over the duration of the evaluation from December 2009 to April 2010, suggesting that the alerts were helping operators post messages more appropriately.
• Adequacy of Messaging Coverage. The second dimension of messaging adequacy related to the posting of messages triggered by sensor alerts on the primary signs near the sensor site. As discussed in this report, there are a number of mitigating circumstances that might reasonably prevent posting weather messages on some of these signs. For example, during road emergencies the signs may be needed for traveler alerts, and during mountain snow events, some valley signs may be used for chain control advisories. Nevertheless, across all these case study events, the number of CMS, EMS and LED signs used was significantly less than the number of signs recommended in the RTMC policy guidelines. For the December fog event, out of 8 opportunities to activate primary signs for significant events, only 2 were used, along with 2 out of 8 secondary signs. For the three wind events taken together, out of 43 opportunities to activate the primary signs for significant events, 12 were used (28%). However, the ratio of messaging on primary signs improved over time, with 7 out of 17 (41%) used in the April wind event. Interpretation: A low number of primary message signs had messages posted during the weather event case studies examined, but coverage improved over the course of the evaluation period, presumably due to the operators’ increasing familiarity with and understanding of the new procedures.
• Timeliness of Message Deactivation. Once a message has been posted advising the traveling public about an existing severe weather condition (e.g., dense fog, high winds, frost on road), the operators need to periodically monitor conditions. They need to remove the message after the weather condition has abated and it is reasonable to assume it is not about to return to threshold conditions soon. This is a judgment call, as the alert notification system does not provide explicit guidance regarding when a weather event is over. The alert notification system will continue to provide warnings and alerts in three hours or longer intervals as long as the weather conditions persist, which is helpful to the operators in maintaining their awareness of the status of these conditions. It is considered prudent to leave a message active for a while as conditions are improving in order to avoid frequently activating and deactivating messages. On the other hand, leaving a weather warning message posted long after the event is over may lead to a loss of public trust in the messages. The RTMC management is considering how their alert notification system might be able to issue an “all clear” signal based either on the length of time that passes after last crossing the threshold with an improving trend or the time when conditions reach a designated level after last crossing the threshold.
  
  For 14 sensor-covered event segments for which messages were activated among the four case studies, the period from the end of the event to message deactivation ranged from 18 minutes to 8 hours and 44 minutes. This extended period of message activation equaled an average lag time of 4 hours and 14 minutes. This is the average period of time during which a message was posted on a roadside message sign indicating an adverse weather condition after that condition no longer exceeded the defined threshold. Without knowing the detailed circumstances associated with each of these events, it is not appropriate to make a definitive judgment about them. For example, operators often rely on CHP in the field to verify conditions for activating messages, but after the weather has improved, CHP is usually no longer available at those locations to advise the operators to deactivate the message.
  Interpretation: While the RTMC has not specified an appropriate amount of time to leave a message actively displaying an advisory about weather that has since subsided, nor implemented alerts to signal that time for the operators, the experience with many of these sensor-covered event segments shows there were a number of periods during which messages were left active much longer than needed or desired.

Findings from the qualitative interviews conducted in the baseline and post-alert periods included the following:

• In the baseline period the operators reported they would like to have more frequent weather updates. They also said they lacked adequate weather readings for some important locations in the valley.
• In both the before and after periods, operators expressed less than high confidence in the quality and accuracy of the weather data they were receiving in the RTMC. They uniformly said it was critical to confirm either data from sensors with human observations or readings from other sensors, or human observations with sensor readings. This follows management guidance that all weather data be verified with one or more additional sources before making a decision to post a travel advisory message. They did, however, perceive that sensor data quality had improved over this period.
• Operators in both periods felt that operating procedures and guidelines for making decisions based on reported weather conditions were less than adequate. Some said they desired additional training and consistent information on how to respond to the weather data (observations and forecasts) that they receive in the RTMC. Other experienced operators felt they knew how best to perform their job responsibilities without need for additional management oversight.
• RTMC operators want to be proactive with regard to weather. They want to be aware of impending weather conditions likely to affect traffic in advance if possible so that they are well prepared to respond in an appropriate and timely way. However, they feel that in practice they tend to primarily be reactive to weather, and as long as they receive the information they need in a clear and timely way, they respond appropriately.
• In the baseline period the operations floor was staffed most of the time with two operators who then could share the workload. In the later period after furloughs and staff reductions, there was typically only one operator. This meant responses to weather events had to take second priority to higher priority safety matters, and there would be response delays due to very busy shift activity.
• The operators valued the warnings and alerts that were available to them in the post-implementation period, though they felt it necessary to verify their accuracy before taking any actions based on them. They said the automated alert system has made them more aware and allowed them to be more responsive to events as they unfold.

Several lessons can be drawn from this evaluation of the experience of the Caltrans District 3 RTMC and their efforts to establish and refine an automated weather alert notification system. What has been learned, and continues to be learned, as this system matures and becomes more a part of the operating procedures of the RTMC, can be helpful to both Caltrans and other TMCs across the country as they explore ways to integrate weather information into their operations. These lessons include the following:

• Operator training is essential for successful weather integration. Both the RTMC management and the operators recognized the importance of training to help assure a well informed and consistent use of the new weather alert notification system. Providing this training to all the operators as a group has been a challenge in the face of the recent staff reductions and furlough policy enacted by the State of California. The training content should include clear operational policy guidance along with conveying to the operators an understanding of the system upgrades and changes, how and why they have been made, and how these affect the weather information flowing into the RTMC. A challenge is to strike an appropriate balance between the level of specificity in operational guidance for taking action in response to weather, and providing flexibility for the operators to use their experience and judgment in making decisions about their advisory and control actions.
• Alert notification procedures need to be clearly and consistently specified. It is important that the thresholds for issuing warnings and alerts that are programmed into the notification system be consistent with the specifications communicated to operators in the written procedures and training jnstructions, and that the operators understand and follow these procedures. The Caltrans District 3 procedures call for operators to verify an alert with information from adjacent RWIS sensors, available third party weather services (NWS, AccuWeather, local weather reports, etc.), and/or field observers (typically CHP, sometimes general public calling in). Reconciling differences among these information sources about a particular weather event condition takes experience and judgment on the part of operators. Procedures and training must account for the complexity of operator decision making based on information of varying accuracy, reliability and geographic focus.
• A successful demonstration of an alert system depends on a well-integrated system. The Caltrans District 3 system is built off of their existing SCAN Web software that monitors data from RWIS sensors and can be programmed to issue warnings and alerts when pre-defined weather condition thresholds are reached. The success of this system depends on accurate and reliable weather data from the sensors, appropriately defined threshold conditions, clear communications of alerts to the operators, procedures in place that guide operator responses, and operator training and buy-in to assure effective use of the information. The RTMC management has remained flexible and responsive throughout this demonstration period to understand where their alert notification system could be fine-tuned and improved as they experienced its use under various weather events. This has provided a foundational experience upon which they can consider a more robust alert system for the future that adds features and capabilities that are not currently available with existing hardware and software. For example, adding new strategically located sensors, upgrading the weather detection capabilities of the sensors, adding better detection and notification of the end of a weather event, adding possible visual and auditory notification in the RTMC, and refining their procedures, are all candidate improvements that have been identified in the course of operating the current system through this weather integration demonstration.
• Time and resource constraints affect the performance of an alert notification system. The State of California is experiencing a severe economic downturn that has resulted in reduced staffing and furloughs among TMC management and operators. This has raised the stresses associated with getting the day-to-day work done and made it more difficult to integrate the weather alert notification system into TMC operations. Operators have competing priorities and fewer staff to meet these responsibilities. Management faces similar constraints, resulting is less time to focus on new weather integration initiatives such as the alert notification system. TMC management also is constrained by time consuming procedural requirements of Caltrans associated with the implementation of new projects. These kinds of constraints need to be anticipated and understood when implementing new systems like this and contingency plans developed to overcome the constraints. There were several planned activities that were not accomplished due to funding or time constraints, including installing new RWIS sites and developing a more sophisticated alert system with enhanced capabilities. Based on the progress made to date, the RTMC management intends to improve their alert system as new funding can be secured.

In implementing the weather alert notification system, management developed a good step-by-step implementation plan that has guided them through the process. A critical task early in this process was to engage a contractor to calibrate the RTMC’s field sensors. In the baseline period the operators reported having very little confidence in the data they were receiving from the sensors. After sensor recalibration and implementation of the alert notification system, operator confidence improved, though there remained some carryover of the perception that these data were still suspect. Operator training can help overcome such skepticism by explaining clearly what has been done to improve the data quality in the system and providing evidence that shows these improvements.

The use of the TMC logs offers a good example of how operators are learning to work with the alert notification system. The new procedures have emphasized the importance of making log entries that document and explain the actions operators have taken in response to receipt of the alerts, and the operators’ logging performance has improved over this period. RTMC management also has made good progress implementing this system and responding in real time to the need for mid-course adjustments, refinement of procedures, and oversight of the operators. The evaluation process has served to identify ways the notification system, and the institutional support for the system, could be refined, and the result of this collaborative interaction with the RTMC is reflected in the benefits being derived from the alert notification system. Ultimately, it is the traveling public that is the beneficiary of these RTMC system innovations in terms of enhanced mobility and safety during periods of inclement weather and dangerous road conditions.

 

---

⁶Cluett, C. and Kitchener, F. (2010). Implementation and Evaluation of the Sacramento Regional Transportation Management Center Weather Alert Notification System. (Report No. FHWA-JPO-10-063. NTL No. 14969). Washington, DC: Federal Highway Administration.

⁷ SCAN Web® is a registered trade mark of Surface Systems, Inc.