5.0 Conclusion
As stated in the Section 3.0 of this final report, the overall goal of the ITS Standards Testing Program is to assess and evaluate the suitability, effectiveness, interoperability and interchangeability of standards. The measure of these three key elements is essential in understanding whether or not a particular standard is ready for field use. The conclusion is therefore stated in terms of these measures.
Additionally, the VDOT deployment and acceptance testing of the DMS devices provided a unique opportunity to not only evaluate and test the standard against the normally prescribed approach, but also to obtain first-hand feedback during the entire process and to report on both the objective and subjective results of this process from the participants. As such, this conclusion has been expanded to include the summary input of the VDOT team.
5.1 ISTT Observations
5.1.1 Suitability
The DMS standard can be considered 'suitable' as it provides the necessary features to satisfy the needs of the deploying agencies and at the same time, provides the technical detail required by the vendors and systems integrators. The suitability of the DMS standard to meet the operational needs of the user was evaluated by both interviewing the stakeholder responsible for selecting and deploying a DMS sign, and by reviewing the feedback of all involved parties associated with the VDOT deployment. The results of both of these activities yielded no significant deficiencies in term of user needs and functional requirement. Determining the level of suitability of the standard at the dialog and object level was done by conducting a static analysis of the standard's content as it was traced to the higher level functional requirements; by witnessing the test activities and the actions of the sign in conjunction with exercising these functional requirements, and in turn, the underlying dialogs and objects; and finally, by examining both the results of the ISTT and VDOT testing activities.
5.1.2 Effectiveness
The DMS standard can be considered 'effective' as it did provide the features necessary to meet the deploying agency needs and those features were presented in such a way as to make their use reasonable. As one example, the various notations used in the standard, for instance, the use of the ASN.1 notation for the object definitions, were appropriate for their intended audience. As another example, the use of UML diagrams to illustrate and support the textual descriptions of both the interfaces and dialogs was also effective and appropriate for the intended audience. Only the use of the PRL, which due to its static representation in the printed form of the standard limits the reader to a top-down approach, was cited as being somewhat less than effective. A suggestion was made that the PRL could be better served by automating the process with a software tool, similar to the Mini-Edit approach used by the TMDD.
5.1.3 Interoperability and Interchangeability
In terms of interoperability and interchangeability, the VDOT DMS deployment project and the process under which the procurement and test was conducted was irrefutable evidence of the standard's contributions to these vital attributes. By specifying the requirement of 1203 v2.25 compliance to each of the vendors as well as to the testing support team, and by maintaining isolation amongst these parties prior to the test and integration activities, both the ability of the standard to be used as a standalone reference by an implementer; an indicator of both its quality and thoroughness; as well as the standard's ability to detail interoperable and interchangeable systems, could be evaluated.
When these systems were brought together, first by testing the sign against the test cases, and then by testing the management station as integrated with the sign against the same test cases, the VDOT team was able to identify and document all discrepancies. Using an iterative and regressive approach, true anomalies in the vendor implementations where then corrected and retested, leaving only those findings against the standard. In the end, the percentage of items directly attributable to deficiencies in the standard itself was minimal, and even those deficiencies were considered.
5.1.4 Application of Systems Engineering Process
In previous versions of the DMS standard, the focus was on the definition of objects associated with the SNMP communications between the DMS field-device, and the central management station. While this practice was fairly well understood by the vendors and systems integrators, it did not readily support the needs of the deploying agency. With the application of the Systems Engineering Process (SEP) and the addition of user needs and functional requirements to the standard, the deploying agency can now use the Procurement Requirements List (PRL) to easily select their needs from a list of well defined, user needs and functional requirements, and through the traceability of the Requirements Traceability Matrix (RTM), the vendors and systems integrators can readily satisfy these needs by implementing the underlying interfaces, dialogs and objects.
The positive results of the VDOT DMS deployment effort and acceptance testing, as documented both herein, and in the VDOT Final Report generated by VDOT/VTTI are proof positive that the application of the SEP process to the standards development lifecycle, and the inclusion of user needs and requirements within the standard, are necessary steps resulting in a higher quality and more useable standard for the transportation community.
5.1.5 Overall
In previous versions of the DMS standard, the focus was on the definition of objects associated with the SNMP communications between the DMS field-device, and the central management
5.2 VTTI Observations
The experiences of specification, procurement, and testing were difficult and each one presented significant roadblocks to an agency with limited NTCIP knowledge, such as VTTI. However, the results of these activities produced many suggestions for improving the processes. Overall, VTTI feels that the project has been a resounding success.
The specification process meets the goals of creating a more user-friendly environment for an agency to develop procurements. However, VTTI feels there is room for improvement in terms of making the PRL electronically based and developing more user decision support to prevent errors. This decision support and electronic PRL can transition from specification directly into test plan development as well as testing.
There has been significant improvement to the specification process between Version 1 and Version 2. This same attention needs to be given to the testing aspect as well. The entire process from specification through testing needs to be developed in concert to ensure successful deployments.
The testing results, coupled with the fact that each component was developed in complete isolation of other components, speak volumes regarding the validity of the Version 2 foundation of traceability. The most significant aspect of the traceability to VTTI was the capability for troubleshooting problems. The testing team was able to quickly identify problems and assign responsibility. This process fostered an amicable environment between the agency and the vendor which produced very fast resolution to problems.
5.3 Trevilon Observations
The specific Trevilon findings are documented in Section 4.3. The following observations as it relates to the evaluated areas of suitability, effectiveness and contribution to interoperability and interchangeability were recorded as part of a post-test interview with Trevilon's Mr. Kenneth Vaughn.
5.3.1 Suitability
No specific issues related to suitability were made.
5.3.2 Effectiveness
The standard was not always efficient. For instance, with respect to scheduling, it was deemed 'overkill' for a message sign, but it is consistent with the NTCIP 1202 Global Objects. With respect to dialogs, there is a need to more clearly define standardized dialogues in order to ensure baseline/consistent interpretations; however as long as people understand them, they make sense and allow flexibility.
5.3.3 Interchangeability/Interoperability
Interchangeability and interoperability were met well by a consistent interface; however, there was some complexities of running the signs and putting tests in place.
5.3.4 Additional Observations
Comments that were provided in Section 4.3 were against the entire standard, the test procedures were generated for the entirely of the v2.25 standard.
5.4 LedStar Observations
The specific LedStar findings are documented in Section 4.4. The following observations as it relates to the evaluated areas of suitability, effectiveness and contribution to interoperability and interchangeability were recorded as part of a post-test interview with LedStar's Mr. Milan Patel.
5.4.1 Suitability
The standard contains all elements/features that would make it usable by end users. It addresses lots of shortcomings from the first version of the standard. Version 2 also allows for better usage of LED technology as opposed to the v1.x standard from 1996. However, certain types of signs are not well addressed by the standard. For example, the application to Hybrid signs and Speed limit + small DMS signs are not obvious. Additionally, auxiliary (trigger) inputs are not currently addressed. The standard indicates that these are outside of the scope of standard at the time it was written.
5.4.2 Effectiveness
Designed to encompass many different sign types and vendors as well as looking to the future. In the case of a basic DMS, there might be a lot of additional controls in the multi objects that do not necessarily need supported but they are there for future usage or more advance features like color signage or graphics. It is the opinion that these are not required by most DOT's for roadway signs, however, the standard appears to include more forward thinking features.
5.4.3 Interchangeability/Interoperability
The use of standard contributes positively to meeting these, but there are still ambiguities and areas left to interpretations.
Additional Observations
As Ledstar had previously implemented v1.x of the DMS standard, they made observations as it related to what they referred to as an Overlooked OID. The specific statement was as follows. "It would be helpful if section 5 (DMS Object Definitions) was available in plain text form for comparing with version 1 using visual file comparison tools". As such, the following list of OIDS presented difficulty in determining what changed between versions of the standard.
| OID | NTCIP1203v02-25 |
|---|---|
| eventClassNumEvents | D.5.2.6 |
| numEvents | D.5.7 |
| eventConfigStatus | D.5.4.9 |
| controllerStandardTimeZone | |
| controllerLocalTime | |
| defaultFlashOnActivate | 5.5.20 |
| defaultFlashOffActivate | 5.5.22 |
| defaultFontActivate | 5.5.24 |
| defaultJustificationLineActivate | 5.5.26 |
| defaultJustificationPageActivate | 5.5.28 |
| defaultPageOnTimeActivate | 5.5.30 |
| defaultPageOffTimeActivate | 5.5.32 |
| defaultBackgroundRGBActivate | 5.5.34 |
| defaultForegroundRGBActivate | 5.5.36 |
5.5 IBI Group Observations
The specific IBI Group findings are documented in Section 4.5. In addition, the following list of general comments was copied from the IBI Group Final Report, as submitted by Mr. Richard Chang.
The NTCIP 1203 v.2.25 standard is an intimidating document at first glance with 349 pages and multiple references to additional documents. It is easy to see how one might want to skip through the document and read only the parts that appear to directly relate to the goals at hand but it is beneficial, if not a necessity, for the Management Station developer to read through the entire document to ensure that a thorough understanding of the document is attained.
The need to reduce pages and redundancy in NTCIP 1203 by referencing other documents forces the reader into obtaining copies of the referenced documents. Difficulties arise when certain objects are referenced from NTCIP 1201 Global Object Definitions and that document is not easily accessible.
There are some instances in the standard where the intentions of the standard are clear, but standard does not explicitly state these intentions. For example, the document does not explicitly state that the Management Station must calculate the dmsMessageCRC when activating a message. The intention was to use this calculated value against the calculated value in the DMS controller to confirm that the message had not been modified or corrupted. This is not the case when the Management Station retrieves dmsMessageCode just prior to activating the message.
One of the major interpretation setbacks involved the standardized dialogs defined in Section 4 DMS Dialogs and Interface Specifications. In the RTM, functional requirements referencing generic dialogs described in sections 4.2.1 – 4.2.3 were interpreted incorrectly. The objects in these functional requirements were retrieved or set with multiple individual GET/SET statements instead of single GET/SET statements with multiple objects. Unfortunately, due to the written style of these sections they were interpreted more as references as opposed to instructions such as in 4.3.1.1 Retrieve a Font Definition. This interpretation variance was discovered after the first Ledstar Testing.
In addition to the writing format of Section 4.2, the opening statements of Section 4 DMS Dialogs and Interface Specifications also cause some uncertainty by stating the following:
“The Requirements Traceability Matrix contained in Annex A provides the formal tracing from each data exchange requirement contained in Clause 3.4 to either the generic dialogs defined in Clause 4.2 or a special dialog in Clause 4.3.”
Section 4.2 is titled SNMP Interface and Section 4.3 is titled Dialogs but should be labeled more appropriately to their previous reference since confusion can develop.
NOTE: The term “dialog” is defined differently for software developers. This term is most often used to describe a window type box that contains data on the screen.
The D.4.2 Global Dialogs should be reference in 4.3 Dialogs so that the reader understands there are more dialogs in another section. If ANNEX D is removed and placed in the standard referenced, this no longer becomes a concern.
The Requirements Traceability Matrix is a great addition to the standard. It allows the reader to quickly reference a functional requirement to the objects involved, interface names, and dialogs.