5.0 System Performance Evaluation

The purpose of the system performance evaluation was to assess how well the ISSES performed in the field, relative to its design and its intended use as described by the system vendor through information such as product literature, specifications, and training materials. The evaluation team attempted to determine the performance of the radiation monitor, the thermal inspection system, and the laser scanner.

As detailed below, the system in Laurel County appears to perform reasonably well in comparison with expectations, and considering that it is the first installation of its kind in the nation. The system displayed real-time visual and digital-format information about the trucks passing through the ISSES portal, permitted users to scan retrospectively through data screens showing visual imagery and digital data on previous passing vehicles, and produced usable data archives from the various subsystems, with some limitations. The user interfaces in the scale house were intuitive and seemed to be easy to learn, given appropriate training.

This assessment was affected by several important factors:

• The local ISSES was not yet integrated with any state or national databases of historical safety, inspection, out of service (OOS), or registration/licensing information, so it was operating in a stand-alone mode.
• The staffing levels at Laurel County were such that no KVE inspectors were assigned to use the ISSES as part of their mainstream job duties. The system was in place and operating during the evaluation period, but as noted in the user acceptance section below, in general no one was attending to the information shown on the ISSES display screens. The inspectors appeared to consider the system to be still something of an experimental or test prototype rather than an integral tool to achieving their day-to-day safety and law enforcement goals.
• Related to the previous factor, the deployment took place in a larger enforcement context that has up to now measured safety improvements (and provided incentives to inspectors) based on the numbers of inspections completed, not based on achieving high rates of OOS orders among a set number of inspections completed. Thus the purpose of the ISSES (to help inspectors focus on the trucks with the worst safety records, and in effect drive upward the rate of OOS orders) is not directly aligned with the current organizational goals of the inspectors in Kentucky. Both approaches, i.e., the productivity of completing a large number of inspections and the efficiency or effectiveness of identifying a high proportion of safety violations per time period, are valid goals of commercial vehicle inspection. This institutional disconnect in Kentucky, however, affected the degree to which the inspectors perceived the ISSES as helping them achieve their personal and organizational job goals.
• At the request of FMCSA, the evaluation team was asked to disregard the performance of the ALPR and USDOT number reader systems. These two subsystems, which if effective could help KVE achieve important safety screening goals, did appear to be operating during the evaluation period. Both the ALPR and the USDOT number reader were capturing and logging visual image and digital-format data on some portion of the commercial vehicles passing through the ISSES portal, but no quantitative evaluation of these subsystems was made. KTC is evaluating the performance of these systems as part of the state's CVISN program.
• Some of the data that were planned to be collected (e.g., electronic screening bypass data for the first week of the field observation and thermal imaging video data from the Laurel County site) were not available, owing to unforeseen hardware problems and communication gaps or misunderstandings between the evaluation team and the system vendor. Also, because no inspectors were using the ISSES thermal imaging system under normal circumstances, the thermal imaging video that was made available to the evaluators was from a special two-day training session, which resulted in the quality of the image data available for analysis being relatively poor and uneven.

5.1 Data Collection

Data were collected on all of the relevant ISSES subsystems from information provided by the system vendor and from site visits to two Kentucky weigh stations. Specifically, training materials and design specifications describing ISSES and its components were provided by the system vendors, Transportation Security Technologies LLC (TransTech), an affiliate of Intelligent Imaging Systems (IIS). Field inspection reports prepared during the ISSES field studies and training exercises in Laurel County and Kenton County were provided to the research team by the Kentucky Justice and Public Safety Cabinet's Department of Vehicle Enforcement (also known as Kentucky Vehicle Enforcement, or KVE). ISSES output generated during the field observation period (June 11 through 22, 2007, at Laurel County) and during a two-day training exercise at Kenton County (July 31 to August 1, 2007) was provided to the research team by TransTech/IIS between June and September of 2007. The research team also visited the Laurel County weigh station in mid-2005 and in June 2007, and visited both Laurel and Kenton in January 2007 to gather feedback from KVE personnel and observe ISSES performance directly.

The following subsections describe the technologies that comprise ISSES, provide examples of output expected to be generated by the system, and summarize the observations made by on-site personnel during the Laurel and Kenton site visits.

5.1.1 System Components, Configurations, and Outputs

ISSES consists of the following component technologies:

• A vehicle detection and classification system, which uses a laser rangefinder to detect commercial motor vehicles (CMVs) and measure their speed, height, width, and length, facilitating the identification of vehicle types based on key characteristics (e.g., number of axles).
• An overhead camera that documents the passing of each CMV by capturing an image of the vehicle (Figure 5-1).
• A radiation detection system, which measures gamma and neutron radiation levels, to help inspectors recognize potentially hazardous material shipments and cargo. Inspectors at the station were also provided with a hand-held radiation detection and identification device, which can be used once a truck has been parked for closer inspection.
• A thermal imaging inspection system, which displays and records IR and visible video of the CMVs as they pass through the sensors, allowing inspectors to detect thermal/visual anomalies.
• An automatic license plate recognition (ALPR) system, which captures and stores wide-angle and narrow-angle digital images of the front of passing CMVs and performs OCR on the tractor front license plate numbers.
• A digital USDOT number recognition system, which captures digital images of the sides of passing CMV tractors and performs OCR on the USDOT number posted on the side of each tractor.

Several of the ISSES subsystems have dedicated computer servers located in the scale house at the weigh station. ISSES is designed for installation at CMV weigh stations, where it can be used by weigh station operators to identify potential problems and/or safety concerns with passing CMVs and to compile CMV traffic data and other statistics. The following sections detail the configurations and outputs of individual ISSES components.

Laser Scanner (Vehicle Detection)

The ISSES laser scanner serves two purposes. First, it acts as the main trigger for the system: when a CMV passes through the detection zone, the laser rangefinder signals the overhead camera, ALPR system, and USDOT number reader to begin collecting data. Second, the scanner generates data on the vehicle's length, width, height, speed, and number of axles as the vehicle passes through its two laser beams. These data can used for vehicle classification. At Laurel, as currently configured by the vendor, the data on length, number of axles, speed, and width is collected by the ISSES laser scanner/sensor system, but is not stored. The vendor determined that such information had limited value in terms of inspection operations at the time of deployment. Once the ISSES is connected to external data sources and automated screening rules are programmed, the state may choose to set alarm thresholds for width, length, etc., and store these data for later analysis if desired.

Under normal conditions, laser scanner output consists of five messages for each vehicle that passes through its field of detection. These messages typically are generated in the following order:

1. Vehicle Detection - Beam #1
2. Vehicle Detection - Beam #2
3. End of Vehicle - Beam #1
4. End of Vehicle - Beam #2
5. Vehicle Classification.

The first four messages detect the presence of a vehicle and its position in the lane. The fifth and final message transmits the vehicle's speed and physical characteristics (length, height, width, and number of axles), classifies the vehicle, and provides a degree of confidence for the classification result. For validation purposes, vehicle classification information can be cross-checked against images from the overhead camera (see example in Figure 5-1).

Figure 5-2 shows some of the ISSES components. The laser scanner apparatus is at right center, aimed downward at a slight angle toward the roadway. Four auxiliary photocell (conventional light beam) emitters/receivers are mounted in an "X" pattern on the upstream support poles of the two square, raised radiation portal monitors on either side of the roadway. These detection devices appear as small gray boxes in Figure 5-2. These electronic beams supplement the laser triggering system that detects the beginning and end of each passing vehicle.

Radiation Portal Monitor

A key role of ISSES is to alert weigh station operators of potentially dangerous vehicles and cargo. The radiation portal monitor (NucSafe Inc.) allows ISSES users to determine whether the bulk levels of gamma and neutron radiation emitted from passing CMVs are above or below a preset threshold. Unlike the Laurel County system, the ISSES deployed at Kenton County has an isotope identification feature in addition to the capability to measure bulk radiation levels against a preset threshold. As shown in Figure 5-2, the portal at Laurel is configured as a pair of sensor panels mounted on opposing towers on either side of the weigh station vehicle lane. The panels face each other and detect the neutron and gamma rays emitted by vehicles that pass between them. If either type of radiation level exceeds a user-defined threshold, the system initiates a neutron or gamma alarm that can be seen and heard by the ISSES user, via the user interface. The gamma alarm sounds different from the neutron alarm, and alarm intensity is governed by the radioactivity of the cargo. For example, items that emit higher levels of gamma or neutron rays produce louder and higher-pitched alarms than items that emit lower levels of radiation. Once an alarm sounds, the user can then decide whether to divert the vehicle that triggered the alarm for inspection. The radiation portal monitor allows weigh station operators to monitor for dangerous cargo without slowing traffic.

A number of output files are generated by the radiation detection system and stored on the radiation (or rad) server. All of these files follow a naming convention that includes the date of "occupancy" (i.e., passage of a single CMV through the portal), the time of occupancy, and the vehicle's sequential occupancy number (i.e., the vehicle's place within the rolling "count" of vehicles that have passed through the portal). An example filename is provided below:

The following file types (extensions) are generated by the radiation monitoring software. In some instances, the sample contents have been edited to protect the vendor's proprietary information, while providing a general idea of the function of each file type.

• OCC – This file type contains details about each CMV as it passes through the ISSES station. Contained within the file are the duration of occupancy in seconds (circled) and the status of both the neutron and gamma alarms for that CMV. Sample contents of an OCC file are provided below:

• JPG – These image files save the count rate and alarm profile for a particular occupancy. Sample JPG files from the Kenton and Laurel sites are shown in Figure 5-3 below. While the underlying data are the same, the two JPG images are configured differently. The Kenton images depict a timeline, whereas the Laurel image superimposes a plan view of a generic commercial vehicle. The superimposed vehicle image remains the same no matter the configuration of the actual vehicle. The computer image is drawn as a semi tractor-trailer to reflect the predominant truck configuration passing through the portal. In the case of alarms, both formats provide the inspector or analyst with a visual cue as to the location of the emitting source relative to the geometry of the vehicle. Note the varying time periods of occupancy, ranging from 3.8 seconds at Laurel (c) to 5.0 seconds on the Kenton gamma alarm image (a), reflecting varying travel speeds of the three trucks through the radiation portal monitor.

• BAL – This NucSafe file type is generated but not normally saved by the system. It is a message file indicating that a radiation alarm has started; the file also records the time at which this has occurred. The following is an example of BAL file contents:

  07:01:45
  Gamma Count Rate Alert

• DATA – This file type, similar to the OCC file, is generated by the ISSES software, not by NucSafe. Thus, DATA files are not associated with the radiation portal monitoring function and are unrelated to safety enforcement.
• ID – This file type, generated only by ISSES when configured with isotope identification (e.g., at Kenton), contains a spectroscopy report. The file is generated only when a radiation alarm is activated and spectroscopy is performed. The contents of a sample ID file are presented below:

• NAF – These NucSafe Alarm Files contain the start and end times of the alarm state for gross counting purposes and for each energy band region of interest. When all alarms clear, the peak dose rate is recorded. Two examples of NAF contents are shown below (one in which no alarm was activated [1] and one in which a gamma alarm was activated [2]):

  [1] 13:10:05 No Alarm
  
  [2] 06-12-2007 05:00:08 End Alarm-CPS: 4464 Lvl: 11.12586
  06-12-2007 05:00:08 End Alarm-CPS: 4503 Lvl: 14.07915
  06-12-2007 05:00:08 End Alarm-CPS: 3284 Lvl: 10.62847
  06-12-2007 05:00:08 End Alarm-CPS: 3323 Lvl: 13.57283
  06-12-2007 05:00:08 End Alarm MAX Dose Rate: 0.00594 mR/hr

• SPC – This file type contains the spectra stored by the spectroscopy package used to generate the ID report. An excerpt from an SPC file (abbreviated for display purposes) is shown below:
  Length : 1024
  Realtime : 1.25
  Livetime : 9.99
  Deadtime :0.999
  CalibPoint1 : 9999999
  CalibCoeff : a=0 b=3 c=0 d=0
  SpectrumText : Nucsafe 07-31-2007 10:01:42

Infrared/Thermal Imaging System

The ISSES infrared (IR)/thermal imaging system uses heat signatures to help weigh station inspectors identify possible heat-related defects or malfunctioning equipment on a CMV that might not be readily visible otherwise. Figure 5-4 shows the thermal image cameras. As described below, one camera captures IR, and the other captures mixed IR/color images. Users of the system are trained to recognize defects or safety concerns characterized by unusually high temperatures (e.g., overheated brakes or tires) or abnormally low temperatures (e.g., brakes that should be generating heat during use but are not engaged), which show up on the video as exceptionally bright or dark spots, respectively. The system can help inspectors identify a variety of problems beyond brake issues, such as overheated wheel bearings, faulty universal joints, leaking (hot) fluids, and flat or damaged tires. The inspector uses the relative differences in brightness and darkness to discriminate between normally operating components (usually wheels and tires) and potentially malfunctioning or unsafe components.

Two video feeds are captured from the thermal inspection device, which houses one full-color video camera and one thermal/IR video camera. Using controls in the scale house, an inspector can "fade" (superimpose or overlay) the screen image from visible/color to IR views, to obtain the optimum image for a given vehicle or ambient lighting condition. A third video, separate from but integrated with the thermal imaging system, is captured by a gable-mounted color overview camera that has pan-tilt-zoom capability, but is normally focused on the ISSES truck portal. The images are archived on a digital video recorder (DVR) for approximately one month or until space is needed. Figure 5-5 provides an example of a still image from the composite video captured at the Kenton site. The three main images show color video from the gable-mounted overview camera (top left); IR (top right); and overlay, or combined color/IR (lower left). In the sample shown in Figure 5-5, the fade control had been set by the user to full IR, so that the two close-up views of the passing truck appear to be identical. This was a user setting and does not represent a hardware fault.

USDOT Number Reader and ALPR System

Although the USDOT number reader and the ALPR subsystems were outside the scope of this evaluation, descriptions of the two subsystems are provided here for reference purposes. ISSES uses OCR systems as the basis for its ALPR system and USDOT number reader. OCR technologies provide digital strings of license plate characters and USDOT numbers-as well as visual imagery-to weigh station inspectors without disrupting the flow of commerce. The ALPR relies on pulsed IR lighting technology, while the USDOT number reader relies on visible lighting, including lights specifically installed for nighttime operation of the USDOT number reader. For the ALPR at Laurel, ISSES uses two cameras to achieve the resolution needed for OCR. Working in tandem, the two cameras provide a greater field of view so that license plate images are captured no matter where they are located on the vehicle, and wherever the vehicle might be in the lane. Figure 5-6 shows the two ALPR cameras, installed with slightly different orientations. One camera is pointed more toward the right side of the vehicle front, and the other camera is pointed more toward the left front. Figure 5-7 shows the configuration of the USDOT number reader camera, located at about the height of the center of the tractor (cab) door. According to the vendor, increases in digital resolution and imaging technology have enabled newer deployments of ISSES to function with a single ALPR camera. Also, a new version of the USDOT number reader using IR lighting is at the testing stage and will be deployed in Kentucky in the future.

ALPR output consists of JPEG images of the target vehicle's front license plate. While the cameras capture four images (two wide-angle and two narrow-angle), the data storage system shows the best two images for each vehicle - one narrow-angle/focused image of the plate and one wide-angle image of the front of the vehicle. The file names of ALPR output files include the date and time of the reading, along with a decoded text string representing the OCR reading of the license plate number, and a confidence level associated with the reading. If the digits on the license plate cannot be read by the system, the file names contain the term "no-read" in place of the license plate number. Examples of images generated by the ALPR system are provided in Figure 5-8.

USDOT number reader output consists of one or two files per CMV, depending on whether the OCR sensor is able to recognize the USDOT number on the side of the vehicle. When the digits can be read, an event file is generated with a file name that includes the date and time of the reading, a decoded text string representing the OCR reading of the USDOT number, and a series of numbers representing the confidence of the reading and status of the record. In such cases (i.e., when the number is legible), a JPEG image of the portion of the vehicle containing the USDOT number is also generated; the date/time stamp in the JPEG file name corresponds with that in the event file name. In these cases only an event file is generated, with the word "Fail" in the file name where the OCR reading would otherwise be. When the digits cannot be read, a JPEG file is created but, based on a decision by the vendor, the file is discarded and is not passed to the storage archive. This decision was driven in part to conserve disk space and to improve operational performance. Another factor at this stage of testing and deployment was that inspectors working under tight time constraints would ordinarily have little time available to review old image files. It would be helpful in future systems to program the system to archive a JPEG file automatically in such cases, so that analysts could attempt to identify trucks retrospectively, even when the OCR system did not read the USDOT number successfully. An example JPEG image generated by the USDOT number reader is shown in Figure 5-9.

5.1.2 Field Data Collection and Direct Observations

ISSES deployments at weigh stations in Laurel County (northbound) and Kenton County (southbound) were included in this evaluation. A description of these sites is provided in Sections 1 through 3 above. The data used to assess system performance were collected during the following time periods:

• Laurel County station - 12:00 AM on June 11, 2007 through 11:59 PM on June 22, 2007 (12 days); and
• Kenton County station - 12:00 AM on July 31, 2007 through 11:59 PM on August 1, 2007 (2 days).

As noted in Section 5.1, the research team visited the Laurel County site several times during the course of the field study. During these visits, members of the research team conferred with KVE CMV inspectors, CMV police officers, and with vendor representatives from TransTech/IIS. Team members also observed the inspection selection process from inside the weigh station scale house, and they observed and photographed several inspections taking place. Observations made during these visits are reported below.

Research team members who visited the two deployment sites noted several structural or design differences between them, as outlined in Table 5-1.

Table 5-1. Comparison of Laurel and Kenton station configurations.

ISSES Feature	Laurel	Kenton
Lighting fixtures for USDOT number reader	On passenger side, facing toward mainline	On driver side, facing away from mainline
Electrical supply conduits for lighting fixtures	Visible	Hidden
Mainline WIM scale	Yes	No
Sorter-Lane WIM scale	Yes	Yes
ISSES location relative to scale house	Closer to scale house	Further upstream from scale house

The location of the visible lighting fixtures was changed at Kenton to reduce the amount of stray light reaching the mainline of traffic. Also, the lights at Kenton are positioned such that the light source is not visible to the approaching driver. The Kenton ISSES equipment was positioned approximately twice as far upstream from the scale house as the ISSES equipment at Laurel, in principle allowing Kenton inspectors more time to make decisions based on the system's output. One other change at Kenton was the placement of all of the ISSES above-ground portal apparatus, except one of the radiation panels, on the driver's (highway) side of the low-speed bypass lane. This change reduces the amount of equipment interfering with the sight lines between the passing vehicle and the inspector in the scale house, which is on the passenger side of the bypass lane.

The ISSES interface operates as follows. The ISSES main system monitor allows the user to view a summary of the seven to ten most recent trucks to pass through the ISSES portal. Each summary includes a color image of the truck, the license plate reading, the USDOT number reading, and the radiation alarm status (if available). The digital readouts on the monitor are color-coded, such that blue means "no data available," green means "good data available," and red indicates an alarm condition. The user can switch this summary view to show all recent trucks or only those recent trucks that triggered an alarm. Figure 5-10 shows an example of the ISSES continuous monitoring interface.

Once the user has chosen a particular vehicle for further analysis, he or she can select any of a series of individual system views, including the radiation (rad) server, the DVR (digital video recorder) server for the thermal imaging system, the ALPR (plate) server, or the USDOT number server. The user chooses these subsystem server views from the management interface screen (Figure 5-11). The various server detail view options, some of which are not yet activated in ISSES, are listed down the left side of the interface screen. For example, the rad server shows individual data for the master and remote panels on either side of the ISSES portal, and a combined or system-level radiation data reading. This view also depicts a generic image of the vehicle's structure (see Figure 5-3 above), showing the relative position of the radiation source and its emission level (indicated by graphic lines that are depicted at various distances from the longitudinal centerline of the vehicle).

The IR/thermal camera server can be adjusted to switch from IR to visible/color output. The USDOT number server shows a large image of the side of the tractor, a date/time stamp, a decoded text string representing the OCR reading of the USDOT number if available, and a series of numbers representing the confidence and status of the record being viewed.

Visitors to the Kenton and Laurel sites made a number of observations about the degree to which ISSES equipment was used by weigh station staff, both in general and with respect to the system's intended usage. On multiple occasions, research team members visiting the Laurel site observed and/or were told by weigh station personnel that ISSES is not used very often. During one site visit, the ISSES monitors were not used at all. As described elsewhere, no one at the Laurel site was assigned to watch the monitors constantly, due partly to manpower and resource constraints, and partly to the enforcement objectives at the site, which emphasize numbers of inspections rather than OOS rates. This prevented the evaluation team from comparing inspection decisions potentially influenced by ISSES data readings with those decisions based on traditional methods such as obvious visual evidence and inspector knowledge or judgment.

The research team also learned that CMVs are only rarely pulled over when an inspector happens to notice a potential problem on the IR screen. Instead of using ISSES, inspections at the Laurel station were typically based on random selection, visual judgment, and the inspector's knowledge of the carriers.

A representative of the KTC, a research organization based at the University of Kentucky and funded in part by the Kentucky Transportation Cabinet (which owns and maintains the weigh stations on Kentucky highways), provided some additional insight into the apparent under-use of the ISSES equipment during the field observation. The KTC representative noted that the system works as designed, but that KVE staff, because of their workloads and primary duties, as well as an institutional emphasis on achieving a required number of inspections per time period, perceive watching the ISSES monitors to be too time-consuming and not worthwhile, in terms of meeting the quotas that make up part of their departmental management objectives and incentive system. For example, an inspector might watch an IR camera for 30 to 40 minutes before seeing a truck he or she wants to inspect, whereas by standing at the window and observing the line of traffic, he or she will likely see a truck of interest in 10 or 15 minutes. That 15 to 30 minute differential is often considered to be wasted time by inspectors (whether justified or not).

With respect to individual ISSES components, the research team made two key observations when visiting the Kenton and Laurel sites. First, the OCR systems are designed for trucks traveling approximately 15 MPH past the cameras. However, observers noted that at both the Kenton and Laurel stations, trucks routinely travel at speeds up to 20 or 30 MPH (despite the 10 MPH speed limit posted on the ramp), reducing the time available for the automated systems to capture, process, and report on the image and the digital output. The vendor indicated that the optimum vehicle speed for the system as designed represents a tradeoff between the interests of the trucking companies, who want to keep their trucks moving quickly through the station, and the state inspectors, who want the trucks to move slowly enough to allow sufficient time to make inspection decisions, without causing a large volume of truck traffic to "tailback" from the station to the mainline. From a technical viewpoint, the limiting factor on portal speed is the radiation detection system. The imaging systems have few restrictions up to highway speeds, but the ability to accurately identify trace radiation sources diminishes as vehicles travel faster than the design speed of 15 MPH. When trucks do so, the ISSES still collects data, but the higher speeds tend to reduce the level of accuracy.

Second, the team members learned that some users are not satisfied with the performance of the radiation monitors for various reasons. At the Kenton site, a KTC staff member indicated that the radiation detection panels are believed to be less than optimal because they are positioned too close to the ground, effectively missing a majority of a vehicle's cargo area (Figure 5-12). The installation at Kenton was made with the intent of eventually installing a second row of sensors at a higher level. Elevating the existing panels-or adding other panels on top of the existing ones—would require a significant capital expenditure. The system vendor indicates that the Kenton configuration is more than sensitive enough to gather accurate radiation data on the vast majority of radiation loads passing through the portal. According to the vendor, raising the Kenton panels or adding a second level would, for the most part, make no difference to what an operator experiences in the weigh station. A similar lower-profile configuration was also used at the Simpson County site, shown in Figure 5-13. For comparison, the higher-profile panels at Laurel County are shown in Figure 5-7 above.

Inspectors at the Laurel site report that the alarm as currently configured is too sensitive to low levels of radiation. As a result, the alarm activates whenever a CMV carrying harmless but gamma ray-emitting materials (e.g., bricks, porcelain, clay, granite, cat litter, ceramic tile) passes through the station, and staff are prone to ignore the numerous gamma alarms. The gamma detector gives more nuisance alarms than the neutron detector.

The system vendor reported that the ISSES radiation detector subsystem will not be optimized until true "risk matrices" are cross-referenced with USDOT hazardous materials rules and remote data to automate useable transportation safety alarms for inspectors. This "rules manager," which is the final stage in the development of ISSES, will cross-reference sensor data from ISSES with remote data stores to give user-defined alerts to operators. For example, the system is being programmed by the vendor to provide an audible alarm when some kinds of radiation-emitting loads are observed being hauled by a carrier whose USDOT number is not associated with the appropriate certificates, credentials, or permits.

According to a TransTech field support representative familiar with the Laurel County and Kenton County demonstrations, the ISSES system has been generally reliable. However, the system has been subject to downtime due to component hardware/software issues (see Appendix E). TransTech informed the research team that the company attempted to use commercial, off-the-shelf technologies whenever possible. While this approach provides advantages with respect to reducing first costs, it also increases the cost and difficulty of integrating disparate systems. Using off-the-shelf technologies allowed KVE to begin using subsystems like the thermal inspection camera and radiation detection system immediately. However, developing a more integrated solution has been a process characterized by:

• Technical challenges associated with hardware/software integration (as of November 2007, reportedly complete save for minor improvements)
• Operational challenges (e.g., how to design the user interface to work for KVE—the remote and on-site viewers have been redesigned since the original installations).

Several specific problems with ISSES and its components were reported to the research team by weigh station and system maintenance personnel. At the Laurel station, site personnel reported that the DOT camera was inoperable for five days when a fiber optic switch needed to be replaced. In general, the system's cameras have proven to be difficult to calibrate and are sensitive to bumps and jars. Some time was required to obtain consistent readings from them. The TransTech maintenance representative reported that ISSES equipment is particularly sensitive to downtime after power interruptions or other system problems. All systems and network connections need to be manually restarted after power is lost, in the version of ISSES that was installed at the Kenton and Laurel sites.

Regarding the manual restart issue, according to the system vendor, the ISSES software and components now deployed-though operational-are considered to be in development mode as of late 2007. The vendor has been upgrading isotope identification sensors in Kenton and Simpson, debugging small anomalies as they appear, and collecting raw data for client studies. In the near future, the vendor plans to turn to production mode, which will incorporate an auto restart function.

5.2 Data Analysis

5.2.1 Data Analysis Methods

The research team analyzed system performance by comparing actual performance data against manufacturer and system integrator specifications, user documentation, and laboratory test data, where available. In essence, evaluators considered how the component technologies performed relative to how they were intended to perform.

Laser Scanner (Vehicle Detection)

The research team considered whether the laser scanner accurately logs the passage of trucks through the ISSES apparatus and signals other subsystems. as part of the inspection efficiency study, the research team manually collected vehicle identifiers during site visits. These counts of identifiers were used to verify the performance of the laser scanner system in recording vehicle events. Although the ALPR and USDOT number reader are not subject to evaluation in this effort, vehicle identification data also were used for quality checking and verification purposes as an independent source to help ensure the completeness of vehicle records at each site.

Radiation Portal Monitor

The research team considered whether the radiation monitor accurately alerts inspectors to potential hazards and minimizes the occurrence of both false positives (alarm is sounded when no significant radiological emission is present from vehicle) and false negatives (alarm is not sounded when there is sufficient radiological emissions present from the vehicle). Graphical representations of the radiological emissions from passing vehicles were archived off of the rad server onto a hard drive and sent to the research team.

Infrared/Thermal Imaging System

The research team evaluated whether the thermal inspection device enables inspectors to see potentially defective or malfunctioning equipment through heat signatures recorded by the device's IR camera. Video feeds captured by the thermal inspection device at the Kenton site were archived off the DVR server onto a hard drive and sent to the research team, which navigated through the video footage to locate specific vehicles and determine whether potential heat-related problems would have been visible to weigh station inspectors. The video footage was also used as a cross-check against Driver/Vehicle Examination Reports prepared by inspectors at the Kenton weigh station to determine if defects noted during the inspection were visible in the IR images.

USDOT Number Reader and Automatic License Plate Reader

These systems were not under evaluation.

5.3 Results

This section presents performance evaluation results for each subsystem, followed by general system performance conclusions.

5.3.1 Laser Scanner (Vehicle Detection)

There was some discrepancy between the number of vehicles counted by the research team staff member during site visits and the number of individual records generated by the software over the same time period. A comparison of records collected over a two-hour period on June 14 uncovered no instances in which the human observer recorded vehicles that were missed by the software. However, the software generated a significant number of records (i.e., distinct rows in the data file output produced by the software, which associates output for each vehicle from the radiation detector, USDOT number reader, overhead camera, and ALPR system in a single line) with "n/a" or "Not Available" values in columns that would normally contain OCR readings, file names, etc., which were not reported by the human observer.

At the Laurel site, for example, 1,769 records were generated by ISSES during a span of approximately 8 hours on June 14. During the same time period, the research team member recorded 1,455 vehicles, a difference of 314 records. While it is possible that the observer missed some vehicles during periods of heavy traffic, the research team attributes much of the inconsistency in observed and detected vehicle counts to the ISSES vehicle detection component, which seems to have produced three distinct types of errors. In some cases ISSES generated two records, or rows, for the same vehicle, as evidenced by the fact that a single overhead camera image is associated with two adjacent records. This occurred 67 times during the 8-hour period on June 14.

In three other instances during the same 8-hour period, a value of "n/a" was generated instead of an overhead camera image, indicating that either the overhead camera was not triggered or that an extraneous record was created. The third type of error occurred when ISSES components (e.g., the overhead camera) were triggered when no vehicle was passing through the portal. This error is evidenced in Figure 5-14, which shows three images recorded in close sequence by the overhead camera. Each of these images is associated with a separate record in the ISSES output file. However, image [2] was taken after the vehicle shown in image [1] had already passed through the portal, but before the vehicle in image [3] reached the sensors. In other words, image [2] should not have been recorded and represents an extraneous record, or row, in the output file. In general, records that contained nothing but "n/a" or "Not Available" values were considered by the research team to be associated with one of these three types of errors.

The vendor acknowledged to the evaluation team that vehicle triggering and ordering the time sequence of actual events in the resulting data files has been a challenge in deployment. It appears that the system generates a certain number of unnecessary records when no vehicle was present. According to the vendor, triggering invariably involves trade-offs between either missing a vehicle or having extra triggers on a single vehicle. Every type of sensor has a chance of either missing a vehicle or having false triggers. In the case of the ISSES, the decision was made to allow a certain number of extra or false triggers, in exchange for having a system that is more likely to collect usable data on any vehicle suspected of posing a radiological threat. The extra ISSES triggers pose a low-consequence nuisance in data analysis. In practice, the vendor contends that these extra triggers should be obvious to operators or analysts, by cross-checking the suspect trigger records against the overview camera image associated with each record. The vendor also indicated that apparent sequencing errors when looking across server time stamps might be caused by fluctuations in computer network traffic volume, and time lags of varying durations required for internal processing and transmitting of signals from ISSES subsystems.

5.3.2 Radiation Portal Monitor

The research team assessed the general output of the radiation portal monitor. During the 2-week field observation, the ISSES recorded nine neutron alarm vehicles and 558 gamma alarm vehicles. Considering the approximate number of vehicles recorded during this period (28,000), the neutron alarm was activated by one out of every 3,111 vehicles, and a gamma alarm was activated by one out of every 50 vehicles. During a standard, 8-hour, daylight shift (8:00 AM to 4:00 PM), the average daily number of gamma alarms activated during the 2-week observation period was 29; and the average daily number of neutron alarms activated was slightly less than 1. The weigh station staff indicated that the gamma alarms sound fairly frequently. The inspectors indicated that staffing levels prevented them from inspecting every truck that tripped a radiation alarm. A tendency for nuisance alarms caused by naturally occurring substances has the effect of making inspectors more likely to ignore all of the bulk gamma radiation monitor alarms, as confirmed in the user acceptance interviews. As indicated in Section 7, the vendor and KTC have confirmed that the Laurel County system was recently modified to greatly reduce the number of nuisance alarms.

The evaluation team intended to make as few changes to the normal operating conditions as possible during the field observation, so no attempt was made to compare the bills of lading or the actual contents of the alarm vehicles with the data recorded by ISSES. No overt or covert field tests of the radiation monitor using known emitters were conducted during the evaluation period. Such tests would have permitted an assessment of false positive or false negative alarms from the radiation detection system. However, as indicated by a number of inspectors at the site, the radiation monitor would benefit from having a mechanism to more finely differentiate potentially dangerous loads from common items such as clay and brick, as evidenced by observations made during field visits. According to KTC, the Laurel County site was adjusted in the fall of 2007, after the time of the field observation, to greatly reduce the frequency of nuisance alarms.

In an attempt to characterize the truck population triggering neutron and gamma alarms during the field observations, USDOT number records and their associated JPG images were extracted from the ISSES records. These USDOT numbers were reviewed against the SAFER database to collect company information. Of the nine neutron alarms recorded by ISSES, only one USDOT number and associated JPG image was available. Of the 558 gamma alarms recorded by ISSES, 151 had valid USDOT number JPG images associated with them. A cursory review of these USDOT numbers showed no clear patterns or trends that would allow the evaluation team to characterize the truck population emitting gamma alarms.

However, it was noted anecdotally that, among the JPG images of the sides of trucks that showed evidence (through graphics or business names painted near the USDOT number) of the type of cargo that might have been carried on trucks triggering gamma alarms, the following commodities were named: marble, building systems, asphalt maintenance, and tile distribution. This unscientific observation would tend to confirm the KVE inspectors' opinions (described in the user acceptance section) that clay and related building products were among the naturally occurring substances that could trigger a gamma alarm.

To assess the internal consistency of the radiation detection data storage system, the final counts of each file type recorded on the RAD server were tabulated. The numbers of each file type are presented in Table 5-2 below for the 2-day Kenton training exercise and the 2-week Laurel field observational studies. As shown in the table, approximately 28,000 vehicles passed through the Laurel site and 4,200 vehicles passed through the Kenton site during the field study periods. While it would be expected that ISSES would generate equal numbers of the main file types for each vehicle (record), represented by the DATA, OCC, and JPG columns, Table 5-2 shows that there is some variability in the numbers when looking across all file types for the periods under study. According to the vendor, as discussed in Section 5.1.1. above, the ID, SPC, NAF, and BAL files would ordinarily be generated and stored only in alarm conditions, so would be expected to be fewer in number. The reasons for this variability are unknown but may be related to the generation of extraneous records by ISSES, as discussed in Section 5.3.1.

Table 5-2. Radiation portal monitor files generated during field studies.

	Number of Files Generated by Type
Site	DATA	OCC	JPG	ID	SPC	NAF	BAL	None
Kenton	4,237	4,207	4,204	4,209	4,210	821	829	1,413
Laurel	28,416	28,006	25,794	0	0	28,004	565	4,260

As shown in Table 5-2, no ID or SPC file types were generated at the Laurel station. TransTech reported that first-generation radiation monitoring software was installed at the Laurel site; therefore, spectroscopy was not performed and the associated SPC and ID files were not created. Theoretically, JPG and OCC files should have been created for each vehicle that passed through the portal. However, Table 5-2 shows that no two file types had the same number of records. It is unclear why this occurred. NAF files, which are supposed to be generated only when an alarm is triggered, were created for nearly all the vehicles that passed through the Laurel site. This could be attributable to the use of first-generation software at Laurel.

The vendor indicated that live testing of the ISSES radiation monitor was conducted through the Domestic Nuclear Detection Office of the US Department of Homeland Security (DHS), and that the systems were confirmed to detect passing loads that were emitting radiation. The vendor reports having seen no evidence through testing that the radiation monitor has issued any false alarms (i.e., an alarm sounds when no radiation source is present). As opposed to false alarms, the tendency for nuisance alarms in the scale house when naturally occurring substances pass through the portal is discussed elsewhere.

5.3.3 Infrared/Thermal Imaging System

The research team reviewed video feeds captured by the thermal inspection device at the Kenton site. The objective was to determine whether potential heat-related defects were visible on the video and to track these defects. Originally, the intent was to compare the evaluator's observations with actual inspection reports provided by KVE to see if the defects found by weigh station inspectors were plainly apparent on the video.

The independent review of the two days' worth of Kenton IR video was hampered by several factors. As indicated by the vendor, the IR camera from this period was used in a training exercise, mainly by untrained inspectors learning and using the system for the first time. The camera was not set up properly for the first seven to eight hours of the Kenton field study. Video images were extremely blurry, and it was difficult to discern the general shape of the vehicle, let alone the specific components that were giving off a particular heat signature (see Figure 5-15). The evaluation team was not present during this scheduled KVE training exercise at Kenton, and no reason was given by the vendor for the video being blurry. It is assumed that the beginning of the recording represents a time when the inspectors were working with other ISSES subsystems, because at one point in the video, the camera settings were noticeably adjusted to provide the appropriate level of contrast between dark and light values. This greatly enhanced the image; however, at the same time that the contrast was adjusted, the operator of the IR camera appeared to zoom in and pan the camera manually to move along with (i.e., track) each passing vehicle. This resulted in only a portion of the vehicle appearing in the IR camera viewer at any given time (see Figure 5-16). Also, camera movement was disorienting to an evaluator with minimal training, trying to focus on possible areas of brightness or darkness. As a result, it was difficult to determine with any certainty that a particular tire, brake, or other component was giving off an unusual heat signature. These difficulties with the video image data appeared to be caused more by operator choices than by any inherent shortcoming with the technology.

The research team also noted a difficulty in correlating the image in the color (gable-mounted overview) camera viewer and the image in the IR camera viewer, on the three-part composite DVR player screen. As a vehicle approached the ISSES portal, it appeared on the IR viewer several seconds earlier than it appeared on the color monitor. This delay—most likely caused by the operator changing the aim of the IR camera while the overview camera remained stationary, or vice versa—was confusing, since the vehicle shown on the top right (IR) screen was often not the same vehicle visible simultaneously on the top left (color overview) screen (see Figure 5-16). The delay or time lag between the images appears to be amenable to correction through more careful operator training and experience. There is a benefit to enabling the user to aim the two camera systems (overview and thermal) independently, but this option carries costs for analysts interested in efficiently matching truck images with data records from the ISSES.

The research team was unable to cross-check video footage against Driver/Vehicle Examination Reports prepared by inspectors at the Kenton weigh station on July 31 and August 1, 2007, due to an inability to accurately identify the inspected vehicles on the IR/color video. USDOT numbers and/or license plate numbers from the inspection reports were used to find the date/time stamp on the USDOT number reader or ALPR output files and identify the time at which the vehicle passed through the ISSES portal. However, a review of the video at the corresponding times failed to identify vehicles with the same physical characteristics as those described in the inspection reports or shown in the still images captured by the ALPR/USDOT number reader. The research team watched video taken several minutes before and after the specified time, but could not conclusively match the images with the paper inspection records. When asked, the vendor indicated that the computer server that synchronizes the system time clocks at the Kenton site was down at the time of the recording, and the DVR clock did not match the ISSES clock. Because of this hardware fault, therefore, a direct, retrospective comparison was not possible, given the state of integration between the thermal imaging subsystem and the other ISSES vehicle identification and triggering subsystems when this sample of image data was stored. Such integration between the truck images shown on the overview color camera and the thermal/IR camera will be critical for enforcement and accurate vehicle identification in future enhancements of the ISSES hardware and software.

5.3.4 USDOT Number Reader and ALPR System

These systems were not under evaluation, so no results are presented.

5.3.5 System Performance Conclusions

The radiation monitor appears to alert inspectors to potential radiation hazards. No attempt was made to simulate radiation-emitting loads to formally test the rates of false positive alarms or false negative (missed detection) alarms. The alarm system produces different kinds of audible signals in the scale house, shows graphic images of the location and strength of the radiation source, and records quantitative information on the alarm conditions for retrospective review. A tendency for nuisance alarms caused by naturally occurring substances, however, has the effect of making inspectors more likely to ignore all of the gamma radiation monitor alarms, which reduces their effectiveness as a tool for identifying true threats. As a rule, the KVE inspectors do attend to neutron alarms, which sound different in the scale house and are much fewer in number than the gamma alarms.

The thermal inspection device enables inspectors to see potential heat-related defective or malfunctioning equipment that might be missed in a visual review. The field of view for the IR image can be manipulated as to direction and width, enabling close-up or wide-angle views of the stream of traffic. The system also records video data (in both IR and color/visible light) for later review. The effectiveness of the thermal inspection system appears to vary depending on the training, experience, and skills of the operator, especially in synchronizing the views of the ground-level IR/color camera and the gable-mounted color overview camera.

The laser scanner appears to log every truck passing through the ISSES apparatus, but its adjustment is such that the system generates a certain number of extra (blank) records or extra trigger events, which is an impediment to later review of traffic data. For the sample of data reviewed for this evaluation, some gaps in the time synchronization were noted.

The ISSES appears to perform with a minimum of unscheduled downtime. Partly owing to the exposed geographic location of the Laurel County weigh station, the hardware has been subject to several outages caused by lightning strikes and other power drops or interruptions. The system has experienced a low rate of hardware failure, other than some events related to the reliability of electrical power to the site. The developmental version of the system software is not equipped with a self-restarting function, which is expected to be included in production versions. Also, the state and the vendor are investigating the installation of an uninterruptible power supply system for the ISSES.

As of mid-2007, the system appeared to be at a late stage in the product development cycle, not completely in full-scale production mode, but well beyond the field test prototype stage. It was not yet integrated with any current or historical state or national databases, which affected its usefulness for real-time enforcement applications, but it appeared to be functioning well in stand-alone mode.

Previous | Next

ITS Home | US DOT Home | Privacy | Feedback

United States Department of Transportation - ITS