User Acceptance and System Cost Evaluation
The user acceptance study focused on the interface between the inspectors and the ISSES equipment, and the more subjective attitudes and contextual environment that affect the adoption or rejection of advanced systems such as the ISSES. For completeness, the user acceptance interview questions were intended to cover all ISSES subsystems, including the ALPR and the USDOT number reader, even though those two systems were not under evaluation. Details on the user acceptance and system cost data collection, including interview guides and transcripts of interview contents, are presented in the Technical Report (USDOT 2008).
User Acceptance Results
The purpose of this section is to summarize the findings of the interviews and site observations and to present a discussion of the prevailing themes that appeared in the user acceptance data. The main two findings are as follows:
- Staffing and training were seen as main barriers to active use of ISSES in everyday KVE inspection operations.
- The majority of inspectors said ISSES appeared to be user-friendly, and that training is necessary to help them make full use of its capabilities.
Several prevailing themes appeared in the data, as described below:
Training. In many replies, respondents cited lack of proper training as being either the main reason or part of the reason they had not used any part of the ISSES equipment. In many instances, respondents indicated that with adequate training and user documentation they could come to appreciate and utilize the equipment. According to KTC, some training and exercises had been conducted at the time of the initial deployment in 2005 and since then; however, training should be offered frequently for all staff, especially new hires. Staffing levels were also seen as an important barrier to using ISSES during daily inspections. There is a perceived scarcity of staff resources to make use of the information being generated by ISSES.
Based on respondent feedback, training should include a discussion of how ISSES can augment current inspection selection practices, which are primarily visual inspection and observation, the use of WIM sensors, and queries of external data sources. For the radiation monitor, training should highlight how to interpret the truck profiles listed on the ISSES screen and how to distinguish and read radiation dose rate values. For the thermal imaging equipment, training should include a thorough explanation of scenarios in order for inspectors to be able to recognize brake violations and other patterns. Three of the six respondents felt that the thermal imaging device should benefit them since it seemed “easier to locate possible brake defects” than working at a location without it.
Equipment. Respondents provided most useful information about two of the ISSES subsystems, the radiation monitor and thermal imaging device. In many replies, respondents considered most ISSES radiation alarms to be caused by routine, naturally occurring substances (e.g., brick, porcelain, clay) or licensed, placarded medical products. Respondents indicated the radiation monitor needs to be fine-tuned to reduce nuisance alarms. The system is perceived as “very sensitive.” Inspectors do not want to waste time chasing down every truck.5 [As a point of reference, during the field observation, approximately 500 gamma alarms and nine neutron alarms were recorded by ISSES in 12 days.] According to the vendor, every ISSES site is provided with a hand-held radiation detector, along with software allowing inspectors to download data from the hand-held detector to the electronic record of the inspection. Several respondents noted that hand-held radiation detectors, while not recognized by them as being part of ISSES, complemented the radiation portal monitor. The hand-held device, which is deployed at every ISSES site, can zero in on a problem when the truck is in the inspection shelter.
In many answers, respondents indicated that they rely on the thermal imaging device with the greatest confidence because they can “actually see trucks on the screen” and believe it enables them to perform their job functions better. It appears to be easy to use, even given little training, and training could only help inspectors make better use of this subsystem. Respondents also said that having the thermal imaging device on site has more benefits than IRISystem vans, although one could complement the other, similar to the combination of the hand-held and fixed portal radiation monitors. On several occasions, respondents raised the point that the thermal camera shows only one side of truck, and they would like to be able to view both sides of the vehicle as it passes the thermal camera location. One characteristic of the thermal imaging system is that it tends to show defects more clearly on the far-side axles of the truck, partly because the tires and rims do not obstruct the line of sight from the camera to the brakes and other components that are most subject to over- or under-heating. Cameras placed on both sides of the lane of travel would thus allow the inspector to view the insides of the wheels on both sides of the truck more clearly.
Lessons Learned. Respondents provided useful lessons learned regarding how ISSES would yield greater benefits for future deployments if it were integrated with state and national systems:
Lesson learned 1: Train early and retrain periodically to account for new staff. Respondents speculated that they could provide additional input or different answers to the user acceptance interview questions if they had had training on the equipment, since many respondents admitted their unfamiliarity with the equipment. Future evaluations could include revised one-on-one interviews as well as focus groups to bring together several trained users in a group setting to discuss and listen to their issues and concerns about the features of the ISSES.
Lesson learned 2: Carefully consider where equipment is sited before installation and obtain input from inspectors. As it is installed now, it appears that the equipment is located too far down the approach ramp from the mainline, at a point that is too close to the scale house. Inspectors need adequate time to interpret information from ISSES and then decide whether to stop a given vehicle. In the current setup, by the time the vehicle arrives, it is often too late; inspectors need more time to visually inspect IR imagery and other ISSES signals. The Kenton site, installed after the Laurel site, provided more distance from the ISSES equipment to the scale house, primarily for the time required for the system to recognize and process the USDOT numbers and license plate numbers.
Changing the siting of the equipment could also help the triggering and correlation process, especially when two trucks are very close together in line. The current ISSES occasionally generates extra data records across the various subsystems, making it difficult to relate, for example, ALPR values with USDOT number values, or with radiation profile values. The topic of triggering issues, including the tradeoffs required when deploying a system that combines both highway safety and homeland security functions, is covered in more detail above in System Performance.
Lesson learned 3: Provide equipment documentation and user guides along with contact information on-site (e.g., if a radiation alarm goes off) that affords inspectors access to personnel with a working knowledge of equipment.6
Overall, the ISSES system works as designed, but KVE staff—because of their workload, primary duties, and enforcement performance measures—perceive watching the ISSES screens to be very time-consuming in terms of meeting the quotas that are set out for them in their jobs.
Deployment and Operating Costs Results
The system cost study focused on the economic dimensions of the deployment, for both one-time start-up costs and recurring (annual) costs to operate and maintain the ISSES. Data on actual costs incurred were supplemented by best estimates for those costs that are not available.
Data collection for the system deployment and operating costs was made via contact with the KTC to identify the various costs associated with purchased and installed materials and system equipment, related software integration, and vendor labor. The KTC has provided a copy of a bill of sale dated 6/2/2005 with cost data and general system specifications. This bill states that the total cost of installing ISSES at the Laurel County weigh station was $350,000. This total cost includes the: radiation detection component; thermal imaging component; license plate reader component; and site preparation and installation. All installed equipment is included in the bill of sale except two rack mount servers. The KTC, which was involved in ISSES contracting between the state and the vendor, reported that funds from Oak Ridge National Laboratory (ORNL) were also used in the original Laurel County installation and deployment, and that subsequent systems installed in other Kentucky counties have actually cost the state approximately $500,000 each to procure and install.
The original budget for the Laurel County ISSES did not provide funding for training or system maintenance. According to the KTC and the vendor, however, recurring (annual) costs for hardware to operate and maintain the equipment have been fairly low. The system is based on low-amperage sensors and communication systems, and does not cause a large electrical current draw. Equipment repairs and replacement of parts, as described below, have been largely due to lightning strikes and electrical power service interruptions, not due to ISSES equipment defects. In November 2006, the KTC entered into a service contract with TransTech to make one field technical support person available at approximately 60% of full-time on-site to cover the three installed ISSES locations for one year, and at about half of the first year’s time commitment for two years thereafter. While the technical support person also participates in client- and vendor-driven data collection projects and other activities outside of this on-site service commitment, his main role is to be available to troubleshoot any maintenance issues, monitor the site remotely, make any repairs on-site as needed or requested by KVE or KTC, provide training to operators/inspectors at each of the sites, and identify and test ISSES enhancements.7 The cost of this maintenance and technical support from November 2006 through August 2007 has been approximately $109,000. This amount has covered the ISSES maintenance duties listed above, but some fraction of the field support technician/analyst’s time within this contract has been devoted to administrative activities, software programming support, and communications protocol development for the nuclear detection subsystem unrelated to the monitoring, repair, and maintenance of the ISSES. Thus, the entire $109,000 has not been attributable to operating and maintaining the ISSES hardware and software.
It appears that ISSES requires frequent maintenance because of system troubleshooting and power interruptions, the latter type being considered unscheduled maintenance. It is difficult to delineate whether the maintenance (both unscheduled and preventive/planned) is monthly, weekly, or daily because of the nature of the troubleshooting (e.g., lightning strike versus software modification).
5 The KTC indicated that the nuclear detection subsystem at the Laurel County ISSES site had been adjusted in the fall of 2007 (after the time of these interviews) to greatly reduce the frequency of nuisance alarms.
6 KTC indicated that, now that a maintenance contract has been established with the vendor, each ISSES site has contact information posted for the on-site technical support person from IIS, giving KVE enforcement personnel consistent access to help if they have a question or a problem with the equipment.
7 The first such training session was a two-day training session held on July 31 and August 1, 2007, provided to personnel at the Kenton County inspection station. The training session focused on the operation of the thermal imaging system.