7.0 User Acceptance and System Cost Evaluation

7.1 Introduction

The purpose of this section is to document specific data collection and analysis methods as well as to report the findings for the remaining objective of Goal Area 2 of the evaluation: "Determine how the ISSES makes the inspection process more efficient and effective, in turn contributing to improved highway safety." (Section 6.0 discussed the other two objectives of Goal Area 2.) Specifically, this section covers the following objective and hypotheses:

Objective 2.1 Determine the degree of user acceptance and the perceived usefulness and usability of the ISSES as deployed, and quantify deployment and operating costs related to the ISSES.

Hypotheses: Inspectors and state transportation managers believe that ISSES enables roadside inspectors to perform their job functions better.

Inspectors believe that ISSES should be deployed more widely.

In deploying similar systems, officials at other sites believe their system enables them to perform their job functions better.

Inspectors found their training and user documentation for ISSES to be helpful to them in their normal course of duties.

In deploying ISSES, Kentucky incurred one-time start-up and recurring costs that were clearly defined and measurable.

This objective was also intended to identify ways that the state and the deployment team can improve the performance and usability of the ISSES as perceived by the inspectors, which should lead to increased benefits as advanced inspection systems are deployed more broadly. In addition, this objective attempted to assess the costs to deploy, operate, and maintain the ISSES.

The user acceptance study focused on the interface between the inspectors and the ISSES equipment, and the more subjective attitudes and contextual environment that affect the adoption or rejection of advanced systems such as the ISSES. As noted in the Evaluation Plan, a natural overlap exists between the system performance and the user acceptance aspects of the independent evaluation. Ideas drawn from the user acceptance study affected the system performance results, and vice versa; however, the evaluation team attempted to maintain a distinction between the two studies.

For completeness, the user acceptance interview questions were intended to cover all ISSES subsystems, including the ALPR and the USDOT number reader, even though those two systems were not under evaluation. However, the evaluation team noted that the KVE respondents tended to say very little about the ALPR and USDOT number reader. This may be because those two subsystems were not integrated with any back-end databases to provide alarms that could aid in the real-time inspection selection process. Alternately, the radiation monitor (because it sounded audible alarms) and the thermal imaging system (because it showed live IR and video of vehicles passing near the scale house) may have been more prominent in the minds of the inspectors when asked about ISSES in general. Also, some of the inspectors may have been familiar with the thermal imaging system because of Kentucky's history of actively deploying the IRISystem vans for mobile enforcement using IR image technology. No conscious attempt was made to steer respondents toward or away from any of the ISSES subsystems.

7.2 Data Collection

Data to address the hypotheses of Objective 2.1 were collected from a variety of data sources, including on-site interviews and site observation visits. Both data sources enabled researchers to test the hypotheses associated with Objective 2.1. The first research hypothesis tested was whether inspectors believed that the ISSES equipment enabled roadside inspectors to perform their job functions better. As noted previously, there was an apparent disconnect between the purpose of the ISSES technology deployment (i.e., increase the rate of OOS orders) and the performance measures for KVE inspectors (i.e., complete inspections at a specified rate, with relatively little regard for the rate of OOS orders).

In order to best address this hypothesis, it was important to understand what the ISSES equipment actually does and how the ISSES could be improved. This was accomplished through in-person and telephone interviews as well as field observations with inspectors and weigh station staff to gauge the ease-of-use of the ISSES, its benefits, disadvantages, or underutilized capabilities, and the lessons that inspectors and other stakeholders have learned through the early deployment and day-to-day use of the ISSES. Additionally, user feedback was useful in documenting the cooperation among stakeholders in the deployment process; how any problems were overcome; what lessons were learned on any technical, business, or institutional issues encountered; and how these issues were handled.

The next research hypothesis tested was whether inspectors believed that ISSES should be deployed more widely. In order to address this hypothesis, it was necessary to understand the potential benefits of broader ISSES deployment. This was accomplished through in-person and telephone interviews as well as field observations with inspectors and weigh station staff to collect opinions on topics such as whether ISSES would yield greater benefits if it were to be integrated with state and national systems, whether the components worked satisfactorily in a stand-alone mode, whether data originating from ISSES in London would be useful if made available to inspectors in other jurisdictions, and which features would motivate inspectors at the London site or other sites to rely on ISSES data.

Another research hypothesis tested whether officials at similar deployment sites believed their system enabled them to perform their job functions better. In order to understand the views of other sites, telephone interviews as well as email exchanges with inspectors and weigh station staff at Kentucky sites similar to the deployment site in London were conducted.

The next research hypothesis tested whether inspectors found the training and user documentation for ISSES to be helpful to them in their normal course of duties. This hypothesis was carried out through in-person and telephone interviews as well as field observations with inspectors and weigh station staff to understand the kinds of training that were offered to KVE inspectors, the shape of the "learning curve" for mastering the various features of the system, and whether the inspectors perceived their training to have been beneficial in using the ISSES. The results from these comments should help Kentucky and other sites develop appropriate training and user documentation as additional systems are deployed.

Another aspect of Objective 2.1 was to quantify deployment and operating costs related to the ISSES. In order to best address this hypothesis, it was important to identify the costs of purchased and installed materials and system equipment, related software integration, and vendor labor. Actual recurring costs to operate and maintain the systems were obtained. Such operations and maintenance costs included technical support, repairs, equipment replacement, software upgrades, and supplies and materials to keep the ISSES in operating order throughout its expected service life.

Several data collection planning activities were completed for the user acceptance aspect of the Kentucky evaluation. First, a set of user acceptance interview questions were created. Second, site visits with various Kentucky Transportation Center (KTC) and Kentucky Vehicle Enforcement (KVE) personnel were conducted to learn about the ISSES equipment and to observe weigh station staff perform their duties in relation to the ISSES in and around the scale house.

An initial planning visit took place on January 24, 2007. Additional user acceptance field observations and interviews were scheduled to take place in June 2007 at the London site. To coordinate the June visit, contact was made with Captain David Marcum of KVE at the London site via email and telephone to schedule a time to conduct the user acceptance interview and to determine what staff, including sworn law enforcement officers and regulatory weight and safety inspectors from KVE, would complete the interview. User acceptance interview questions were sent to KVE personnel prior to the site visit.

For the user acceptance study, interviews were conducted during a site visit to the London site on June 20, 2007, near the end of the two-week field observation period. The interview was presented as part of a research effort to assist Kentucky and the deployment partners in measuring the degree of user acceptance and the perceived value and usability of the ISSES. For the London site, four individuals were identified as participants. The user acceptance site visit and all interviews were conducted on the same day.

Additionally, weigh station staff from other weigh stations in Kentucky were contacted to participate in the user acceptance interview. Kenton County and Simpson County weigh station facilities each had one KVE staff member participate in the one-on-one interview portion of the study. These additional interviews were administered via email during the July and August 2007 timeframe.

The target as given in the Test Plan (USDOT 2007c) was to conduct three to four individual interviews for the user acceptance portion of the evaluation. This goal was met. Focus groups were not conducted as part of the user acceptance study; however, during the user acceptance site visit and during other visits during June 2007, researchers held informal discussions with several weigh station staff in the scale house to glean general information about the inspection operations of the weigh station as well as the ISSES equipment.

7.3 Data Analysis

Following on-site data collection, researchers compiled participant feedback from the field observations and one-on-one user acceptance interviews into a table that enabled them to analyze and document the degree of user acceptance and the perceived usefulness and usability of the ISSES technology deployment. The original user acceptance interview guide is attached to this report (see Appendix B) and consists of 29 questions divided up into six sections, A through F. Some of the questions or interview prompts consisted of several statements or subquestions. Table 7-1 shows the question groups, topics, and the numbers of questions in each group.

Table 7-1. User acceptance interview question topics.
Section	Topic	Number of Questions
A	General questions about ISSES	6
B	Use of the ISSES equipment	7
C	Training	3
D	Inspection selection efficiency	5
E	Future deployments	5
F	Hypothesis evaluations (true/false)	3
TOTAL		29

All questions were intended to give participants the opportunity to provide their candid opinions about the ISSES and their experiences with ISSES to date.

As noted, six participants from three Kentucky weigh stations provided input to the user acceptance interview questions. A complete transcript of the responses by question is presented in Appendix C.

7.4 User Acceptance Results

This section summarizes the user acceptance interview outcomes and identifies trends observed among the responses. Following the summary, responses to individual questions are presented, providing count and percentage data to aid in analyzing responses. Selected responses (in both paraphrase and direct quotations) are included below, when appropriate and relevant, to highlight views representative of a number of respondents.

7.4.1 Prevailing Themes Among Respondents

The purpose of this section is to summarize the findings of the interviews and site observations and to present a discussion of the prevailing themes that appeared in the user acceptance data. The main two findings are as follows:

Staffing and training were seen as main barriers to active use of ISSES in everyday KVE inspection operations.
The majority of inspectors said ISSES appeared to be user-friendly, and that training is necessary to help them make full use of its capabilities.

Several prevailing themes appeared in the data, as described below:

Training. In many replies, respondents cited lack of proper training as being either the main reason or part of the reason they had not used any part of the ISSES equipment. In many instances, respondents indicated that with adequate training and user documentation they could come to appreciate and utilize the equipment. According to KTC, some training and exercises had been conducted at the time of the initial deployment in 2005 and since then; however, training should be offered frequently for all staff, especially new hires. Staffing levels were also seen as an important barrier to using ISSES during daily inspections. There is a perceived scarcity of staff resources to make use of the information being generated by ISSES.

Based on respondent feedback, training should include a discussion of how ISSES can augment current inspection selection practices, which are primarily visual inspection and observation, the use of WIM sensors, and queries of external data sources. For the radiation monitor, training should highlight how to interpret the truck profiles listed on the ISSES screen and how to distinguish and read radiation dose rate values. For the thermal imaging equipment, training should include a thorough explanation of scenarios in order for inspectors to be able to recognize brake violations and other patterns. Three of the six respondents felt that the thermal imaging device should benefit them since it seemed "easier to locate possible brake defects" than working at a location without it.

Equipment. Respondents provided most useful information about two of the ISSES subsystems, the radiation monitor and thermal imaging device. In many replies, respondents considered most ISSES radiation alarms to be caused by routine, naturally occurring substances (e.g., brick, porcelain, clay) or licensed, placarded medical products. Respondents indicated the radiation monitor needs to be fine-tuned to reduce nuisance alarms. The system is perceived as "very sensitive." Inspectors do not want to waste time chasing down every truck.⁶ [As a point of reference, during the field observation, approximately 500 gamma alarms and nine neutron alarms were recorded by ISSES in 12 days.] According to the vendor, every ISSES site is provided with a hand-held radiation detector, along with software allowing inspectors to download data from the hand-held detector to the electronic record of the inspection. Several respondents noted that hand-held radiation detectors, while not recognized by them as being part of ISSES, complemented the radiation portal monitor. The hand-held device, which is deployed at every ISSES site, can zero in on a problem when the truck is in the inspection shelter. (See the response to Question #A4 below.)

In many answers, respondents indicated that they rely on the thermal imaging device with the greatest confidence because they can "actually see trucks on the screen" and believe it enables them to perform their job functions better. It appears to be easy to use, even given little training, and training could only help inspectors make better use of this subsystem. Respondents also said that having the thermal imaging device on site has more benefits than IRISystem vans, although one could complement the other, similar to the combination of the hand-held and fixed portal radiation monitors. On several occasions, respondents raised the point that the thermal camera shows only one side of truck, and they would like to be able to view both sides of the vehicle as it passes the thermal camera location. One characteristic of the thermal imaging system is that it tends to show defects more clearly on the far-side axles of the truck, partly because the tires and rims do not obstruct the line of sight from the camera to the brakes and other components that are most subject to over- or under-heating. Cameras placed on both sides of the lane of travel would thus allow the inspector to view the insides of the wheels on both sides of the truck more clearly.

Lessons Learned. Respondents provided useful lessons learned regarding how ISSES would yield greater benefits for future deployments if it were integrated with state and national systems:

Lesson learned 1: Train early and retrain periodically to account for new staff. Respondents speculated that they could provide additional input or different answers to the user acceptance interview questions if they had had training on the equipment, since many respondents admitted their unfamiliarity with the equipment. Future evaluations could include revised one-on-one interviews as well as focus groups to bring together several trained users in a group setting to discuss and listen to their issues and concerns about the features of the ISSES.

Lesson learned 2: Carefully consider where equipment is sited before installation and obtain input from inspectors. As it is installed now, it appears that the equipment is located too far down the approach ramp from the mainline, at a point that is too close to the scale house. Inspectors need adequate time to interpret information from ISSES and then decide whether to stop a given vehicle. In the current setup, by the time the vehicle arrives, it is often too late; inspectors need more time to visually inspect IR imagery and other ISSES signals. The Kenton site, installed after the Laurel site, provided more distance from the ISSES equipment to the scale house, primarily for the time required for the system to recognize and process the USDOT numbers and license plate numbers.

Changing the siting of the equipment could also help the triggering and correlation process, especially when two trucks are very close together in line. The current ISSES occasionally generates extra data records across the various subsystems, making it difficult to relate, for example, ALPR values with USDOT number values, or with radiation profile values. The topic of triggering issues, including the tradeoffs required when deploying a system that combines both highway safety and homeland security functions, is covered in more detail above in System Performance.

Lesson learned 3: Provide equipment documentation and user guides along with contact information on-site (e.g., if a radiation alarm goes off) that affords inspectors access to personnel with a working knowledge of equipment.⁷

Overall, the ISSES system works as designed, but KVE staff—because of their workload, primary duties, and enforcement performance measures—perceive watching the ISSES screens to be very time-consuming in terms of meeting the quotas that are set out for them in their jobs.

7.4.2 Question-by-Question Summary

This section provides counts of responses per question (out of a total of six interviews conducted with KVE personnel) and, following a recap of each question, a qualitative description and interpretation of the responses collected.

A. General Questions about ISSES

(Q#A1) HAVE YOU USED ANY PART OF THE ISSES TECHNOLOGY? IF SO, PLEASE LIST THE SUBSYSTEMS YOU HAVE USED. ALSO NOTE ABOUT HOW MUCH YOU HAVE USED THE SUBSET AND FOR HOW LONG.

Of the six respondents, five reported being familiar with some part of the ISSES technology, namely the radiation monitor and thermal imaging camera.⁸ Respondents noted they had used the subsystems anywhere from several months to eight months. There appeared to be less familiarity with the laser scanner. One respondent had seen the license plate reader and USDOT reader, but had never seen it in operation.

(Q#A2) DOES THE ISSES EQUIPMENT APPEAR TO BE USER-FRIENDLY?

Of the six respondents, six reported essentially that "when it's working it's friendly." Four respondents noted that either no initial training was provided or with training it could be considered user-friendly. One respondent noted that different vendor personnel visit the site frequently to make repairs. The same respondent estimated that the ISSES equipment worked 20% of the time and said the equipment screens were frequently password-protected.

(Q#A3) IF YOU HAVEN'T USED ANY PART OF THE ISSES EQUIPMENT, CAN YOU EXPLAIN WHY YOU AREN'T USING IT? (E.G., TIME ISSUES, STAFFING, TRAINING ISSUES)

Of the six respondents, all six reported that lack of training was either the main reason or part of the reason they hadn't used any part of the ISSES equipment. In addition, staffing issues (being short staffed) and timing were also identified as factors. Inspectors have to conduct a designated number of inspections per week. They reported having enough to do without watching the ISSES screens.

(Q#A4) WHAT DO YOU WANT FROM THE ISSES EQUIPMENET (OR ANY TECHNOLOGY FOR THAT MATTER)? AND, IS THERE INFORMATION THAT IS DIFFICULT FOR YOU TO OBTAIN THAT A TECHNOLOGY WOULD MAKE EASIER?

Each of the six respondents provided varying responses. Two respondents indicated they wanted training on the ISSES equipment. One of these respondents would like the hand-held radiation detector to be more integrated with ISSES because it complements the radiation portal monitor. Two respondents also found that the radiation monitor needs to be finely tuned or have an alarm cutoff to limit nuisance alarms on common, natural materials (e.g., kitty litter, bricks). As one respondent put it, "You want to hear every alarm, but you don't want to chase down every truck." Another respondent would like for the thermal imaging camera to view both sides of the vehicle, as it would make the inspection process easier. Two other respondents indicated that the record of trucks coming through is beneficial, and that real-time information would be useful in locating violations.

(Q#A5) WHAT WOULD MAKE YOU, OR OTHERS YOU WORK WITH, USE ISSES OR UTILIZE IT MORE THAN YOU ARE CURRENTLY USING IT?

Of the six respondents, five noted that they would use ISSES more than they are currently using if they had training on the equipment. One respondent stated that staffing was more of an issue than training. If someone was hired to sit and solely monitor ISSES, it would be used more than it is currently being used.

B. Questions Specific to any Portion of the ISSES Equipment

(Q#B1) DO YOU RECOGNIZE NOTICEABLE GAPS IN THE DATA PROVIDED BY THE ISSES (OR A PARTICIULAR SUBSYSTEM)?

Of the six respondents, five provided responses. Two respondents said they did not recognize noticeable gaps in the data provided by ISSES. Notably, the thermal imaging equipment was singled out as a system that generated very few data gaps, which was viewed as a positive feature. However, one respondent was not sure what to even look for. This uncertainty relates to another respondent comment, not knowing when the equipment is working because it is password-protected and the individual had evidently not been trained in login procedures.

(Q#B2) DOES THE ISSES GENERATE TOO MUCH INFORMATION FOR WEIGH STATION STAFF TO APPRECIATE AND USE IN THE TIME AVAILABLE FOR AN INSPECT/BYPASS DECISION?

Of the six respondents, five provided responses. Two respondents said they could not answer because they do not know enough about the system. Notably, one respondent said, "Let's have this conversation again in six months after training." One respondent said that it generated too many audible radiation alarms, while two respondents commented that they utilized the thermal imaging equipment. One of these respondents said that in the time inspectors have available for an inspect/bypass decision, the thermal imaging device could be sited further away to work better and allow them to make a decision in the time allotted.

(Q#B3) ARE THERE ISSES FEATURES OR FUNCTIONS THAT COULD BE CHANGED OR THAT FUTURE UPGRADES COULD OFFER?

Of the six respondents, five provided responses. In response to features that could be changed, one respondent would like to retrieve information (e.g., location and quantity) from the radiation portal monitor and have that information e-mailed to the appropriate party. Another respondent would like to change the thermal imaging device, by incorporating a camera to capture IR data from the other side of the vehicle. Two respondents commented on the license plate reader operation and integration into other databases (e.g., the National Crime Information Center, or NCIC).

(Q#B4) WHAT BENEFITS DOES THE ISSES OFFER? DO THESE BENEFITS MAKE YOUR JOB MORE CONVENIENT/EASIER COMPARED TO THE LEGACY SYSTEM?

Of the six respondents, four provided responses. Two respondents said that ISSES did not make their job easier; one respondent added the caveat that ISSES would offer benefits if weigh station staff knew how to utilize the equipment. One respondent does not use ISSES enough to comment on the offered benefits. One respondent said ISSES gives a heads up with respect to the thermal imaging and radiation monitor. It is good to have both (a) the hand-held detector to verify truck contents at the inspection shed and (b) the stationary portal monitor at the scale house.

(Q#B5) WHAT ASPECT OF THE ISSES EQUIPMENT ENABLES YOU TO PERFORM YOUR JOB FUNCTION BETTER?

Of the six respondents, five provided varied responses. Three respondents felt that the thermal imaging device should benefit them since it seemed "easier to locate possible brake defects" than working at a location without it. Another respondent said the "heads up" aspect of the ISSES equipment enabled him to perform his job function better. In this context, "heads up" meant that the system gave the inspector advance notice of a potential problem with a vehicle entering the station and additional information about that vehicle (see question #D6 below).

(Q#B6) COMMENT ON THE SYSTEM. DOES THE ISSES EQUIPMENT PERFORM AS EXPECTED, BASED ON THE SPECIFICATIONS OR PRODUCT LITERATURE? IF NOT, ELABORATE ON THE PERFORMANCE OF THE PARTICULAR SUBSYSTEM.

Of the six respondents, two provided responses. One respondent essentially said he could not appropriately answer the question because they have not received adequate training. The other respondent said the thermal imaging device performs satisfactorily, while the other subsystems were unsatisfactory and unreliable in their daily functions. The four remaining respondents indicated that they could not comment on the system.

(Q#B7) DOES ONE SUBSYSTEM ADD MORE VALUE THAN ANOTHER, OR DO ALL SUBSYSTEMS EQUALLY HELP YOU PERFORM YOUR JOB FUNCTION BETTER? (E.G., "SUBSYSTEM X HELPS STAFF PERFORM THEIR JOB FUNCTIONS BETTER, BUT SUBSYSTEM Y IS DIFFICULT AND TIME-CONSUMING TO INTERPRET")

Of the six respondents, four provided comments. One respondent said he could not respond to the question because he does not know enough to answer appropriately. Another respondent said that none of the subsystems assisted him or helped him do his job better. Another respondent said the radiation monitor provides value, while two of the respondents said that the thermal imaging device adds value and provides more information.

C. Training

(Q#C1) HAS TRAINING BEEN PROVIDED FOR THE USE OF THE ISSES EQUIPMENT? IF SO, HOW LONG DID THE TRAINING LAST? IF NOT, HOW MUCH TRAINING WOULD BE NEEDED TO BECOME PROFICIENT IN ANY OF THE ISSES SUBSYSTEMS?

Of the six respondents, a 100% response rate was received. All six respondents said that training had not been provided for the use of the ISSES equipment. Personnel turnover and hew hires, however, may have contributed to this reported gap in training.⁹ Two respondents felt that two days of training on the ISSES equipment could be an appropriate length of time for training. One respondent noted that he had received informal, on-the-spot training as technicians worked on the system, and considered it self-motivated training.

(Q#C2) ARE SPECIFICATIONS OR DOCUMENTATION (E.G., USER'S MANUAL) ON THE ISSES EQUIPMENT AVAILABLE ON-SITE? WAS IT DETAILED ENOUGH? WHAT DETAILS WERE MISSING? WHAT KIND OF ADDITIONAL DOCUMENTATION WOULD BE USEFUL?

All six respondents seemed to be very unclear as to what specifications and documentation were available for the ISSES equipment on-site. Some respondents mentioned basic response guides and flowcharts being available as well as radiation portal monitoring information.¹⁰ Additional documentation that would be useful to respondents could include user-friendly training manuals, (i.e., "Cliffs Notes" format guidance), flowcharts, or a quick reference guide.

(Q#C3) HOW DO INSPECTORS OR MANAGERS DEAL WITH THE POTENTIAL LIABILITY FOR MISSED DETECTIONS OF UNSAFE OR HIGH-RISK TRUCKS, WHICH MIGHT TRAVERSE THE WEIGH STATION AND THEN BECOME INVOLVED IN A CRASH CAUSED, FOR EXAMPLE, BY FAULTY BRAKES? IS THERE AN ISSES OPERATING PROTOCOL THAT HELPS INSPECTORS DETECT AS MANY UNSAFE TRUCKS AS POSSIBLE OR PRACTICAL, AND IF SO, HOW EFFECTIVE IS THAT PROTOCOL?

All six respondents interpreted this question differently. None of the respondents had seen an operating protocol, and one respondent directed researchers to check policies and procedures through KVE HQ. One respondent said that manpower is an issue, because no personnel are solely assigned to monitor the ISSES screens. Another respondent said that ISSES does not show him everything that could be faulty on the truck (e.g., the thermal imaging device does not identify broken air reserve tanks, things under the truck that could fall out, straps, etc.).

D. Selection Efficiency

(Q#D1) OTHER THAN INFORMATION FROM THE ISSES, WHAT SPECIFIC DATA IS COLLECTED FROM THE COMMERCIAL VEHICLE PRIOR TO MAKING A DECISION ON WHETHER TO INSPECT?

Of the six respondents, five responses were provided. Visual inspection occurs at the officer's or inspector's discretion. Specific data collected from the vehicle prior to making a decision on whether to inspect includes general appearance or condition of vehicle, USDOT number, company name, tire condition, past experience with a particular carrier, and vehicle weight and tax information.

(Q#D2) HOW IS THIS INFORMATION COLLECTED (SENSORS, WIM, CAMERAS, EYESIGHT, ETC.)?

Of the six respondents, five responses were received. All five respondents indicated that eyesight is the primary means to make a decision on whether to inspect. Four of the respondents also added that the WIM sensor coming off the interstate was utilized. One respondent also added data entry as a means to make vehicle decisions.

(Q#D3) ARE ANY EXTERNAL DATA SOURCES (SAFER, SAFESTAT, QUERY CENTRAL) USED TO SUPPLEMENT DATA COLLECTED AT THE SITE?

Of the six respondents, five provided responses. To supplement data collected at the site, two respondents use all three external data sources: SAFER, SafeStat, and Query Central. Three respondents said they only use SAFER and Query Central, and do not use SafeStat.

(Q#D4) WHAT ARE THE MAIN PIECES OF INFORMATION COLLECTED FROM THE KENTUCKY CLEARINGHOUSE DATABASE TO HELP WITH INSPECTION SELECTION DECISIONS?

Of the six respondents, two provided detailed responses. Both respondents use the Kentucky Clearinghouse to check whether the vehicle has current tax or insurance credentials. One respondent added that this check took place after the inspection decision had been made and the vehicle had been stopped.

(Q#D5) BASED ON ALL DATA COLLECTED, HOW ARE DECISIONS RELATED TO INSPECTIONS MADE? WHAT METHODOLOGIES PLAY A ROLE IN THE DECISIONS (ISS ALGORITHM, INSPECTOR JUDGMENT, ETC.)? HOW MUCH IS BASED ON DATA COLLECTED AND HOW MUCH IS BASED ON INSPECTOR OBSERVATION AND JUDGMENT?

Of the six respondents, five provided responses. All five respondents noted that inspector observation and judgment come into play, sometimes solely, to determine obvious violations (e.g., flat tires, broken headlights, placard violations, or when alarms sound). One respondent estimated that 75% of the decision is driven by inspector observation, while the remaining 25% is data-driven. One respondent noted that the SAFER database played a role in inspection selection process.

(Q#D6) HOW HAS THE INSPECTION SELECTION PROCESS CHANGED WITH THE INTEGRATION OF ISSES AT THE LONDON [KENTON/SIMPSON] SITE?

Of the six respondents, five provided responses. Four respondents indicated that the inspection selection process has not changed much or at all with the integration of ISSES. One respondent noted that WIM, not ISSES, has increased the number of overweight vehicles pulled over for inspection. The ISSES change is that it provides a "heads up" to inspectors.

E. Future Deployments

(Q#E1) WOULD THE ISSES YIELD GREATER BENEFITS IF IT WERE MORE FULLY INTEGRATED WITH STATE AND NATIONAL SYSTEMS, SUCH AS QUERY CENTRAL, STATE INSPECTION OR LICENSING DATABASES, SAFER, COMMERCIAL DRIVER LICENSE INFORMATION SYSTEM (CDLIS), NATIONAL LAW ENFORCEMENT TELECOMMUNICATION SYSTEM (NLETS), ETC.?

Five of the six respondents indicated that ISSES would yield greater benefits or would be more useful if it were integrated with state and national systems or licensing databases. Of those respondents, one said that the current process relies on going to one to two places to get information (e.g., to run a USDOT number), so having everything tied together would make an inspector's job easier. The sixth respondent did not have an idea as to whether ISSES would yield greater benefits if more fully integrated with other systems.

(Q#E2) DOES EACH ISSES SUBSYSTEM WORK SATISFACTORILY IN A STAND-ALONE MODE?

Of the six respondents, five provided responses. One respondent indicated that from what he knows, each ISSES subsystem works satisfactorily in a stand-alone mode. Another respondent indicated ISSES should be integrated with what is already available on-site for inspectors and "weaved into databases" (e.g., Query Central). Two respondents were unable to comment on whether each ISSES subsystem worked satisfactorily in a stand-alone mode.

(Q#E3) WOULD THE ISSES DATA BE USEFUL IF MADE AVAILABLE TO INSPECTORS IN OTHER JURISDICTIONS (E.G., OTHER PARTS OF THE STATE OR SIMILAR ROADSIDE SYSTEMS IN OTHER STATES)?

Of the six respondents, all six provided a response. Five respondents indicated that it would indeed be useful if ISSES data were made available to inspectors in other jurisdictions. A respondent commented that the more people who know how to operate the system, "the better."

(Q#E4) WHAT ASPECTS OF THE ISSES DATA DO YOU RELY ON WITH THE GREATEST CONFIDENCE? IF THERE ARE NO ASPECTS THAT YOU RELY ON, WHAT CHANGES TO THE SYSTEM MIGHT MOTIVATE YOU TO USE AND RELY ON THE DATA?

Of the six respondents, all six responded to various degrees. Two respondents rely on thermal imaging data because you can "actually see trucks on the screen." Other respondents would like to gain familiarity with the equipment before commenting on what aspects they rely on or what changes to the ISSES equipment might motivate them to use and rely on the data.

(Q#E5) CAN YOU SHARE ANY LESSONS LEARNED THAT WOULD PERHAPS BE USEFUL TO OTHER STATES CONSIDERING THE DEPLOYMENT OF SIMILAR EQUIPMENT? (E.G., WORKING WITH EQUIPMENT, TRAINING, LOCATION OF THE EQUIPMENT ON-SITE, ETC.)

Of the six respondents, all six provided a response. In terms of sharing lessons learned, two respondents said that part of the cost of the ISSES equipment should include the cost of training and to make sure that both initial and follow-up training are provided to introduce new staff to the equipment, especially the operation of the system. Two respondents said that as far as a lesson learned, the location of equipment should be considered carefully before installation. Inspectors have to be given sufficient reaction time to stop a vehicle. The two remaining respondents did not report any lessons learned.

F. User Acceptance Hypotheses

(Q#F1) INSPECTORS BELIEVE THAT ISSES ENABLES ROADSIDE INSPECTORS TO PERFORM THEIR JOB FUNCTIONS BETTER. TRUE OR FALSE

All six respondents provided a response. Three respondents answered true, two respondents answered false, and one respondent was unable to answer. One respondent who answered false indicated it was because they don't rely on the ISSES all the time because they do not have the time or staffing to do so. The other respondent who answered false said that because he did not know how to use the equipment, he did not believe it helps him perform his job functions better.

(Q#F2) INSPECTORS BELIEVE THAT ISSES SHOULD BE DEPLOYED MORE WIDELY. TRUE OR FALSE

Of the six respondents, all participants provided a response. Three respondents answered true (with the caveat that proper training would be necessary), two respondents answered false, and one respondent was unable to answer.

(Q#F3) INSPECTORS FOUND THEIR TRAINING AND USER DOCUMENTATION FOR ISSES TO BE HELPFUL TO THEM IN THEIR NORMAL COURSE OF DUTIES. TRUE OR FALSE

Of the six respondents, five participants provided a response. One respondent answered true (again with the caveat of proper training), two respondents answered false, and two respondents did not provide a true or false response because they have not been trained yet to answer the question as to whether training and documentation for ISSES is helpful to them in their normal course of duties.

7.5 System Deployment and Operating Costs Results

The system cost study focused on the economic dimensions of the deployment, for both one-time start-up costs and recurring (annual) costs to operate and maintain the ISSES. Data on actual costs incurred were supplemented by best estimates for those costs that are not available.

Data collection for the system deployment and operating costs was made via contact with the KTC to identify the various costs associated with purchased and installed materials and system equipment, related software integration, and vendor labor.

The KTC has provided a copy of a bill of sale dated 6/2/2005 with cost data and general system specifications. This bill states that the total cost of installing ISSES at the Laurel County weigh station was $350,000. This total cost includes the: radiation detection component; thermal imaging component; license plate reader component; and site preparation and installation. All installed equipment is included in the bill of sale except two rack mount servers. The KTC, which was involved in ISSES contracting between the state and the vendor, reported that funds from Oak Ridge National Laboratory were also used in the original Laurel County installation and deployment, and that subsequent systems installed in other Kentucky counties have actually cost the state approximately $500,000 each to procure and install.

The original budget for the Laurel County ISSES did not provide funding for training or system maintenance. According to the KTC and the vendor, however, recurring (annual) costs for hardware to operate and maintain the equipment have been fairly low. The system is based on low-amperage sensors and communication systems, and does not cause a large electrical current draw. Equipment repairs and replacement of parts, as described below, have been largely due to lightning strikes and electrical power service interruptions, not due to ISSES equipment defects. In November 2006, the KTC entered into a service contract with TransTech to make one field technical support person available at approximately 60% of full-time on-site to cover the three installed ISSES locations for one year, and at about half of the first year's time commitment for two years thereafter. While the technical support person also participates in client- and vendor-driven data collection projects and other activities outside of this on-site service commitment, his main role is to be available to troubleshoot any maintenance issues, monitor the site remotely, make any repairs on-site as needed or requested by KVE or KTC, provide training to operators/inspectors at each of the sites, and identify and test ISSES enhancements.¹¹ The cost of this maintenance and technical support from November 2006 through August 2007 has been approximately $109,000. This amount has covered the ISSES maintenance duties listed above, but some fraction of the field support technician/analyst's time within this contract has been devoted to administrative activities, software programming support, and communications protocol development for the nuclear detection subsystem unrelated to the monitoring, repair, and maintenance of the ISSES. Thus, the entire $109,000 has not been attributable to operating and maintaining the ISSES hardware and software.

TransTech is providing periodic maintenance status reports to the KTC (see Appendix E). According to the KTC, TransTech is supposed to submit periodic status reports, based on daily status reports. These reports are obtained by a TransTech field support technician who can connect remotely to see if the ISSES systems are up and running. When a problem arises, the TransTech technician attempts to troubleshoot the problem remotely, and on some occasions makes site visits to troubleshoot problems. Between May and August 2007, four maintenance records had been sent to the KTC.

It appears that, based on respondent input, observation, and other correspondence (see Appendix E), ISSES requires frequent maintenance because of system troubleshooting and power interruptions, the latter type being considered unscheduled maintenance. It is difficult to delineate whether the maintenance (both unscheduled and preventive/planned) is monthly, weekly, or daily because of the nature of the troubleshooting (e.g., lightning strike versus software modification).

⁶ The KTC indicated that the nuclear detection subsystem at the Laurel County ISSES site had been adjusted in the fall of 2007 (after the time of these interviews) to greatly reduce the frequency of nuisance alarms.

⁷ KTC indicated that, now that a maintenance contract has been established with the vendor, each ISSES site has contact information posted for the on-site technical support person from IIS, giving KVE enforcement personnel consistent access to help if they have a question or a problem with the equipment.

⁸In response to this finding from the interviews, the KTC indicated that there would have been no reason to expect enforcement personnel to be familiar with or to have used the LPR and USDOT number reader, because KTC is still evaluating the performance of these systems as part of the state's CVISN program. Additionally, since the laser scanner does not have a user interface, and given that it is used to trigger other systems, enforcement personnel would not be expected to be familiar with this system.

⁹ In response to this finding from the interviews, the vendor indicated that training had been provided to Laurel County inspectors by ORNL at the time of the system commissioning in 2005. Later, training was provided in Frankfort by the Domestic Nuclear Detection Office (DHS), on response protocol, the use of the hand-held radiation detector, and how to transmit the radiation profile data to the appropriate entities.

¹⁰ In response to this finding from the interviews, the KTC indicated that a response protocol for weigh station personnel to use when a radiation alarm was activated by a truck was developed by the Department of Vehicle Regulation and the Kentucky Office of Homeland Security. After the training in Frankfort, an exercise was conducted by the Domestic Nuclear Detection Office (DHS), in which Laurel County weigh station personnel responded to an alarm, followed the protocol, conducted the inspection using the hand-held detector, and notified the appropriate agencies of the problem.

¹¹ The first such training session was a two-day training session held on July 31 and August 1, 2007, provided to personnel at the Kenton County inspection station. The training session focused on the operation of the thermal imaging system.

Previous | Next