TECHNICAL REPORT ON THE VALIDITY OF THE SURVEILLANCE AND MONITORING ASSESSMENT EXERCISE (SAMAE)
REPORT NUMBER 2/1998
Dr Craig Donald (PhD) - Leaderware
This research report reviews the effectiveness of the Surveillance and Monitoring Assessment Exercise (SAMAE) in predicting CCTV operator performance. SAMAE scores demonstrate strong and significant relationships with actual reported incidents observed on the job including suspicious incidents, management and performance ratings, and the detection of non-verbal behaviour as measured in pre- and post-training assessment. SAMAE also shows the ability to differentiate between predicting performance factors related to what it is measuring and others which should show no relationship (eg. report writing). Selection is emphasised as a critical factor in attaining effective CCTV system performance, and the use of SAMAE as part of an approach to managing human factors in the CCTV operation is strongly supported.
The introduction of closed circuit television (CCTV) into communities, business operations, and policing and government operations has been rapid and sweeping. Social needs, shortages of manpower in relation to areas covered, and the enabling effects of technology have contributed to the widespread enthusiasm of adoption. However, this growth has also been characterised at times by shortfalls in the quality and suitability of equipment, suitability for purpose issues and implementation difficulties, and a lack of effective usage of the equipment. Shortfalls in this regard have become the focus of a number of standards bodies and institutions who have endeavoured to define the technology requirements in terms of the operational requirements and human factors of the environments in which they are being introduced (eg., see Aldridge, 1995). This has represented a significant step forward in the quality and application of the technology in the last few years and to a lesser extent the man/environment interface and working procedures.
A significant area of weakness of CCTV systems remains the mindset that the technology has sufficient impact by itself. Achieving system performance through people is often seen as a routine matter and the requirements in terms of the type of people to staff these highly expensive investments have been largely neglected. It has been clearly demonstrated, however, that even in the highest technology environments such as aviation, human factors relating to the person responsible for system performance are critical in achieving effective outcomes (Wickens, 1992). With this in mind, industrial psychologists from Leaderware with experience in the security industry initiated a project to identify core CCTV competencies and develop a method of assessing the suitability of personnel to operate effectively in such an environment. It was felt that through the effective assessment, placement and development of personnel, the full potential of the CCTV technology systems could be better realised.
An extensive job analysis was initiated in CCTV operations within three countries to profile the competencies and work factors required for CCTV operators (see Donald, 1997 for a brief description of these competencies). This job analysis provided the basis for the development of an assessment instrument. The instrument, the Surveillance and Monitoring Assessment Exercise (SAMAE), has been designed to evaluate the capacity of personnel to effectively perform on a sustained and consistent basis within the CCTV environment. SAMAE thus represents part of the solution to ensure that the human component of the security system can fulfil the potential of the technology systems that are implemented. The job analysis indicated a need for the development of three exercises, Scanning, Dynamic Attention, and Incident Detection to address the range and combination of competencies seen as necessary to job performance. The background to these is discussed below.
The Nature of the SAMAE Exercises
The Scanning Exercise evaluates a person's ability to rapidly scan a situation and detect a range of subtle to obvious deviations from a defined standard. The exercise consists of a constantly moving object which can change in a number of ways at any stage and which needs to be continually compared to a standard image. Elements of colour, positioning of objects relative to one-another, rotation, loss of information, and the ability to evaluate the situation as a whole are incorporated into the exercise. The availability of a standard representation of the target helps by refreshing memory and keeping the person sensitive to requirements. Aspects evaluated by the exercise include:
The Dynamic Attention exercise evaluates a person's ability to effectively monitor and deal with a situation where attention must be allocated to independent and changing sources of information. The exercise consists of a number of objects which move around on predetermined paths and which change at specific intervals. Candidates are required to constantly track and prioritise their attention between these objects in order to detect changes in colour, perspective, speed and the visual presence of the objects. Aspects examined during the exercise include:
The incident detection exercise evaluates a person's sensitivity and capacity to identify and respond to specific cues within a short time period. The exercise requires the identification of basic representations of behaviour. Continual changes in the postures of the figures of people in the tests, deliberate distractors, and the intermittent showing of incident activities allows an evaluation of the detection capacity as well as the ability to differentiate between incident and near incident activity. Aspects assessed by the exercise include:
The Validation Site and Sample
Fundamental to the use of an assessment instrument is the confidence that it is a valid and relevant measure of the qualities that do in fact lead to performance. With this in mind, Leaderware initiated an in-depth validation process to demonstrate that the instrument can in fact differentiate between effective and non-effective personnel. This validation process is complicated in any situation when one is reviewing performance because of the difficulty in obtaining a criterion measure of performance that is in itself valid and justified. Further, in the surveillance area, one must find a large enough sample which has criteria in common and justifies being able to draw meaningful conclusions.
The site from which the validation sample was drawn was an internationally recognised operation which met all the validation requirements and which has comprehensively addressed the performance measurement issue as a conscious part of establishing benchmarks of system effectiveness. Further, the operation was willing to allow the assessment of personnel with additional instruments. The sample involved all surveillance personnel and support personnel working in the security department. These personnel had all been selected using standard selection screening procedures before SAMAE was available. Support personnel were involved in training but performed only occasional stand in surveillance duties. They were included in the review of how SAMAE scores relate to training performance. The sample size used for validation varies depending on the time period under consideration and the criterion under consideration. The typical sample size is 14 but full details of sample size are given with each statistical procedure.
SAMAE and Criterion Performance Measures
The full SAMAE battery was administered to surveillance personnel together with the Sixteen Personality Factor Questionnaire (16PF) as part of a concurrent validation study. Individual measures of variables covered by the SAMAE battery were recorded, as well as composite measures which could include combinations of variables within an exercise (eg., accuracy and reaction speed) as well as across exercises (eg., cumulative optimal accuracy for each exercise). The performance of personnel was measured in the following ways:
Monthly recorded detection rates (actual number of incidents) for the following :
- Procedural violations detected including loss control and safety violations.
- Procedural violations detected including loss control and safety violations which were confirmed by the supervisor after video review.
- Suspicious incidents indicating possible criminal activity.
Performance appraisal. Based on a single time period and reflecting whether the person exceeded, met or was sub-standard for:
- A composite rating of confirmed detection ability reflecting the number and quality of confirmed and suspicious incidents
- Public relations feedback.
- Report writing.
Overall management evaluation score by the manager of the surveillance operation based on a one to seven scale representing a range from excellent (7) to poor (1).
Non-verbal behaviour recognition training assessment (evaluation of detection rates of non-verbal behaviour samples at the start and end of a non-verbal behaviour recognition training course).
- Precourse detection rate.
- Postcourse detection rate.
Detection rates of actual incidents were recorded over an eighteen month period. Because of the extended time period, there was some movement of staff as well as some refinements in the way in which incidents were measured. To control for these effects, time periods which reflected common staff membership and which had identical measures within that period were used in analysis to establish trends.
The Validation Process
The validation process took two major principles into consideration:
a) Results had to be replicable - ie., there is a consistent trend that SAMAE predicts performance. This is to ensure that the result is not just a once off anomaly.
b) Results must be generalisable - ie., SAMAE must predict a range of criterion reflecting performance and not just an isolated measure of performance that may not be relevant to the real world.
In line with these principles, the following approach was taken:
1. SAMAE results were compared to performance across a time period which represented the best continuity of people and where the same measuring technique was used. In addition, SAMAE results were compared to relevant performance measures across different time periods to see if there was a consistent relationship within each of these time periods. Establishing relationships across different time periods indicates that the small sample size is not creating a misleading conclusion. The fact that the composition of the sample changes slightly across time periods leads to further support for the validity of the findings as results are consistent not only across time but also with sample alterations.
2. SAMAE results were compared to a number of different measures of performance. In this way, it would be possible to evaluate the extent to which SAMAE identifies a full range of factors relevant to performance which enable surveillance personnel to be effective.
3. SAMAE results were compared to performance on non-surveillance tasks. The rationale for this is that relationships should exist between SAMAE results and measures of surveillance performance, but not necessarily for criteria that do not require those competencies. If there is this distinction, it shows that SAMAE is measuring specific surveillance based competencies and not just a general condition within the workplace.
Prior to the establishment of relationships between SAMAE and performance criteria, an analysis was conducted of the reliability of the three core exercises. Reliability was assessed using the Kuder Richardson formula for internal consistency. Reliability coefficients were generally high and consistent with other previous samples tested with SAMAE. Kuder Richardson reliability indices were as follows: Scanning exercise (.83), Dynamic Attention (.86), and Incident Detection (.74). These reflect good reliability for the exercises.
The relationships between SAMAE and the performance variables were then examined. These relationships are illustrated using correlations (Pearson) between the variables. Given the small sample sizes, it is difficult to do more sophisticated analyses. However, analysis of the consistency of similar results demonstrated across smaller time periods provides confirmation of the relationships with performance variables where this is applicable. As indicated above, periods were determined by the consistency of the sample of personnel within a period and the existence of a common measurement. During the initial stages, there was some revision of how incidents were defined although the total number remained the same, and later the detection of suspicious incidents was introduced.
SAMAE results are reported in three ways. Firstly, composite variables are used which involved the combination of scores across two or more exercises. Secondly, composite variables which combine scores within an exercise are used. Thirdly, discrete scores produced by the tests are recorded in some instances. These are briefly explained below:
The performance measures are reported as general variables (eg., the management evaluation), for optimised periods of time which allow sample size and consistency of measures (eg., combined periods), and specific periods which are broken down into durations of two to four months (where people and measures are common within these periods).
SAMAE results are initially compared to overall measures of performance in order to outline the consistency across different performance measures. The relationship is illustrated in Table 1. For incident rates (reported, confirmed and suspicious), a nine month period is used where the sample size is ten people. This was the longest time period which accommodated the need for consistency of the sample and measures. The longer time period is seen as important in that detection of incidents may occur infrequently and longer time periods give a better sampling of the relevant behaviours. Shorter time periods provide larger samples and this is illustrated in later tables which show SAMAE relationships with performance outcomes across several time periods.
Table 1. Correlations between SAMAE Variables and Work Performance Measures
Note: * p<0.05 ** p<0.01
The correlations shown in Table 1 indicate that SAMAE scores consistently and strongly relate to a number of performance indicators. The size of these correlations is substantial when compared to the general reported usefulness of other assessment instruments. The scope of relationships is also significant as it covers the full performance spectrum. SAMAE results relate to effectiveness in training, several measures of direct work performance, and overall performance and general managerial ratings.
Another of the essential validation principles covered within this study was whether the relationships between SAMAE variables and performance criterion were consistent over time. For purposes of this document, the relationship between SAMAE scores and confirmed incidents are used. This is because confirmed incidents are a higher quality measure as the observed incidents are substantiated using video tape review. The relationships to non-confirmed incidents follow a similar pattern. Correlations between SAMAE scores and the number of confirmed incidents are shown in Table 2.
Table 2. Correlations between SAMAE variables and Confirmed Incidents Across Time
Note: * p<0.05 **p<0.01 1 Combined period is consecutive nine month period.
The above table shows that the SAMAE relationships with performance is maintained different time periods. Further, even for shorter periods where incident rates are typically lower, the relationships SAMAE/performance remain strong. Further analysis of SAMAE performance over different time periods was made using the detection rates for suspicious incidents. Suspicious incidents detected are those which directly involve the observable theft of product or constitute behaviour which prepares conditions for the theft of product. Results show similar trends to that with confirmed incidents and are reported in Table 3 below.
Table 3. Correlations between SAMAE variables and Suspicious Incidents Across Time
Note: * p<0.05 **p<0.01
Towards the end of the period during which data was gathered, the security department concerned initiated a performance management system involving the rating of personnel on key output dimensions. This data was incorporated into the validation as a composite performance management dimension derived from performance ratings of confirmed incidents and suspicious incidents. This composite measure was used as it was seen to represent the critical outputs of the performance management. However, the performance appraisal also incorporated non-surveillance dimensions that reflected abilities that were not seen to be measured by SAMAE. A further principle covered in this article is that an instrument's validity can be supportedwhen it is demonstrates that it predicts that which one would expect it to, and does not predict criterion which are not related to the factors that it is measuring. This relates to the ability to differentiate in predicting relevant versus non-relevant criteria. A detailed review of the relationships between SAMAE variables and performance management dimensions is illustrated below in Table 4. As discussed after the table, SAMAE successfully differentiates between surveillance and non-surveillance related performance.
Table 4. The Relationship and Differentiation of SAMAE results and Performance Management Dimension
Note: * p<0.05 **p<0.01 Sample size = 15
The introduction of CCTV systems has largely emphasised the capacity of the technology to perform the task of protecting people or products. This has led to an artificial belief that given cutting edge technology and sophisticated software systems, desired results will automatically be accomplished. There has been an increasing emphasis on designing systems which are appropriate to operational requirements and a growing awareness of ergonomic factors in such areas as control room environments. However, the operator competencies that are required for effective operation and the implementation of appropriate selection instruments has been largely neglected. This study shows that SAMAE measures qualities that are necessary for effective operator performance. Further, it indicates that selection of personnel plays a major role in achieving the desired outputs from the CCTV system. Results clearly indicate that to produce a comparable level of increase in system performance to that of selecting appropriate personnel by investing in additional technology is likely to be prohibitively expensive and unrealistic. The impact that the relationships between SAMAE and performance variables have on system effectiveness is illustrated in Table 5. This provides the contrast between the actual detection rates of personnel scoring in the top 50% of SAMAE versus the bottom half of those tested. Using three different SAMAE scores, those scoring in the top 50% consistently detected more than double the number of confirmed incidents than those scoring below the average.
SAMAE results strongly and consistently relate to actual detection rates on the job including the detection of potentially illegal activities. These relationships are significant in their impact. The fact that they occurred consistently across time periods and with slightly differing sample compositions across these periods reinforces the predictive validity of SAMAE scores. SAMAE also relates well to management ratings of ability, a criterion that is typically used in most validation studies in the absence of hard performance data such as the incident detection rates used in this research.
The case for SAMAE's predictive validity is strengthened by the fact that is clearly differentiates between surveillance ability and other job related activities. The strong relationship between SAMAE and performance ratings on detection can be expected and one would similarly expect SAMAE not to correlate with public communication and report writing - something that proves to be the case. In this sense, SAMAE can be seen to be measuring what it sets out to do.
The relationships between SAMAE and training performance in an area essential to performance, that of non-verbal behaviour recognition, also has fairly major implications. The fact that SAMAE results relate to the identification of non-verbal behaviour in both pre- and post-test training evaluation appears to indicate that there is an underlying set of competencies within people that make them naturally more suited to surveillance and detection work. Training is likely to enhance people's ability, but people who have natural potential for the work appear likely to maintain the performance gap after training. This reinforces the need to have an appropriate selection instrument to identify people who will be able to operationalise the training inputs more effectively than others.
The review of SAMAE scores and their relationship to performance indicates that surveillance is composed of a number of abilities that combine to show a particular style. In this context, the combination of variables within SAMAE is seen to provide the most effective way of identifying personnel. For instance, efficiency for the scanning exercise is based on a combination of accuracy and false alarms. Taken separately, a person may have high accuracy indicating high detection rates,
but this may in fact be due to indiscriminate identification of all items. Using a combination of both scores as an efficiency index provides one with a measure that takes both accuracy and false alarms into consideration within the same score. Use of combinations of scores across exercises also seems to work well, providing slightly more predictive utility than single exercises by themselves. Among single instruments, the scanning exercise provides the highest utility. This possibly reflects that situational analysis reflected in the ability to analyse the visual components of a situation relative to one another is one of the main factors in surveillance success. The status of the incident detection exercise needs further consideration. Although significant with the detection of non-verbal behaviour rates in training, it was not a strong identifier of job performance in this context. This may mean that it has more specialised application in certain contexts, for example casino tables where quick actions are characteristic of the type of behaviour under surveillance. This will be followed up in research currently in progress.
This research included a review of the relationship between personality factors as measured by the Sixteen Personality Factor Questionnaire (16PF). This information will be reported in a subsequent research report. Although some trends emerged, there were only a few significant relationships with detection rates. These included characteristics such as people who tend to be reserved (A), people with higher levels of energy and enthusiasm (F), and more practical people (M). People who were more emotionally stable (C) showed an appreciable but non-significant relationship. While there is support for the contention that personality does have an impact on suitability for surveillance, it appears from these results that the physical and mental competencies as measured by SAMAE have a stronger impact on determining performance.
Given the strong trend across different measures of performance, as well as training outcomes, it is clear that the relationships shown in this research are not a "situational coincidence" but are in fact a clear reflection of the effectiveness of SAMAE in measuring potential performance levels of personnel. Although these results have been gathered from a relatively small sample group, the consistency across time periods and different measures including on the job incident rates addresses some of these sample limitations. There are nevertheless a number of human factors that will impact on CCTV operator performance (see Donald, 1997a). SAMAE should therefore form part of systematised approach to the management of these human factors in the CCTV system, including aspects of selection, training, performance management, motivational inputs, and ergonomic system design to allow the strengths of personnel to be realised and system performance to be optimised.
Aldridge, J. (1995). CCTV Operational Requirements Manual - Publication No 17/94. Police Scientific Development Branch. Home Office.
Donald, C.H.M. (1997). Job competencies required for effective CCTV operation. Electronic Security Systems SA, 3, 43-44.
Donald, C.H.M. (1997a). The Impact of human factors in CCTV. Hi-Tech Security Systems SA, 3, 6, 54-62.
Donald, C.H.M. (1998). Never underestimate the human factor. International Security Review, 104, May/June.
Wickens, C.D. (1992). Engineering Psychology and Human Performance. Champaign: Harper Collins.