P A G E   &   AS S O C I A T E S Human Resource & Organizational Consultants

Hiring the right people directly impacts productivity, quality, customer service and the bottom-line profitability of the organization. Moreover, bad hiring decisions can be very costly to the organization. The cost of replacing employees that have turned over is estimated to be 1½ times annual salary. This figure includes the costs associated with recruitment, selection, training and lost productivity while the new hire is learning the job.

The following sections address the key issues in Test Validation.
 

• Selection (pre-employment) tests — tests that are administered to job applicants prior to being hired.

 
• Promotion tests — tests that are administered to employees who are being considered for promotion.

 
• Certification tests — tests that are administered to individuals who wish to be certified as possessing a specific level or type of expertise. 

Test Validation addresses the extent to which tests actually measure what they are purported to measure. Do the tests actually identify the most effective employees?

The concept underlying tests is this: If an individual scores higher in a test than other individuals, then he or she should be a more effective employee than the other individuals. To what extent is this true? Is it a valid inference that the higher scorer will in fact become a more effective employee? 

In work settings, Test Validity addresses the existence and nature of evidence demonstrating that employment tests assess job relevant behaviors that relate to job performance outcomes.
 

In an era where an organization’s competitive advantage is achieved through its people, it is important to have highly effective staff. Organizations need to hire the best candidates. Test Validation is the process that pinpoints those pre-employment assessment techniques that identify the best candidates for the organization’s jobs.

There are three primary reasons for doing a validation study:
 

• Defining key success factors — identifying and defining the key competencies that drive success in the organization. By knowing precisely what to look for, your chances of hiring high-performance candidates are greatly increased.

 
• Identifying the best candidates — objectively assessing candidates and assuring that the organization selects those with superior talent. By selecting the best candidates, you can build a superior organization.

 
• Assuring legal defensibility — assuring that your hiring process is legally defensible. By following the prescriptions of the EEOS’s Uniform Guidelines for Employee Selection Procedures, you can provide documentation for the legal defensibility of your testing system, if it were to be legally challenged.

 

There are several important benefits from performing a validation study. These include:
 

• Precise definition of the competencies that drive job success — assuring that you can target those behaviors that truly drive superior performance.

 
• Hiring superior talent — because the most competent candidates are identified through an objective assessment process.

 
• Greater objectivity and fairness in the job applicant screening process — to assure accurate comparisons between candidates.

 
• Significantly Improved productivity and profitability — because of reduced errors in selection decisions, reduced training time and costs, reduced staff turnover, and improved productivity and effectiveness of workers.

 
• Legal defensibility of the employee selection process — providing peace of mind regarding hiring decisions and the avoidance of potentially costly litigation.

 

There are different approaches to validating tests. The most commonly uses approaches in employment settings include:

1. Content Validation — A technique that assures that there is overlap in key aspects of job and test content. Content validity is shown by carrying out a detailed analysis of job content followed by a matching of this information to test content. If there is overlap, indicating that test content is directly related to job content, it is asserted that the test is job related, thus valid.

 
2. Concurrent Validation — A technique that demonstrates, for a sample of employees, that there is a significant relationship between test scores and job performance scores. For a sample of current employees, test scores are obtained and, concurrent in time, job performance scores are obtained. Then, the test scores are statistically correlated with the job performance scores. The resulting correlation coefficient is an index showing the degree of validity for the tests. This approach empirically demonstrates that those identified by the tests as likely to be superior performers are truly superior performers.

 
3. Predictive Validation — A technique that demonstrates, between time of hire and a later point in time, that there is a significant relationship between test scores and job performance. Typically, job applicants are tested and the test results are not used in making hiring decisions. Then, at a later point in time, job performance scores are obtained. The test scores are then correlated with job performance scores to identify the extent to which the tests predict job performance.

Content validity shows that the content of the test mirrors important work behaviors of the job. The process involves an analysis of job content (through job analysis) and information on the test content. Job content is linked to test content. If there is overlap between the test content and important aspects of job content, then the test is determined to be content valid. 

Content validity is a particularly appropriate approach for certain types of knowledge and skill tests that are designed to be a sample of actual work behaviors. Examples include tests that assess word processing skills, alphabetical filing skills, math skills, and knowledge of a job-related technical area.

The process for demonstrating content validity involves the following steps:

1. Job Analysis. An analysis of job content is performed to identify the essential responsibilities of the job and to specify the job competency requirements for job success. This is accomplished by any of a number of different approaches, which include: review of job descriptions, review of training materials, interviews with job holders and supervisors, focus group sessions with subject matter experts (SMEs), and/or completion of job analysis questionnaires.

 
2. Identification of job-related tests. The job content information is matched to tests. Those tests that have a high degree of overlap with job content and the underlying job competency requirements are identified as being content valid. Where multiple tests are identified for a specific skill area, the organization should select the one test that has the closest match to the job. The test that is closest to job content in terms of level of difficulty, format of items and scope of skills assessed should be selected. Additionally, consideration should be given to selecting tests that are easy to administer in a consistent way and are fair and equally relevant for diverse applicant group. Often combinations of different tests are identified, each test assessing a different aspect of job content. These tests are combined together into a test battery. 

 
3. Validation Report. A final report is developed that documents the validity of the testing process. It reports the results of the job analysis and the matching of job content to test content. Also, procedures for the implementation of the test battery in a fair and consistent manner are described. This report has two key purposes: 1) describing how to implement the testing program in an appropriate manner, and 2) providing legal documentation in the case of a legal challenge to the use of the testing process.

 

The concurrent validation process shows that, for a sample of employees, test scores and job performance scores, measured concurrently in time, are statistically related. The statistical measure that is used is the correlation. It measures the degree of relationship between test scores and job performance scores and has a range from plus one to minus one. 

Positive correlations indicate that individuals with higher test scores have higher job performance scores. Correlations of zero indicate that there is no relationship between test scores and job performance. Negative correlations indicate that people with higher test scores tend to have lower job performance scores. The "size" of the correlation shows the degree to which test scores and job performance scores are related. However, if the magnitude of the correlation is statistically significant, the test is typically judged to have met the technical requirements for being valid.

Concurrent validity is one of the most accepted means of demonstrating test validity. It is used for all types of tests, including knowledge and skill tests, but also of tests assessing general mental ability, work styles (i.e., personality), and tests that measure abstract constructs that are not effectively addressed through content validity. 

An important prerequisite to undertaking a concurrent validity study is having a sufficiently large sample. A rule of thumb is that a minimum of 35 subjects be involved in the study. However, the larger the sample the better and it is often desirable to have 100 or more subjects. The larger the sample, the greater the accuracy and precision in asserting the validity of the tests.

A key aspect of the concurrent validity study is the job performance measures used for employees. These are referred to as the "criterion measures." For certain jobs these may be objective measures. For a Data Entry Operator, this may be a measure of the number of error-free keystrokes per hour. However, for most jobs there is no such objective measure and supervisor ratings provide the most direct and accurate measure of job performance.

It is critically important to use accurate and fair job performance measures. These are typically developed for the validation study and typically consist of ratings on separate job performance dimensions. The best rating forms use behavioral descriptors along a scale for each dimension, what is referred to as behaviorally anchored rating scales (BARS). These rating scales are typically developed as a part of the job analysis phase of a concurrent validation study.

The process for demonstrating concurrent validity involves the following steps:
 

1. Job Analysis. An analysis of job content is performed to identify the essential responsibilities of the job and to specify the job competency requirements for job success. This is accomplished by any of a number of different approaches, which include: review of job descriptions, review of training materials, interviews with job holders and supervisors, focus group sessions with subject matter experts (SMEs), completion of job analysis questionnaires. Criterion rating scales are then developed.

 
2. Identification of job-related tests. The job content information is matched to tests. Those tests that appear to most directly relate to the job requirements are selected for inclusion in the concurrent validity study. This may include skills tests, ability tests, work styles (personality) tests, etc.

 
3. Validation Study. A sample of employees are administered the tests and are assessed in terms of job performance. Test scores are correlated with job performance scores.

 
4. Validation Report. A final report is prepared that documents the validity of the testing process. It reports the results of the job analysis and the development of the criterion assessments. It describes the validation study process including the sample, tests, data collection process, and correlations between test scores and job performance. Finally, the report documents procedures for the implementation of the test battery in a fair and consistent manner. This report has two key purposes: 1) describing how to implement the testing program in an appropriate manner, and 2) providing legal documentation in the case of a legal challenge to the use of the testing process.

The Predictive validity process is very similar to the concurrent validity process. The difference is that the validation study is carried out over time. Job applicants are administered the tests, but the test results are not used in selecting employees. At a later point in time, say a year after the hiring of the job applicants, job performance measures are gathered.

Test scores are used to predict job performance scores. This process most directly matches the actual hiring process, where test scores at the time of hire are used to predict future job performance. Consequently, this method provides the most accurate assessment of test validity. However, it is the most difficult to implement because of the challenges in gathering the data over time and assuring that the test scores do not inadvertently influence the job performance scores.

The process for demonstrating content validity involves the following steps:
 

1. Job Analysis. As in content validity, an analysis of job content is performed.

 
2. Identification of job-related tests. The job content information is matched to tests. This process is the same as the process used in concurrent validity.

 
3. Validation Study. A set of job applicants are administered the tests. The test results are not used in selecting employees; individuals with the full range of high and low scores are hired. After a period of time, such as one year, the individuals are assessed in terms of job performance. Test scores are then correlated with job performance scores.

 
4. Validation Report. A final report is prepared. This report has the same content and format as the report for concurrent validity.

The Federal Government, in Title VII of the Civil Rights Act of 1964, as amended, forbids discrimination on the basis of race, color, religion, gender, or national origin. The act prohibits any conditions from being put on employment unless they are a business necessity. Conditions of business necessity are referred to as bona fide occupational qualifications (BFOQ).

The EEOC and three other Federal Government agencies published the Uniform Guidelines on Employee Selection Procedures in 1978. This document defines the nature of discriminatory practices in HR management and provides a set of guidelines for demonstrating the validity of selection procedures. In subsequent legal cases, the courts have used the Uniform Guidelines as the standard against which validity evidence is determined to be legally defensible. Consequently, it is important to assure that validation procedures meet the standards identified in the Uniform Guidelines.

Content validity is addressed in the Uniform Guidelines. They state: 
 

"To demonstrate the content validity of a selection procedure, a user should show that the behavior(s) demonstrated in the selection procedure are a representative sample of the behaviors(s) of the job in question or that the selection procedure provides a representative sample of the work product of the job." (Section 14C(4)).

1. Perform a careful job analysis.

 
2. Document the validity of the selection procedure.

 
3. Implement the selection procedure in a fair and consistent manner.

 

P A G E   &   AS S O C I A T E S Human Resource & Organizational Consultants