Loading...
HomeMy WebLinkAboutTable of ContentsTABLE of CONTENTS METHODOLOGY: Qualifications, Capabilities & Experience A) Uniform Guidelines Experience 3 B) Sworn Service Test Development Experience 3 C) Validation Study Reporting & Adverse Impact Analyses 4 D) Expert Witness / Test Validation Litigation Support 4 E) Test Validation Training & Professional Development 6 F) Sample Final Report 23 TECHNICAL APPROACH 7 IMPLEMENTATION STRATEGY 11 REFERENCES 12 COST 13 RESUME 14 WORK SAMPLE 23 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) Jim(c)JimsharLcom METHODOLOGY: Qualifications, Capabilities & Experience A) Describe proposer's experience that demonstrates a depth of understanding and practical application of the Uniform Guidelines on Employee Selection Procedures issued by the U.S. Department of Justice in applying the Uniform Guidelines on Employee Selection Procedures. 1974 - 1978: Chief Psychologist, Equal Employment Opportunity Commission, Washington, DC Assisted EEOC attorneys in: 1) assessing prima facie adverse impact statistics, and 2) testifying as expert witness in Title VII litigation; Represented EEOC in drafting 1978 Uniform Guidelines on Employee Selection Procedures. 1985 to 1991: Personnel Research Psychologist, U.S. Office of Personnel Management, Washington, DC Used criterion -related cross -validation design to develop biodata selection procedure for Federal civil service. (Gandy,J., A.Outerbridge, J.Sharf & D.Dye. Development and Initial Validation of the Individual Achievement Record. Washington, DC: U.S. Office of Personnel Management, 1989)-for which I received the Received Director's Award. Represented OPM on the National Academy of Science study of validity generalisation for the U.S. Employment Service's General Aptitude Test Battery. 1991 to 1993: Special Assistant to the Chairman, Equal Employment Opportunity Commission, Washington, DC Advised EEOC Chairman on: 1) the generally accepted principles & practices of industrial psychology governing employee selection & appraisal procedures; 2) the precedent of Title VII employment testing case law in relation to the EEOC's 1978 Uniform Guidelines on Employee Selection Procedures; and 3) Drafted "race norming" prohibition in the Civil Rights Aetof 1991. B) Describe proposer's competency and experience constructing valid selection instruments and testing processes for classifications in the sworn service. 1994 to 1998: Vice President, Human Resources Consulting Group, Aon Consulting, Washington, DC Developed biodata selection test for Louisiana State Police under terms of consent decree to which the U.S. Department of Justice was a party. 2002: Consultant to Transportation Security Administration, U.S. Department of Homeland Security, Washington, DC Consulted with TSA in developing Federal Air Marshall selection tests, although this work is classified as "security sensitive" and not available for public review. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) jim@jimshartcom C) Describe proposer's competency and experience in compiling final reports, statistical analyses following administration of competitive examination processes. 2003: Alabama Department of Transportation On the basis my affidavit, the Federal District Court both changed a consent decree and approved my content validity reports of the State's minimum qualification requirements for several dozen jobs in Reynolds v. Alabama Department of Transportation. (U.S. District Court for the Middle District of Alabama. Case No. CV-85-T-665-N. Affidavit: Feb. 2003; Testified before Special Master: March 2003). - 2002: Consultant to Transportation Security Administration, U.S. Department of Homeland Security, Washington, DC Developed the on-line prerequisite screening criteria used to pre -qualify 1.7m applicants; Developed the 3.5 hour computer -administered selection test battery given to 355k applicants at 84 locations nation-wide over an 8-month period -which resulted in the hiring of fifty-five thousand airport security screeners. (See attached DRAFT Transportation Security Screener Validation Study Report, July 2004). The Society for Industrial and Organizational Psychology recognized my TSA test development accomplishment in awarding me the M. Scott Myers Award for Applied Research in the Workplace in 2006. 2001: California Attorney General, CALTRANS Wrote content validity report of the performance appraisal and promotion criteria for CALTRANS, the largest public sector employer in California in successfully defending an age class action brought in State court. 1989: Gulf Oil Corp. On the basis of my validation study of numerous refinery jobs, my successful defense of validity generalization was affirmed by the Fifth Circuit (Bernard-v. Gulf Oil Corp. 890 F.2d 735 (5`h Cir. 1989), the highest Federal Court V-G decision to date. D) Describe proposer's competency and experience serving as an expert witness in employment litigation cases in state, local, and circuit court, particularly as it concerns the validity of competitive selection processes. U.S. Department of Justice Testified on behalf of the DOJ's Employment Section of the Civil Rights Division challenging the validity of selection tests used by the New York State Police (United States v State of New York Testimony, Fall 1978); and challenged DOJ in Federal court defending validity evidence for selection tests used by Nassau County NY Police Department (United States v Nassau County: Technical report, May 1995, CV-92 5728 (ARR). 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 6844886 (o) jim a@jimsharicom FBI & Secret Service Presently serving as employment testing expert for: 1) the Black FBI (in DOJ) agents under terms of the. BADGE consent decree challenging the validity evidence for the FBI's promotion exams, and 2) the Black Secret Service agents in Moore v. Chertoff (Civil Action # 00-953 (RWR/DAR) challenging the validity evidence for the Secret Service's promotion criteria. 2002: New York Attorney General, New York State Board of Regents Defended the content validity of teacher competency exams developed by the Educational Testing Service and National Evaluation Systems in Gulino v. The Board of Education of the City School District of the City of New York and the New York State Education Department. U.S. District Court for the Southern District of New York. Case No. 96-Civ. 8414 (CBM). Report: May 2002. 2002 — present: New York City Board of Education Presently representing a class of intervenors defending the content validity of promotion tests in opposition to DOJ's consent decree which ignored validity evidence and implemented quota relief (United States v. New York City Board of Education. Civil Action No. 96-0374. Report (intervenors) Jan. 2003; Deposition; Oct. 2003; Testimony, Feb. 2007. Sharf & Associates, 1999 - present; Test Validity Litigation Support Coch v. The Detroit Edison Company. State of Michigan in the Circuit Court for the County of Wayne. Case No. 98-841455 NO. Affidavit: May 2000. Weaver v. Techneglas, Inc. Case No. C-2-99-086. Declaration: Sept. 2000. Lotharp v. The Detroit Edison Company. State of Michigan in the Circuit Court for the County of Wayne. Case No. 00-007464 NO. Affidavit: Oct. 2000. Pfohl v. Western Kentucky University. Report: Nov. 2000_ Johnson v. City of Memphis. U.S.D.C. No. 00-2608. Report: June 2001. McCoy v. Willamette Industries, Inc. Civil Action No. CV-401-75 (class action). U.S. District Court for the Southern District of Georgia, Savannah Division. Deposition: Nov. 2001. SUMMARY JUDGMENT Williams v. PPG Industries, Inc. U.S. District Court for the Middle District of North Carolina. Case No. 1:01CV00459. Affidavit; Nov. 2001. SUMMARY JUDGMENT Gulino v. The Board of Education of the City School District of the City of New York and the New York State Education Department. U.S. District Court for the Southern District of New York. Case No. 96-Civ. 8414 (CBM). Report: May 2002. Maxey v. ALCOA, Inc. U. S. District Court for the Northern District of Ohio, Eastern Division / Cleveland. Civil Action No. 1:02CV 280. Affidavit: Oct. 2002. Albrecht v. Leland Stanford Jr. University. Superior Court of the State of California, County of Santa Clara. Case No. CV802456. Declaration: Jan. 2003. 211 North Union Street, Suite 100, Alexandria, VA 22314 7113! 684-4886 Oa) Jim@jimsharf cam United States v. New York City Board of Education. Civil Action No. 96-0374. Report (intervenors): Jan. 2003; Deposition, Oct. 2003. Reynolds v. Alabama Department of Transportation. U.S. District Court for the Middle District of Alabama. Case No. CV-85-T-665-N. Affidavit: Feb. 2003; Testified before Special Master: March 2003. Mitchell v. DaimlerChrysler Corporation. USDC Case No. 5:02CV1465. Report: March 2003. DISMISSED WITH PREJUDICE United States v. New York City Board of Education. Civil Action No. 96-0374. Report: April 2004. Testified: March 2007. E) Describe proposer's competency and experience in providing assessor training and professional development to practitioners in the area of testing and validation. Conducted Professional Association Seminars on Uniform Guidelines Conducted three-day seminar series in 14 cities nation-wide entitled "Federal Guidelines on Employee Selection Procedures, Interpretation and Application" for the American Society for Personnel Administration (now Society for Human Resource Management), 1978-79. Conducted "Developing Sound Employee Selection Procedures: A Detailed Look at Job Analysis and the Requirements of the Uniform Guidelines on Employee Selection Procedures" for the Equal Employment Advisory Council, January '82, Conducted three-day seminar series in 5 cities nation-wide "Appraising Performance and Productivity" for the International Personnel Management Association, 1983. "EEO Legal Developments" gave invited talks to industrial psychologists at Personnel Testing Councils in New York (2x), Washington (6x), Chicago, St. Louis, Houston, Los Angeles (5x), and San Francisco (2x): 1977 — 2006. Conducted Society for Industrial / Organizational Psychology Workshops (participants received professional continuing education credit) Designed, Produced and Conducted: "The Tightening Legal Strangle -hold on Employee Selection Procedures" for the Society of Industrial & Organizational Psychology Workshop at the Annual Convention of the American Psychological Association, Aug. '76. "Uniform Guidelines: Basic Issues and Practical Implications" for the Society of Industrial & Organizational Psychology Workshop at the Annual Convention of the American Psychological Assn., Aug. '79. "EEO/Fair Employment Update" with private sector attorney for the Society of Industrial & Organizational Psychology Workshop at the Annual Convention of the American Psychological Assn., Aug. '82. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (a) pri charCcnm "EEO Update" with private sector attorney at the Society of Industrial & Organizational Psychology Workshop at Annual Convention of the American Psychological Association, Aug. '87. "EEO: Legal and Regulatory Developments" workshop with private sector attorney at Annual Meeting of Society of Industrial & Organizational Psychology, April '96. "Defending Certification & Licensing" workshops with private sector attorneys at the Annual Meeting of the Association of Test Publishers, April '05 & Feb. '06. "Defending Minimum Qualifications for the Internet Applicant" workshop with private sector attorneys at Annual Meeting of Society of Industrial & Organizational Psychology, May '06. Invited University Lectures on EEO & Employment Testing Duke University George Washington University Memphis State Univ. Law School Pennsylvania State University Stetson University Law School University of Chicago University of Houston Univ. of Louisville School of Law University of Maryland Univ. North Carolina Law School Univ. South Carolina Law School USC School of Public Admin. University of Tennessee Univ. Texas School-ef Public Admin. Univ. Wisconsin Law School Vanderbilt University Law School F) Sample of Final Report: Provide the City with one sample of a final report prepared following the administration of a competitive examination administration. See p.23: DRAFT Transportation Security Screener Validation Study Report, July 2004. TECHNICAL APPROACH My technical support of the City of Miami's test validation efforts will be based on my extensive experience both in developing and successfully defending employment testing for both public and private sector employers nationwide -as well as my experience teaching fellow industrial psychologists. I bring to the City of Miami extensive experience both a student of the Federal Court testing decisions and a scientist - practitioner in developing and implementing objective selection procedures. First of all, I have written extensively on the Federal Courts' resolution of issues related to adverse impact, less -adverse "alternatives" and the so-called "bottom line": 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) jim@jimsbarf com Supreme Court Teal decision: Good "bottom line" does not preclude a prima facie case nor does it provide a defense to a prima facie case. The Industrial/Organizational Psychologist, 1982, 20(2), 33-36. Legal and EEO issues impacting on personal history inquiries. In G. Stokes, M. Mumford & W. Owens (Eds.), The Biodata Handbook: Theory, Research & Application. Palo Alto, CA: Consulting Psychologists Press, Inc., 1992. Statistics of prima facie case examined in Hazelwood School District v. United States. The Industrial/Organizational Psychologist, 1977, 15(1), 15-16. The prima facie case: Keeping Title VII honest. In K. McGovern (Ed.), An Equal employment Opportunity Practice Guide. Washington, D.C.: Subcommittee on Publications of the Committee on Equal Employment and Collective Bargaining of the Council on Labor Laws and Labor Relations, Federal Bar Association, 1978. Court of Appeals rejects EEOC's claim that everyone -in -labor market is equally qualified for all jobs (...and but for discrimination, the workforce would reflect parity with the labor market): The Industrial/Organizational Psychologist, 1980, 17(3), 60-61. Third Circuit's Lanning v. SEPTA Decision: `Business necessity' requires setting minimum standards. The Industrial/Organizational Psychologist, 37(2), 138-149. Industrial/Organizational Psychologist, 40(3), 3 1-40, , Courts limit use of cutoff scores in employment testing. Employment Testing Law & Policy Reporter, June 1997, 6(6), 89-93, 103. I have also written extensively on the Federal Courts' addressing test validation issues: Supreme Court to examine criterion adequacy and job analysis requirement in Moody v. Albemarle Paper Co. The Industrial/Organizational Psychologist, 1975, 12(3), 30-33. Supreme Court decides Washington v. Davis involving training criteria. The American Society for Training and Development Journal, 2(8), June 1, 1976. A lawyer's guide to testing practices and procedures. In K. McGovern (Ed.), An Equal Employment Opportunity Practice Guide. Washington, D.C.: Subcommittee on Publications of the Committcc en Equal Employment and Collective Bargaining of the Council on Labor Laws and -Labor Relations, Federal Bar Association, 1977. Content validity: Whither thou goest? The Industrial/Organizational Psychologist, 1980, 17(3), 8-14. Validity generalization: Round one (Pegues v. Mississippi State Employment Service). The Industrial/Organizational Psychologist, 1980, 17(4), 36-38. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) jim@jimshartcom Landmark appeals decision on content validity: Guardians Assn.. of New York City Police Department v. Civil Service Commission of the City of New York. The International Personnel Management Association Assessment Council News, Winter 1980, 4(4), 1-3; The Industrial/Organizational Psychologist, 1981, 18(2), 21-23. With Ad Hoc Group on Uniform Guidelines: A Professional and Legal Analysis of the Uniform Guidelines on Employee Selection Procedures. In V. Day, F. Erwin & A. Koral (Eds.) Alexandria, VA: The American Society for Personnel Administration, 1981. National Research Council of the National Academy of Sciences report: Ability Testing: Uses, Consequences and Controversies. The Industrial/ Organizational Psychologist, 1982, 19(3-, 34-35. Personnel testing and the law. In J. Ferris & K. Rowland (Eds.), Personnel Management: New Perspectives. Boston, MA: Allyn and Bacon, 1982. The Uniform Guidelines' paradox persists. New Perspectives, The U.S. Commission on Civil Rights, Winter/Spring 1986, 18(1), 53. With Ad Hoc Committee on Employee Selection. Employee Selection: Legal and Practical Alternatives to Compliance and Litigation, Second Ed. In E. Potter (Ed.), Washington, D.C.: National Foundation for the Study of Equal Employment Policy, 1986. Validity generalization: Round two. The Industrial/Organizational Psychologist, 1987, 24(3), 48-50. Gottfredson, L. & Sharf (guest Eds.). Fairness in employment testing: Litigating personnel measurement policy. Journal of Vocational Behavior, 1988, 33(3), 235-271 Gandy,J., A.Outerbridge, J.Sharf & D.Dye. (1989). Development and Initial Validation of the Individual Achievement Record. Washington, DC: U.S. Office of Personnel Management. Employment testing and the -law in the United States. In J. Jones, B. Steffy, and D. Bray (Eds.), Applying Psychology in Business. Lexington, MA: Heath & Co., 1991 Valid employment testing lowers litigation risks. The California Psychologist, October 32(10), 22-25. Third Circuit's Lanning v. SEPTA Decision: `Business necessity' requires setting minimum standards. The Industrial/Organizational Psychologist, 37(2), 138-149. SharfJ. and Jones, D. "Managing Employment Risk." In Kehoe, J. (ed). Managing Selection Strategies in Changing Organizations. Society for Industrial and Organizational Psychology Professional Practice Series. San Francisco, CA: Jossey-Bass, 1999. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) Jim@a jtmsharf.com Lanning revisited: The Third Circuit again rejects relative merit. The Industrial/Organizational Psychologist, 40(3), 31-40. Borden, L & J. Sharf (in press) "Developing Legally Defensible Content Valid Selection Procedures." In Whetzel, D. & G. Wheaton (Eds.), Applied Measurement: Industrial Psychology in Human Resources Management. My intent is to teach the City of Miami's test development staff using the most relevant of my published articles describing Federal Courts' adverse impact and test validation decisions. I also will be teaching City Employee Relations staff using the 2007 edition of Whetzel, D. & G. Wheaton (Eds.), Applied Measurement: Industrial Psychology in Human Resources Management in which my chapter on defending content validity can be found. The chapters most relevant to professional development of the City's staff include: Job Analysis L. Borden & J. Sharf. Developing Legally Defensible Content Valid Selection Procedures Harvey, J., L. Anderson, L. Baranowski, & R. Morath. Job Analysis: Gathering Job -Specific Information. Peterson, N. & R. Jeanneret. Job Analysis: Overview and Description of Deductive Methods. Item Writing Drasgow, F., D. Whetzel & S. Oppler. Strategies for Test Validation and Refinement. Wheaton, G. & D. Whetzel. Contexts for Developing Applied Measurement Instruments. Ree, M. & T. Carretta. Tests of Cognitive Ability. Tsacoumis, S. Assessment Centers Test Development Felker, D., P. Curtin & A. Rose. Tests of Job Performance Huffcutt, A. & S. Youngcourt. Employment Interviews. McDaniel, M. & D. Whetzel. Situational Judgment Tests. McGonigle, T. & C. Curnow. Measures of Training and Experience. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) Jim@Jimsharf.coat Statistical Analyses Mumford, M., D. Whetzel, S. Murphy & D. Eubanks. Background Data. Pulakos, E. Performance Measurement. Russell, T. & N. Peterson. Measurement Plans and Specifications. Finally, I will bring my experience having written/edited content validation study reports in successfully defending selection tests for dozens of jobs at the Alabama Department of Transportation and for promotion exams at CALTRANS, the largest public -sector employer in California. IMPLEMENTATION STRATEGY Since incorporating my own consulting business in 1999, my professional time has been focused on working side -by -side with employment attorneys, HR managers and fellow industrial psychologists on developing, implementing and defending selection and appraisal systems that minimize the risk of employment litigation. My success as a consultant has been due, in part, to my willingness to work closely with client I/Os as my references below will attest. Secondly, as described in Section D above, I have extensive public -sector experience largely challenging DOJ's employment testing agenda and, as my record shows, I have gone into Federal Court on numerous occasions in which opposing council was from DOJ. I am confident that my considerable DOJ experience will well serve the City of Miami's interest in resolving issues related to the consent decree. Finally, my experience in reducing adverse impact is undoubtedly best illustrated in the "bottom line" results we achieved nation-wide for the Transportation Security Administration. After nine months of testing, the newly -hired fifty-eight thousand federal transportation security screener workforce was comprised of 35% women and 44% minorities (25% Black, 13% Hispanic and 5% Asian American / Pacific Islander). These EEO results were obtained by giving careful attention both to the competencies measured and to the setting of cut scores for component tests in the 3.5 hour computer - administered test battery. I will bring this "bottom line" expertise of mine to minimizing adverse impact at each step of both test development and implementation. 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 6844886 (a). Jim©jimeharf.com REFERENCES Wrote/edited content validity reports of the State's minimum qualification requirements for several dozen jobs in Reynolds v. Alabama Department of Transportation. (U.S. District Court for the Middle District of Alabama.. Case No. CV- 85-T-665-N. Affidavit: Feb. 2003; Testified before Special Master: March 2003). (References: Maury Buster, PhD, 334/353-8579; Lisa Borden,. JD, outside counsel to State of Alabama, 205/244-3803). Developed biodata selection test for Louisiana State Police under terms of consent decree to which the U.S. Department of Justice was a party. (Reference: Keith Pyburn, JD, outside counsel at New Orleans law firm Fisher & Phillips, 504/592-3838). Consulted with TSA in developing both airport security screener computer - administered test battery and Federal Air Marshall selection tests, although the FAM work is classified as "security sensitive" and not for public distribution (Elizabeth Kolmstetter, PhD, Dept. of Homeland Security, 571/227-1804): Wrote content validity report of the performance appraisal and promotion criteria for CALTRANS, the largest public sector employer in California in successfully defending an age class action brought in State court. (Reference: Marty Hagan, JD, Asst. Attorney General, Office of California Attorney General, 619/645-2094). Presently representing a class of intervenors defending the content validity of pioutotion tests in opposition to DOJ's consent decree which ignored validity evidence and implemented quota relief. (Reference: Michael Rosman, JD, General Counsel, Center for Individual Rights, 202/833-8400). Presently defending Black Secret Service agents in Moore v. Chertoff (Civil Action # 00-953 (RWRIDAR) challenging the validity evidence for the Secret Service's promotion criteria. (Reference: Debbie Boardman, JD., Hogan & Hartson, 202/637- 6620. Presently defending Black FBI agents under terms of the BADGE consent decree challenging the validity evidence for the FBI's promotion exams. (Reference: David Shaffer, JD, 202/960-2820). 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (a) lim jimchartrnm COST Hourly rate for professional time: $400/hr ($3,200/day) + travel expenses. Year 1 Training / professional development 4 days $12,800 Content validation study final report, police officer 5 days $16,000 Review validation study report, police lieutenant 2 days $6,400 Test Development Sign Off: police officer & lieutenant test plan, job analysis, job analysis results, exam design, candidate preparation manuals, exam administration instructions, exam content: Final Written Reports 8 days $25,600 Annual Status Report w/recommendations 2 days $6,400 Criterion -related validation study final report, probationary police officer 5 days $16,000 $83,200 Year 2 Training / professional development 4 days $12,800 Content validation study final report, police officer 5 days $16,000 Review validation study report, police lieutenant 2 days $6,400 Test Development Sign Off: police officer & lieutenant test plan, job analysis, job analysis results, exam design, candidate preparation mane als, exam administration instructions, exam content: Final Written Reports 8 days $25,600 Annual Status Report w/recommendations 2 days $6,400 $67,200 DOJ consultation (years 1 & 2) 3 days $9,600 Books for professional development (15 x $86.87@a) $1,200 Travel expenses (12 trips x $750/trip) $9,000 TWO YEAR TOTAL: professional time + expenses $176,600 211 North Union Street, Suite 100, Alexandria, VA 22314 703 / 684-4886 (o) jim@Jtmsharf.com 14 James C. Sharf, PhD PROFESSIONAL EXPERIENCE 1999 to present. President, 703.684.4886 (o) SHARF & Associates, Employment Risk Advisors, Inc. 703.838.5574 (0 211 North Union Street, Suite 100, Alexandria, VA 22314 jim jimsharfcom Jim Sharf (1) advises employment attorneys, HR managers and fellow industrial psychologists on developing, implementing and defending selection and appraisal systems that minimize the risk of employment litigation, and (2) provides expert testimony regarding: Age discrimination challenges to downsizing and reengineering decisions involving corrective actions and terminations; Age, race and. gender discrimination challenges to employment testing, assessment, and performance appraisal systems; Disparate impact claims challenging minimum education & experience requirements and employment tests; Disparate treatment claims challenging subjective employment decisions,' unstructured interviews and performance appraisals; Go* & timetables used in managing diversity under the Dept. of Labor's Office of Federal Contract Compliance Programs' Executive Order 11246. 1994 to 1998: Vice President, Human Resources Consulting Group, Aon Consulting, Washington, DC Headed EEO compliance, risk management, and litigation support services within the Human Resources Consulting Group. Provided expert testimony and litigation support in both public and private sectors. Developed and defend job -related selection, appraisal and downsizing procedures against: EEOC's enforcement of Title VII of the Civil Rights Act of 1964, the Age Discrimination in Employment Act, the Americans with Disabilities Act; and the Office of Federal Contract Compliance Programs' enforcement of Executive Order 11246. 1991 to 1993: Special Assistant to the Chairman, U.S. Equal Employment Opportunity Commission, Washington, DC Advised Chairman on the principles & practices of industrial psychology governing employee selection & appraisal procedures, and the precedent of Title VII employment testing case law in relation to the EEOC's 1978 Uniform Guidelines on Employee Selection Procedures. Drafted "race norming" prohibition in the Civil Rights Act of 1991. 1985 to 1991: Personnel Research Psychologist, U.S. Office of Personnel Management, Washington, DC Developed job -related selection procedures for Federal civil service. Represented OPM on the National Academy of Science study of validity generalization for the U.S. Employment Service's General Aptitude Test Battery. Received Director's Award for developing `biodata" exam used to screen entry-level professional and career candidates nation-wide. 1978 t9 1984: Vice President, Richardson, Bellows, Henry & Co., Inc., Washington, DC Designed and conducted multi -employer validation studies utilizing objective background & experience biodata tests; Provided expert testimony in Title VII litigation. 15 1974 - 1978: Chief Psychologist U.S. Equal Employment Opportunity Commission, Washington, DC Assisted EEOC attorneys as expert witness in Title VII litigation; Represented EEOC in drafting 1978 Uniform Guidelines on Employee Selection Procedures. Spring 1977: Team Leader, President's Reorganization Project Executive Office of The President, Washington, DC Led team which analyzed management decision -making structure and redefined unit responsibilities at the Office of Management and Budget in the Carter Administration. 1986 — 1994; Associate Professorial Lecturer, The George Washington University, Washington, DC Taught employment testing and assessment courses for both PhD students in the Department of Psychology and MS degree students in the Administrative Sciences Program. 1970 - 1977: Assistant & Adjunct Professor, College of Business Administration The American University, Washington, D.C. Taught in undergraduate, MBA and PhD degree programs; Directed doctoral dissertations. Served on College of Business Rank & Tenure Committee. 1971 - 1974: Senior Associate Leadership Resources, Inc., Washington, D.C. Developed, implemented and evaluated comprehensive Management by Objectives (MBO) work planning & appraisal system for the National Aeronautics and Space Administration. 1969 - 1970; Campus Manager, A.C. Nielsen, Knoxville, TN. Supervised market research data collection evaluating the effectiveness of corporate campus recruiting programs. 1967 - 1970: Assistant to Director, Career Planning & Placement Office The University of Tennessee, Knoxville, TN. Counseled undergraduate and graduate students in career planning; Coached students on improving interviewing skills; Provided feedback to corporate college relations programs. EDUCATION 1970: PhD in Organizational Psychology, University of Tennessee, Knoxville, TN. 1965: BS in Chemistry, Dickinson College, Carlisle, PA, PROFESSIONAL ORGANIZATIONS American Psychological Association Elected to Board of Professional Affairs,1986; elected Fellow, 1995, Association of State and Provincial Psychology Boards Member of exam committee, 1989-93. Wrote test items for professional exam required for licensure as a psychologist in every state in the U.S. and all provinces in Canada. Society for Industrial and Orgnizational Psychology Chairman, Ad hoc Committee on Legal Issues, 1978-79; Membership Chairman, 1983-85; 16 Professional Affairs Committee, 1975-76, 1985-87; awarded the M. Scott Myers Award for Applied Research in the Workplace it 2006 based on developing TSA selection tests used to hire fifty-five thousand airport security screeners in 2002; elected Fellow, 1995. EDITORIAL BOARDS Personnel Psychology, 1987-88; The Industrial/Organizational Psychologist, 1977-1998. BIBLIOGRAPHY 1974 Behavioral dimensions of the job interview. The Journal of College Placement, Spring 1974, 34(3), 17-22. 1975 A psychologist's guide to Title VII legal language. The IndustriaVOrganizational Psychologist, 1975, 13(1), 30-31; also reprinted in The Training and Deyelopment Journal, February 1977, 31(2), 20. Bibliography on personnel testing. Daily Labor Report, 1975, 66 (April 4), 18-22. Supreme Court to examine criterion adequacy and job analysis requirement in Moody v. Albemarle Paper Co. The Industrial/Organizational Psychologist, 1975, 12(3), 30-33. 1976 "Do's and Don'ts in the Employment Process: The Current Status of Testing," 1 Employee Relations Law Journal 579-589 (1976); also reprinted in T. Siegel (Ed . ), Equal Employment Opportunity Law Reader 1976. New York: Executive Enterprise -Publications Co., Inc. 1976. Bibliography of FEP testing. Daily Labor Report, 1976, 195 (October 6), Fl -F7 . Supreme Court decides Washington v. Davis involving training criteria. The American Society for Training and Development Journal, 2(8), June 1, 1976. The influence of lawyers, legal language and legal thinking. Personnel Psychology, 29(4), 541-549, Winter 1976. 1977 A lawyer's guide to testing practices and procedures. In K. McGovern (Ed.), An Equal Employment Opportunity Practice. Guide. Washington, D.C.: Subcommittee on Publications of the Committee on Equal Employment and Collective Bargaining of the Council on Labor Laws and Labor Relations, Federal Bar Association, 1977. Facing reverse discrimination. The American Psychological Association "Monitor," Mar. 1977, 8(3), 6, 21. Fair employment implications for HRD: The case of Washington v. Davis. The American Society for Training and Development Journal, February 1977, ?j. (2), 16-21 . Second-guessing the legal fair employment system. The American Psychological Association "Monitor," April 1977, 8(4), 4. Statistics of prima facie case examined in Hazelwood School District v. United States. The Industrial/Organizational Psychologist, 1977, J5(I), 15-16. 17 1978 The impact of litigation on the personnel manager of the future. In Schneier (Ed.), Personnel Administration Today. New York: Addison-Wesley, 1978. The prima facie case: Keeping Title VII honest. In K. McGovern (Ed.), An Equal employment Opportunity Practice Guide. Washington, D.C.: Subcommittee on Publications of the Committee on Equal Employment and Collective Bargaining of the Council on Labor Laws and Labor Relations, Federal Bar Association, 1978. 1979 EEO issues: Proposed Federal "Truth in Testing" bills. The Industrial/Organizational Psychologist, 1979, 17(1), 25-26. The impact of competency assessment on minorities: A Title VII perspective. In J. Towslee-Collier (Ed.), Issues in Competency -Based Education; Conference Proceedings. Atlanta, GA: Georgia State University, 1979. Uniform Guidelines: Competence or numbers? In J. Erickson & K. McGovern (Eds.), Equal. Employment Opportunity Practice Guide, Vol. 1, Washington, D.C.; Subcommittee on Publications of the Committee on Equal Employment and Collective Bargaining of the Council on Labor Laws and Labot Relations, Federal Bar Association, March 1979. �980 Content validity: Whither thou goest? The Industrial/Organizational Psychologist, 1980, 17(3), 8-14. Court of Appeals rejects EEOC's claim that everyone in labor market is equally qualified for all. jobs (...and but for discrimination, the workforce would reflect parity with the labor market). The Industrial/Organizational Psychologist, 1980, 17(3), 60-61 Landmark appeals decision on content validity: Guardians Assn. of New York City Police Department v. Civil Service Commission of the City of New York. The International Personnel Management Association Assessment Council News, Winter 1980,A(4), 1-3. Validity generalization: Round one (Pegues v. Mississippi State Employment Service). The Industfial/Organizational Psychologist, 1980, 17(4), 36-38. 1981 "Business necessity" clarified and "bottom line" confounded in two Appeals decisions: Contreras v. City of Los Angeles, & Teal v. State of Connecticut. The IndustrialOrganizational Psychologist, 1981, 18(4), 41-42. Good validity work and good legal defense withstand challenge: Cornierier v. PPG. The Industrial/Organizational Psychologist, 1981, 19(1), 9-11. Landmark Appeals decision on content validity: Guardians Association of New York City Police Department v. Civil Service Commission of City of New York. The Industrial/Organizational Psychologist, 1981, 18(2), 21-23. Unanimous Supreme Court reaffirms legal burdens in individual case under Title VII: Texas Department of Community Affairs v. Burdine. The Industrial/organizational Psychologist, 1981, 18(4), 41-42. 18 With Ad Hoc Group on Uniform Guidelines: A Professional and Legal Analysis of the Uniform Guidelines on Employee Selection Procesiures. In V. Day, F. Erwin & A. Koral (Eds.) Alexandria, VA: The American Society for Personnel Administration, 1981. 1982 "Bottom line" before Supreme Court: State of Connecticut v. Teal. The Industrial/Organizational Psychologist, 1982, 19(2), 19-21. National Research Council of the National Academy of Sciences report: Ability Testing: Uses, Consequences and Controversies. The Industrial/ Organizational Psychologist, 1982, 19(3), 34-35. Personnel testing and the law. In J. Ferris & K. Rowland (Eds.), Personnel Man4gement: New Perspectives. Boston, MA: Allyn and Bacon, 1982. Supreme Court Teal decision: Good "bottom line" does not preclude a prima facie case nor does it provide a defense to a prima facie case. The Industria.1/9rganizational Psychologist, 1982, 24(2), 33-36. 1983 Reagan administration underscores opposition to affirmative action quotas. The Industrial/Organizational Psychologist,1983, 2Q(4), 61-63. The new Joint Technical Standards: Legal problems for clinical practice? The Industrial/Organizational Psychologist, 1983, 20(2), 56-57. 1984 Department of Justice asks Supreme Court to review Constitutionality of voluntarily adopted racial quota in Detroit police promotion case. The Industrial/Organizational Psychologist, 1984, 21(2), 53-58. Office of Technology Assessment reviews polygraph validity evidence. The Industrial/Organizational Psychologist, 1984, 21(3), 53-56. - Relief limited to individuals, not merely members of affected class, says Supreme Court in Memphis firefighters decision. The Industrial/Organizational Psychologist, 1984, 21(4), 62-5. 1985 Segar v. Smith: Employers now burdened with defending the "business necessity" (validity?) of the total personnel decision making system! The Industrial/Organizational Psychologist, 1985, 23(1), 29-34. 1986 The Uniform Guideliines' paradox persists. New Perspectives, The U.S, Commission on Civil Rights, Winter/Spring 1986, 1(1), 53. With Ad Hoc Committee on Employee Selection. Employee Selection: Legal and Practical Alternatives to Compliance and Litigation, Second Ed. In E. Potter (Ed.), Washington, D.C.: National Foundation for the Study of Equal Employment Policy, 1986. 1987 Industrial/Organizational psychology: Today's critical issues. Board of Professional Affairs Symposium, annual meeting of the American Psychological Association, 1985. (ERIC 19 Document Reproduction Service No. ED 263 467); also reprinted in The Professional Practice of Psychology, 7(1), January, 1987. Lower court decisions on the legality of preferential treatment and amicus positions, pro & con, before the Supreme Court in Johnson v, Transportation Agency, Santa Clara County. The IndustriaiIOrganizational Psychologist, 1987, 24(2), 68-73. Validity generalization. The Industrial/Organirational Psychologist, 1987, 24(3), 48-50. 1988 Gottfredson, L. & Sharf (guest Eds.). Fairness in employment testing: Litigating personnel measurement policy. Journal of Vocational Behavior, 33(3), 235-271. 1989 Gandy,J., A.Outerbridge, J.Sharf & D.Dye. Development and Initial Validation of the Individual Achievement Record. Washington, DC: U.S. Office of Personnel Management. 1991 Employment testing and the law in the United States. In J. Jones, B. Steffy, and D. Bray (Eds.), Applying Psychology in Business. Lexington, MA: Heath & Co., 1991. 199Z Legal and EEO issues impacting on personal history inquiries. In G. Stokes, M. Mumford & W. Owens (Eds.), The Biodata Handbook: Theo _, Research & Application. Palo Alto, CA: Consulting Psychologists Press, Inc., 1992. 1997 The Clinton administration's opposition to the California Civil Rights Initiative argues for affirmative action status quo. The Industrial/Organizational Psychologist, 1997, 34(4), 57-68. Courts limit use of cutoff scores -in -employment testing. Employment Testing Law & Policy Reporter, June 1997, 6(6), 89-93, 103. - 1998 U.S. Department of Justice's Supreme Court amicus brief in Taxman flip flops on affirmative action but favors diversity. The Industrial/Organizational Psychologist, 1998, 25(3), 34-50. Shari, J. and Wolf, P. University of Michigan's affirmative action case: A good bet for the Supreme Court. The Industrial/ Organizational Psychologist, 35(4), April 1998, p.85-88; also reprinted in Personnel Testing Council of Metropolitan Washington Newsletter, XXI(7), July 1998, p.3-5. Use of contingent workers: What you should know. The Industriall-Organizational Psychologist, 1998, 36(2), 30-32. 1999 Testing as an employment risk strategy. Employee Rights and Employment Policy Journal, 3(1), 133-145. President Clinton proposes government "solution" for "gender pay gap." The Industrial/Organizational Psychologist, 1999, 36(4), 59-63. Valid employment testing lowers litigation risks. The California Psychologist, October 32(10), 22-25. 20 Third Circuit's Lanning v. SEPTA Decision: `Business necessity' requires setting minimum standards. The Industrial/Organizational Psychologist, 37(2), 138-149. 2000 Sharf, J. and Jones, D. "Managing Employment Risk." In Kehoe, J. (ed). Managing Selection Strategies in Changing Organizations. Society for Industrial and Organizational Psychology Professional Practice Series. San Francisco, CA: Jossey-Bass, 1999. 2003 Lannin revisited: The Third Circuit again rejects relative merit. The Industrial/Organizational Psychologist, 40(3), 31-40. University of Michigan's admissions challenged by Bush Justice Department. The Industrial/Organizational Psychologist, 40(4), 49-58. 2007 Borden, L. and Shari J. Developing legally defensible content valid selection procedures. In D. Whetzel & G. R Wheaton (Eds.), Applied Measurement: Industrial Psychology in Human Resources Management. Mahwah, NJ: Lawrence Erlbaum, 2007. "Protecting Intellectual Property." In Borman, W. (ed). The 1/0 Consultant's Handbook. Washington, DC: American Psychological Association Books (in press). Slippery Slope of "Alternatives" Altering Topography of Testing? The Industrial/Qrganimtional Psychologist (in press). PROFESSIONAL ASSOCIATION SEMINARS Conducted three-day seminar series in 14 cities nation-wide entitled "Federal Guidelines on Employee Selection Procedures, Interpretation and Application" for the American Society for Personnel Administration (now Society for Human Resource Management), 1978-79; Conducted "Developing Sound Employee Selection Procedures: A Detailed Look at Job Analysis and the Requirements of the Uniform Guidelines on Employee Selection Procedures" for the Equal Employment Advisory Council; January'82; Conducted three-day seminar series in 5 cities nation-wide "Appraising Performance and Productivity" for the International Personnel Management Association., 1983. "EEO Legal Developments" talks to industrial psychologists at Personnel Testing Councils in New York (2x), Washington (5x), Chicago, St. Louis, Houston, Los Angeles (5x), and San Francisco (2x): 1977 — 2006. PROFESSIONAL SOCIETY WORKSHOPS (continuing education credit to participants) Designed, Produced and Conducted: "The Tightening Legal Strangle -hold on Employee Selection Procedures" for the Society of Industrial / Organizational Psychology Workshop at the Annual Convention of the American Psychological Association, Aug. "76; "Uniform Guidelines: Basic Issues and Practical Implications" for the Society of Industrial / Organizational Psychology Workshop at the Annual Convention of the American Psychological Association, Aug. "79; 21 "EEO/Fair Employment Update" with private sector attorney for the Society of Industrial / Organizational Psychology Workshop at the Annual Convention of the American Psychological Association, Aug. '82; "EEO Update" with private sector attorney at the Society of Industrial / Organizational Psychology Workshop at Annual Convention of the American Psychological Association, Aug. '87; and "EEO: Legal and Regulatory Developments" workshop with private sector attorney at Annual Meeting of the Society of Industrial / Organizational Psychology, April '96. "Defending Minimum Qualifications: The e-Applicant" with private sector attorneys at Annual Meeting of the Society of Industrial / Organizational Psychology, May '06. UNIVERSITY LECTURES ON FAIR EMPLOYMENT LAW PUBLIC POLICY: Duke University George Washington University Pennsylvania State University Stetson University Law School University of Chicago University of Houston Univ. Louisville School of Law SECURITY CLEARANCE TOP SECRET Univ. Maryland Univ. North Carolina Law School USC School of Public Admin. Univ. Tennessee Univ. Texas School of Public Admin. Univ. Wisconsin -Law School Vanderbilt University Law School CIVIC Chairman, Waterfront Committee, Old Town Civic Association, Alexandria, VA Coach, Naval Academy Sailing Squadron, U.S. Naval Academy, Annapolis, MD LITIGATION SUPPORT & EXPERT TESTIMONY (partial) Attorney General, State of California San Diego (Caltrans) Attorney General, State of Washington Seattle (Washington Highway Patrol) Attorney General, State of New York New York (NYC Board of Education & State Education Deptartment) Baker Donelson Birmingham (State of Alabama Dept. of Transportation) Center for Individual Rights State of Washington (University of Washington Law School) New York (NYC Board of Education: intervenors) Eastman & Smith Toledo OH (DaimlerChrysler) 22 Ellzey & Brooks Columbia SC (Willamette Industries, Inc.) Fisher & Phillips New Orleans (Weyerhaeuser) Foley & Lardner Detroit (Kmart) Friday Eldridge Little Rock (EN 1'F,RGY) Garvey Schubert Washington (FBI, plaintiffs) Hogan & Hartson Washington (Secret Servie; plaintiffs) Homes Roberts Denver (Martin Marietta) Jones Day Cleveland and DC (American General, GM, Bridgestone -Firestone, Westinghouse) Kaye Scholer New York (Texaco) Kienbaum, Opperwall Detroit (FORD) Kilpatrick & Cody Atlanta (Krnart) Latham & Watkins Los Angeles (City of Garland) Lawyers' Committee for Civil Rights Under Law Washington (U.S. Civil Service Commission (plaintiffs) LeBoeuf Lamb Pittsburgh (Alcoa) Littler Mendelson Pittsburgh (PPG Industries) McCalla Thompson New Orleans -(PPG, Louisiana State Police, Willamette Industries) McGuire Woods Richmond (James River Corp.) Miller Canfield Detroit and- Washington (Detroit Edison; Lockheed -Martin) Morgan Lewis Philadelphia (Allied Signal, CONRAIL) Ogletree Deakins Washington (NISSAN); Morristown (Timken) Semmes Bowen & Semmes Baltimore (MBNA, Thomson Consumer Electronics) Seyfarth Shaw Chicago and San Francisco (Borg-Warner) Sidley & Austin Chicago (Norfolk Southern) Snitow & Cuningham New York (Nassau County Police) Steptoe & Johnson Washington (Burlington Northern) Thelen Reid Washington (Miero-Strategy) U.S. Department of Justice Washington (State of New York, plaintiffs) Venable, Baetjer and -Howard- Baltimore and -Washington (MBNA, Mirage Resorts) Woods & Aitken Lincoln (Ritz Carlton) Wilmer, Cutler & Pickering Washington (Educational -Testing Service, Fannie Mae) Wilson Sonsimi Palo Alto (Home Depot) CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Transportation Security Screener Validation Study Report Uniform Guidelines on Employee Selection Procedures, Sec.1607.15(B). DOCUMENTATION: Criterion -related validity studies 1) User(s), locations, and date(s) of study.' Dates and Location of Job Analysis The study of transportation security screeners was conducted at the Baltimore Washington International Airport (BWI) in February 2002, utilizing the U.S. Department of Labor's National Skill Standards Board (NSSB) job analysis methodology. The NSSB job analysis methodology was instrumental in first determining -what the newly federalized transportation security screener needed to do to perform competently and second what knowledge and skills enabled a screener to carry out these duties. Dates and Locations of Employment Testing The transportation security screener selection procedures reviewed in this report were assembled in February and March 2002, based on the NSSB job analysis information developed at BWI. Vacancy announcements for the screener job were first posted in early March and the first applicant testing was piloted in mid -March in Los Angeles, Memphis and Chicago. The results of the pilot test battery were analyzed in terms of both variance and average score differences (adverse impact) for classes covered by the Civil Rights Act of 1964. By April, tests to be included in the final test battery were specified. Over the following eight months, over 1.8 million applicants were screened and 340,000 tested at 84 locations nationwide.' By the end_ the calendar year 2002, almost sixty thousand federal -transportation security scrccncrs had been hired based on the selection procedures described in this report. - 2) Problem and setting!' GAO Describes Circumstances in Which The Validation Study Was Conducted Almost 30 years ago, the FAA mandated the screening of passengers and baggage with the objective of deterring hijackers. Approximately 15,000 private -sector screeners were employed in the U.S. prior to 9/1-1. In 1995, it is estimated they -screened approximately 1.3 billion persons at some 700 screening checkpoints.2 In 1978, screeners were not ' Kolmstetter, E. (April 2003). I-O's making an impact: TSA Transportation Security Screener Skill Standards, Selection System, and hiring process. The Industrial - Organizational Psychologist, 40(4), 39-46. 2 National Academy of Science (1999) Assessment of Technologies Deployed to Improve Aviation Security: First reportff999). Commission on -Engineering and Technical Systems. Washington, DC: National Academy Press, p.20. 23 CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study detecting 13% of potentially dangerous objects FAA agents carried through checkpoints. In 1987, screeners were not detecting 20%.3 According to the GAO: "Test data for the 1991 to 1999 period show that the declining trend in detection rates continues. Information on FAA test results (for the 1991- 99 period) is now designated as sensitive security information and cannot be publicly released. Furthermore, the recent tests show that as tests become more realistic and more closely approximate how a terrorist might attempt to penetrate a checkpoint, screeners' ability to detect dangerous objects declines even further" (emphasis added).4 "In a high percentage of our tests involving explosives detection equipment, threat items were not identified or looked for by the equipment operators....we found operators who could not properly operate the equipment to resolve a threat."5 "(A)ccording to FAA research officials, screeners are placed on the job who do not have the necessnry knowledge, skills, or abilities to perform the work effectively and who then find the duties tedious and unstimulating. The President's Commission on Aviation Security and Terrorism and the White House Commission on Aviation Safety and Security both concluded that better selection, training, and testing of screeners could improve their performance. The 1999 National Research Council's report assessing explosive detection equipment also found that FAA needed to improve the training, testing, and qualification procedures for the screeners operating the equipment."6 "FAA's regulations establish minimum employment and training standards for screeners. Among other things, these standards specify that screeners possess basic aptitudes and physical abilities, including color perception, visual and aural acuity, physical coordination, and motor skills; know how to read, write, and speak English; and complete 12 hours 3 U.S. General Accounting Office (April 6, 2000). Aviation Security; Vulnerabilities Still Exist in the Aviation Security System, Testimony before the Subcommittee on Aviation, Committee on Commerce, Science, and Transportation, U.S. Senate. GAO/T- RCED/AIMD-00-142, p.7. 4 U.S. General Accounting Office (Sept. 25, 2001). Aviation Security: Vulnerabilities in, and Alternatives for, Preboard Screening Security Operations, Testimony before the Committee on Governmental Affairs and its Subcommittee on Oversight of Governmental Management, Restructuring and the District of Columbia, U.S. Senate, GAO-01-1171 T, p.5. 5 Office of Inspector General, Aviation Security, Federal Aviation Administration, U.S. Department of Transportation. Report #AV-1999-068, March 24, 1999, p.7. 6 U.S. General Accounting Office. Aviation Security: Long -Standing Problems Impair Airport Screeners' Performance, Report to Congressional Requesters, GAO/RCED-00- 75, June 2000, p.26. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study of classroom training and 40 hours of on-the-job training...Currently, almost 100 security companies, employing an estimated 18,000 screeners, are operating at U.S. airports."7 "Screening duties require repetitive tasks as well as intense monitoring for the very rare event when a dangerous object might be observed. Too little attention has been given to factors such as (1) improving individuals' aptitudes for effectively performing screening duties, (2) the sufficiency of the training provided to screeners and how well they comprehend it, and (3) the monotony of the job and the distractions that reduce screeners' vigilance. As a result, screeners are being placed on the job who do not have the necessary aptitudes, or sufficient knowledge to perform the work effectively, and who then find the duties tedious and dull."8 Congress Passes and President Signs Aviation and Transportation Security Act In response to the events of 9/11, 2001, on November 19, 2001, the Aviation and Transportation Security Act (ATSA) (P.L. 107-71) was signed by President Bush with the objective of federalizing aviation security at 429 commercial airports. The Transportation Security Administration (TSA) was created to carry out the Aviation and Transportation Security Act (Act). Section 111 of the Act addressed "Training and Employment of Security Screening Personnel" and specified the minimum qualifications required of security screening personnel_ Congress directed TSA to staff security personnel at commercial airports nationwide by Fall 2002. Purpose of Validation Study In January 2002, TSA had fewer than 50 full-time employees most of whom had transferred from the Federal Aviation Administration (FAA) and other federal law enforcement agencies. By the end of calendar 2002, TSA had over 1,000 employees at their Washington DC HQ and almost 60,000 transportation security screeners at 429 airports nationwide. TSA's immediate priority that January was to establish valid, legally defensible qualification standard for deciding who would become a federal transportation security screener. Instrumental in addressing that goal was the NSSB job analysis the objective of which was to determine what duties and tasks were to be performed by screeners in a job that in the early months of 2002 had yet to be defined in other than general salary and benefit terms. U.S. General Accounting Office. Aviation Security: Long -Standing Problems Impair Airport Screeners' Performance, Report to Congressional Requesters, GAO/RCED-00- 75, June 2000, p.18. 8 U.S. General Accounting Office. Aviation Security: Vulnerabilities in, and Alternatives for, Preboard Screening Security Operations, Testimony before the Committee on Governmental Affairs and its Subcommittee on Oversight of Governmental Management, Restructuring and the District of Columbia, U.S. Senate, GAO-01-1171 T, Sept. 25, 2001, p.5. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Existing Job Analyses and Employment Standards for Security Screeners In early February, 2002, one of the private security vendors at BWI agreed to participate with TSA in conducting the critical job analysis. As was typical of commercial airports nationwide, at BWI the largest airline carrier at each of the five concourses contracted with a private -sector, lowest -bid vendor who staffed checkpoints and checked passengers and carry -on bags. Each of the five BWI concourses had a different vendor supplying minimum -wage screeners. When the Act was signed into law, there were neither job analyses nor formal, standardized employment standards used by any of the five private - sector vendors staffing BWI passenger checkpoints. A check with security vendors at other airports found the same lack of employment standards. The BWI NSSB job analysis identified critical skills and abilities which informed the assembling of three hours of valid employment tests measuring those same critical skills and abilities. The objective of this report is to document in conformance with EEOC's 1978 Uniform Guidelines on Employee Selection Procedures the validity of those skill and ability tests used from March thru December 2002, to select transportation security screeners, Employment Standards Required by the Aviation and Transportation Security Act The Act was unusual in detailing the minimum qualifications required of transportation security screeners who would now be hired as federal employees. The Act also delegated authority to the Under Secretary of the Transportation Security Administration to establish other qualifications required for employment. The basic employment standards required by Section 111 of the Aviation and Transportation Security Act (ATSA) (P.L. 107-71) are the following: ■ To have a satisfactory or better score on a Federal security screening personnel selection examination • To be a U.S. citizen ■ To possess a high school diploma, general equivalency diploma or experience that the Under Secretary has determined to be sufficient for the individuals to perform the duties of the position ■ Basic aptitudes and physical abilities, including color perception, visual and aural acuity, physical coordination, and motor skills including being able to: Distinguish on screening equipment monitor the appropriate imaging standard Distinguish each color displayed on every type of screening equipment and explain what each color signifies Hear and respond to the spoken voice and to audible alarms in an active checkpoint environment Perform physical searches by efficiently and thoroughly manipulating and handling baggage, containers, and other objects CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Perform pat -downs or handheld metal detector searches of individuals with sufficient dexterity and capability to thoroughly conduct the procedures over an individnal's entire body • Proficiency in reading, speaking, and writing in English including being able to: Carry out written and oral instructions regarding proper performance of screening duties Read English language identification media, credentials, airline tickets, and labels on items normally encountered in screening process Provide direction to and understand and answer questions from English- speaking individwils undergoing screening Write incident reports and statements and log entries into security records in the English language ▪ To have the ability to demonstrate daily a fitness for duty without impairment due to illegal drugs, sleep deprivation, medication, or alcohol ■ To successfully pass an employment investigation background check (including a criminal history record check) ■ Tv not pose a national security risk or threat • To satisfactorily complete all initial, recurrent, and appropriate specialized training required by the security program. 3) Job analysis or review of job information."' Existing Job Analyses Most job analysis methodologies presume that there is an extant job with both job descriptions and incumbents who can be observed and surveyed. Although the several X- ray baggage -screening technologies used at all checkpoints nationwide were required to have been approved by the FAA prior to 9/11, for the existing private -sector screener job there were no standard job descriptions, no minimum training requirements and no skills certification required of either applicants or incumbents. At this stage, an extensive literature search was conducted focusing on security job analyses, job descriptions and valid selection procedures in FAA and aviation databases worldwide. It was found that FAA had developed a valid simulation of the X-ray monitoring tasks common to both the former, private -sector screener and the new, federalized transportation security screener jobs:9 9 Neiderman & Forbes, Computer-Baased Training Performance Reports, DOT/FAA/AR- 98/14, March 1998)." Development and Validation Plan for Screener On -the -Job Training Assessment, DOT/FAA/AR-98/49, Sept. 1998, p.1. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study "According to Federal Aviation regulation (FAR) ¶108.17, Use of X-ray Systems, there shall be a program for initial and recurrent training of operators of X-ray systems. This includes training in efficient use of X- ray systems, and identification of weapons and other dangerous articles. Section XIII of the Air Carrier Standard Security Program (ACSSP) presents the standards for training and testing of persons performing screening and security functions. For many years, the only FAA -approved training was developed by the Air Transport Association. Their 12-hour initial screener training program includes 40 multiple-choice questions and 40 X-ray images to assess mastery prior to On -the -Job Training (OJT). In April 1997, the FAA also approved the use of a Computer -Based Training (CBT) system for initial training prior to OJT. The Safe Passage International,Limited, CBT system has a library of test questions and the trainee is presented with unit tests, a 50-item content mastery final exam, and a 50-item threat image interpretation final exam to assess mastery. Both approaches stress knowledge acquisition whose mastery will be assessed as part of another effort," Although this valid X-ray test had been available to private -sector security vendors prior to 9/11, it had not been implemented because of labor market pressures. To be competitive as the lowest -bid vendor, private -sector vendors could not afford to be selective when hiring minimum wage employees. National Skill Standard Board Job Analysis Methodology Used Generally accepted principles and practices of industrial psychology recognize job analysis as the appropriate first -step data -collection tool -to- develop information describing the requirements of jobs. Neither the generally accepted principles and practices of industrial psychology nor EEOC's Uniform Guidelines proscribe any one specific job analysis methodology. t° 11 1° Uniform Guidelines, Sec. 14(A). Validity studies should be based on review of information about the job. "Any validity study should be based upon a review of information about the job for which the selection procedure is to be used. ... Any method of job analysis may be used if it provides the information required for the specific validation strategy used." 11 Society for Industrial and Organizational Psychology, Inc. (1987). Principles for the Validation and Use of Personnel Selection Procedures. "In conducting job analyses, the important consideration is understanding the organization's needs as they relate to the selection problem so that the researcher can formulate sound hypotheses about relationships among predictors and criteria. A systematic job analysis should provide the investigator with information to develop relevant criteria and make good judgments about predictors. However, a less detailed job analysis may be all that is required because there is so much previous job analytic work on the occupation of interest or because past research on the job allows the generation of sound hypotheses concerning predictors and criteria can be developed with little reference to a specific job analysis in a particular CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study The BWI study used the job analysis methodology developed by the U.S, Department of Labor's National Skill Standards Board (NSSB). The NSSB job analysis defines skill standards in terms of two parts: work- and worker -oriented components.12 Work -oriented Skill Standards The work -oriented job analysis component is used to describe what the security screener needs to do on the job and what the screener needs to do in order to perform competently. Three elements make up the work -oriented component of a skill standard: 1. Critical Work Function (CWF) — The major responsibilities of a security screener. 2. Key Activity (KA) — The major duties or tasks used by a security screener in carrying out a critical work function. 3. Performance Indicators (PI) — Information on when a security screener is performing each key activity competently. Upon describing the work of the security screener, the next job analysis step identified the competencies associated with the screeners and linked those competencies to the work requirements. Worker -oriented Skill Standards The worker -oriented job analysis component described the knowledge and skills a security screener needs to posses in order to perform competently. The security screener's knowledge and skills are broken down into three categories: 1. Academic— Knowledge and skills associated with the academic disciplines of reading, writing, mathematics, and science. 2. Employability- Applied knowledge and skills used to perform effectively across a broad range of occupations, such as listening, speaking and decision making. 3. Occupational and Technical — The specific technical and occupational knowledge and skills needed for security screener work, such as skill in using an x-ray machine, knowledge of security protocols, or knowledge security procedures. Once the security screener's overall skills and knowledge were defined and linked with the work requirements, an additional step was taken to define further the academic and employability knowledge and skills in terms of the complexity level for each element. An invited team of 15 industrial psychologists with job analysis expertise took part in the organization. When a systematic new job analysis is not completed, the researcher should compile reasonable evidence which establishes that the jobs in question are similar in terms of work behavior and/or required knowledge, skills, and abilities." 12 National Skill Standards Board (2000), Built to Work: A Common Framework for Skill Standards & Skill Scales Companion Guide. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study "leveling" exercise which required estimating the complexity and criticality of worker - oriented competencies taking into consideration the work -oriented critical work functions. The resulting NSSB transportation security screener job analysis information identified how critical a skill is to the work and how much skill is needed to perform work competently. Job Analysis Findings Appendix A reports the job analysis information describing transportation security screeners in terms of five critical work functions each of which is described in terms of several key activities (KS) and performance indicators (PI) describing competent performance of each of the key activities. Appendix A reveals that there were 23 key activities defining the five critical work functions and a total of 119 performance indicators defining the key activities. Of this NSSB job analysis data, the most significant information identified the five following critical work functions: 1. Control Entry and Exit Points 2. Perform Routine Security Screening of Persons 3. Perform Security Screening of Property and Baggage 4. Continuously Improve Security Screening Processes 5. Continuously Improve Own Performance The team of industrial psychologists then used the NSSB methodology to identify the critical knowledge and skills and to link them to the five critical work functions identified above. The critical knowledge and sklLs included (see Appendix B for criticality rating): Visual observation, English (speaking, listening, reading, writing), Maintaining command posture, Stress tolerance, Ability to learn, Using social skills (customer service skills), Making decisions and judgments, Gathering and analyzing information, Self and career development, Using information and communications technology, Analyzing and solving problems, and Adaptability. It was the ranking of critical skills in Appendix B that informed the assembling of tests shown to be valid measures of these critical skills. 4) Jab titles and codes." CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study There is no O*NET Code for Transportation Security Screeners. The closest job description is that of Guards and Watch Guards (63047) i3 which is relaxed to Police Patrol Officer (63014A). The Dictionary of Occupational Titles has no occupational code for Transportation Security Screeners. The closest is that of Surveillance -system Monitor (379.367-010)14 which is related to Public Safety Officer (379.263-014). 5) Criterion measures: Criterion measures were not developed for this validation study. 6) Sample description."' 7) Description of selection procedures.v° The selection procedure for transportation security screeners is a multiple -hurdle consisting of three phases which must be passed in order. Potential applicants were informed of the availability of transportation security screener jobs with the announcement found in Appendix C. Phase I: On-line Minimum Qualification Pre-screen The first step required of an applicant was completion of an on-line "quick -screen" questionnaire to establish that the interested person met minimum qualifications stipulated by Congress in the Act including: U.S. citizenship, and High school (or GED) diploma OR one year of relevant experience. Additional questions dealt with: 13 Stand guard at entrance gate or walk about premises of business or industrial establishment to prevent theft, violence, or infractions of rules. Guard property against fire, theft, vandalism, and illegal entry. Direct patrons or employees and answer questions relative to services of establishment. Control traffic to and from buildings and grounds, Include workers who perform these functions using a car patrol. 14 Monitors premises of public transportation terminals to detect crimes or disturbances, using closed circuit television monitors, and notified authorities by telephone of need for corrective actidn: Observes television screens that transmit in sequence views of transportation facility sites. Pushes hold button to maintain -surveillance of location where incident is developing, and telephones police or other designated agency to notify authorities of location of disruptive activity. Adjusts monitor controls when required to improve reception, and notifies repair service of equipment malfunctions. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Conditions of employment such as availability for full-time, shift, overtime and extended hours; Related work history, and Military service (veteran's preference). Those meeting minimum qualifications were bona fide applicants who were then invited to the next phase of assessment based on several factors including: veteran's preference awarded by statute, geographical (i.e., candidate's choice of a specific airport) and shift - work preferences designated by the applicant. Phase II: Computer -administered Test Battery Although a number of tests were administered for research pruposes, this report deals only with the scored scales of the computer -administered test battery which comprised two modules: a competency inventory and a screener aptitude test. Competency Inventory Service Orientation Interpersonal Sensitivity Service Attitude Readiness Drug Avoid:3nce Integrity / Honesty Stress Tolerance Applied Thinking Communication / Listening Reading Writing Screener Object Recognition Test (SORT) The SORT (based on signal detection theory) assesses an applicant's aptitude for visually identifying objects embedded in other objects in terms of X-ray images of objects in suitcases. SORT requires the following skills: visual observation, vigilance, attention to detail, concentration, mental focus, visual and mental rotation, discerning hidden figures and discerning hidden patterns. Phase III: Structured interview Following the structured interview, a conditional offer of employment is made (or not). Phase IV; Physical / Medical / Drug / Background Physical Abilities test Baggage Lift Baggage Search Medical Examination Drug Tesl Urinalysis Background Investigation Credit CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Criminal Phase IV: Meeting with HR Representative A conditional hire letter is issued (pending satisfactory drug and background checks). A salary offer is made, questions are answered and forms are checked for completion. 8) Techniques and results."`a Method Used in Analyzing Data The research design supporting TSA's use of tests for transportation security screeners is that of transporting and generalizing previously developed validity evidence. Appendix D is a contemporary overview of this research methodology known as `validity generalization." Additionally, TSA's implementation of a test without having conducted a local, criterion -related validation study is supported by the "interim use" provisions found in EEOC's 1978 Uniform Guidelines on Employee Selection Procedures. EEOC's Uniform Guidelines Permits TSA's "interim use" of Selection Procedures Without Having Conducted a Local, Criterion -related Validation Study Sec.5J "Interim use of selection procedures" in the Uniform Guidelines states: "Users may continue the use of a selection procedure which is not at the moment fully supported by the required evidence of validity, provided: 1) The use has available substantial evidence of validity, and 2) the user has in progress, when technically feasible, a study which is designed to produce the additional evidence required by these guidelines within a reasonable period of time._ ."ls 15 Uniform Guielines Q&A#59 Q: Section 5J on interim use requires the user to have available substantial evidence of validity. What does this mean? A: For purposes of compliance with 5J, "substantial evidence" means evidence which may not meet all the validation requirements of the Guidelines but which raises a strong inference that validity pursuant to these standards will soon be shown. Section 5J is based on the proposition that it would not be an appropriate allocation of Federal resources to bring enforcement proceedings against a user who would soon be able to satisfy fully the standards of the Guidelines. For example, a criterion -related validity study may have produced evidence which meets almost all of the requirements of the Guidelines with the exception that the gathering of the data of test fairness is still in progress and the fairness study has not yet produced results. If the correlation coefficients for the group as a whole permits the strong inference that the selection procedure is valid, then the selection procedure may be used on an interim basis pending the completion of the fairness study. Uniform Guidelines Q&A#60 CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Validity Generalization Is Supported by the National Research Council Validity generalization as a research strategy was commented upon favorably the National Research Council of the National Academy of Sciences as follows:16 "We accept the general thesis of validity generalization, that the results of validity studies can be generalised to many jobs not actually studied, but we urge a cautious approach of generalizing validities only to appropriately similar jobs." Validity Generalization Also Supported by Generally Accepted Principles and Practices of Industrial Psychology The generally accepted principles and practices of industrial psychology state the following with regard to transporting and generalizing validity evidence: "Validity generalization is a demonstration that a selection procedure, or kind of selection procedure permits valid inferences about job behavior or job performance across given jobs or groups of jobs in different settings.17 "The cumulative validity evidence may indicate generalizability of validity for a selection procedure only for particular kinds of jobs or job families In such a case, reliance on validity generalization results for jobs in new settings or organizations should meet two conditions: (1) The selection procedure to be used... is a representative example of the type of selection procedure included in the validity generalization study..., and (2) The job in the new setting is similar to the job, or a member of the same job family, included in the validity generalization study... If sufficient information is available -on the new job families- to accurately assign it to a Q: What are the potential consequences to a user when a selection procedure is used on an interim basis? A: The fact that the Guidelines permit interim use of a selection procedure under some conditions does not immunize the user from liability for back pay, attorneys fees and the like, should use of the selection procedure later be found to be -in -violation of the Guidelines, Sec. 5J. For this reason, users should take steps to come into full compliance with the Guidelines as soon as possible. It is also appropriate for users to consider ways of minimizing adverse impact during the period of interim use. 16 Hartigan, J. & Wigdor, S. (1989). Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery, National Research Council, National Academy of Sciences. Washington, DC: National Academy Press, p. 8, 143-144. 17 Society for Industrial and Organizational Psychology, Inc. (1987). Principles for the Validation an4 Use of Personnel Selection Procedures. (Third Edition) College Park, MD: Author p.26. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study relevant occu1ational category, classification can be made without a formal job analysis." g Uniform Guidelines Requirement for Generalizing Validity Evidence to Support TSA's Use of Tests With regard to TSA's transporting and generalizing validity evidence developed elsewhere, the Uniform Guideline requires three lines of evidence: validity, similarity and fairness.19 Validity The first line of evidence called for in Sec. 7B(1) of the Uniform Guidelines in order to transport and to generalize validity evidence is to establish that the test is valid for its intended purpose. Job Similarity The second line of evidence called for in Sec. 7B(2) of the Uniform Guidelines in order to transport and to generalize validity evidence is to establish that: "The incumbents in the user's job and the incumbents in the job or group of jobs on which the validity study was conducted perform substantially the same major work behaviors, as shown by appropriate job analyses both on the job or group of jobs on which the validity study was performed and on the job for which the selection procedure is to be used." Fairness The third line of evidence called for in Sec. 7B(3) of the Uniform Guidelines in order to transport and to generalize validity evidence is to investigate "fairness." However, the generally accepted principles and practices of industrial psychology state the following with regard to investigating "fairness":20 "We find little convincing evidence that well -constructed and competently administered tests are more valid predictors for one population subgroup than for another: individuals with higher scores tend to perform better on the job, regardless of group identity." The National Research Council of the National Academy of Sciences supports the generally accepted principles and practices of industrial psychology with regard to investigating "fairness" as follows:21 18 Society for Industrial and Organizational Psychology, Inc. (1987). Principles for the Validation and Use of Personnel Selection Procedures, p. 28. 19 Uniform Guidelines, Sec. 7B(1-3). 20 Society for Industrial and Organisational Psychology, Inc. (1987). Principles for the Validation and Use of Personnel Selection Procedures, p. 18. 21 Hartigan, J. & Wigdor, S. (1989). Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery. National CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study "The use of a total -group regression equation generally would not give predictions that were biased against black applicants. If the total -group equation does give systematically different predictions than would be provided by the equation based on black employees only, it is somewhat more likely to overpredict than to underpredict. These results are generally consistent with results that have been reported for other tests." As was noted by the Committee on Ability Testing in the National Research Council of the National Academy of Sciences: "Predictions based on a single equation (either one for whites or for a combined group of whites and blacks) generally yield predictions that are quite similar to, or somewhat higher than, predictions from an equation based only on data from blacks. In other words, the results do not support the notion that the traditional use of test scores in a prediction yields predictions for blacks that systematically underestimate their performance." Because of the phenomenon of "overprediction" referred to in the generally accepted principles and practices of industrial psychology, if literally followed, Sec. 7B(3) of the Uniform Guidelines would require the setting of higher cutoff scores on tests for minority candidates in order to get job performance comparable to nonminorities. This conclusion as to the lack of professional support for investigating "fairness" as "required" in Sec. 7B(3) of the Uniform Guidelines is supported by both the generally accepted principles and practices -of industrial psychology and the National Academy of Sciences both of which were quoted above. Uniform Guidelines Fall Short of Generally Accepted Principles and Practices of Industrial Psychology The primary shortcomings of the Uniform Guidelines that no longer reflect generally. accepted principles and practices of industrial psychology are the incorrect presumptions that validity is "situationally specific," and that there is `differential prediction." The first refers to the incorrect presumption that validity evidence must be developed for each new location where a test is to be used even though the profession -recognizes that validity evidence is transportable and generalizable across situations. The Uniform Guidelines' "requirement" to investigate "test fairness" refers to the incorrect presumption that even though a test may be shown to be valid for one group in the population, it can not be presumed to be valid for other subgroups. Research Council, National Academy of Sciences. Washington, DC: National Academy Press, p. 185. 22 Wigdor, S. & Garner (1982). Ability Testing, Parts I & II. Committee on Ability Testing, National Research Council, National Academy of Sciences, Washington, DC: National Academy Press, p.77. CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study The profession has now recognized, as revealed in the Congressional testimony on behalf of the Society for Industrial and Organizational Psychology and the American Psychological Association, and that if a test is valid for one group in the population, it is valid for all subgroups. Finally, there is substantial precedent of favorable Title VII case law supporting transporting and generalizing validity evidence.23 9) Alternative procedures investigated.' The pilot test battery administered mid -March in Los Angeles, Memphis and Chicago contained a number of tests which could have been scored and used, but were dropped because less adverse (i.e., lower average minority/non-minority score differences) tests measuring the same critical skills and abilities were chosen. 10) Uses and applications.' In Phase 1, because Congress so stipulated in statutory Iangtuage of the Act, lack of either citizenship or a high school diploma (or one year of relevant experience) were knockout factors. In Phase II, the Service Orientation, Readiness and Applied Thinking tests were each scored with using a minimum cut score (i.e., pass/fail). Both the SORT and the structured interview were similarly scored on a pass/fail basis. Phase IV components were scored on a pass/fail basis (except for credit and criminal checks which were discussed before a decision was made). 11) Source data.'6 12) Contact person." 13) Accuracy and completeness.' ""1" 23 Friend v. City of Richmond (588 F.2d 61 (1978); Watson v. Fort Worth Bank and Trust (108 S.Ct. 2777 (1988); EEOC v. Atlas Paper Box Company (680 F.Supp. 1184 (E.D. Tenn. (1987) & (868. F.2d. 1487 (6th Cir. 1989); Pegues v. Mississippi State Employment Service (488 F.Supp, 239 (N.D. Miss:1980), of d in part and rev'd in part, 699 F.2d 760 (5th Cir.), cert. denied, 4E4 U.S. 991 (1983); Bernard v. Gulf Oil Corp. 890 F.2d 735 (5`h Cir. 1989); Taylor v. James River Corp. (51 FEP Cases 893 (1988); Bruckner v. Goodyear Tire and Ruhr Co. 339 F.Supp. 1108-(N.D. Ala. 1972), afi'd per curium, 476 F.2d 1287 (5th Cir. 1973); Rivera v. City of Wichita Falls 665 F.2d 531, 538 at n.10 (5th Cir. 1982); Brunet v City of Columbus 642 F.Supp-__ um (S.D. Ohio 1986); Agulera v. Cook County Police & Corrections Merit Board 582 F.Supp. 1053, 1057 (N.D. Ill, 1984), aff'd 760 F.2d 844, 847-48 (7th Cir. 1985), cited with approval in Davis v. City of Dallas, 777 F.2d 205, 212-13, n.6 (5th Cir. 1985), cert. Denied, 476 U.S. 1116, 106 S.Ct. 1972, 90 L.Ed.2d 656(1'986). CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Appendix A Skill Standards for Transportation Security Screeners Critical Work Function 1: Control Entry and Exit Points KA 1.1: Examine tickets and other documentation of individuals prior to entering security checkpoints PI 1.1.1 PI 1.1.2 PI 1.1.3 PI 1.1.4 PI 1.1.5 PI 1.1.6 PI 1.1.7 PI 1.1.8 Individual clearance to enter security checkpoint (i.e., airport workers with identification, airline workers with identification, or passengers with valid ticket information) is given or denied Ticket information indicates the ticket/travel is valid Entry to security checkpoint is appropriate for travel date and time, and Pier/Concourse, indicated on ticket information (ticket, boarding pass, e-ticket, etc.) Name on ticket information matches the photo identification of the individual Individuals are instructed to return to ticket counter to resolve discrepancies in ticket information Individual does not exceed allowable carry -on baggage limit Examination is conducted in a professional and respectfiil manner Individuals with cultural and/or language differences are treated with respect KA 1.2: Ensure individuals correctly place personal property onto x-ray conveyor belt PI 1.2.1 PI 1.2.2 PI 1.2.3 PI 1.2.4 Directions are stated to the individual clearly Individuals are observed placing items onto x-ray belt Items are properly and safely placed onto the x-ray belt Breaches of security are communicated to supervisor immediately KA 1.3: Prepare individuals and airport personnel to enter through metal detector PI 1.3.1 PI 1.3.2 PI I.3.3 PI 1.3.4 PI 1.3.5 Individual.S are told of procedures regarding examination of pocket content and other personal items Pocket contents are examined according to protocol Prohibited items are identified and removed Individuals are directed to proceed through metal detector Breaches of security are communicated to supervisor KA 1.4: Prevent individuals from entering through the exit lanes PI 1.4.1 PI 1.4.2 Individuals are monitored as they exit sterile area into the airport's public area Individuals attempting to enter through exit lanes are identified CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study PI 1.4.3 Individuals attempting to enter through exit lanes are instructed on proper procedures and requested to return to checkpoint for screening PI 1.4.4 Security breach procedures are followed when un-screened individuals refuse or continue into sterile area PI 1.4.5 Breaches of security are immediately reported to supervisor PI 1.4,6 Visual contact is maintained with any individual making a security breach PI 1.4,7 Un-screened individwiis are kept from having physical contact with screened individuals PI 1.4.8 Armed law enforcement officers (LEO) entering through exit area are referred to supervisor PI 1.4.9 Individuals with cultural and/or language differences are treated with respect KA 1.5: Facilitate orderly flow of individuals through security checkpoints PI 1.5.1 Security lines are processed as smoothly and quickly as possible PI 1.5.2 Security procedures and directions are communicated to individuals in a timely manner PI 1.5.3 Language barriers interfering with the individual's ability to follow directions are communicated to supervisor immediately PI 1.5.4 Individuals with special needs or needing extra assistance are instructed of procedures and given assistance as appropriate PI 1.5.5 Upset and difficult individuals are managed to ensure that screening- functions are not disrupted and security is maintained CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study [ �.. .i?S xbS4..YR"1C �KPSA I�H�����,t`i.lslmbMGi"�,SAM Criticality Rank Skill Criticality Complexity 1 LISTENING: DISTRACTIONS 4.7 4,3 2 SPEAKING: TACT AND SENSITIVITY REQUIRED 4.6 3.6 2 MAINTAIN COMMAND POSTURE 4.6 4.3 3 VISUAL OBSERVATION 4.5 4.4 3 INTEGRITY/HONESTY 4.5 4.1 4 SPEAKING: DIVERSITY OF AUDIENCE 4.4 4.6 5 SPEAKING: DISTRACTIONS 4.2 4.3 6 STRESS TOLERANCE 4.1 3.6 6 4.1 2.4 LISTENING: DEMANDS ON At IENTION 7 USING SOCIAL SKILLS 4.0 3,8 8 LISTENING: OVERALL 3.9 2.9 8 SPEAKING: OVERALL 3.9 3,6 8 SPEAKING: LISTENER RESISTANCE 3.9 2.9 9 INITIATIVE/MOTIVATION 3.8 3.1 10 READING 3.7 1.7 11 GATHERING AND ANALYZING INFORMATION 3.5 2.1 11 MAKING DECISIONS AND JUDGMENTS 3.5 2.0 12 MATHEMATICS 3.2 1.0 13 LISTENING: COMMUNICATION INDIRECTNEss 3.0 2.8 Criticality Scale: 1 = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) Complexity Scale: 1=Low; 2=Low Moderate; 3=Moderate; 4=Moderate High; 5=High CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Critical Work Function 2: Perform Routine Security Screening of Persons KA 2.1: Ensure that individuals who walk through the metal detector and set off the alarm undergo additional metal detection screening with hand-held wand PI 2.1.1 PI 2.1.2 PI 2.1.3 PI 2.1.4 PI 2.1.5 Individuals are immediately detained for hand-held screening or "wanding" Visual contact with individual is maintained Individuals are told of the process for using hand held -metal detectors including front and back scans Upset or difficult individuals are managed to ensure that screening functions are not disrupted and security is maintained Language barriers interfering with the individual's ability to follow directions are communicated to supervisor immediately KA 2.2: Ensure that hand-held metal detectors are operational PI 2.2.1 PI 2.2,2 PI 2.2.3 Detectors are calibrated or tested prior to the start of the shift Malfunctioning detectors are immediately turned off, discontinued or otherwise removed from service Malfunctioning detectors are reported to supervisor KA 2.3: Use band -held metal detector identify source of alarm P12.3.1 PI 2.3.2 PI 2.3.3 PI 2.3.4 PI 2.3.5 PI 2.3.6 KA 2.4: Perform PI 2.4,1 PI 2.4.2 PI 2.4.3 PI 2.4.4 PI 2.4.5 PI 2.4.6 Use of hand-held metal detectors follow specified protocol Sources of alarm (i.e., jewelry, shoes, etc) are identified If cause of alarm is not identified, individuals are referred to pat down station Breaches of security are reported to supervisor immediately Examination is conducted in a professional and respectful manner Cultural and language differences are considered while using held held devices pat -down searches to resolve "wanding" alarms Pat -down searches are consistent with proper protocol Pat -down searches are conducted in a professional and respectful manner Physical searches of disabled and impaired individuals include search of assistive devices (i.e., wheelchair, Seeing Eye dog equipment, etc.) Breaches of security are reported to supervisor immediately Upset or difficult individuals are managed to ensure that screening firnctions and security is maintained Cultural and language differences are considered whle performing pat -downs CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study KA 2.5: Perform physically sensitive searches of individuals PI 2.5.1 PI 2.5.2 PI 2.5.3 PI 2.5.4 PI 2.5.5 Physically sensitive searches are consistent with protocol Physically sensitive searches are conducted in a professional and respectful manner (e.g., in private when appropriate) Breaches of security are reported to supervisor immediately Upset or difficult individuals are managed to ensure that screening functions are not disrupted and security is maintained Cultural and language differences are considered while performing physically sensitive searches CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study ;art iN k ..na .� 4.4', $ f�. • ., ,. .... '� i6® ti r°`,&... § 4�b r7Ti. r it;. :. Criticality Rank Skill Criticality Complexity 1 VISUAL OBSERVATION 4.9 4.7 2 SPEAKING: TACT AND SENSITIVITY REQUIRED 4.8 4.3 2 MAINTAIN COMMAND POSTURE 4.8 4.5 3 LISTENING: DISTRACTIONS 4.6 4.3 3 INTEGRmr/HONESTY 4.6 4.3 4 SPEAKING: DIVERSITY OF AUDIENCE 4.5 4.6 4 STRESS TOLERANCE 4.5 4.6 5 USING SOCIAL SKILLS 4.4 4.4 6 SPEAKQNG: DISTRACTIONS 4.3 4.2 6 SPEAKING: OVERALL 4.3 3.8 6 MAKING DECISIONS AND JUDGMENTS 4.3 2.6 7 4.1 2.6 LISTENING: DEMANDS ON A n.ENTION 7 SPEAKING: LISTENER RESISTANCE 4.1 3.4 8 LISTENING: OVERALL 4.0 2.8 9 INITIATIVE/MOTIVATION 3.9 3.4 10 GATHERING AND ANALYZING INFORMATION 3.8 2.6 11 ANALYZING AND SOLVING PROBLEMS 3.3 2.3 12 SPEAKING: CONSTRAINTS ON PREPARATION 3.1 3.8 12 LISTENING: CONTENT COMPLEXITY 3.1 1.9 12 USING INFORMATION AND COMMUNICATIONS TECHNOLOGY 3.1 2.2 13 LISTENING: COMMUNICATION INDIRECTNESS 3.0 2.7 13 ADAPTABILITY 3.0 2.9 Criticality Scale: 1 = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) Complexity Scale: 1= Low; 2= Low Moderate; 3= Moderate; 4= Moderate High; 5=High CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Critical Work Function 3: Perform Security Screening of Property and Baggage KA 3.1: Operate x-ray device controls and monitor screen to detect dangerous, deadly, incendiary, and explosive articles in property and baggage PI 3.1,1 PI 3.1.2 PI 3.1.3 PI 3.1.4 PI 3.1.5 KA 3.2: Request PI 3.2.1 PI 3.2.2 PI 3.2.3 PI 3.2.4 PI 3.2.5 PI 3.2.6 KA 3.3: X-ray equipment is tested for proper functioning prior to the start of the shift Use of x-ray device follows proper protocols Screening identifies disguised objects, improvised explosive devices (IED), and combined threat images Potential threats are referred for baggage search Calibration of equipment is checked as appropriate hand search of property and baggage when necessary Individual is immediately detained for additional property and baggage screening Individual is told of process for additional screening Security breaches are reported to supervisor immediately Physical control of the suspect property and baggage is maintained Visual contact with individual is maintained Upset or difficult individuals are managed to ensure that screening functions are not disrupted and security is maintained Conduct Explosive Trace Detection (ETD) inspection PI 3.3.1 PI 3.3.2 PI 3.3.3 PI 3.3.4 PI 3.3.5 KA 3.4: ETD inspection follows proper protocol Graphic output is interpreted correctly to identify articles that require additional examination Suspicious output and articles are reported to supervisor immediately Purge and decontamination procedures are performed according to protocol Baggage and property is returned to individual if no prohibited articles are detected Conduct random physical check of property and baggage at gate PI 3.4.1 PI 3.4.2 PI 3.4.3 PI 3.4.4 PI 3.4.5 Random physical checks follow protocol Individual is told of process for additional screening Baggage and property is handled neatly and carefully Baggage and property is returned to individual if no prohibited articles are detected Security breaches are reported to supervisor immediately CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study ,� df +�� rr ?r ^ aBIM'• ps.�,. 4 r_ 4#¢ 'CY �..{ N j�R' �Y.tom 3W..d J.' aA44. �h�.iC2 ��.P u7.c�`,F „V . ' Criticality Rank Skill Criticality Complexity 1 VISUAL OBSERVATION 4.9 4.9 2 MAINTAIN COMMAND POSTURE 4.6 4.3 2 INTEGRITY/HONESTY 4.6 4.4 3 SPEAKING: TACT AND SENSITIVITY REQUIRED 4.4 4.0 3 SPEAKING: DIVERSITY OF AUDIENCE 4.4 4.6 4 LISTENING: DISTRACTIONS 4,3 4.1 4 S TRES s TOLERANCE 4.3 4.3 4 4.3 4.1 SPEAKING: DISTRACTIONS 4 MAKING DECISIONS AND JUDGMENTS 4.3 3.1 5 GATHERING AND ANALYZING INFORMATION 4.1 3.3 6 USING SOCIAL SKILLS 3.9 4.4 6 LISTENING: DEMANDS ON ATTENTION 3.9 2.4 6 INITIATIVE/MoIVATIoN 3.9 3.3 7 SPEAKING: LISTENER RESISTANCE 3.8 3.4 8 SPEAKING: OVERAL 3.7 3.4 8 USING INFORMATION AND COMMUNICATIONS TECHNOLOGY 3.7 3.0 9 LISTENING: OVERALL 3.5 2.4 9 ANALYZING AND SOLVING PROBLEMS 3.5 2.9 10 SPEAKING: CONSTRAINTS ON PREPARATION 3.1 3.7 11 LISTENING: COMMUNICATION INDIRECTNESS 3.0 2.5 11 READING 3.0 2.7 11 LISTENING: LIMITATIONS ON INTERACTION 3.0 2.3 Criticality Scale: 1 = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) Complexity Scale: 1=Low; 2=Low Moderate; 3=Moderate; 4=Moderate High; 5= High CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Critical Work Function 4: Continuously Improve Security Screening Processes KA 4.1: Report security weaknesses, vulnerabilities, or other problem situations related to specific areas of work PI 4.1.1 PI 4.1.2 PI 4.1.3 Reports on safety hazards, equipment problems, security breaches, FAA tests, injuries, etc. are made immediately to appropriate personnel Reports are written clearly and follow appropriate format Follow-up on reported weaknesses is performed in a timely manner KA 4.2: Respond to potential or actual emergency situations PI 4.2.1 Supervisor and/or law enforcement officials are notified immediately of emergency situations PI 4.2.2 Visual contact is maintained with individuals involved in the situation PI 4.2.3 Verbal detainment, rather than physical restraint, is made when appropriate, without jeopardizing his/her safety or that of others in the immediate vicinity PI 4.2.4 Supervisor and/or law enforcement official given detailed description of suspect and situation PI 4.2.5 Safe and proper handling of deadly or dangerous weapons and dangerous objects which are discovered -is ensured until supervisor/law enforcement officer arrives KA 4.3: Assist in inquiries/investigations or security issues related to airports PI 4.3.1 PI 4.3.2 PI 4.3.3 PI 4.3.4 Requested information is complete and accurate Requested information is provided in an appropriate written or oral format Requested information is discussed only with appropriate personnel Assistance follows established protocol KA 4.4: Make oral and written formal reports such as incident reports and witness statements PI 4.4.1 PI 4.4.2 PI 4.4.3 PI 4.4.4 P14.4.5 PI 4.4.6 Reports include all pertinent information Reports are provided in the appropriate format Reports are provided to appropriate personnel Passengers are provided with appropriate incident reports for lost or damaged property or personal injury Reports are discussed only with appropriate personnel Reports are made using correct English CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study KA 4.5: Understand written and verbal classified or security sensitive information PI 4.5.1 PI 4.5.2 PI 4.5.3 Information is processed to determine applicability to current job function Information received is discussed only with appropriate personnel Breaches of security and/or confidentiality are reported to appropriate personnel 'rR e �ry - ,��4 q� ,¢li V ^� \ ,. `, d �w _ . p M .., �t�Y�< 3� � dA S pyy l % �4 o, �r .1 4�".19 Criticality Rank Skill Criticality Complexity 1 INTEGRITY/HONESTY 4.3 4.4 1 INTIIATIVE/MOT VATION 4.3 3.8 2 VISUAL OBSERVATION 4.2 4.4 3 MAKING DECISIONS AND JUDGMENTS 3,9 2.8 3 ANALYZING AND SOLVING PROBLEMS 3.9 3.1 3 WRITING 3.9 2.6 4 READING 3.8 2.8 5 LISTENING: OVERALL 3.7 2.9 6 LISTENING: DISTRACTIONS 3.6 3.6 6 STRESS TOLERANCE 3,6 3.8 6 3.6 2,9 LISTENING: DEMANDS ON A1IENTION 6 SPEAKING: OVERALL 3.6 3.4 7 MAINTAIN COMMAND POSTURE 3.3 3.5 7 SPEAKING: TACT AND. SENSITIVITY REQUIRED 3.3 3.3 7 3.3 2,6 LISTENING: CONTENT COMPLEXITY 8 SPEAKING: DISTRACTIONS 3.2 3.4 8 USING SOCIAL SKILLS S 3.2 3.1 9 ABILITY To LEARN 3.1 3.1 10 SPEAKING: DIVERSITY OF AUDIENCE 3.0 2.7 10 SPEAKING: CONSTRAINTS ON PREPARATION 3.0 3.0 10 3.0 2.7 SPEAKING: CONTENT COMPLEXITY Criticality Scale: 1 = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) Complexity Scale:l=Low; 2=Low Moderate; 3=Moderate; 4-Moderate High; 5=High CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Critical Work Function 5: Continuously Improve Own Performance KA5.1: Identify gaps in personal skill and knowledge PI 5.1.1 PI 5.1.2 PI 5.1.3 PI 5.1.4 PI 5.1.5 PI 5.1.6 Performance evaluations are reviewed Opportunities for training are identified Feedback on skill proficiency is obtained from colleagues and supervisor Action plan is created and implemented to close gaps A Career Development Plan is created Personal experience in non -routine situations is utilized to identify additional training needs KA5.2: Attend job -related training sponsored by employer PI 5.2,1 PI 5.2.2 PI 5.2.3 PI 5.2.4 PI 5.2.5 PI 5.2,6 PI 5.2.7 Training sessions are attended Passing scores on all tests are obtained Training on key equipment (i.e., x-ray machine, ETD machine, etc) is successfully completed On -going training on customer service is obtained On -going training on stress management is obtained Information gathered in training is utilized on the job Training opportunities are identified KA5.3 Utilize additional training materials to augment professional knowledge PI 5.3.1 Security -related periodicals are identified and, where appropriate, reviewed PI 5.3.2 Safety notices, pictures, and postings are reviewed on a regular basis PI 5.3.3 Changes to Federal regulations and requirements are reviewed on a regular basis PI 5.3.4 Information gathered is utilized on the job and shared with co workers KA5.4: Obtain and maintain necessary certifications PI 5.4.1 PI 5.4.2 PI 5,4,3 PI 5.4.4 Certification requirements for job are identified Certification exams are taken at the appropriate time Passing score on certification test is obtained Certifications are kept up-to-date CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study GIC l.1{byac p fir,.-c, ,¢�.r W'f'r+.: �l Y:Jt '(1 .iyy a? �..; It it 3i k e at 3g F: q« t. .. pn r ,y � r a� �: � �'' Criticality Rank Skill Criticality Complexity 1 INITIATIVE/MOTTVATION 4.5 3.8 1 ABILITY To LEARN 4.5 3.7 2 SELF AND CAREER DEVELOPMENT 4.1 3.9 3 READING 3.8 3.0 3 3.8 3.1 LISTENING: DEMANDS ON AIIENTION 4 INTEGRrIY/HONESTY 3.7 3.2 4 LISTENING: OVERALL 3.7 3.1 5 LISTENING: CONTENT COMPLEXITY 3.6 3.2 5 GATHERING AND ANALYZING INFORMATION 3.6 2.9 6 MAKING DECISIONS AND JUDGMENTS 3.4 2.4 6 USING INFORMATION AND COMMUNICATIONS TECHNOLOGY 3.4 2.8 7 WRITING 3.3 2.4 8 ANALYZING AND SOLVING PROBLEMS 3.2 2.5 9 VISUAL OBSERVATION 3.0 2.9 9 STRESS TOLERANCE 3.0 2,6 9 ADAPTABILITY 3.0 2.4 Criticality Scale: 1 = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) Complexity Scale: 1=Low; 2=Low Moderate; 3=Moderate; 4=Moderate High; 5= High CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study Appendix B Overall Critical Knowledge and Skills Required for Screeners CONFIDENTIAL: Attorney - Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study CRITICALITY RANK VISUAL OBSERVATION Sim" COMPLEXI 4.9 2 2 SPEAKING: TACT AND SENSITIVITY REQUIRED 4.8 4.3 3 2 MAINTAIN COMMAND PosTURE 4.8 4.5 4 3 LISTENING: DISTRACTIONS 4.7 4.3 4 INTEGRTIY/HONESTY 4.6 4.4 5 SPEAKING: DIVERSITY OF AUDIENCE 4.5 4.6 7 5 STRESS TOLERANCE 4.5 4.6 5 INITIATIVE/MOTIVATION 4.5 3.8 9 5 ABILITY To LEARN 4.5 3.7 10 6 USING SOCIAL SKILLS (CUSTOMER SERVICE SKILLS) 11 7 SPEAKING: DISTRACTIONS 4.4 4.3 4.4 4,3 12 7 SPEAKING: OVERALL 4.3 3.8 13 7 MAKING DECISIONS AND JUDGMENTS 4.3 3.1 14 8 LISTi NINGG DEMANDS ON ATTENTION 4.1 3.1 15 8 SPEAKING LISTENER RESISTANCE 4.1 3.4 16 8 GATHERING -AND ANALYZING INFORMATION 4.1 3.4 17 8 SELF AND CAREER DEVELOPMENT 4.1 3.9 18 9 LISTENING: OVERALL 4.0 3.1 19 10 WRITING 3.9 2.6 20 11 READING 3.8 3.0 21 12 USING INFORMATION AND COMMUNICATIONS TECHNOLOGY 3.7 3.0 22 13 ANALYZING AND SOLVING PROBLEMS 3.6 3.1 23 13 LISTENING: CONTENT COMPLEXITY 3.6 3.2 24 14 MATHEMATICS 3.2 1.6 25 15 SPEAKING: CONSTRAINTS ON PREPARATION 3.1 3.8 26 16 LISTENING: COMMUNICATION INDIRECTNESS 3.0 2.8 27 16 ADAPTABILITY 3.0 2.9 28 16 LISTENING: LIMITATIONS ON INTERACTION 3.0 2.4 29 16 SPEAKING: CONTENT COMPLEXITY 3.0 Criticality Scale: = Not Critical (Not at all); 2 = Slightly Critical (Low); 3 = Somewhat Critical Medium); 4 = Very Critical (High); 5 = Extremely Critical (Very High) 2.7 CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study variability in test validity have been put forth (e.g., the reliability of performance measures is higher is some organizations than in others, which can lead to apparent differences in validity), but much of the literature dealing with validity generalization has focused on the simplest and probably the most important explanation for differences in validity coefficients across studies (Le., simple sampling error). Many validity studies, particularly studies from the early 1970s and before, used small samples, and it is well known that statistical results of all sorts, including validity coefficients, are highly unstable when samples are small. Corrections for sampling error and for other artifacts that artificially inflate the variability in test validities often suggest that much of the apparent instability of validities is a reflection of weaknesses of validity studies (small samples, variation in the degree of unreliability and range restriction) rather than a reflection of true differences in the validity of tests across settings" (p.8-9). "Meta -analytic methods are tremendously useful describing general trends in the research literature, and these trends often give selection practitioners a very good idea of what will or will not work. However, it is easy to over interpret VG and to make inferences about how well particular test will work in particular settings that are not empirically justified. One of the greatest challenges in this sort of analysis is determining what inferences can be made (e.g., the inference that cognitive ability tests are at least minimally valid in most jobs seems a safe one) and what sorts cannot be made on the basis of meta -analyses of validity studies" (p.23-24). Endnotes Dates and location(s) of the job analysis or review ofjob information, the date(s) and location(s) of the administration of the selection procedures and collection of criterion data, and the time between collection of data on selection procedures and criterion measures should be provided (essential), If the study was conducted at several locations, the address of each location, including city and state, should be provided. An explicit definition of the purpose(s) of the study and the circumstances in which the study was conducted should be provided. A description of existing selection procedures and cutoff scores, if any, should be provided. u; A description of the procedure used to analyze the job or group of jobs, or to review the job information should be provided (essential). Where a review ofjob information results in criteria which may be used without a full job analysis... the basis for the selection of these criteria should be reported (essential). Where a job analysis is required a complete description of the work behavior(s) or work outcome(s), and measures of their criticality or importance should be provided (essential). The report should describe the basis on which the behavior(s) or outcome(s) were determined to be critical or important, such as the proportion of time spent on the respective behaviors, their level of difficulty, their frequency of performance, the consequences of error, or other appropriate factors (essential). Where two or more jobs are grouped for a validity study, the information CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study called for in this subsection should be provided for each of the jobs, and the justification for the grouping... should be provided (essential). `° It is desirable to provide the user's job title(s) for the job(s) in question and the corresponding job title(s) and code(s) from U.S. Employment Service's Directory of Occupational Titles. ° The bases for the selection of the criterion measures should be provided, together with references to the evidence considered in making the selection of criterion measures (essential). A full description of all criteria on which data were collected and means by which they were observed, recorded, evaluated, and quantified, should be provided (essential). °` A description of how the research sample was identified and selected should be included (essential). The race, sex, and ethnic composition of the sample, including those groups set forth in section 4A above, should be described (essential). This description should include the size of each subgroup (essential). A description of how the research sample compares with the relevant labor market or work force, the method by which the relevant labor market or work force was defined, and a discussion of the likely effects on validity of differences between the sample and the relevant labor market or work force, are also desirable. Descriptions of educational Ievels, length of service, and age are also desirable. Any measure, combination of measures, or procedure studied should be completely and explicitly described or attached (essential). If commercially available selection procedures are studied, they should be described by title, form, and publisher (essential). Reports of reliability estimates and how they were established are desirable. °Il' Methods used in analyzing data should be described (essential). Measures of central tendency (e.g., means) and measures of dispersion (e.g., standard deviations and ranges) for all selection procedures and all criteria should be reported for each race, sex, and ethnic group which constitutes a significant factor in the relevant labor market (essential). The magnitude and direction of all relationships between selectin procedures and criterion measures investigated should be reported for each relevant race, sex, and ethnic group and for the total group (essential). Where groups are too small to obtain reliable evidence of the magnitude of the relationship, need not be reported separately (sic). Statements regarding the statistical significance or results should be made (essential). Any statistical adjustments, such as for less than perfect reliability or for restriction of score range in the selection procedure or criterion should be described and explained; and uncorrected correlation coefficients -should also be shown (essential). Where the statistical technique categorizes continuous data, such as biserial correlation and the phi coefficient, the categories and the bases on which they were determined should be described and explained (essential). Studies of test fairness should be included where called for by the requirements of section 14B(8) (essential). These studies should include the rationale by which a selection procedure was determined to be fair to the group(s) in question. Where test fairness or unfairness has been demonstrated on the basis of other studies, a bibliography of the relevant studies should be included (essential). If the bibliography includes unpublished studies, copies of these studies, or adequate abstracts CONFIDENTIAL: Attorney — Client Work Product (July, 2004) DRAFT TSA: Transportation Security Screener Validation Study or summaries, should be attached (essential). Where revisions have been made in a selection procedure to assure compatibility between successful job performance and the probability of being selected, the studies underlying such revisions should be included (essential). All statistical results should be organized and presented by relevant race, sex, and ethnic group (essential). The selection procedures investigated and available evidence of their impact should be identified (essential). The scope, method, and findings of the investigation, and the conclusions reached in light of the findings should be fully described (essential). The methods considered for use of the selection procedure (e.g., as a screening device with a cutoff score, for grouping or ranking, or combined with other procedures in a battery) and available evidence of their impact should be described (essential). This description should include the rationale for choosing the method for operational use, and the evidence of the validity and utility of the procedure as it is to be used (essential). The purpose for which the procedure is to be used (e.g., hiring, transfer, promotion) should be described (essential). If weights are assigned to different parts of the selection procedure, these weights and the validity of the weighted composite should be reported (essential). If the selection procedure is used with a cutoff score, the user should describe the way in which normal expectations of proficiency within the work force were determined and the way in which the cutoff score was determined (essential). xl Each user should maintain records showing all pertinent information about individual sample members and raters where they are used, in studies involving the validation of selection procedures. These records should be made available upon request of a compliance aency. In the case ofindividual sample members these data should include scores on the selection procedure(s), scores on criterion measures, age, sex, race, or ethnic group status, and experience on the specific job on which the validation study was conducted, and may also include such things as education, training, and prior job experience, but should not include names and social security -numbers. Records should be maintained which show the ratings given to each sample member by each rater. "" The name, mailing address, and telephone number of the person who may be contacted fur further information about the validity study should be provided (essential). The report should describe the steps taken to assure the accuracy and completeness of the collection, analysis, and report of data and results. '°° The report should describe the steps taken to assure the accuracy and completeness of the collection, analysis, and report of data and results.