HomeMy WebLinkAboutExhibit 2ATTACHMENT "A"
2.2 TEST DEVELOPMENT REQUIREMENTS (From RFP 04-05-085)
The Consultant shall:
1. Abide by timetables outlined herein over the life of the contract and for each classification,
with the understanding that timetables outlined herein may be superseded by provisions
outlined in the current Fraternal Order of Police (FOP) Labor Agreement. These timetables are
estimates only and subject to change. The City shall provide reasonable written notice of any
changes to specified timetable.
2. Conduct a legally defensible job -analysis according to current professional and legal standards
within 90 calendar days after the execution of the Agreement for both classifications of Police
Lieutenant and Police Sergeant.
3. Develop a written examination that is appropriate to directly measure the knowledge, skills,
abilities, and other characteristics (KSAOs) identified in the job analysis as necessary to
perform the job functions within 180 calendar days of the completion of the job analysis for
each classification of Police Sergeant and/or Police Lieutenant. The written examination may
act as a hurdle for the classification of Police Sergeant.
4. Develop no less than three (3) examination exercises for each examination and for each
classification that is appropriate to directly measure the knowledge, skills, abilities, and other
characteristics (KSAOs) that are identified in the existing job analysis within 180 calendar
days after the completion of the job analysis necessary to perform the job functions for the
classifications of Police Lieutenant and Police Sergeant. Consultant will have input from
subject matter experts (SME) from the Miami Police Department including final review of
exercises. The Department of Employee Relations (DER) will also have input into the
development of examination measures and must approve exercises after Police Department
SME approval. Each examination process must have a preparation manual/study guide
published and delivered to DER in electronic format by the test Consultant, with sample
exercises, at least 60 days prior to the administration of the examination. All exercises must
require responses which are recorded by the candidate on paper and/or verbally to live
assessors.
5. Develop a parallel form for each exam exercise, for each classification and for each day of
testing that exceeds a one (1) day administration period. In addition, develop one (1) extra
form for each and every exercise beyond the number of days of testing that will not be used
during the test administration process.
6. Work with the Miami Police Department and Department of Employee Relations to develop
one (1) book list for each examination and for each classification based upon a current Job
Analysis, if not already developed. The book lists must be the sole source of examination
material for each classification. Each list for each examination and for each classification must
be published a minimum of 90 days prior to the administration of the exam. Source materials
are to be compared with the job analysis data to ensure that all items, exercises, etc. are drawn
13
from included source materials and are representative of the job content. Parallel exercises
must also be developed by the Primary Consultant's firm or agency from the booklist.
7. Identify percentage of job domain measured by each customized examination process and for
each classification (e.g., simulation, role-play) within 15 days prior to the first day of the
administration of the exam. Identify area(s) of job domain not measured by each examination
process and identify appropriate measures for assessing that portion of job domain for the
classification. Reviews must be based on the current job analysis for the appropriate
classification.
8. Prepare and submit for approval to the Department of Employee Relations all scripts for all
appropriate exercises for all appropriate classifications in electronic format at least 21 calendar
days prior to the first examination date.
9. The Project Manager, or the individual as proposed and responsible to work directly with the
City on an as -needed basis throughout the life of the contract to perform the scope of services
and administer the examination(s) (identified herein as the Primary Consultant), and members
of the Consultant's firm or agency shall be present, complete with materials needed during the
duration of the test administration process for each exam, for each exercise and for each
classification. Consultant will be in possession of all materials needed for administration of
each exercise for each examination and for each classification. Location of test sites is the
responsibility of the City. City staff approved by the Department of Employee Relations may
be used as proctors for the examination process.
10. Primary Consultant's firm or agency, under the direct supervision of the Primary Consultant,
shall score each examination for each classification and generate descriptive statistics such as
means and standard deviations. Primary Consultant shall analyze the scores from each
examination process to determine the extent to which group means differ by ethnicity/race and
by gender for each classification, present said analyses to DER for review, and include said
analyses in the final report. Analyses to include, but not limited to, adverse impact, validity,
and exercise and examination reliability by sub -dimension and overall, item difficulty, item
descriptives, and item homogeneity, etc. where applicable.
11. Primary Consultant shall ensure that all scoring calculations and results are error free and
completed within 14 days after the last exam administration for each classification. Data entry
and scoring is to be completed by at least two different individuals working independently with
results matched for agreement. Any and all discrepancies must be resolved quickly in a fair
and unbiased manner.
12. Primary Consultant shall recommend a cutoff score within 21 days after the administration of
the exam for each customized examination process, for each classification and provide written
rationale for each recommendation given including the establishment of the cutoff score. All
cutoff scores recommended by the Consultant must be based on the results of an appropriate
validation study conducted according to professional standards. The Department of Employee
Relations reserves the right to review and approve the recommended cutoff point for each
examination process.
14
13. Primary Consultant or qualified consultant's representative shall provide an on -site orientation
program for each examination, for each classification to ensure that all candidates receive
uniform information for the applicable examination within 40 days prior to the administration
of the exam. The orientation will include written guidelines and rules that will govern each
exam administration of each examination process for each classification and a "Questions and
Answers" packet for each exam process for each classification. Each "Questions and
Answers" packet must be submitted by the Primary Consultant to the Department of Employee
Relations, Testing and Validation Section staff at least 30 days prior to each test administration
for each classification and will summarize the questions and issues raised at the on -site
orientation with acceptable responses to each question.
14. Consultant shall contact, screen, select, secure, obtain, train and ensure the presence of
individuals from the law enforcement community to serve as assessors/role players/raters for
each exercise, for each exam for each classification, where applicable, pending approval of the
Department of Employee Relations, Testing and Validation Section. All assessors will be
from (a) United States agencies, (b) comparable in size and/or selected demographics (e.g.,
ethnicity, race, gender) to the Miami Police Department, (c) outside of the State of Florida, and
(d) of the same rank or higher rank as the classification being tested. Every effort must be
made by the Primary Consultant to ensure that each exam process for each classification has
appropriate ethnic/racial and gender representation among assessors used in exercises for each
classification. The selection and training of potential assessors/role players/raters and the
ethnic/racial and gender background making up the assessor panel must be approved by the
Department of Employee Relations for each classification process a minimum of thirty (30)
days prior to each exam administration for each classification. The Department of Employee
Relations reserves the right to require that consultants use separate role players from that of
assessors for the Subordinate Counseling or Subordinate Conference exercise. Assessors may
serve as a role player for other exercises (e.g., Citizen Meeting). A minimum of three (3)
assessors must participate in training and assessment for all exercises with the exception of the
Letter or Memo Writing, Simulation, or other DER approved exercise(s) for which a minimum
of two (2) assessors must participate in training and assessment. The City reserves the right to
require that the Primary Consultant contact, screen, select, secure, obtain, train and ensure the
presence of one additional assessor and/or role player and/or rater for each examination and for
each classification to serve as an alternate should a primary assessor and/or role player and/or
rater be unable to carry out their duties as an assessor and/or a role player and/or a rater,
respectively. Primary consultant will develop, photocopy, collate, and otherwise prepare and
administer assessors/role players/raters training materials and administer training for
assessors/role players/raters. The City reserves the right to have representation present from
the Department of Employee Relations at any and all processes that involve assessors,
including but not limited to orientations and/or training sessions and/or selection proceedings
and/or assessment processes and/or scoring. The same number of assessors/role players/raters
must be used to assess everyone participating in a specific exercise requiring assessors/role
players/raters for each classification, respectively and as applicable. Candidates will be scored
by the same assessors/role players/raters for each exercise of each exam for each classification
barring any unforeseen circumstances where an assessor becomes i11, or for other reasons
cannot participate in the entire rating process.
15
15. The Primary Consultant shall provide candidates with proper feedback to include a written
profile of candidate's performance from their examination processes, and a plan for improving
performance that shall be developed from candidate score(s). This feedback must be specific
to their performance in each exercise and the scoring guidelines for each exercise, in order for
the candidates to learn what could have been said/done that would have improved his or her
score(s); this will be provided so that candidates can improve on their performance on future
examinations. If a hurdle examination process is administered, only those who competed in
the entire process shall receive written feedback within 45 days after the exam administration.
16. The Primary Consultant shall provide evidence (e.g., statistics) supporting the content validity
and reliability of each exam for each classification in a separate final report to the Department
of Employee Relations, Testing and Validation Section. Actual scores for each candidate
including, but not limited to raw, standardized, and computed, must be provided to the City
for each candidate for each exercise for each exam in each classification within 14 days
following completion of the exam administration. In addition, the Primary Consultant shall
provide recommendations in the final report to assist the City with preparing examination
candidates for future processes. This report should include sufficient information to assist the
City in developing future training programs for promotional candidates.
17. The Primary Consultant shall establish the content validity for each exam for each
classification by documenting the links between the Job Analysis and examination process and
exercises in accordance with the Federal "Uniform Guidelines on Employee Selection
Procedures" and current professional standards including, but not limited to, the Society for
Industrial and Organizational Psychology "Principles for the Validation and Use of Personnel
Selection Procedures", and/or American Educational Research Association/American
Psychological Association/National Council on Measurement in Education "Standards for
Educational and Psychological Testing". This information shall be included in each final
report along with documentation of exam development, administration, scoring for all exams,
exercises, components, etc. The final report shall also include a detailed explanation of any
and all controls and safeguards used to eliminate rater errors or other biases and ensure
accurate analytical interpretation and completeness and all documents provided to candidates
and assessors, role players and raters (e.g., study/prep guide, Questions and Answer Packet,
assessor/role player/rater training manuals, assessor/role player/rater rating materials).
18. The Primary Consultant or qualified individual from the Consultant's firm or agency shall
provide three (8 hours) days of post-test consultation inclusive in proposed fees, as well as
maintaining fixed and firm fees throughout the initial term of the contract for the
Specifications/Scope of Work cited herein, including any additional work as necessary. The
Primary Consultant shall, as part of the Contract, agree to appear on the City's behalf as an
expert witness in the event of any litigation arising from any of the examination processes for
each classification. The Primary Consultant agrees to appear for a total time of three (3) days
up to eight (8) hours per day without any additional financial costs to the City. The Consultant
shall be responsible for transportation, lodging, and meals. After the three (3) days has been
utilized, the Primary Consultant shall be allowed to charge the City his or her reasonable daily
rate at time of Contract for any additional days for the services relating to litigation and/or
appearing as an expert witness; said appearance and resulting costs have to be approved by the
16
City in advance of travel. The Consultant shall be responsible for travel time, transportation,
lodging and meals during the first three (3) days of post-test consultation. Reasonable travel
expenses, a rental car allowance, hotel accommodations, and a meal allowance for appearances
arising from litigation shall be paid to the Primary Consultant by the City as additional costs
upon approval by the City and in accordance with §112.061, Florida Statutes.
19. The Primary Consultant shall propose a fixed and firm lock rate for all exams within the time
period of the contract.
20. The Primary Consultant shall furnish all equipment, materials, supervision or supplies
necessary for completion of all the services specified in the Solicitation Documents. Failure to
possess the necessary materials and equipment in order to satisfactorily perform the services
may result in the default/termination of contract and/or postponement of test. Consequently,
Primary Consultant must develop, validate, administer, etc. a new examination with all
applicable exercises at no cost to the City within an expedient time frame. Should a re -
administration occur as a result of default herein, the Primary Consultant agrees to waive any
and all fees related to Item #18 and Item #19 regarding fee charges beyond the three days of
service for expert testimony and/or litigation arising as a result of the re-examination for each
classification.
21. All work shall be performed by, or under the direct supervision of, the Primary Consultant. All
associates used on each project must have a minimum of a Masters degree in Industrial and
Organizational Psychology or closely related field. All work performed must conform to the
ethical and professional guidelines, principles, and standards outline in the Federal "Uniform
Guidelines on Employee Selection Procedures" and current professional standards including,
but not limited to, the Society for Industrial and Organizational Psychology "Principles for the
Validation and Use of Personnel Selection Procedures", and/or American Educational
Research Association/American Psychological Association/National Council on Measurement
in Education "Standards for Educational and Psychological Testing".
22. Should the Primary Consultant leave the employ of the Successful Proposer during the initial
and extension term(s) of the contract, as applicable, the City reserves the right to approve, or
not approve, its replacement. In addition, the City reserves the right to seek a replacement of
the Primary Consultant, should the City determine he/she is not performing in accordance with
the terms and conditions of the scope of work/services.
17