HomeMy WebLinkAboutExhibit 2ATTACHMENT "A"
2.2 TEST DEVELOPMENT REQUIREMENTS (From RFP 04-05-121)
The Consultant shall:
l Abide by timetables provided by the City outlined herein over the life of the contract and for
each classification, with the understanding that timetables are estimates only and subject to
change. The City shall provide reasonable written notice of any changes to specified timetable.
2. Conduct an initial and legally defensible iob-analysis according to current professional and
legal standards within 60 calendar days after the execution of the Agreement for each the
classification of Chief Fire Officer, within 90 days after the execution of the Agreement for the
classification of Fire Captain and within 120 days after the execution of the Agreement for the
classification of Fire Lieutenant.
3. Develop a written examination that is appropriate to directly measure the knowledge, skills,
abilities, and other characteristics (KSAOs) identified in the job analysis(es) as necessary to
perform the job functions within 180 calendar days of the completion of the job analysis(es)
for each of the classifications of Chief Fire Officer and Fire Captain. Develop no less than
three (3) examination exercises for each examination and for each classification that is
appropriate to directly measure the knowledge, skills, abilities, and other characteristics
(KSAOs) that are identified in the job analysis(es) within 180 calendar days of the completion
of the job analysis(es) necessary to -perform the job functions for the classifications of Chief
Fire Officer and Fire Captain. Consultant will have input from subject matter experts (SME)
from the Department of Fire -Rescue including final review of exercises. The Department of
Employee Relations (DER) will also have input into the development of examination measures
and must approve exercises after Fire -Rescue Department SME approval. Each examination
process must have a preparation manual/study guide published and delivered to DER in
electronic format by the test Consultant, with sample exercises, at least 60 days prior to the
administration of the examination. All exercises must require responses which are recorded by
the candidate on paper and/or verbally to live assessors.
4. Develop no less than three (3) written examinations for each examination that are appropriate
to directly measure the knowledge, skills, abilities, and other characteristics (KSAOs) that are
identified in the job analysis within 180 calendar days of the completion of the job analysis
necessary to perform the job functions for the classification of Fire Lieutenant. The written
examinations shall assess the job knowledge of candidates and will be based on the sources
identified by the book committee comprised of an equal number representatives from Fire -
Rescue Management and the Bargaining Unit. Consultant will have input from subject matter
experts (SME) from the Department of Fire -Rescue including final review of exercises. The
Department of Employee Relations (DER) will also have input into the development of
examination measures and must approve exercises after Fire -Rescue Department SME
approval. Each examination process must have a preparation manual/study guide published
and delivered to DER in electronic format by the test Consultant, with sample exercises, at
least 60 days prior to the administration of the examination. All exercises must require
13
responses which are recorded by the candidate on paper in multiple-choice and/or matching
and/or scenario format(s) with at least four (4) reasonable alternative answers. Exam questions
from dosed -book reading list sources are to be verbatim with source name and page number,
listed.
5. Develop a parallel form for each exam exercise for each exam for each classification for each
day of testing that exceeds a one (1) day administration period for the classifications of Chief
Fire Officer and Fire Captain. In addition, develop one (1) extra exam form for each and every
exercise beyond the number of days of testing that will not be used during the test
administration process. Develop a parallel form for each written exam for each day of testing
that exceeds a one (1) day administration period for the classifications of Fire Lieutenant. In
addition and for the classification Fire Lieutenant, develop one (1) extra exam form for each
and every administered written examination beyond the number of days of testing. Extra exam
form for each and every administered written examination for the classification of Fire
Lieutenant shall include the same number, and type, of questions each of which must be unique
(or different) from the first examination. Provide written rationale to the Department of
Employee Relations, Testing and Validation Section for each recommendation as to the
number of additional job knowledge questions generated for each written exam and exam
exercise also to be included in the Final Report.
6. Work with the Department of Fire -Rescue and Department of Employee Relations to develop
one (1) book list for each examination and for each classification based upon a current Job
Analysis, if not already developed. The book lists must be the sole source of examination
material-for-eachrclassification. Each -list for each examination and foreach classification must
be published a minimum of 90 days prior to the administration of the exam. Source materials
are to be compared with the job analysis data to ensure that all items, exercises, etc. are drawn
from included source materials and are representative of the job content. Parallel exercises
must also be developed by the Primary Consultant's firm or agency from the booklist.
7. Identify percentage of job domain measured by each customized examination process and for
each classification (e.g., simulation, role-play) within 15 days prior to the first day of the
administration of the exam. Identify area(s) of job domain not measured by eachexamination
process and identify appropriate measures for assessing that portion of job domain for the
classification. Reviews must be based on the current. job .analysis for the appropriate
classification.
8. Prepare and submit .for approval to the Department of Employee Relations all scripts for all
appropriate exercises for all appropriate classifications in electronic format at least 21 calendar
days prior to the first examination date.
9. The Project Manager, or the individual as proposed and responsible to work directly with the
City on an as -needed basis throughout the life of the contract to perform the scope of services
and administer the examination(s) (identified herein as the Primary Consultant), and at least
one additional member from the Primary Consultant's firm or agency shall be present for all
days and exercises of the on -site test administration for each exercise, for each examination
process and for each classification. Both the Primary Consultant and the one additional
member from the Primary Consultant's firm or agency shall have a minimum of a Masters
14
degree in Industrial and Organizational Psychology or closely related field. The number of
staff from the Consultant's firm or agency shall be appropriate in a number commensurate with
the logistical needs of each situation. The Project Manager will be in possession of all
materials needed during the duration of the test administration of each exercise for each
examination for each classification. Location of test sites is the responsibility of the City. City
staff approved by the Department of Employee Relations may be used as proctors for the
examination process. The number of City staff approved by the Department of Employee
Relations may be used as proctors for the examination process will vary with a minimum of
four (4) City staff members available for each exercise, for each examination process and for
each classification.
10, Primary Consultant's firm or agency, under the direct supervision of the Primary Consultant,
shall score each examination for each classification and generate descriptive statistics such as
means and standard deviations. Primary Consultant shall analyze the scores from each
examination process to determine the extent to which group means differ by ethnicity/race and
by gender for each classification, present said analyses to DER for review, and include said
analyses in the final report. Analyses to include, but not limited to, adverse impact, validity,
and exercise and examination reliability by sub -dimension and overall, item difficulty, item
descriptives, and item homogeneity, etc. where applicable.
11. Primary Consultant shall ensure that all scoring calculations and results are error free and
completed within 14 days after the last exam administration for each classification. Data entry
and scoring is to be completed by at least two different individuals working independently with
results matched for agreement. Any and all discrepancies in data -entry, scoring calculations,
and scoring results shall be resolved quickly in a fair,; and unbiased manner by the Primary
Consultant who shall have a minimum of a Masters degree in Industrial and Organizational
Psychology or closely related field.
12. Primary Consultant shall recommend a cutoff score within 21 days after the administration of
the exam for each customized examination process, for each classification and provide written
rationale for each recommendation given including the establishment of the cutoff score. All
cutoffscores recommended by the Consultant must be based on the results of an appropriate
validation study conducted according to professional standards. The Department of Employee
Relations reserves the right to review and approve the recommended cutoff point for each
examination process.
13. Primary Consultant or qualified consultant's representative shall provide an on -site orientation
program for each examination, for each classification to ensure that all candidates receive
uniform information for the applicable examination within 40 days prior to the administration
of the exam. The orientation will include written guidelines and rules that will govern each
exam administration of each examination process for each classification and a "Questions and
Answers" packet for each exam process for each classification. Each "Questions and
Answers" packet must be submitted by the Primary Consultant to the Department of Employee
Relations, Testing and Validation Section staff at least 30 days prior to each test administration
for each classification and will summarize the questions and issues raised at the on -site
orientation with acceptable responses to each question.
15
14. Consultant shall contact, screen, select, secure, obtain, train and ensure the presence of
individuals from the Fire -Rescue community to serve as assessors/role players/raters for each
exercise, for each exam for each classification, where applicable, pending approval of the
Department of Employee Relations, Testing and Validation Section. All assessors will be
from (a) United States .agencies, (b) comparable in size and/or selected demographics (e.g.,
ethnicity, race, gender) to the Department of Fire -Rescue, (c) outside of the State of Florida,
and (d) of the same rank or higher rank as the classification being tested. Every effort must be
made by the Primary Consultant to ensure that each exam process for each classification has
appropriate ethnic/racial and gender representation among assessors used in exercises for each.
classification. The selection and training of potential assessors/role players/raters and the
ethnic/racial and gender background making up the assessor panel must be approved by the
Department of Employee Relations for each classification process a minimum of thirty (30)
days prior to each exam administration for each classification. The Department of Employee
Relations reserves the right to require that consultants use separate role players from that of
assessors. It is preferred that individuals serve as either an assessor or a role player but not in
both capacities. A minimum of three (3) assessors must participate in training and assessment
for all exercises with the exception of the Letter or Memo Writing, Simulation, or other DER
approved exercise(s) for which a minimum of two (2) assessors must participate in training and
assessment. The City reserves the right to require that the Primary Consultant contact, screen,
select, secure, obtain, train and ensure the presence of an additional assessor and/or role player
and/or rater for each exercise for each exam for each classification to serve as an alternate
should a primary assessor and/or role player and/or rater be unable to carry out their duties as
..an assessor and/or. a .role player and/or a. rater, .respectively.. Primary consultant will develop,
photocopy, collate, and otherwise prepare and administer assessors/role players/raters training
materials and administer training for assessors/role players/raters. The City reserves the right
to have representation present from the Department of Employee Relations at any and all
processes,_ that involve assessors, including _ but not limited to orientations and/or training
sessions and/or selection proceedings and/or assessment processes and/or scoring. The same
number of assessors/role players/raters must be used to assess everyone participating in a
specific exercise requiring assessors/role players/raters for each classification, respectively and
as applicable. Candidates will be scored by the same assessors/role players/raters for each
exercise of each exam for each classification. The cost of one alternate assessor for each
unique exercise and a sufficient number of role players shall be included in the pricing, should
the Proposer's submission require the use of assessors and/or role players.
1.5..__The_Pximary_ Consultant _shall_provide candidatesWith. Qper _feedback to include a written
profile of candidate's performance from their examination processes, and a plan for improving
performance that shall be developed from candidate score(s) for each exam and for the
classifications of Chief Fire Officer and Fire Captain. This feedback must be specific to the
performance in each exercise and the scoring guidelines for each exercise, in order for the
candidates to learn what could have been said/done that would have improved his or her
score(s); this will be provided so that candidates can improve on their performance on future
examinations.
16. The Primary Consultant shall provide evidence (e.g., statistics) supporting the content validity
and reliability of each exam for each classification in a separate final report to the Department
16
of Employee Relations, Testing and Validation. Section. Actual scores for each candidate
including, but not limited to raw, converted and computed, must be provided to the City for
each candidate for each exercise for each exam in each classification within 14 days following
completion of the exam administration with consultation with the Department of Employee
Relations.. In addition, the Primary Consultant shall provide recommendations in the final'
report for each exam for each classification to assist the City with preparing examination
candidates for future processes. This report should include sufficient information to assist the
City in developing future training programs for promotional candidates after for each exam for
each classification.
17. The Primary Consultant shall establish the content validity for each exam for each
classification by documenting the links between the Job Analysis and examination process and
exercises in accordance with the Federal "Uniform Guidelines. on Employee Selection
Procedures" and current professional standards including, but not limited to, the Society for
Industrial and Organizational Psychology "Principles for the Validation and Use of Personnel
Selection Procedures", and/or American Educational Research Association/American
Psychological Association/National Council on Measurement in Education "Standards for
Educational and Psychological Testing". This information shall be included in each final
report along with documentation of exam development, administration, scoring for all exams,
exercises, components, etc. The final report shall also include a detailed explanation of any
and all controls and safeguards used to eliminate rater errors or other .biases and ensure
accurate analytical interpretation and completeness and all documents provided to candidates
and assessors, role players and raters (e.g., study/prep guide, Questions and Answer Packet,
assessor/role player/rater training manuals, assessor/role player/rater rating materials).
18. The Primary Consultant or qualified individual from the Consultant's firm or agency shall
provide... three (8 hours) .days of post-test consultation inclusive in proposed fees, as well as
maintaining fixed and firm fees throughout the initial term of the contract for the
Specifications/Scope of Work cited herein, including any additional work as necessary. The
Primary Consultant shall, as part of the Contract, agree to appear on the City's behalf as an
expert witness in the event of any litigation arising from any of the examination processes for
each classification. The Primary Consultant agrees to appear for a total time of three (3) days
up to eight (8) hours per day without any additional financial costs to the City. The Primary
Consultant shall be responsible for transportation, lodging, and meals. After the three (3) days
has been utilized, the Primary Consultant shall be allowed to charge the City his or her
reasonable daily rate at time of Contract for any additional days for the services relating to
litigation and/or appearing as an expert witness; said appearance and resulting costs have to be
approved by the City in advance of travel. The consultant shall be responsible for travel time,
transportation, lodging and meals during the first three (3) days of post-test consultation.
Reasonable travel expenses, a rental car .allowance, hotel accommodations, and a meal
allowance for appearance(s) arising from litigation shall be paid to the Primary Consultant by
the City as additional costs upon approval by the City and in accordance with §112.061,
Florida Statutes.
19. The Primary Consultant shall propose a fixed and firm lock rate for all exams within the time
period of the contract.
17
20. The Primary Consultant shall furnish all equipment, materials, supervision or supplies
necessary for completion of all the services specified in the Solicitation Documents. Failure to
possess the necessary materials and equipment in order to satisfactorily perform the services
may result in the default/termination of contract and/or postponement of test. Consequently,
the Primary .Consultant must develop, validate, administer, etc. 'a new examination with all
applicable exercises at no cost to the City within an expedient time frame. Should a re -
administration occur as a result of default herein, the Primary Consultant agrees to waive any
and all fees related to Item 418 and Item 419 regarding fee charges beyond the three days of
service for expert testimony and/or litigation arising as a result of the re-examination for each.
classification.
21. All work shall be performed by, or under the direct supervision of, the Primary Consultant. All
associates used on each project are to have a minimum of a Masters degree in Industrial and
Organizational Psychology or closely related field. Work of a clerical nature or involving data -
entry shall not require associates with a minimum of a Masters degree in Industrial and
Organizational Psychology or closely related field. Scoring calculations and results shall be
ensured by the Primary Consultant who shall have a minimum of a Masters degree in Industrial
and Organizational Psychology or closely related field. All work performed must conform to
the ethical and professional guidelines, principles, and standards outline in the Federal
"Uniform Guidelines on Employee Selection Procedures" and current professional standards
including, but not limited to, the Society for Industrial and Organizational Psychology
"Principles for the Validation and Use of Personnel Selection Procedures", and/or American
Educational Research., Association/American Psychological Association/National Council on
--Measurement--in Education `°Standar-ds for -Educational -and -Psychological Testing".
22- Should- he-P-�imary-Consultannt leave--the-employ of -the -Successful -Proposer during the initial
-and extension term(s) of the contract, as applicable, the. City reserves the right to approve, or
not approve, its replacement. In addition, the City reserves the right to seek a replacement of
the Primary Consultant, should the City determine he/she is not performing in accordance with
the terms and conditions of the scope of work/services.
18