HomeMy WebLinkAboutEB Jacobs' Proposal�.�. (1GICD,LS)
Prepe,s-A
July 11, 2012
VIA EMAIL (krobertson@miamigov.com) Mr. Kenneth Robertson,
Director / Chief Procurement Officer, City of Miami, Florida
City of Miami - Purchasing Department
Miami Riverside Center
444 S.W. 2nd Avenue - 6th Floor
Miami, FL 33130
Re: FOIA Request for EB Jacobs LLC Proposal for City of Miami RFP 316283
Dear Mr. Robertson:
a m
c 0 4
We are in receipt of a communication from the City of Miami's Purchasing Department informing EB Jacobs that a
Freedom of Information (FOIA) request has been made for our recent proposal for fire testing services for the City of
Miami for the positions of Fire Lieutenant, Captain, and Chief Fire Officer (RFP 316283). We would like to request
that all materials that involve test materials or trade secrets in our proposal be exempted from disclosure.
With regard to exempting trade secrets we reference statute 119.071(1)(f) which states "Data processing software
obtained by an agency under a licensing agreement that prohibits its disclosure and which software is a trade secret,
as defined in s. 812.081, and agency -produced data processing software that is sensitive are exempt from s.
119.07(1) and s. 24(a), Art. I of the State Constitution. The designation of agency -produced software as sensitive
shall not prohibit an agency head from sharing or exchanging such software with another public agency."
In addition, we reference the Florida Senate Government — Oct. 2008: General Public Records Exemption for Trade
Secrets, Issue Brief by the Florida Senate 2009-325, which clarifies the Florida Senate's intent and supports the
exclusion of trade secrets from the public record and clarifies the applicability of existing state statutes in the
protection of trade secrets with respect to FOIA requests. In this Florida Senate Issue Brief for public record
requests, a trade secret is defined by two different existing statutes. The first definition is in statute 688.002(4)
and defines trade secret as "...information, including a formula, pattern, compilation, program, device, method,
technique, or process that: a) Derives independent economic value, actual or potential, from not being generally
known to, and not being readily ascertainable by proper means by, other persons who can obtain economic value
from its disclosure or use; and (b) Is the subject of efforts that are reasonable under the circumstances to maintain
its secrecy.
The second definition is found in statute 812.081(1)(c) and defines trade secret to mean "... the whole or any
portion or phase of any formula, pattern, device, combination of devices, or compilation of information which is for
use, or is used, in the operation of a business and which provides the business an advantage, or an opportunity to
obtain an advantage, over those who do not know or use it. "Trade secret" includes any scientific, technical, or
commercial information, including any design, process, procedure, list of suppliers, list of customers, business code,
or improvement thereof."
Statute 815.045 also supports the concept that trade secrets should be protected to be compliant with the rules
public organizations must follow and to protect the economic interests of private organizations:
The Legislature in making disclosure of trade secrets a crime has clearly established the importance attached
to trade secret protection. Disclosing trade secrets in an agency's possession would negatively impact the
business interests of those providing an agency such trade secrets by damaging them in the marketplace,
and those entities and individuals disclosing such trade secrets would hesitate to cooperate with that
agency, which would impair the effective and efficient administration of governmental functions. Thus, the
public and private harm in disclosing trade secrets significantly outweighs any public benefit derived from
disclosure, and the public's ability to scrutinize and monitor agency action is not diminished by
nondisclosure of trade secrets.
Our City of Miami fire testing proposal contains the following materials associated with the proven processes,
product templates, client information, project information, pricing information that we have developed and
validated as part of our promotional public safety selection criteria and we request that they be excluded from the
public records submission by the City. This information is not available on our web site or on any publically available
materials. The materials represent a competitive advantage to EB Jacobs and are released only to our prospective
clients.
• Detailed staffing plan which contains detailed information about our staff (Proposal Text, Section II, E)
• Current and previous clients which can be targeted (Proposal Text, Section II, G)
• Recent clients and projects for primary consultants which includes prices for other clients (Proposal Text,
Section III, C)
• Cut scores which contains our proprietary approach (Proposal Text, Section IV, A3)
• Adverse impact reduction which contains our proprietary approach (Proposal Text, Section IV, A11)
• Information about our past and current clients that we release only to prospective clients (Appendix A)
• Detailed information about our employees that we release only to prospective clients (Appendix B)
• Client contact information that we release only to prospective clients (Appendix C)
• Detailed reference letters from our clients that we release only to prospective clients (Appendix D)
• Detailed project experience that we release only to prospective clients (Appendix E)
• Sample test materials/exercises we provide only to prospective clients: Appendix 1, Appendix J,
Appendix K. Appendix L, Appendix M, and Appendix N
• Detailed feedback report that we release only to prospective clients and candidates (Appendix P)
• Detailed litigation experience that we release only to prospective clients (Appendix S)
• Cost proposal details we provide only to prospective clients (Cost Proposal)
The information listed above represents and reflects our trade secrets, intellectual property, and competitive
advantage. This information is only presented to others who are interested in acquiring our services in order to
convey the quality and nature of the services and products we provide so that they can make a fully informed
decision.
A fundamental tenet of all testing is that every candidate is provided the same information about the test at the
same time. Since our testing processes follow a specific framework, revealing the details of the framework
through the release of these materials to a commercial organization that would consolidate and sell this information
to those candidates that would pay it a fee, would only serve to provide an advantage to those candidates that are
willing to pay for the firm's service. This is a concern for all testing firms and should also be a concern for all
organizations that undertake promotion testing processes for their employees.
To the extent that any of the materials listed above convey details about our proprietary examination processes,
they should be exempt from public disclosure to avoid any perception of unfairness in terms of the nature and
timing of access to exam -related information. This applies not only for candidates in the City of Miami but for all
candidates tested using EB Jacobs processes across the United States. We believe this is a critical factor that
should be considered whenever open records requests of this nature arise. When information is released to the
general public, it is impossible to maintain control over who gains access and when that access occurs. We have no
guarantee that the individual or entity requesting our proposals and reports shares our concerns for test security
and fairness and understands the important role that perceptions can play in the public safety testing arena.
Our proposals also contain information concerning our clients and personnel that we feel should also be exempt
from disclosure. We provided this information to the City of Miami so that they could contact any clients referenced
and evaluate the credentials of our staff. We believe this was the best way for the City to find out about our record
of performance as well as our capabilities as they related to our proposal. If this information were to be made
available to competitors they would have the opportunity to contact any and all of the clients or personnel listed.
We feel this would be inconvenient to our past and present clients and employees. Our clients and personnel give us
permission to provide their information as references in our proposals but not to provide access to others who
might solicit their business or employment.
For the above reasons, we seek to have the above listed information from our proposals exempted from being
provided under the public records request. None of this information is publicly available in this level of detail on
our website or through any other public mechanism.
For all of the above reasons, we urge the City to deny disclosure of any information contained within the sections of
our proposals that reveal test material or trade secrets.
Janet Echemendia can be reached at Janet.Echemendia@eblacobs.com or 814-237-5997 if you have any questions
or need further information. Jay Silva can be reached at Jay.Silva@eblacobs.com or 352-332-9708.
Sincerely,
Janet M. Echemendia
President
Jay M. Silva
Managing Senior Consultant
June 11, 2012
Ms: Maritza Suarez
City of Miami — City Clerk
3500 Pan American Drive
Miami, FL 33133
Re: Proposal in Response to RFP 316283
Request for Consulting Services for Request for Proposals for Chief Fire Officer, Capt. & Lt.
Examination Processes
Dear Ms. Suarez:
EB Jacobs, LLC is very pleased to respond to your Request for Proposals for the development and
administration of promotional exams for the positions of Chief Fire Officer, Captain, and Lieutenant in
the City of Miami. EB Jacobs has supported the City and the Department of Fire -Rescue's
promotional examination needs for the past six years and hopes to continue and expand upon the
effective collaborative relationship that has developed during this period. To this end, the current
cost proposal is slightly below the cost of the fire promotional testing services we provided in 2011-
2012 (adjusting for the fact that in 2011-2012 work did not include job analyses for the positions).
This proposal represents a comprehensive solution for your fire -rescue promotional testing needs,
one that builds on what you have specified in the RFP and our experience in testing in the fire
service.
EB Jacobs specializes in the development, validation and defense of human resource programs for
public safety agencies. We excel in identifying the top performers within candidate pools while
minimizing potential adverse effects on members of protected classes. For nearly 30 years our staff
of industrial/organizational psychologists, statisticians, exercise physiologists, educational
psychologists, and clinical psychologists has brought their combined expertise to hundreds of
contracts in municipal, county, state, and federal agencies across the country. We believe that your
review of our proposal will clearly convey the breadth and depth of our expertise and experience in
the realm of public safety assessment. In addition to relying on your own experience, we encourage
you to contact our other references for further insight into the quality of our work.
As you review our proposal, we would like you to keep in mind that it is just that — a proposal. We
welcome the opportunity to address the City and the Department of Fire -Rescue to answer any
questions arising from our proposal and/or discuss possible modifications that would better suitthe
needs of the City and Department. We are available to discuss our proposal and ideas via an
interview or conference call should you desire this as part of your decision process.
Please accept this letter as official notification that I will serve as the primary contact for negotiations.
For more information on our organization and services, please access our website at
http://www.ebjacobs.com. Please feel free to contact me if any questions arise at (800) 367-5214
(phone), 814-237-4886 (fax), or at Janet.Echemendia@ebjacobs.com.
Respectfully,
net M. Echemendia, Ph.D.
resident
300 South Burrowes Street • State College, PA 16801
phone: 814.237.5997 • 800.367.5214 • fax: 814.237.4886
web: www.ebjacobs.com
•
Proposal in Response to Request for Proposal (RFP) No. 316283
Request for Proposals for Chief Fire Officer, Capt. & Lt. Examination
Processes
EB Jacobs acknowledges receipt of two RFP 316283 Addenda. Addendum 1 was received May 30,
2012. Addendum 2 was received on June 7, 2012. All information in the addenda was considered in our
response.
Confidentiality Statement:
This proposal includes data that shall not be disclosed outside of the proposal review
process of the City of Miami Government, Florida (the City) and shall not be duplicated,
used or disclosed, in whole or in part, for any purpose except for use in the
procurement process, and as mandated by applicable law. If, however, a contract is
awarded to EB Jacobs as a result of, or in connection with, the submission of these
data, the City shall have the right to duplicate, use or disclose the data to the extent
consistent with the City's needs in the procurement process. This restriction does not
limit the City's right to use information found in this document, if it is obtained from
another source. The data subject to this restriction are contained on every page of the
proposal and related appendix material.
Submitted by:
EB Jacobs, LLC
300 South Burrowes Street
State College, PA 16801
(800) 367-5214
Closing Date: June 13, 2012
Authorized Signature:
net M. Eche(vnendia, President
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified In the Confidentiality Statement of this proposal,
Certification Statement
Please quote on this form, if applicable, net prices for the ite n(s) listed. Return signed original and
retain a copy for your files. Prices should include all costs, including transportation to destination. The
City reserves the right to accept or reject all or any part of this submission. Prices should be firm for a
minimum of 180 days following the time set for closing of the submissions.
In the event of errors in extension of totals, the unit prices shall govern in determining the quoted
prices.
We (I) certify that we have read your solicitation, completed the necessary documents, and propose to
furnish and deliver, F.O.B. DESTINATION, the items or services specified herein.
The undersigned hereby certifies that neither the contractual party nor any of its principal owners or
personnel have been convicted of any of the violations, or debarred or suspended as set in section
18-107 or Ordinance No. 12271.
All exceptions to this submission have been documented in the section below (refer to paragraph and
section).
EXCEPTIONS:
We (I) certify that any and all information contained in this submission is true; and we (I) further certify
that this submission is made without prior understanding, agreement, or connection with any
corporation, firm, or person submitting a submission for the same materials, supplies, equipment, or
service, and is in all respects fair and without collusion or fraud. We (I) agree to abide by all terms and
conditions of this solicitation and certify that I am authorized to sign this submission for the submitter.
Please -print the following and sign your name:
SUPPLIER NAME' EB Jacobs LLC
ADDRESS' 300 South Burrowes Street
PHONE: (800) 367-5214 FAX (814) 237-4886
EMAIL, j anet . echemendia@ebj acobs . corn BEEPER'
SIGNED BY.
TITLE' Pr
"s ident
DATE' June 11, 2012
FAILURE TO COMPLETE. SIGN. AND RETURN i'HIS FORM SHALL DISQUALIFY THIS BID.
Page 2 of 48
Certifications
Legal Name of Firm:
EB Jacobs, LLC
Entity Type: Partnership, Sole Proprietorship, Corporation, etc.
Limited Liability Corporation
Year Established:
2004
Office Location: City of Miami, Miami -Dade County, or Other
State College, PA, Gainesville, FL, and Seattle, WA
Occupational License Number:
Not Applicable
Occupational License Issuing Agency:
Not Applicable
Occupational License Expiration Date:
Not Applicable
Respondent certifies that (s) he ha a read and understood the provisions of City of Miami Ordinance No.
10032 (Section 18-110 of the City Code) pertaining to the implementation of a "First Source Hiring
Agreement.": (Yes or No)
Yes
Do you expect to create new positions in your company in the event your company was awarded a
Contract by the City? (Yes or No)
No
In the event your answer to question above is yes, how many new positions would you create to
perform this work?
None
Please list the title, rate of pay, summary of duties, number of positions, and expected length or duration
of all new positions which might be created as a result of this award of a Contract.
Not applicable
Will Subcontractor(s) be used? (Yes or No)
No
Page 3 of 48
Line: 1
Description: Disregard this line item. Please refer to the Submission Requirements, Section 4.1.,
12. Price Proposal
Category: 91885-00
Unit of Measure: Dollar
Unit Price: $ Number of Units: 1 Total: $
Page 4 of 48
TABLE OF CONTENTS
Completed City Documents Inserted Prior to the Table of Contents or with Cost Proposal
1. EXECUTIVE SUMMARY: Supporting the City of Miami's Objectives 1
1I. PROPOSER'S QUALIFICATIONS: Experience and Qualifications 2
A. Organization 3
B. Incorporation 4
C. Organization Description and Previous Projects With the City of Miami 4
D. List of Staff to Use on the City's Project 6
E. Detailed Staffing Plan 6
F. Locations Where Work Will Be Performed 11
G. Current and Previous Clients 11
H. Letters of Reference 11
1. Lawsuits 11
J. Sub -Consultants 11
HI. PRIMARY CONSULTANT'S EXPERIENCE 12
A. Experience 12
B. Qualifications of Lead Consultants 12
C. Recent Clients and Projects for Primary Consultants 13
D. Primary Consultant Expert Witness Experience 15
E. Lawsuits 15
V. DESCRIPTION OF PROPOSED TESTING PROCESSES 16
A. Exam Processes 16
1. Job Analysis Process 17
2. Exam Plan Specification and Exam Development 18
a. Written ln-Basket. Exercise 20
b. Open and Closed -Book Technical Knowledge Tests 20
c. Situational Judgement Test 21
d. Assessment Center 21
1. Subordinate Conference 22
2. incident Command 22
3. Cut Scores 23
4. Assessors 24
5. Parallel Forms 25
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
ii
TABLE OF CONTENTS
(continued)
6. Exam Review 26
7. Candidate Preparation 26
8. Exam Administration 29
9. Exam Security, Scoring, Appeals, and Feedback 29
a. Exam Security 29
b. Exam Scoring 3031
c. Appeals
d. Candidate Feedback Report Preparation 31
10. Utilization of Subject Matter. Experts for Exam Development 32
11. Adverse Impact Reduction 32
12. Reports 33
B. Validity Process 33
C. Project Staffing 34
V. IMPLEMENTATION STRATEGY 36
A. Exam Administration 36
B. Project Resource Requirements 36
C. Project Staffing 40
VI. ABILITY TO PERFORM WORK WITHIN THE CONTRACT TIME FRAME... 42
A. Schedule 42
B. Concluding Statement 42
VII. WORK SAMPLES. 43
VII1. LOCAL PREFERENCE 0 44
IX. TRADE SECRET EXECUTION TO PUBLIC RECORDS DISCLOSURE 45
X. COST/FEE PROPOSAL See Sealed Envelope Containing Cost/Fee Proposal
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
Appendix A:
Appendix B:
Appendix C:
Appendix D:
Appendix E:
Appendix F:
Appendix G:
Appendix H:
Appendix I:
Appendix J:
Appendix K:
Appendix L:
Appendix M:
Appendix N:
Appendix 0:
Appendix P:
Appendix Q:
Appendix R:
Appendix S:
APPENDICES
Project List
Resumes
References
Letters of Recommendation
Similar Project Experience
Sample Excerpt from a Task Analysis
Sample Excerpt from an Ability Analysis
Sample Excerpt from a Knowledge Analysis
Sample Written In -Basket Exercise •
Sample Open -Book Technical Knowledge Test
Sample Closed -Book Technical Knowledge Test
Sample Situational Judgement Test
Sample Subordinate Conference Role -Play Exercise
Sample Incident Command Exercise
Sample Table of Contents for Candidate Preparation Guide
Sample Candidate Feedback
Project Staffing Plan
Project Timeline
Litigation Experience
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
I. EXECUTIVE SUMMARY: Supporting the City of Miami's Objectives
We are pleased to provide a proposal in response to the City of Miami's (City) request for the
development and administration of promotional examination processes for Lieutenant, Captain, and
Chief Fire Officer within the City of Miami's Department of Fire -Rescue. This proposal summarizes our
cost effective, administratively efficient approach to meeting the City's needs and represents our
interest in continuing and expanding our relationship with the City. EB Jacobs has developed a system
to meet the City's needs that is driven by three goals:
o select the best candidate(s) for the job
o minimize adverse impact
o maximize test standardization and perceived faimess for all candidates
Our recommendations are based upon information provided in the RFP, our previous work experience, .
and our standard methodology. Due to the nature of promotion testing, we propose the use of tailored
tests for each position. However, since the same methodology is used to develop and administer such
tests -for all positions, we will present our general model within this proposal. The proposed work plan
includes the following general steps for each position:
❑ Conduct orientation (kick-off) meeting with members of the City of Miami Department of Human
Resources, Department of Fire -Rescue and other personnel identified by the City to make
introductions and discuss project plans, timelines and resource needs.
❑ Conduct additional meetings as warranted with members of the City and Department to discuss
project status and address issues that arise.
❑ Review existing job analysis data and gather additional/updated data as warranted.
❑ Conduct a valid job analysis for each position.
❑ Develop and validate test plan specifications and reading lists.
❑ Develop and review test components including written technical and applied knowledge
examinations, written in-basket/work sample examinations and oral examinations (please see
Appendices F through P for samples of our work).
❑ Prepare and deliver candidate preparation program including reading lists, written candidate
preparation guides, orientation sessions for each position, and detailed questions and answers
following the tutorial sessions.
o Administer all examinations.
❑ Develop and conduct appeal process for multiple-choice examinations.
o Score all examinations.
❑ Prepare a final, rank -ordered promotional list for each position and provide summary statistics
as requested
❑ Prepare individual feedback notification reports.
❑ Document all project activities in the form of a final validation report.
We believe that the same steps that one might use to reduce adverse impact can also play an
important role in helping to distinguish the outstanding candidates from the less capable candidates.
EB Jacobs will help identify the best candidates for each position by eliminating any artificial barriers to
effective test performance. Our position is straightforward: the more we do prior to testing, the better.
That is why we include (and are pleased to see the City has requested) candidate preparation guides,
tutorial sessions, and post -tutorial question and answer materials.
Specifically, throughout the project, EB Jacobs will implement a series of actions that will benefit all
candidates, including:
o stratified sampling of subject matter experts (SMEs) during job analysis activities
CONFIDENTIAL 1
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
❑ conducting technical as well as cultural/gender bias reviews on all test material
o use of reading -level analyses to ensure test questions do not artificially depend upon reading
comprehension and speed
❑ use of multiple testing modalities
❑ measurement of a broad range of knowledge and abilities
o use generous time limits on all tests
o implement candidate preparation programs
o provide detailed feedback to candidates
All work will be managed by Drs. Echemendia and Silva as'the primary consultants. Drs. Echemendia
and Silva have over 55 years combined experience in job analysis and employment test development
and validation and 41 years of experience in job analysis and employment test development and
validation in the public sector. Drs. Echemendia and Silva have worked in over fifty public safety
organizations throughout their tenure at EB Jacobs. Over the past fourteen years, Drs. Echemendia and
Silva have worked. closely together with many public safety agencies on large-scale job analyses, test
development, and administration processes, including the City of Miami's police and fire -rescue
promotional processes.
Based on our previous experience, we are convinced that a successful testing program will be greatly
enhanced by our efforts along these lines. Our response takes into consideration the provisions of the
Americans with Disabilities Act, the Civil Rights Acts of 1964 and 1991, the Age Discrimination in
Employment Act, relevant case law and authorities (e.g., Uniform Guidelines), and local case law and
statutes. All test components will be developed and administered in accordance with the above
guidelines including any applicable rules and regulations in the City. This proposal offers a rational,
data -driven approach to the provision of promotion testing services for multiple positions within the City
of Miami's Department of Fire -Rescue.
For nearly 30 years and more than 300 contracts, we have provided public safety testing services for
police, fire, corrections, and other public safety organizations across the country. Our projects are
known for efficiency, accuracy, and adherence to timelines. Our numerous contracts with the same
organizations over periods extending since the mid 1980's are evidence of our efforts in meeting the
needs all of the needs of our clients.
We are excited at the possibility of continuing and expanding our collaborative relationship with the City
of Miami and the Department of Fire -Rescue. As always, our proposal is just that — a proposal. We
would be happy to consider alternative specifications that would be of best interest to the City and D.
Vendor Information
EB Jacobs
300 South Burrowes Street
State College, PA 16801
(800) 367-5214
Fax: (814) 237-4886
Authorized Signature:
ianet M. Ec
Primary Consultants
Dr. Janet Echemendia
(800) 367-5214
janet.echemendia@ebjacobs.com
Dr. Jay Silva
(352) 332-9708
jay.silva@ebjacobs.com
mendia, President
l� -I I - la
Date
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
II. PROPOSER'S QUALIFICATIONS: Experience and Qualifications
A. Organization
In January of 2004, EB Jacobs purchased the public safety division of SHL Landy Jacobs to enable the
staff to focus exclusively on the public safety market in the US. This move has enabled the
professional staff in State College, PA and at the two regional locations to create a new initiative of
growth within the public sector research arena, as well as a new vitality for continuing its 30-plus-year
history of supporting law enforcement, fire and other public sector agencies in HR programs.
Contact information
Name: EB Jacobs, LLC
Contacts:
Dr. Rick Jacobs, Chief Executive Officer
Dr. Janet M. Echemendia, President
(800) 367-5214 (toll free)
(814) 237-4886 (facsimile)
300 South Burrowes Street
State College, PA 16801
Dr. Jay M. Silva, Managing Senior Consultant
(352) 332-9708
3325 SW 91st Drive
Gainesville, FL 32608
EB Jacobs is a Limited Liability Corporation (LLC) with three principals, owners, and members of the
Board of Directors:
❑ Dr. Rick Jacobs
o Dr. Janet Echemendia
o Mr. Charles Boyle
CONFIDENTIAL 3
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentially Statement of this -proposal
Below is the breakdown of professional and other employees by EEO classification:
❑ Managers/Owners:
❑ Professionals:
❑ Technical:
❑ Administrative Staff:
3 (2 White males, 1 White female)
6 (3 White males, 1 Hispanic male, 2 White females)
2 (1 White male, 1 White female)
3 (3 White females)
B. incorporation
EB Jacobs was incorporated in the State of Pennsylvania in January 2004.
C. Organization Description and Previous Projects With the City of Miami
EB Jacobs specializes in helping departments develop public safety professionals through human
resource initiatives. Our staff has worked in many jurisdictions, involving diverse stakeholders (e.g.,
departments, solicitor's offices, unions, plaintiff organizations, the Department of Justice, EEOC),
diverse organizational levels and job titles, and often in contexts of great legal jeopardy for the client.
In every instance, work has been carried out in a timely and professional manner, without undue
dependence on jurisdiction personnel, but with heavy conceptual guidance from all affected parties.
EB Jacobs has used, and is proficient in, all forms of validation — criterion -related, content -oriented and
construct. We have used our expertise to develop and validate:
❑ Cognitive ability tests
o Technical knowledge exams
o Structured interviews
o Oral and written work samples
o Competency based assessments
❑ Assessment centers
o Background investigation processes
o Medical Standards
o Personality/Biodata inventories
o Internal review boards
o Training curricula & tests
o Physical ability tests
o .Performance management systems
o Licensing examinations
o Psychological screening standards
The process used to develop legally defensible tests is the same process required to:
o Identify top performers
o Produce tests that are embraced and accepted by candidates
Therefore, whether or not the agency operates in.a highly litigious environment, it is important that the
consultant be experienced in the development of effective solutions. We excel in developing business -
oriented solutions based on science, research and the law with in-depth knowledge of:
CONFIDENTIAL 4
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
❑ Age Discrimination in Employment Act (ADEA)
❑ Americans with Disabilities Act (ADA)
o Civil Rights Acts of 1964 and 1991
Our public safety tests have been scrutinized by plaintiffs' experts, judges, trial attomeys and federal
agencies (e.g., the Civil Rights Division of the Department of Justice, EEOC) and found acceptable. In
addition, many of the entry-level public safety projects conducted over the past seven years (see
Appendix A), have been performed in a court -oversight context. Given the depth and breadth of our
testing and litigation experience, we are confident that our assessment design and processes conform
to legal and professional best practices.
EB Jacobs has successfully executed 25 fire -rescue and police promotion exams for the City of Miami
over the past 14 years. The following City fire promotion projects were performed by EB Jacobs (2012
promotion projects are in progress and nearing completion):
❑ 1999 (Feb) Fire -Rescue Lieutenant
❑ 1999 (Feb.) Fire -Rescue Chief Fire Officer
❑ 1999 (Jun) Fire -Rescue Captain
❑ 2007 Fire -Rescue Lieutenant
❑ 2007 Fire -Rescue Chief Fire Officer
❑ 2008 Fire -Rescue Captain
❑ 2012 Fire -Rescue Lieutenant
❑ 2012 Fire-Rescue.Chief Fire Officer
❑ 1999 (Feb) Fire -Rescue Captain
o 1999 (Jun) Fire -Rescue Lieutenant
o 1999 (Jun) Fire -Rescue Chief Fire Officer
❑ 2007 Fire -Rescue Captain
o 2008 Fire -Rescue Lieutenant
o 2008 Fire -Rescue Chief Fire Officer
❑ 2012 Fire -Rescue Captain
The following City police promotion projects were performed by EB Jacobs:
❑ 1998 Police Captain
❑ 1999 Police Lieutenant
❑ 2003 Police Sergeant
❑ 2004 Police Captain
❑ 2006 Police Lieutenant
o 1998 Police Sergeant
❑ 2000 Police Captain
❑ 2003 Police Lieutenant
❑ 2006 Police Sergeant
o 2008 Police Sergeant
A number of these projects were performed under Department of Justice oversight and were aimed at
releasing the City of Miami from the oversight of the Department of Justice.
CONFIDENTIAL 5
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
D. List of Staff to Use on the City's Project
The following proposed individuals will be used to perform the work on the City's project. Detailed
biographies of the individuals listed below and their roles in the City's project are provided in the next
section:
❑ Dr. Rick Jacobs (Chief Executive Officer); manager/owner; White male)
o Dr. Janet Echemendia (President; manager/owner, White female)
o Dr. Jay Silva (Managing Senior Consultant; professional; Hispanic male)
❑ Joseph Hinish (Managing Senior Consultant; professional; White male)
o Nicole Howard (Consultant; professional; White female)
❑ Brian Himelright (Consultant; professional; White male)
❑ John Evans (Consultant; professional; White male)
❑ Dr. Harold Cheatham (Specialist; professional; African American male)
❑ George Conrad (Data Analysis Manager; technical; White male)
❑ Chris Williams (Data Coordinator; technical; White female)
CONFIDENTIAL • 6
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
'CONFIDENTIAL 7
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of Phis proposal
CONFIDENTIAL 8
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
CONFIDENTIAL 9
Use or disclosure of data contained on this page is subject to the restriction specked in the Confidentiality Statement of this proposal
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
10
F. Locations Where Work WiII Be Performed
The work will be performed at three offices. The Project Manager, Dr. Jay Silva, will work in the
Gainesville, Florida office; Ms. Nicole Howard will work in the Seattle, Washington office, and the
remaining staff will work from the State College, Pennsylvania office.
H. Letters of Reference
As requested in the RFP, we have included two letters of recommendation from current clients. These
letters can be found in Appendix D.
1. Lawsuits
No lawsuits have been -filed against EB Jacobs. As is common in in the public safety arena, over the
last 30 years some lawsuits have been filed against agencies that use EB Jacobs tests. Appendix S
lists lawsuits filed against agencies that use EB Jacobs tests. In addition, Appendix S also lists our
expert witness experience in litigation that does not involve EB Jacobs tests.
J. Sub -Consultants
No sub -consultants will be used on the City's project.
CONFIDENTIAL 11
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
III. PRIMARY. CONSULTANT'S EXPERIENCE
A. Experience
EB Jacobs conducts all projects using a team -based approach. All team members are fully qualified to
conduct test development, content validation, and/or associated test reviews and statistical analyses
(see resumes in Appendix B). While project oversight will be maintained by Co -Director Dr. Janet
Echemendia, and day-to-day activities are controlled by Co -Director and Project Manager Dr. Jay Silva,
all team members are key resources to the projects success.
As outlined in Appendices A and E, EB Jacobs has extensive experience in conducting promotional
testing for fire departments. With respect to testing for fire promotional positions, our most recent
experience includes the City of Buffalo, Commonwealth of Massachusetts; City of Miami; and City of
Madison, Wisconsin (see Appendix E for more detailed descriptions of recent project experience).
B. Qualifications of Lead Consultants
Drs. Echemendia and Silva have over.55 years combined experience in job analysis and employment
test development and validation and 41 years of experience in job analysis and employment test
development and validation in the public sector. Drs. Echemendia and Silva have worked in over fifty
public safety organizations throughout their tenure at EB Jacobs. Over the past fourteen years, Drs.
Echemendia and Silva have worked closely together with many public safety agencies on large-scale
job analyses, test development, and administration processes, including the City of Miami's police and
fire -rescue promotional processes. Specifically, with respect to the City of Miami, since 1998 Drs. Silva
and Echemendia have co -directed and provided daily project management in the development and
administration of 25 promotional exams in both the Miami Department of Fire -Rescue and.the Miami
Police Department. For further information on the primary consultant (i.e., Dr. Janet Echemendia and
Dr. Jay Silva) please reference Section 2 of this proposal. Their resumes are also provided in
Appendix B. Prior to joining EB Jacobs, Dr. Jay Silva headed numerous large-scale test development
projects for the Army in his former capacity as a senior research scientist at the US Army Research
Institute and as a senior scientist at Personnel Decisions Research Institute. Dr. Silva has also
contributed to the research literature in selection related areas, including test faimess and impact of
increased minority hiring, through professional publications and presentations. His research since
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
2010 has been in the area of adverse impact and has yielded two joumal articles and a book chapter
with his co-authors.
Dr. Silva will manage all of the day to day activities on the City's project. In addition to being in near
daily communication with the City's representatives, Dr. Silva will be involved in designing the job
analysis methodology, implementing the job analysis data collections, identifying the exam components
to employ in the promotion testing process using job analysis results, overseeing the development of
exam materials, ensuring a good fit between the developed exam materials and the job duties and
abilities and knowledge required to'perform them (i.e., content validation) through the use of subject
matter expert (SME) data collections, developing and delivering candidate tutorial sessions, on -site
administration of the exam components, overseeing the scoring of candidate responses, and writing
the final report. Dr. Echemendia will review all proposed methodologies and reports and issue
revisions. Dr. Echemendia will also maintain oversight over the project.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
• CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
14
D. Primary Consultant Expert Witness Experience
Of the primary consultants (Drs. Echemendia and Silva), only Dr. Silva occasionally performs as an
expert witness. His last and only expert witness testimony was provided in 2009 through 2010 in the
Bradley et al v. City of Lynn et al case (see Appendix S for more details). This was a case where the
Commonwealth of Massachusetts hired Dr. Silva as an expert witness to defend their intemally
developed police sergeant promotional exam process.
Expert witness work is highly demanding and is typically of an urgent nature. Such work would
jeopardize the smooth execution of selection and promotion projects. Although Drs. Silva and
Echemendia provide limited expert witness testimony, from time to time both Drs. Echemendia and
Silva provide guidance and specialized analysis to assist clients in defending against lawsuits for work
performed by other firms. In such cases, Drs. Echemendia and Silva typically perform specialized
analyses and provide reports to the agencies requesting the counsel.
Dr. Rick Jacobs, however, has substantial expert witness experience and performs that role for EB
Jacobs with support from Drs. Echemendia and Silva.. We will provide a detailed synopsis of Dr.
Jacobs' expert witness experience upon request.
E. Lawsuits
No lawsuits have been filed against the primary consultants.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
IV. DESCRIPTION OF PROPOSED TESTING PROCESSES
A. Exam Processes
Our Primary Goal
Our review process is
directed at achieving
valid, job -related
selection instruments
with superior technical
qualities, that will identify
the most qualified
candidates for the
position while minimizing
impact on protected
classes.
This Technical Proposal outlines the procedures EB Jacobs will follow to
conduct job analyses, develop exam plans, and administer and score
promotional examinations for the City of Miami Department of Fire -
Rescue.
Early in the project, EB Jacobs and the City team members will
participate in a series of communications with the various involved
stakeholders. The stakeholders may include representatives from the:
Department of Human Resources
Department of Fire -Rescue
City's Counsel
IAFF 587
The purpose of these communications is to outline project goals and activities, timelines, logistics, and
resource needs. This phase of the project gives affected parties the opportunity to ask questions and
make suggestions related to project activities. The City and EB Jacobs will make a joint determination
regarding who should be involved in these initial communications.
The testing process will include a job analysis to determine the duties, abilities, and knowledge areas
involved in successfully performing in the three roles (i.e., Lieutenant, Captain, and Chief Fire Officer).
Following the job analysis, an exam development plan will be constructed and it will target the most
central tasks, abilities, and knowledge areas identified in the job analysis. The development of the test
materials will then follow the blueprint set forth in the exam development plan. Once the test materials
are developed, they will be reviewed by three or four subject matter experts (SME) selected by the
Department of Fire -Rescue (bound by a confidentiality agreement). The review will determine which
tasks, abilities, and knowledge areas are covered in the test materials and they will be compared to
those identified in the job analysis. A strong overlap between the job analysis results and the content of
the test materials will serve to establish the content validity of the promotional exams..
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction.specitied in the Confidentiality Statement of this proposal
16
1. Job Analysis Process
Our approach to testing involves a content -oriented strategy of test validation. A content strategy rests
upon a careful and detailed job analysis of the position, in which essential functions (e.g., important and
frequently performed tasks) are identified, and the knowledge, skills and abilities (KSAs) that are
prerequisites to task performance, are delineated. This information enables us to:
o determine essential functions and job descriptions
o identify the knowledge and ability areas critical to job performance
o determine precise specifications for each test component
o specify the weights to be allocated for each test component
We will begin by reviewing the existing job analysis reports and surveys and expect to spend one to two
days on -site per job title collecting job analysis information from subject matter experts (SME) and
constructing an updated written job analysis survey. The survey will identify task, ability and knowledge
areas that describe each position. The three targeted job analysis areas will be collected in the following
detail:
Task Analysis Survey
o Do you perform this task? If so....
• How frequently do you perform this task?
■ How important is this task to successful job performance?
• Is this task required upon promotion, or taught on the job?
Ability Analysis Survey
❑ How important is this ability for successful job performance?
❑ The relevance of each ability to each task group.
Knowledge Analysis Survey
❑ Do you use this knowledge source on the job?
o How important is this knowledge to successful job performance?
o Does this knowledge need to be memorized for successful job performance?
o The relevance of each knowledge source to each task group.
• With the forthcoming job analyses we plan to move away from an approach used by Core Corporation in
CONFIDENTIAL
.: Use_ or_ disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
17
the late 1990s. The Core Corporation approach required incumbents to rate nearly 100 knowledge areas
and nearly 150 abilities (it varied by rank). Incumbents found this overly cumbersome and we agree.
Our job analysis for each fire -rescue rank, will for the first time in City of Miami, employ a more traditional
EB Jacobs approach that usesa much smaller number of broader ability and knowledge areas. Not only
will the new approach reduce incumbent time needed to respond to job analysis surveys, it will yield
more meaningful results that can be more strongly linked to the exam content.
Appendices F, G, and H contain excerpts from the task analysis, ability analysis, and knowledge
analysis portions of the survey, respectively. EB Jacobs will provide a camera-ready copy of the job
analysis survey to the respective Department of Fire -Rescue or Department of Human Resources for
duplication and distribution to a stratified representative sample of incumbents. Completed surveys,
based on client preference, will at the City's option be returned either to the Department of Human
resources or mailed directly to EB Jacobs for analysis.
2. Exam Plan Specification and Exam Development
Based on the job analysis results, we will identify the most important and frequently occurring job tasks,
as well as the most important underlying knowledge and abilities. This information will serve as the basis
for the exam plan and the exam plan in tum will drive the development of exam materials.
Our extensive experience with the positions covered in the RFP supports the measurement of job
knowledge and managerial abilities (e.g., information analysis, decision making, planning and organizing,
resource management) to promote individuals. Based upon the specifications in the RFP, and our
experience with the City, we are proposing the following exam components:
_4„-'�1
�-.- •T:a;�=4'�""' `-'W f -a` �..
v
fi i12 s};,--- F.may'
-
—a
tien' Ezerctse
dr _
or�fes
4 _:—
,-
-
Oralu_ ork
_ ..„ to r
sip_: _.
W
--,-,
�.,
S i
-
�`-J
mac,__
=-r
.k^Z i
-�"
- '�a',�=-
-..m.S- :F 3T
SF .�}y�
S '�.
.Clo S'e �.
-,Closed-�-�t
T.' _.���I 0 2r
.�. ,
F
4
.�.. E
..� -� .. _ .
r
;may c��
Exam
ling"Er+
L"s. .�.f �L�i
Open
:tom �. .�
f
--z-
3,�`�,
- s
i
l�
i lk "�
- Book
�,�r
Te h a `#
FCnowl ge
o k
_--�
-
�-
--E�'s - .17
echrncal
K owiedge --
,
—'r
Situaatu al
trJudgemeim-,,
q� , __Test rm
.
Subo
r -3r�P'•-
{-Inciden t
i Command
.Exercise r
Writtene
d a
Conferenc
i.3Exercise
I
E
Testz .,_>,
,,= Test-,
Lieutenant
✓
V'
✓
Captain
✓
V
V
Chief Fire
Officer
✓
✓
V
✓= Preliminary Recommendation
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
18
The relative weight of these components will be dependent upon the job analysis results and/or
Department or City contract or policy. Detailed descriptions of these exam components are provided in
the next section of the proposal. In the case that Department or City policy deviates substantially from
job analysis guidance, EB Jacobs will engage the Department or City in discussions to determine the
best course of action.
EB Jacobs promotional processes typically require supervisory/managerial candidates to participate in
multiple exercises consisting of written and oral job simulations. This is also our approach for the
Captain and Chief Fire Officer assessment processes proposed for the City. For the Lieutenant
assessment process, only written multiple-choice oriented testing is being proposed at the request of the
City. Although we would typically still recommend the use of at least one job simulation for the
Lieutenant rank exam, probably a Subordinate Conference Role -Play exercise, we are not doing so in
this instance since the Department and IAFF 587 membership, based on our past experience in the
Department, are strongly against any non -written testing exercises for the Lieutenant promotion process.
Our references will vouch for the effectiveness of job simulation exercises in identifying the best
candidates via the vetting of a more complete candidate skill set. In addition, job simulations typically
demonstrate lower levels of adverse impact compared to written tests. The job simulation exercises will
involve essential higher level skills, as determined through the job analysis. Based upon our experience
with similar positions and with positions in the City of Miami Department of Fire -Rescue, it is anticipated
that the following assessment dimensions will be evaluated:
. -......a ... ,. •,-..-._� �.a��: ....... ...:. -�-• -g4-.,
:'Id' � +# '�ltr.,.� P's f twa 7'17 ii � i m�i
6
rfi lyr tif,
[1 •y , ,, �.;: x —'m!ftWia..
Ta TLq�iAj'Y 3� y' d, 4"-`;
z W
$
Fn ["x -16' i u iH -
-ti' xq
� �,Wr(tfen °z
d ,.
,' _ _-- ;, ^''
m,r In Basket
I -' . -. 1
J'A'r i .P
�,�s-4aExerctse4q=m_.
rxi >>:w7i t _ _ : ''5;r5•� i'.,
t=r2 P at nsessyf' y-1 �,f ` � N;n Tt,,. -. r{4i-' � t �,
h
`* � � Assessment Center-_ ' i ��
"'1`.1.a.7-:.. " .1itee,e'ofrr .1.1. _ K• , .. :.4- `2v t,, Yrtl ii. t 1
4 ejtr� .�
7 r2r - L 1'- t i 4..- .
�b v! i1 ( -Y •" .r o, Ex L �r
, . K x� r y
o.! vi r r%.r ''
t ^-� } Y r : a
�v
Y ; 416f{i mil..
?
.s��a.incident �;yw;;
;t-:s
Command
* c5 rs
.-:; ,.. 6: , 0 ,,.., ;� �•,. „.
l4
_;:.
a
„ � �t ,�
je a, r ,f�H„� �� r
�, nl .. r , :� <
Fe �i , _. r �� - �,
Assessment Dimension � A.,-., �
a,w� 5liS dtnate �� {
�_ �,,�
�
. Conference t -
w.L z -, 4z.,°,r.r..... b
Written Expression
✓
Oral Expression
interpersonal Relations
✓
Information Analysis
✓
✓
✓
Judgement and Decision Making
✓
✓
V
Planning and Organizing
✓
V
✓
Resource Management
✓
V
✓
CONFIDENTIAL
- - Use -or -disclosure ofdata-contained.on.(his.page.is subject_to the restricfion.specified-in_the Confidentiality Statement of this.proposal
19
9
a. Written In -Basket Exercise •
We propose that a written in -basket exercise be used to test for the Chief Fire Officer position.
Although we could have proposed a less expensive multiple-choice testing modality and lowered our
bid price, we believe that it is important that candidates for this position demonstrate high-level
managerial skills and not just technical knowledge. As current Captains, most candidates for this
position would already be expected to possess superior technical knowledge. As a result testing on
technical knowledge alone, would be much less effective in differentiating the readiness of
candidates to step into the Chief Fire Officer role.
As candidates begin this exercise, they will be presented with an in -basket containing paperwork
(e.g., incident forms, performance appraisal forms, training requests, telephone call messages, e-
mails, public requests, etc.) that must be completed, responded to, or otherwise acted upon. Other
items may relate to the preparation of an annual budget, revisions or additions to department
policies and other reports required of Captains and Chief Fire Officers. The items will reflect a
number of underlying issues and the response to any given item may require candidates to consider
information from other items. Given the importance of written communication skills, we suggest that
this exercise be developed such that candidates provide their answers in written format. The
candidates will need to refer to information provided in their in -basket to complete their responses.
The written responses will be rated by a panel of trained assessors, using a scoring guide based on
the source documents and verified, if possible, by a panel of City of Miami Department of Fire -
Rescue personnel (at the rank equal to or higher than Captain or Chief Fire Officer). An example of
a Written In -Basket Exercise item is included in Appendix 1.
b.. Open and Closed -Book Technical Knowledge Tests
We propose that both open and closed -book written technical knowledge tests be used to test for
the positions of Lieutenant and Captain. The closed -book portion of the exam usually contains
questions based on department policies, procedures and orders that must be memorized on the job.
The open -book portion of the exam typically contains items from policies, procedures and orders
that can be `looked up" on the job and are determined to be open -book reading list sources. The
open- and closed -book sections would be delivered via a paper -and -pencil, multiple-choice format
with at least four response.options. Based on past requirements we envision producing:
o 125 questions for the Lieutenant exam to be distributed between the open and closed -book
technical knowledge tests.
) CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specfied in the Confidentiality Statement of this proposal
20
o 100 questions for the Captain exam to be distributed between the open and closed -book
technical knowledge tests.
The number of questions can be increased or decreased based on the Department's preference.
The proposed question counts are the same as were used in the last exam. In the past we have
negotiated, as needed, the final number of questions on the exams with the City. The same
approach will be used this time if the City would prefer to reduce or increase the number of
questions. A final determination of the actual sources and the proportion of questions from each
source will be based on the results of the job analysis and discussions with the City. The question
content will be based on task statement ratings and knowledge analysis results. For the closed -
book Technical Knowledge Test, questions from reading list sources can be verbatim with source -
specific reference listed on the exam or not verbatim, based on the City'spreference. Examples of
open -book, multiple-choice questions are included as Appendix J. Examples of closed -book,
multiple-choice questions are included as Appendix K.
c. Situational Judgement Test
We propose that a situational judgement test be used to test for the position of Lieutenant. This type
of exercise requires candidates not only to have the technical knowledge to address the issues in
question but also to know what knowledge, procedures, and rules to apply in realistic, contextual
situations. We recommend this type of test for the Lieutenant position since the City has opted to
not test Lieutenant candidates using any job simulations. Approximately 25 multiple-choice
questions (with at least four response options) will be asked of candidates. using a series of
approximately 4 to 6 scenarios. The number of questions can be increased or decreased based on
the Department's preference. In the past we have negotiated, as needed, the final number of
questions on the exams with the City. The same approach will be used this time. A final •
determination of the actual sources and the proportion of questions from each source will be based
on the results of the job analysis and discussions with the City. The question content will be based
on task statement ratings and knowledge analysis results. Appendix .L.contains excerpts from a
multiple-choice style situational judgement test.
d. Assessment Center
We propose that assessment center exercises be used to test candidates for the positions of
Captain and Chief Fire Officer. Although using only written tests would be less expensive, the
positions involved are heavily comprised of supervisory activities and higher level application of
CONFIDENTIAL
_ _ _ _ _Use or.disclosure_of data_confainedon_thts page is subject to the restriction specked in the Confidentiality Statement of this proposal
21
technical skills. Without job simulation exercises used in the assessment center, the City would
ally
potentially be subject to a much greater level of scrutiny and liability if the exam results were
leg iy
challenged. The assessment center will consist of "intact work activities carried out by
e.g., an incident command and a conference session with a problem subordinate). The job analysis
data will determine the topics to be covered and will guide the selection of assessment dimensions
to be examined. We recommend that all oral work sample exercises be evaluated by a panel of
three trained assessors to ensure excellent reliability and mitigate against unforeseen circumstances
(e.g., assessor illness in the middle of the job simulation exercise process). We recommend that the
assessors independently evaluate candidate performance initially using a series of behaviorally
anchored rating scales. If the evaluation on a performance dimension differs by more than twoand
points on our nine -point rating scales, then assessors would discuss their independent ratings
attempt to identify the reasons for the difference and resolve the difference in the ratings. The
proposed exercises for the assessment center are as follows:
1) Subordinate Conference. This exercise is designed to be a role-play between the
candidate (who assumes the role of a newly promoted incumbent), and the actor playing the
role of the subordinate. A script is prepared far the actor/role-player containing a series of
questions or statements to which the candidate should respond. Candidates are given a
description of the scenario prior to their meeting with the subordinate. sbordinaie conference
interaction will occur in a ten minute period. An exampleof a
scenario is provided in Appendix M.
2) Incident Command. This type of exercise requires candidates to present a plan for
handling a major incident. To simulate the job, candidates are given a brief description of an
incident scene prior to appearing in front of the assessment panel. The candidate is given
up to six minutes to describe how they would handle the scene. Assessorswillthen
pired rovide
the candidates with a short update as the incident develops. The candidates areq
to
use this new information to modify their approach for the next four minutes. An example of
an incident command scenario is provided in Appendix N.
We propose testing all Chief Fire Officer candidates on the assessment center exercises since that thee
expected number of candidates is likely to' be no more than 40 candidates. However, given
expected number of Captain candidates is likely to reach as high as 110,we This represents estinalingso onl 68.18
y a
maximum of 75 to 80 Captain candidates on the assessment center.Pr
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
percent of the candidates tested on the written test portion of the exam. We examined the impact
that this approach would have had if this approach had been used in the last Captain examination
process. We discovered that if the promotion rate had been no higher than 16.36 percent, there
would have been no impact whatsoever, on who was promoted. Thus, even if the lowest performing
18 candidates on the written testing had not been allowed to test on the assessment center
exercises in 2008, the same candidates would have been promoted. When the 16.36 selection rate
is applied to the foreseen 110 candidates it means that more than 18 would need to be promoted to
have an impact under the same circumstances.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
23
4. Assessors
EB Jacobs would recruit assessors according to a plan in which the gender and racial make-up of the
assessor boards matches the gender and racial make-up of the applicable position, or possibly over -
represents protected classes. In the past, in the City of Miami, we have convened three -assessor oral
board panels that are composed of one White, one Hispanic, and one African American subject matter
expert. In addition, in the past, one of the three assessors is a female and the other two are males. For
written in -basket panels, in the City of Miami, we have typically convened a two -assessor panel
composed of one majority group member and one minority group member, one of which is male and the
other female. We recommend continuing this composition in future testing. Using one oral board panel
of assessors per exercise type, we can typically process 15 to 16 candidates per day. For this proposal
we assume that the same panel will assess all candidates for the same exercise.
Assessors will be procured from fire -rescue departments across the US (except Florida to avoid the
likelihood of familiarity with the candidates) of comparable size and demographics. All assessors will be
at the same rank or higher as the classification being tested.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
Assessor Training
We will train all assessors in assessment techniques and administrative logistics including:
❑ tasks and responsibilities of the job(s)
❑ proper observation techniques
❑ use of rating scales
❑ potential errors and biases
❑ observing and rating "mock candidates"
❑ development of a common frame of reference
❑ preparation of developmental feedback for candidates
For the purpose of the Cost Proposal, we assume that one backup assessor per exercise will be
available and that any role-play required in the exercises will be performed by someone other than the
assessors. EB Jacobs will procure and train the role-player(s).
5. Parallel Forms
For test security purposes, we will devise a system of parallel oral exercise forms to accommodate three
days of oral board testing for Chief Fire Officer candidates and seven days of testing for Captain
candidates. We will also develop one extra parallel version of each exercise (Le., written and oral) for
each of the three positions. Each oral board exercise type (e.g., subordinate conference and incident
command) will be developed in such a manner that multiple forms of the problem can be constructed.
The versions will vary with respect to the specific details presented, but the overall problem and difficulty
of each version will remain the same (e.g., subordinate is performing poorly). We suggest that a panel of
command level personnel (SMEs) for each position (if possible) be convened to examine the parallel
forms and help ensure that each problem is the same with respect to difficulty and that the only
difference lies in the details. Candidates will be told that parallel forms have been developed and that
the set of exercises they receive will be different than the set received by someone else. The use of
parallel forms minimizes the chance that candidates will gain an advantage by.sharing information.
CONFIDENTIAL 25
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
6. Exam Review
The exams undergo a series of reviews to ensure:
o Technical accuracy
o Cultural fairness
o Appropriate reading level
❑ Organizational Appropriateness
o Job Relevance (links to job analysis)
First, each test component will be reviewed to ensure that the correct item construction has been used, it
has been keyed correctly and professional testing procedures have been employed. Second, Dr. Harold
Cheatham will review all components for cultural fairness (e.g., eliminate wording and characteristics that
might pose a problem for minority candidates). Third, a reading level analysis of all exam materials will
be conducted using an accepted readability program. Examination items and text are targeted to be at a
level that is equal to or slightly lower than the average reading level of the materials used in public safety
jobs. Bytargeting materials at these levels, we are able to restrict the effect that reading speed and
comprehension might have on test scores. This ensures that the relevant KSAs, and not reading ability,
are being measured by the test.
In addition to reviews by our experts, whenever possible we also like to assemble a small, 2 to 4 person
panel of superior officers to conduct a final review of all exam materials. The advantage of such a review
is obvious. First, the integrity of the test will only be enhanced since the best items in the pool will be
used to construct the final test. Second, each item will have been examined by the department on those
dimensions that might raise objections before being used. Third, the internal SMEs can help establish
links between the test content and job domain. We are confident that the quality of our tests will not be
compromised, if an internal review is not possible however.
7. Candidate Preparation
Just as world -class athletes work with personal trainers to prepare for competition, today's public safety
professionals invest large amounts of personal time and energy preparing for promotional assessments.
Providing information regarding the content and logistics of each examination component helps ensure
that all candidates are on equal footing when they take the exam, thereby representing an effective step
toward the reduction of adverse impact. Agencies that acknowledge the commitment of their candidates
and recognize the criticality of ensuring equal access to information often see an increase not only in the
caliber of candidates, but also in the general force. Formal candidate preparation programs minimize
26
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement ofihis proposal
superficial performance differences, such as mistakes made because of being unfamiliar with a type of
test, careless errors, or improperly allocating study time based on rumor and/or "others' experiences.'
Our recommended program includes a description of the strategies that candidates can use to develop
their KSAO's to enhance exam (and job) performance and are typically designed around three basic
objectives:
Content of Exam. We provide information regarding the abilities, knowledge and personal
characteristics that will be assessed and the types of questions and exercises that will be used to
assess them. Providing such information can be particularly helpful in ensuring appropriate
allocation of study time and equal access to exam -related information and preparation strategies.
Administrative Procedures. Candidates are given detailed information regarding the number of
questions on each exam, the length of the testing sessions, the scoring guidelines, the weight of
each test component and other general instructions. Such information is particularly helpful in
minimizing the number of errors candidates make as a result of carelessness, anxiety, or failure to
follow instructions.
Sample Questions/Exercises. Candidates can benefit enormously from exposure to the actual
type of test they'll be taking. Therefore, we provide them with sample questions/exercises that
closely approximate those they will encounter on the examination, including answer keys and
suggested responses.
The Purpose of Candidate Preparation Activities
Candidates may perform poorly on promotion examinations for a variety of reasons,
including:
o unfamiliar with the type of test
o making careless errors
o improperly allocating study time
This is inefficient for the candidate and organization. If an individual's scores are too
low for promotion for superficial or trivial reasons, in spite of the fact that the candidate
would have been a good supervisory or command officer, the department is deprived of
the services of a potentially outstanding performer.
Formal candidate preparation programs can minimize these superficial reasons for
performing poorly. Candidate preparation programs ensure equal access to all
candidates and are an effective step toward the reduction of adverse impact.
27
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
This information is typically presented in a self -paced Candidate Preparation Guide (see table of
contents of a typical Candidate Preparation Guide included in Appendix 0) provided in camera-ready
and/or electronic form by EB Jacobs. EB Jacobs will prepare a camera-ready copy of the candidate
preparation guide for each position tested. The City/Department will be responsible for the reproduction
and distribution of these guides.
We would also provide additional on -site orientation and tutorial programs as has been requested in the
RFP. Up to three sessions per position would be provided to accommodate different candidate work
shifts. In the past we have conducted two-hour orientation sessions and we propose continuing with the
same approach which has worked well. The sessions would be conducted in a classroom -style setting.
The content of these sessions is described below:
The candidate orientation is designed to help candidates digest the information provided to them in
the written preparation guide as well as to provide a forum for asking questions. This portion of the
on -site candidate preparation would be a maximum of 2 hours in duration and is presented via
PowerPoint. The orientation program is designed to help candidates focus on the key points
presented in the guide, to help them understand how to apply the test -taking and preparation
strategies suggested, and to answer questions they have about the examination process.
Once all sessions are concluded, EB Jacobs would provide a follow-up question and answer document
to ensure all candidates are aware of any questions raised and answers provided in any given session.
We encourage the City to prepare a videotape of the orientation session so that the videotape may be
used to facilitate future sessions and/or may be made available to candidates who cannot attend the
regularly scheduled sessions. in similar projects, we have made both the PowerPoint slides and .a
videotape prepared by the City available for continuous access to candidates via the City's internal
website. We can work with the City to make the orientation information easily available to the candidates
in the City of Miami. •
We are very pleased to see that the City is including an on -site preparation program, including a follow-
up question and answer summary, to its promotion processes to ensure that all candidates are aware of
any questions raised and answers provided in any given session, even if they did not attend any of the
sessions. We feel that such programs go a long way toward:
❑ de -mystifying the examination process and thereby reducing test -taking anxiety,
❑ helping candidates to more effectively prepare,
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
28
❑ contributing to the motivation levels of candidates, and
❑ alerting the City and EB Jacobs to any lingering misperceptions or concerns that may exist in
sufficient time to allow them to be addressed.
8. Exam Administration
Two representatives from our staff with at least a Master's degree in Industrial and Organizational
Psychology will be on -site to administer the written exams (assuming one day of written examinations
per job title). The cost proposal assumes the provision of two EB Jacobs test administrators per
administration with the assistance of City or Department monitors (EB Jacobs recommends one
administrator or monitor per 30 candidates). EB Jacobs will also provide two representatives from our
staff with at least a Master's degree in Industrial and Organizational Psychology to oversee the
administration of the oral exercises for each rank for all administration days. We also provide a detailed
set of administrative instructions to assist monitors before testing commences.
For each position, we recommend that the written exams (Le., Written In -Basket exercise, Situational
Judgement Test and Open and Closed -Book Technical Knowledge Tests) be administered to all
candidates simultaneously in a single testing session for each position. Oral exercise administration
would follow on subsequent days for the Captain and Chief Fire Officer positions. We will work with the
City to determine the most appropriate schedule for testing. With respect to the distribution of
responsibilities for the examinations, the City will be responsible for securing and compensating written
test monitors and for securing appropriate space for testing (written and oral). EB Jacobs will provide
procedures and training for monitors, be present during written test administration, and read all
necessary instructions to candidates. Per the RFP, EB Jacobs will also provide similar services for the
oral assessment center portions of the exams.
9. Exam Security, Scoring, Appeals, and Feedback
a. Exam Security
A prescribed set of steps will be followed to ensure the security of test materials developed for client
use, including having all personnel (EB Jacobs, City) sign a. confidentiality agreement. We assume
responsibility for test security before and during test administration. Specifically, we:
❑ password protect all electronic files
❑ . secure test materials in locked storage rooms at EB Jacobs facilities
❑ personally transport tests in locking cases
❑ number and count test booklets before, during, and following test administration
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified, in the Confidentiality Statement of this proposal
29
❑ record test booklet numbers on candidate answer sheets
❑ train test monitors in appropriate test material accounting procedures
In short, we feel that we can ensure the security of the materials developed for this project. in the
25 exams that EB Jacobs has developed and administered for the City of Miami, there have been no
security breaches.
b. Exam Scoring
We have complete data processing capabilities including keypunch and optical "scanning" input
formats, as well as virtually unlimited software and report -generating capabilities. Software includes
all common packages (e.g.,.SPSS and SAS) as well as tailored packages that we can modify to
meet all data processing needs (e.g., Access, Iteman, Astec, and Excel). We have full-time
programmers and data analysts on our staff. Our hardware capacity is virtually unlimited in size and
flexibility. Scoring procedures will be developed for all test components and will be carried out by
our personnel within the timeframes dictated outlined in the RFP (Le., within 14 days).
All data entry and scanning will be done twice and compared. Any differences in the two entries will
be examined and corrected. All scoring code will be developed by George. Conrad who is highly
experienced in all aspects of test scoring and has scored exams for many years. His scoring code
will be reviewed by Dr. Jay Silva and approved before scoring proceeds. Once scores are
generated, the top five candidates and bottom five candidates will be hand -scored and compared to
the computer generated scores. Only when the hand -scores match the automated scores will the
scores be certified. If an error is encountered at the hand -scoring phase, the process would start
anew with Dr. Jay Silva taking over the scoring process.
Test results will be analyzed using our standard procedures. We will generate descriptive statistics
such as means and standard deviations for each component and classification and analyze results
for differences by ethnic group and/or gender. We will also calculate the reliability for each of the
test components for reporting purposes.
When Department policy allows, we recommend that each test component be standardized prior to
being combined to produce the final promotion scores to ensure the appropriate weighting of these
components in determining the placement of candidates on the final eligibility lists. We will utilize
weights for the various test components as determined through the job analysis and joint
discussions between EB Jacobs, the City and Department(s). The final scores can be transformed
CONFIDENTIAL 30
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
to a 100-point scale to make the final scores more meaningful to candidates. We will be responsible
for all such calculations and the production of the final eligibility list in a rank -order format. Seniority
points will be incorporated into the final candidate scores when they are provided by the City. In the
past the City has opted for non -standardized raw scores (i.e., number of items answered correctly)
on the individual tests for Department of Fire -Rescue candidates. EB Jacobs will continue to
accommodate any such request as we have done in the past.
c. Appeals
EB Jacobs will establish a process for handling appeals for the multiple-choice tests (Le., Lieutenant
Open and Closed -Book Technical Knowledge Tests and Situational Judgement Test and Captain
Open and Closed -Book Technical Knowledge Tests). The questions and answers would be
available for review at the Department of Human Resources or other location designated by the City
starting the morning after testing for Lieutenant candidates and after the assessment center
exercises are complete.for Captain candidates. The materials would be reviewable by candidates
(without knowledge of their own responses) for three days. All appeals would be submitted to the
same office on an official form (provided by the consultant) by 5 PM of the third day of review. All
appeals would be reviewed by the consultant and a summary of the appeals will be provided to the
City along with the consultant's recommendations. The summary would include item statistics (e.g.,
.proportion who selected the various options, point biserial correlations, etc.) as well as a logical
analysis of the item itself. The Department of Fire -Rescue would review the summary and
recommendations and make a final determination.
d. Candidate Feedback Report Preparation
We will produce a written feedback report for each Captain and Chief Fire Officer candidate to be
distributed by the City. The feedback for each candidate can be provided printed on paper or
electronically (one Acrobat PDF document per candidate). As a group, the assessors will provide 2-
4 positive and/or negative statements per dimension per assessment center exercise. The
statements will give behavioral examples of the candidate's strengths and/or weaknesses in each of
the exercise (e.g., written in -basket exercise, subordinate conference exercise and incident
command exercise). Appendix P contains a sample feedback form resulting from an assessment
center containing an in -basket exercise, a subordinate conference exercise and an incident
command exercise.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to (he restriction specified in the Confidentiality Statement of this proposal
31
10. Utilization of Subject Matter Experts for Exam Development
The test development strategy we employ relies heavily on the input and expertise of job incumbents and
supervisors. We refer to this collection of personnel as subject matter experts (SMEs). During the test
development effort, we will spend time with all available SMEs. We will engage in direct observations of
work activities and interviews. We will perform our work around the scheduled activities of the SMEs.
While some departmental time is required, we keep it at a minimum and only collect data necessary to
competently develop the tests and document our actions.
We will also use outside SMEs to assist in the test development process. EB Jacobs has relationships
with a number of experts in the areas of protective services covered by this proposal. With the approval
of the City, we will contact appropriate SMEs (outside the state of Florida) and enlist their help in the
creation and/or review of test items. We have used this approach for more than 30 years and have
found that the face validity of the resulting test items is increased and that candidates react favorably to
such items. All external SMEs: 1) sign a confidentiality agreement; 2) are trained by EB Jacobs on test
item construction; and 3) submit their work directly to EB Jacobs testing experts for review and approval.
EB Jacobs accepts complete responsibility for all work conducted by our external SMEs.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this Proposal
32
12. Report
The project reports will document all project activities and will be structured around the legal
requirements of Title VII, the Uniform Guidelines, and professional testing practices. In projects similar to
this one, clients have used our project reports as a first line of defense against legal challenges to test
validity.
B. Validity Process
As highlighted throughout this proposal, we will develop a strong content validity argument for the testing
process. Any content -oriented strategy of validation rests upon the notion that important elements of the
job are measured in the promotion exam and that links are made between the test material and the
target content domain (i.e., job tasks and needed abilities and knowledge). The job analysis is the basis
for identifying the target content domain and it will be .the first element of the content -validity strategy.
Subsequent elements are:
❑ develop an exam development plan based on the job analysis and follow that exam development
plan to develop exam materials
❑ have exam materials reviewed and evaluated by Department of Fire -Rescue subject matter
experts (SME) to determine the following characteristics of the exam:
V to ensure that the exam covers central aspects of the position and not esoteric or outdated
procedures
CONFIDENTIAL
Use or disclosure of data contained on this page is .subject to the restriction specified in the Confidentiality Statement of this proposal
33 .
v" to compare parallel versions of the assessment center exercises to ensure that they target
similar abilities and are of equal difficulty
/ to determine what task groups, abilities, and knowledge areas are being targeted by the
exam
❑ compare the tasks, abilities, and knowledge areas identified through the job analysis to those
identified by SMEs as covered on the exam and determine whether sufficient overlap exists to
provide content validity for the exam
Dr. Jay Silva, who has extensive experience in validation efforts (see Appendix B), will oversee the
entire content validation effort.
C. Project Staffing
Substantial detail has already been provided on project staffing throughout the proposal. To summarize:
❑ Dr. Rick Jacobs will hold overall responsibility for this project and will serve as the project's
technical contact on issues that may arise during the course of the contract.
❑ The specific responsibilities for the remaining EB Jacobs staff members will follow each
individuals areas of expertise.
❑ As Project Directors, Drs. Janet Echemendia and Jay Silva will be responsible for project
planning and oversight on a general and long term basis.
❑ Dr. Echemendia will have a focus on the overall defensibility of the promotional process and its
development and will oversee all project activities and provide technical guidance.
o As Co -Director and Project Manager, Dr. Jay Silva will be responsible for the day-to-day
implementation of the project, and oversight and review of the accuracy and quality of materials
produced by project staff.
o Dr. Silva will work with the City project manager to coordinate a regular project update process to
be provided through conference calls and/or memorandums. Dr. Jay Silva will also serve as
Technical Advisor regarding statistical, IT, procedural and policy issues that arise in the course of
performing the project activities.
o Nicole Howard will provide the majority of project support with assistance from other consultants
as needed.
o Dr. Harold Cheatham will review all test material for gender and cultural bias.
34
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
❑ Dr. Jay Silva, George Conrad and Chris Williams are responsible for computer programming,
data analysis, and data entry and data scanning at EB Jacobs. On this project, they will be
responsible for all aspects of data entry and analysis.
For a detailed view of personnel usage by project task please see Appendix Q. Resumes of major
project staff are included in Appendix B.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
35 ..
V. IMPLEMENTATION STRATEGY
A. Exam Administration
Efficiency, security and standardization are the primary test administration goals that EB Jacobs will help
you achieve. EB Jacobs staff will work closely with the project team to develop the most efficient
administration process for the promotional examinations. Whenever possible, we recommend that all
candidates complete all phases of the testing process. The proposed process achieves this goal under
most circumstances. We recommend administering the written exams to all candidates in a single
testing session. This approach maximizes test security. This approach has not been a problem for the
City in the past but if resources are a concern, we will work to develop a multi -session approach, where
two testing sessions will overlap slightly, thereby ensuring that candidates from the first session do not
interact with candidates from the second session.
For the assessment center phase, we will conduct assessor training immediately prior to the
administration of the exercises. Per the RFP, two EB Jacobs representatives (Dr. Jay Silva and Nicole
Howard) will be on -site during the administration of the assessment centers to handle any issues that
may arise. City personnel will be responsible for assisting with the administration process.
B. Project Resource Requirements
Mutual understanding and clarity of each party's responsibilities ensures effective execution of the
project. Once the project plan is fine-tuned, a final resource requirements list will be provided. Until that
time, for the purpose of the Cost/Fee Proposal, the following assumptions regarding the responsibilities "
of EB Jacobs and the City have been made (begins on next page).
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
36
Initia|Communications
Initial Communications
Provide arecommended timeline and
Identify asingle poindofoontaotbosunxyasthe
resource list.
Internal project manager.
|dont�vna|�vant�arbesfortheproject ata�-up
Identify
Suggest stakeholders hopad)oipata in project
kick-off call.
call.
Revimvvthensoornn��ndednasuuroe
Padic|patainpn�actkic�offcall ««�h
nequirame�aandtinne|inosand»vorkvvith�B
stakeholders toensure mutual undmmta»dinQ
Jacobs bodevelop �na|versions that «viUba
ofQoo|sondd�|ivanab|ea�
followed bvboth parties.
Provide EBJacobs will updated copies of
Provide amended scope ofwork, timeline and
organizational charts, mivaion/va|uaa, and all
onot proposal, if necessary.
available job -related information /a.g., job
descriptions, job analysis data, description of
existing testing prooaoo\.
Provide EBJacobs with demographic profile of
the Department and incumbent population -
Jo b Analysis/Exam Phan Specification
Job Ano|ysis/Ezamn Plan Specification
Meet with SK8Estogather information onthe
Arrange and Coordinate SK8Esfor all job
job.
analysis activities.
Construct job
'
Copy, d\atdbuh*.gather job analysis surveys,
and forward nurvny'neoponoestothe City
Analyze job analysis survey data and create
Department ofHuman Resources orEB
exam plans that|inktoetoontanttothejob
analysis nsou|to.
Jacobs.
Prepare final exam plans.
CONFIDENTIAL m�mxmmmn^»ovmvumomconnxonm��o��men sm���muo
_
\
/
)
.Test Development
Prepare all exam material according to
guidelines set.forth in proposal..
Provide draft versions of appropriate materials
to the City for review.
Test Development
Provide necessary SMEs to review exam
Provide atimely review ofexam materials
by City.
Prior to Test
Provide guidelines for the test administration
facility and staff.
Provide camera-ready copy ofcandidate
preparation guide todistribute prior tothe exam
Provide tutorial session Question and Answer
padket to the City.
recommend to City, and secure all
assessors (3 per oral panel and 2 per written.
in -basket panel) and make travel arrangements
for assessors.
Prior TmTest
Locate and secure all testing facilities.
Handle all candidate registration activities
including accepting and screening
applications. ,
Schedule and notify candidates for all exam
components.
Copy and distribute candidate preparation
guide and tutorial Question and Answer
packet to candidates.
Review and approve all assessors
recommended by EB Jacobs.
Provide BBJacobs with list ofall scheduled
applicants, seniority points, and
demographic information onthe
Locate and secure all test monitors using
the following guidelines (at least one -to -
thirty monitor candidate ratio for written
tests and necessary monitors for the
assessment center).
'
CONFIDENTIAL 38
t I I l ;ICY 1 'y .h.w*1 /J fi M '- J
7C 1 rt 7 -;wi :e I er ''' l fl Tln L ��
J ray;, � �; �rJEi37Yiacobs:v+nll � .7'•rwJ:. Y �, ,.,F.�
l 1 ^Y i`...'J( c' TT'i. . -liii r} , (. ry �J Alms--:'.
C 1I .J LS ^Mv�:.,+'(JI l• S* l i. J 0
J,< ��fJ„;..,,.r^�-�-�:The:.City wilt :. u,r,;:���i� .::,:�';;
Training
Train assessors on -site over course of 1.5 days
.at City provided facilities.
Training
Provide facilities and meals (if appropriate) for
assessor training.
•
Administration
Provide all exam materials.
Provide two Masters level I/O Psychologists to
be on -site for all exam component •
administrations.'
Administration
Provide staff and guidelines for candidate
check -in.
Reproduce all candidate answer/data sheets
and send originals to EB Jacobs for data
processing according to agreed upon schedule.
Maintain copies at least until EB Jacobs
.confirms receipt of originals.
Maintain security of all test related materials
while in their possession.
Scoring
Produce any necessary interim and/or final lists
within 14 days of exam administration for each
classification.
Prepare relevant adverse impact analyses
and subgroup statistics. .
Scoring
Have representative available for consultation
as needed on data and analysis decisions or
issues with. candidate IDs or names.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified In the Confidentiality Statement of this proposal
39
r ,i n i, r wfA 'ai r j,.Q . i am `xatut ..yam rwu
��w rr, �G'F� � y �,'�dlw _. r i4} 1� � ry tiNF 1 wI.A
h -,.:4 -" fit EB �JacobsiwiII �� i" , ;� ,; ,- 4 . � i�,
�..,�„n_--r}i�rV__,wr,..t.,._.t.�...._.
,. .,ONgJrf 'r.r y-.1—c�€,i I a) - l'1`A lr. F ' 1'iy,�l s ,. A.`''i'
i� }, � , t r ��� -.1,�.cw G r t C .,,, .
,....Tv�r'���.�. „!r�.._ ..:. r�4.,�tr�,"ri��i��_.ari�.,..��.:y:r'-•!t
ig :=.1, 0..The:. Cify,.wilt
,r�_,a.. s,..r� _.
Reports/Debriefing
Prepare a final report outlining all project steps
and results for each classification.
Prepare written feedback for all assessment
center candidates.
Reports/Debriefing
If desirable, participate in a post-test
conference call with stakeholders to
discuss project and suggestions for
future administrations.
Due to security concerns arising from multiple testing sessions, we suggest that the City find test facilities
that can accommodate all candidates (per classification) for the written examinations. If we must run two
testing sessions, it is possible to use one test version without compromising test security. This is done
by holding all candidates from session 1 in the room until candidates for session 2 have checked in. We
suggest having a 3D-45 minute "overlap period" where the candidates for test 2 must report 30-45
minutes prior to the scheduled end of test session 1. If simultaneous administration or overlapping
sessions cannot be used, an amended cost proposal can be prepared to allow for parallel versions of
each applicable written test to be developed. If the City is interested in alternate arrangements, we
would be happy to provide a modified cost estimate. These represent the major categories of resources
necessary for the completion of this project. Additional resource needs, that may arise while finalizing
the project plan, will be mutually agreed upon by EB Jacobs and the City. Every effort will be made to
minimize the demand on City resources.
C. Project Staffing
Maintaining open dialog and updates on project activities are core components of our approach to
project management. Staff members are assigned to project roles based on their expertise.
o Co -Project Directors
Drs. Jay Silva and Janet Echemendia will provide technical guidance and determine policy -
related standards for the accomplishment of project tasks and objectives.
o Technical Advisor
Dr. Rick Jacobs provides input on standard procedures as well as policy issues that may arise
during the contract.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
40
❑ Project Manager
Dr. Jay Silva is directly' responsible for the day-to-day implementation of the project, service
quality and accuracy.
❑ Project Team
Nicole Howard will be the primary support consultant on the City's project. Joe Hinish, Brian
Himelright, and John Evans will provide any additional needed project support.
❑ Analysis Team
Dr. Jay Silva and George Conrad will conduct all statistical analyses necessary for the project.
Chris Williams will provide data scanning and data entry services
o Cultural Bias Reviewer
Dr. Harold Cheatham reviews test material to ensure they are free of unintended and unwanted
racial/gender bias.
Dr. Rick Jacobs, Chief Executive Officer of EB Jacobs, holds overall project responsibility.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
41
Vl. ABILITY TO PERFORM WORK WITHIN THE CONTRACT TIME FRAME
A. Schedule
Delivering quality services on -time is central to the EB Jacobs reputation. We have assembled an initial
time line for the completion of the final list as it has been defined in this proposal (Appendix R). The
proposed timeline will achieve the target test dates (e.g., April 2013 for initial Chief Fire Officer exam,
May 2013 for initial Captain exam, and May 2013 for initial Lieutenant exam) assuming that a contract
will be in place by September 1, 2012. It also assumes timely compliance with our requests for
assistance from the 'City project team and SMEs. Upon contract award, EB Jacobs will provide a more
detailed timeline for the project that includes dates for all deliverables (including those required by both
EB Jacobs and the City project team). If this information and/or our assumptions are deemed
inappropriate, we will modify and finalize this timeline to meet your needs and expectations.
B. Concluding Statement
We develop solutions that identify the best candidates for the position while gaining candidate trust and
minimizing adverse impact. Since 1972, the identification of top performers through job -related
comprehensive assessments has been our legacy. Developing effective human resource management
systems based on best practice, research, and the law is our hallmark. We offer the breadth of services
(e.g., competency modeling, internet-based recruiting, managerial development, performance
management) of an international consulting firm and the depth of knowledge of highly specialized and
trained public safety staff. Our proposal is just that —a proposal. We welcome the opportunity to discuss
possible modifications to better suit your needs. We look forward to the opportunity to work with the City
once again on this project.
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
42
Vll. WORK SAMPLES
Job analysis survey excerpts, sample assessment materials, and candidate materials can be found in the
following appendices:
o Appendix F: Sample Excerpt From a Task Analysis
o Appendix G: Sample Excerpt From an Ability Analysis
❑ Appendix H: Sample Excerpt From a Knowledge Analysis
o Appendix I: Sample Written In -Basket Exercise
o Appendix J: Sample Open -Book Technical Knowledge Test
o Appendix K: Sample Closed -Book Technical Knowledge Test
o Appendix L: Sample Situational Judgement Test
o Appendix M: Sample Subordinate Conference Exercise
o Appendix N: .Sample Incident Command Exercise
o Appendix 0: Sample Table of Contents for Candidate Preparation Guide
❑ Appendix P: Sample Candidate Feedback
CONFIDENTIAL
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
43
VIII. LOCAL PREFERENCE
EB Jacobs is not seeking local preference consideration.
CONFIDENTIAL 44
-Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
IX. TRADE SECRET EXECUTION TO PUBLIC RECORDS DISCLOSURES
The entire proposal including appendices and itemized cost proposal are designated as "trade secret."
The cost itself can be released per the City's rules.
CONFIDENTIAL 45
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal
Appendix F
• Sample Excerpt from a Task
Analysis
EXCERPTS FROM TASK SURVEY
.'n } i:1 M'�'1' '�:iAV, i!?�,4'7Jf� ibf �S A.. L�`�,.�fytr rY�15t.i�,' i,-.:.js.. .�•4-�£��•SH 1 x't" t—Y'rYy�?i 1x sY•,�',�� ��,(f %..
,stet .. x 5 ' ltr4;`rz;." "k01" !°njb7. 2 f4. f t , f [,r i3 i �iWW <'; .' , �,.y r1,11, . + '� 'i k{�4 i1s , Liv `it t ,0 r
«S.tem" -0 Revlewther llowing xa.,mple"lof how;yougw111suseaallfour scaales4to teleach, ask ,ATiASKAX �t5
i t„-%,-iks ...7,q;,11.iirbtog, ,at Vw i "i'r; S0.F.ki t j�N,,. t.+ i • iv.i tP.ven. . 7+'' '+j
ai dF.�TASK YAare sampI sithalwereipul11ed�fro fhre4listtsof ksttoillustrate howmresponsestII 1
?*.0 i` i = �G �R e' ' ,fir' th
shouitlibeirecordeddr the JAC P .esponse Booklet # •` )11 � r A
F Y ��� li"' x r ilf; v°i�''xrA,j� 1• t,,4' 't�Il "'°� , - t `-" ` v,4 ¢ t Vi4
f y `� R �b�� i S� ��Y 1 ' '�f u sr
1' M ?� �.ttlf �L t?�t� 'hF� b1 (AIL;
�+stSZ'� :MUM � i9 i'rn.f°avekr' i t�5irr�1A)tt�'tw��t'�yc tlt a I`�. r�ry Ju�=ci �.r�•�5'•.: k3.,1� S1�il r�i.'. ': �rn'.i'��tt#.4 �'l.'.',•..
THESE ARE ONLY EXAMPLES; THE RATINGS SHOWN HAVE BEEN PROVIDED FOR
ILLUSTRATIVE PURPOSES AND MAY NOT BE APPROPRIATE FOR YOU.
SCALE A
Performance
A = Yes
B = No
SCALE B
Frequency
A = Part of job, but never performed.
B = Previous years, but not this oast year.
C = Less than once a month.
D = Monthly; once ortwice a month.
E = Weekly; once or twice a week.
F = Once or more per shift.
SCALE C
Im_portance
A = Not Important
B = Minor Importance
C = Important
D = Very Important
E = Critical
SCALE D
Day One
A = Yes - Responsible on Day One.
B = No - Not Responsible on Day One.
TASK X: Obtain information„by telephone or radio on existing situation from supervisors of
responding units at the scene to assuming charge at the scene.
Scale
Response
EvIt
A A: Yes, if the appropriate situation occurs, 1 am responsible for performing this
task.
B D: Monthly; once or twice a month (over the past year).
C B: Minor Importance.
D A: Yes - Responsible on Day One.
TASK Y: Assesses scene to determine if the area is unsafe and if actions such as evacuation,
traffic re-routing, and crowd control procedures should be taken or if other agencies
(e.g., MPD, FP&L, water department) are needed.
Scale Response
A B: No, if the appropriate situation occurs, I am NOT responsible for performing
this task.
(Since response B was chosen, this task is not rated on SCALES B, C, and D.)
- -CONFIDENTIAL- .Appendix F_- Rage 1 _ _..,_....
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
SCALE A
Performance
A = Yes
B=No
SCALE B
Importance
A = Not important
B = Minor Importance
C = Important
D = Very Important
E = Critical
Task Ratings
Use numbers 1 through 166 of Part II of the JAQ Response Booklet to complete Step 7.
TASK LIST
A. ALARM TO ARRIVAL:
1. Listens for alarm and radio communications in order to respond to call, learn the address,
as well as other pertinent information.
2. Reads dispatch describing alarm in order to determine the nature of the alarm.
3. Directs subordinates to verify address by reading maps and/or other sources to determine
most efficient route and possible hazards for driving to alarm.
4. Directs subordinates to acknowledge to alarm office that unit is responding to the incident.
5. Observes firefighters to verify they have donned protective clothing/equipment to prepare
for emergency.
6. Directs subordinates to notify Alarm Office. by radio to preempt traffic lights in order to clear
driving route.
7. Confers with driver about route to be followed to avoid delay due to traffic jams,
construction, or other transporting apparatus.
8. Listens to radio while responding to scene to learn of emergency conditions in order to plan
what actions should be taken for the crew and apparatus.
9. En route to emergency incidents, provides the crew with information (verbally or with
signals) relevant to the company's response to prepare the crew for the incident.
10. Steers tiller of truck to assist driver in positioning truck.
11. Directs subordinates to notify alarm office of problems encountered (e.g., accident or
breakdown) on the way to the scene.
PLEASE CHECK TO ENSURE THE NUMBER OF THE ITEM YOU ARE RATING MATCHES THE
NUMBER IN THE RESPONSE BOOKLET AND MAKE SURE YOU ARE MAKING COMPLETE DARK
MARKS.
CONFIDENTIAL Appendix F - Page 2
- -Use or -disclosure of data-contained-omfhis page -is -subject -to the restriction-specified-in-the-Confidentiality-Statement-of_this.proposaI__
Use numbers
SCALE A
Performance
A = Yes
B=No
SCALE B
Importance
A = Not important
B = Minor Importance ,
C = Important
D = Very Important
E = Critical
Duty Ratings
167 through 181 of Part II in your JAQ Response Booklet to complete Step 9.
���:w �ria to -r—". r.�^sr•�. � �1u�Te
eviemtf e duies-on pagesi,222throug_ antl ecord-youfi rating;foraeacf xo,tt . es o scaies
DUTY/DEFINITION LIST
167. RECORD AND REPORT MANAGEMENT: This duty involves reviewing, preparing and/or
maintaining logs, records, forms, memos, reports and other field and administrative documents
and correspondence used in the course of performing the job. This includes reviewing
documents prepared by subordinates or other personnel for completeness and accuracy,
reviewing and authorizing personnel requests and integrating information from multiple sources
into summary documents.
168. INCIDENT COMMAND: This duty involves assuming responsibility for incidents or special
operations, to ensure the efficient and safe handling of the incident/operation and the
preservation of life and property. This also includes conducting follow-up, debriefings and
critiques of the incidents or operations.
169. INVESTIGATIONS/INSPECTIONS: This duty involves conducting or supervising the conduct
of arson -related or internal investigations. This includes the coordination of the investigation
as well as participation in field work and interviews.
170. OFFICIAL REPRESENTATION: Acts as the official representative of the. Department. In this
capacity you may meet with other fire agencies or the media. Departmental representation can
include personal appearances, television interviews and formal written correspondence.
171. PERSONNEL ALLOCATION: This duty involves planning and assigning personnel to various
functions -within your command. This takesthe -form.. of eitherplanning schedules yourself or
ensuring that plans developed by your subordinates conform to the manpower needs of the
department.
PLEASE CHECK TO ENSURE THE NUMBER OF THE ITEM YOU ARE RATING MATCHES THE
NUMBER IN THE RESPONSE BOOKLET AND MAKE SURE YOU ARE MAKING COMPLETE DARK
MARKS.
CONFIDENTIAL Appendix F - Page 3
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
OVERALL JOB RATING
Use number 182 of Part II in your JAQ Response Booklet to complete Step 10.
You have lust reviewed the Itst of ge•neral job duties and tasksrfor the' rank of Lieutenari
the Department We would like you:to provide one:.flnal rating to‘indicate theidegree,to
which youlfee! thtsttask Itst encompasses the ac ivities performed by Lieutenants m thef
l]eoartment
182. How much of the job, as you see it, has been covered by the tasks and general job duties
listed? (By job, we mean the complete set of duties and responsibilities which are performed by
those in the same job classification.) When making this judgment, consider only duties and
responsibilities. Knowledge and abilities will be addressed in another survey,
)B DESCRIPTION COVE
5a`ctiviiti.es
Please fill in your response to Question 182 under Scale A.
COVERAGE
182
183
184
me- oriDet oca i cD(_L'1Cr;!titCT CQw5 CaDeorma)ce,ez
%h'!ocommki f,C)'II C6)mCw imf'J'( GS-'iriC'iCG%omwi
czo o m o f m G G`J e CJ ti) (!s malomoo.s
Eieuterian
u'wCtsiE=u„.tmry w;tl
a) f1i3
Va ..ifr)r.g)'G) 0Cuo
C eT) cr
t?)4..r3 ao CF3
CONFIDENTIAL Appendix F - Page 4
Use or disclosureof-data contained on this page is subject to the restriction specified -in: the Confidentiality Statement-of-this-proposal.-
J
CONFIDENTIAL Appendix F - Page 5
Use or disclosure of data contained on this page is subject to the restriction specked in the Confidentiality Statement of this proposal.
Appendix G
Sample Excer.pt from an
Ability Analysis
Sample Fire Job Analysis Ability Survey
The Ability Rating Scales
SCALE A: importance
Record your ratings for the Importance of each ability in the column labeled SCALE A.
Here, importance should be viewed broadly in terms of your responsibility to preserve life and
property and ensure that subordinates are performing effectively and efficiently. When rating
importance, consider how critical each ability is to performing your job as a Lieutenant. Please refer
to the following scale when assigning your IMPORTANCE ratings.
When you consider and rate the importance of each ability, do NOT consider how frequently you use
the ability. For this set of ratings, we are only concerned with importance.
CONFIDENTIAL Appendix G - Page 1
Use or disclosure of data conlained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
Scale
A
ABILITY Y:
Sample Fire Job Analysis Ability Survey
Record your ratings for the Importance of each ability in the column labeled SCALE A of your JAQ
Response Booklet as shown below:
IMPORTANCE
:t474' Y lnhi r✓r�.h O(�EYk;;4 P /,
�.?1 f.. x,:.'1P R*. f1
2
3
%i MP$ rT` r'' a 00Yan Qzs ; p 3 * pp f .�'r,.'1.` sir
• Ck3%F .es?u?t.2s. 1 tlk7''i T1 7.°s'-;ieCv{.ffar
Er. t.•t 1,?I
7ftr Er~, c�UFi �G�i1
r, ca cioi;r?L £'srmiat
4 co.ciis:. c iS J
•5viy��rii.Sig4txi`::W(. ,v u�ci �vi
exampleoe ewefllov}v�'�.- rrr•�FM}I4'�'1'+j� CRN+)?Y.•;�: .? •�
you,wtli use h�s�one�s�cale to rate eachrabtlrr
h,4B1d':4Ijl C*andf'ABILiTYAYsare�sample�st hat w retpul edfrornahe actual �s�tofxab
illUs't•ate�howiresponses'rshouldberecortle�tlr.inathe•JAQi3Response,Bootklet`}�s3",;
�} t t rk".r, .Its{ c Nr '2,,,i., . l� ; , qP f I"t•z .rA;49'
t4 a2 `tY �k����f' j, �,..;�- �r t t��. r ='dl
d.,r., N .��a,.e4 "'�,uFk 4 �.'�"".,,5.11 .�rt!�,�..P:• r.1.�- . �,;a.�'+' � f ...
THESE ARE ONLY EXAMPLES; THE RATINGS SHOWN HAVE BEEN PROVIDED FOR
ILLUSTRATIVE PURPOSES AND MAY NOT BE APPROPRIATE FOR YOU.
SCALE A
Importance
How important is this ability to performing your job?
A = not very important to the performance of my job.
B = helpful, but not necessary, for the performance of my job.
C = important to the performance of my job.
D = critical to the performance of my job.
ABILITY X: Ability to communicate orally, in an effective manner, to respond to questions or
concerns, from subordinates, superiors, attorneys, judges, community members, etc.
t+1r .,. t%c. .lrF ) rfzOf.r•LrjN.Lr�m.,
mwrn.rr�s>co 1 c rctuOCIDGDC xak i„asaD tIE=McDt i s iI)Ctie tBITYCM, rrr
Response
C: Important to the performance of my job.
Ability to communicate ideas and required information in an organized, clear, and
concise written style, in order to prepare reports, letters and memoranda.
:44r a'qicfjPAggiffgrear4
Scale Response
A B: Helpful, but not necessary, for the performance of my job.
CONFIDENTIAL. Appendix G - Page 2
Use or disclosure.ordata-contained.on.lhis.page.issubjecLlo the restriction specifiedJn_the.Confidentiality Sfatemenf of this proposal.
Sample Fire Job Analysis Ability. Survey..
SCALE B: Relevance
2
= ESSENTIAL - This ability is essential to
IAlarm to Arrival
Size -Up and Incident Command
Search and Rescue
Ventilation, Salvage, and Overhaul
Emergency Medical Services
Tactical Familiarizations and Inspections
Station Policy -Making
Monitoring, Evaluating, and Counseling
Subordinates
the performance of most of the tasks in
this group. Without this ability you
would not be able to perform these
1
tasks.
= HELPFUL - This ability is helpful in
.3 w
c m
m:=
performing most tasks in this group,
but not essential. These tasks could be
D
o 0
performed without this ability, although
o g_
it would be more difficult or time
` a)
o fl
consuming.
i•- a)
o—
0
= NOT NEEDED - This ability is not
cts
needed to perform most tasks in this
°'
group. Having this ability would make
a .c
no difference in the performance of
m E
these tasks.
a_ Q
1. Ability to communicate orally, in
an effective manner, to respond
to questions or concerns, from
subordinates, superiors,
attorneys, judges, community
members, etc.
2
2
2
0
0
_
2
2
•
2
The ratings selected above indicate that oral communication ability is essential for six of the nine job
tasks, helpful for one task, and not needed for two tasks.
CONFIDENTIAL Appendix G - Page 3
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
Appendix H
Sample Excerpt from a
Knowledge. Analysis
Sample Fire Job Analysis Knowledge Survey
The Knowledge Source Rating Scales
�Yf�' 1irj--^•aafl�Z S�VStl�"•r��` pn� ills �+t` }P�SfrFt�iS�ri .11...".ri�r._h�J%i,1'•'ti t A�a�x�n �Q ,
rr N•. 'N ia1'f �1f I In g r f , ;� .;
y6:�t'tReulew=the followln ,,exarnplefeflaowryoutrwill ue�all threescales1rate{eachtknowledgersource
}�'. , E„ A: 3 nd�KNOWLEDGE1YTacesam Ies that�wereufledf om�tFa't ry _�•
s u � e111st9of�sources�to•'�
KNOW11110. E X firtx�tXVIRRA ,zV-Viefike- , ,1ret a'er•r p;.i;� l' { t, 4iar�,'<< 1 `
lilllusiratellioWr,esponses Should be-recordedlm e JAQ ResponseNBsogklet + �
yr y , o r 9 .. , .�aP rJ� r rr,
�tv. cc3 '4 t fir �y�?- ,
rl Y . ig� Sl N a,i f{ j � t �c�'i 5;, ,,tC " S�/�,1•43� � 4f�c�l�. '�j�''�"�� ����r�-{f �f!"3'✓�{' �.v; , 4 � xu,, �
, t�f rt ,,� Its (.
�F,� d�, r � S s»"i4^'.Iri •� .. t. ; f��?�rxr �i,rr, .,3�f . , . ,...
ri1Y,M,�ttka�i)�r'6.Ls�.ihT]� .o't�0' 1�7r-PS�llt�rt•�vJ�.'?"n •.r��
THESE ARE ONLY EXAMPLES; THE RATINGS SHOWN HAVE BEEN PROVIDED FOR ILLUSTRATIVE
PURPOSES AND MAY NOT BE APPROPRIATE FOR YOU.
SCALE A
SCALE B
SCALE C
Use
Importance
Manner of Use
A = Yes
A = not very important to
A = 1 am always able to refer to the source; I never need to
recall the information from memory.
B = No
the performance of my
job.
B = 75% of the time I am able to refer to the source; 25%
of the time I must recall the information from memory.
C = 50% of the time I am able to refer to the source; 50%
B = helpful, but not necessary,
for the performance of my
of the time I must recall the information from memory.
job.
C = important to the •
D = 25% of the time I am able to refer to the source; 75%
of the time I must recall the information from memory.
•
performance of my job.
D = critical to the
E = I am never able to refer to the source; I must recall the
information from memory all the time.
performance of my job.
KNOWLEDGE X: Knowledge of common prevalent violations that are likely to exist in building (e.g.,
blocked areas, illegal wiring) and thus to look for them in inspections.
vi•i rV .t,;,% r
Scale Response
{
s.k4r.,
A A: Yes. I use the information in this source to perform my job.
B C: Important to the performance of rnyjob.
C A: I am always able to refer to the source; I never need to recall. information from
memory.
KNOWLEDGE Y: Knowledge of hydrant maps and hydrant notice regarding water availability,
emergency sources of water and auxiliary water supply systems.
Scale Response
A B: No. I do not use the information in this source to perform my job.
(Because response B was chosen, this source is NOT rated on Scales B and C.)
.SCALE D: Relevance
CONFIDENTIAL Appendix H - Page 1
Use or disclosure of data contained on this page is subject to -the restiicffon specified in -the -Confidentiality Statement of this proposal.
Sample Fire Job Analysis knowledge Survey
2
= ESSENTIAL - This knowledge area is
Preparing for Tour of Duty and
Administrative Responsibilities
Alarm to Arrival
Size -Up and Incident Command
Search and Rescue
Ventilation, Salvage, and Overhaul
Emergency Medical Services
Tactical Familiarizations and Inspections
Station Policy -Making
Monitoring, Evaluating, and Counseling
Subordinates
essential to the performance of most of
the tasks in this group. Without this
knowledge area you would not be able
to perform these tasks.
1
= HELPFUL - This knowledge area is
helpful in performing most tasks in this
group, but not essential. These tasks
could be performed without this
knowledge area, although it would be
more difficult or time consuming.
0
= NOT NEEDED - This knowledge area is
not needed to perform most tasks in this
group. Having this knowledge area
would make no difference in the
performance of these tasks.
1.Knowledge of case assignment
criteria to apply in determining
o
0
2
1
0
1
0
0
0
. whether a case should be
investigated by the unit.
The ratings selected above indicate that knowledge of case assignment criteria is essential for one
job task, helpful for two tasks, and not needed for six tasks.
--CONFIDENTIAL-- - ._ ... _ _ _..__ . _... _-. . _.._ __ _ ... __ _... ._ ._..---AppendixH-Page 2
Use or disclosure of data contained on this page is subject to the restriction specked in the Confidentiality Statement of this proposal.
Appendix 0:
Sample Table of
Contents for Candidate
Preparation Guide
TABLE OF CONTENTS FOR SAMPLE CANDIDATE PREPARATION GUIDE
I. INTRODUCTION 1
II. DESCRIPTION OF THE EXAM 2
A. Date, Time, and Location 2
B. Confidentiality 2
C. Content 2
1. Written Test Components 2
A. Open -Book Technical Knowledge Test 2
B. Closed -Book Technical Knowledge Test 3
C. Situational Judgment Test 4
2. Oral Board Exercises 6
A. incident Command 6
B. Subordinate Conference Role -Play 7
D. Administrative Logistics 8
1. Written Tests ...8
2. Oral Board Exercises 9
A. Preparation 9
—) B. Participation 10
E. Evaluation 11
1. Exam Component Weights 13
2. Exam Answer Key Challenge Process 14
3. Assessors 14
4. Procedure 15
F. •Parallel Forms 15
i11. PREPARATION AND TEST -TAKING STRATEGIES FOR MULTIPLE-CHOICE TESTS 16
A. How to Study 16
B. General Written Multiple -Choice Test -Taking Strategies 20
C. Error Analysis 22
D. General Test -Taking Suggestions 24
IV. PREPARATION AND PARTICIPATION STRATEGIES FOR THE ORAL BOARD EXERCISES 26
A. Oral Board Exercises Preparation Strategies 26
B. Oral Board Exercises Participation Suggestions 29
. Appendix O - Page 1
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
V. SAMPLE WRITTEN TEST ITEMS 30
A. Sample Technical Knowledge Test Items 30
B. Sample Situational Judgment Test Scenario and Items 31
VI. SAMPLE ORAL BOARD EXERCISES AND SUGGESTED RESPONSE
APPROACHES • 33
A. Sample Oral Board Exercises 33
B. Suggested Approaches to Oral Board Exercises 35
VII. CONCLUSION 37
CONFIDENTIAL
_. __ _ .. - APPen ix- _ __. Page-2- -
Use or disclosure of data contained on this page is subject to the restriction specked in the Confidentiality Statement of this proposal.
Appendix Q:
Project Staffing Plan
CITY OF MIAMI
FIRE -RESCUE LIEUTENANT, CAPTAIN, AND CHIEF FIRE OFFICER
EXAMINATION PROCESSES
PROJECT STAFFING PLAN
s.• �� tG
-Y 't,.-4
'�' ` ' im
.'.x� y
,,J'-d-E' -f �e i.m w
..tJ
i.
a.
Job nalysis/
�
_lam
Development}lAd
Rick Jacobs (PhD)
X
X
P
X
Janet Echemendia (PhD)
X
X
X
Jay Silva (PhD)
X
X
X
X
X
Nikki Howard (MS)
X
X
X
X
X
George Conrad (MS)
X
P
Chris Williams
X
Harold Cheatham (PhD)
X
Joe Hinish (MS)
P
P
Brian Himeiright (BS)
P
P
John Evans (BS)
P
P
Key for Usage: X = Key Resource P = Resource May be Used as Needed
CONFIDENTIAL Appendix Q - Page 1
_ ._Use.or_disclosureof data contained on_this page is subject to the restriction specked in the Confidentiality Statement of this proposal.
Appendix R:
••
Project Timeline
Proposed City of Miami Lieutenant. Captain, and Chief Fire Officer Project Timeline
Lieutenant Project Timeline
t
jjpxe 5 i+' j � '"T
a -1 a ,d `He K*.,- rm .w vc" k c gyp
- _ %ggy. ., ' ws ^
'"
t.' T1" � -'�S_r1 - E �y � Ii
Week 1
Kick-off conference call
Weeks 2 & 3
Job analysis SME interviews / task list reviews
Week 4
Finalize job analysis survey and provide City with camera-ready copy to
distribute to Lieutenant incumbents
•
Weeks 5 & 6
Provide two weeks for survey completion
Weeks 7 to 12
Receive job analysis survey responses and analyze data and produce job
analysis report
Week 13
Prepare and finalize book list (jointly with City) and complete exam plan
based on job analysis results
Week 13
Begin exam material development; conference call with City SMEs
Week 16
Provide City with camera-ready copy of Candidate Preparation Guide for
distribution to candidates -
Week 21
SME review of exam materials
Week 21
Provide City with information on the percentage of the job domain measured
by the exam materials
Weeks 25 & 26
Deliver candidate tutorial sessions & prepare questions and answers packet
Week 28
Provide City with all exam instructions for review
Week 31
Administer written tests and collect written test appeals by candidates
Week 35
Complete review of appeals, analyze test results, and provide
recommendations to appeals; wait for final decision by City on appeal
recommendations
Week 37
Provide final score list within 14 days of final decision on appeals by the City
Week 40
Final report provided to City
CONFIDENTIAL Appendix R - Page 1
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
Proposed City of Miami Lieutenant, Captain. and Chief Fire Officer Project Timeline
Captain Project Timeline
`. ft* tAtdkVi.,?", o
C1h��
t T
�
11.4�
;'�r
'r.n, k' P
�-„.1
y..,,_AIctvi. y_E;,sF,�4Lt��.
r,,
t_.,;.
Y.i.., ..r!�.a�
(rL ++4..-
r12I,..,, ..at,._ . ,e 4....i ...�. r 1.. .
r
Week 1
Kick-off conference call
Weeks 2 & 3
Job analysis SME interviews / task list reviews
Week 4
Finalize job analysis survey and provide City with camera-ready copy to
distribute to Captain incumbents
Weeks 5 & 6
Provide 2 weeks for survey completion
Weeks 11 to 12
Receive job analysis survey responses and analyze data and produce job
analysis report
Week 13
Prepare book list (jointly with City) and complete exam plan based on job
analysis results
Week 13
Begin exam material development; conference call with City SMEs
Week 16
Provide City with camera-ready copy of Candidate Preparation Guide for
distribution to candidates
Week 21
SME review of exam materials
Week 21
Provide City with information on the percentage of the job domain measured
by the exam materials
Weeks 25 & 26
Deliver candidate tutorial sessions & prepare questions and answers packet
Week 26
Provide City with list of proposed assessors
Week 28
Provide City with all exam instructions for review
Week 31
Train assessors, administer written tests and assessment center
Week.35
Complete review of appeals, analyze test results, and provide
recommendations toappeals; wait for final decision by City on appeal
recommendations
Week 37
Provide final score list within 14 days of final decision on appeals by the City
Week 38
Provide City with camera-ready candidate feedback reports to distribute to
candidates
Week 40
Final report provided to City
Note. Schedule would be modified if the number of scheduled candidates required a hurdle approach.
CONFIDENTIAL Appendix R - Page 2
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.
Proposed City of Miami Lieutenant. Captain, and Chief Fire Officer Project Timeline
Chief Fire Officer Project Timeline
41- C ,yF igl
43 ' - =' e' ;::.r'64' rt ,hi..-=-Vil n'.y t 3 t-2' .�.i.-'>A, s�+:.lisR sr°- 5 -A ',, 4 # r ?``� vva
.,+.. 'r., '� { ± di. -.4..,3irs ,?: +RF.: .r 5T,4 3:r aar`,�- nri Vr.s.: .. •.r
r{ c1 .t. `�-
Week 1
Kick-off conference call
Weeks 2 & 3
Job analysis SME interviews / task list reviews
Week 4
Finalize job analysis survey and provide City with camera-ready copy to
distribute to Chief Fire Officer incumbents
Weeks 5 & 6
Provide 2 weeks for survey completion
Weeks 11 to 12
Receive job analysis survey responses and analyze data and produce job
analysis report
Week 13
Prepare book list (jointly With City) and complete exam plan based on job
analysis results
Week 13
Begin exam material development; conference call with City SMEs
Week 16
Provide City with camera-ready copy of Candidate Preparation Guide for
distribution to candidates
Week 21
SME review of exam materials
Week 21
Provide City with information on the percentage of the job domain measured
by the exam materials
Weeks 25 & 26
Deliver candidate tutorial sessions & prepare questions and answers packet
Week 26
Provide City with list of proposed assessors
Week 28
Provide City with all exam instructions for review
Week 31
Train assessors, administer written in -basket and assessment center
Week 33
Provide final score list within 14 days of last oral board administration
-Provide
Week 38
Gity-with camera-ready candidatefeedback-reports-to-distribute-to .....
candidates
Week 40
Final report provided to City
CONFIDENTIAL Appendix R - Page 3
Use or disclosure of data contained on this page is subject to the restriction specified in the Confidentiality Statement of this proposal.