July-September 2008 (Quarterly Newsletter)
Design & Construction
Building Transparency into Qualifications-Based Selections 
by John A. Carter 1
“I know of no safe depository of the ultimate powers of the society but the people themselves; and if we think them not enlightened enough to exercise their control with a wholesome discretion, the remedy is not to take it from them but to inform their discretion.” Thomas Jefferson
Thomas Jefferson may have been talking about education in this quote, but his underlying theme of the importance of an informed citizenry still resonates through our political culture. Arizona has historically favored open government and an informed citizenry. For confirmation, one need look only to Arizona's public records law 2 that dates to territorial days and presumptively opens to public inspection any record “reasonably necessary to maintain an accurate knowledge of official activities” and Arizona's Open Meetings Law, 3 enacted in its original form more than 45 years ago. These enactments reflect Arizona's clear recognition of the importance of transparency in the conduct of public business.
Growing public agency use of alternative project delivery methods (APDM) in Arizona has been shadowed by parallel growth in complaints about a lack of openness in the associated selection processes and an absence of meaningful debriefings. Addressing these criticisms requires that agencies shed the “black-box” approach to qualifications-based selections and crack open the inner workings of the evaluation process in a way that transparently discloses not just the result, but how and why that result was derived. Transparency is the key.
This article discusses the process implemented by the Design and Construction Division of the Pima County Procurement Department specifically to incorporate greater transparency into qualifications-based selections. 4 For readability, it assumes the reader is already familiar with Arizona's APDM legislation. References to Exhibits are hot-linked; clicking on “ Exhibit X ” should take you directly to the example document.
Identifying, developing and implementing a transparent qualifications-based evaluation process can be a daunting task, particularly where it represents a shift from past practice. Agencies that ordinarily have not disclosed the inner workings of their evaluations may experience some discomfort in the transition. The overall objective, however, should be the institution of an open, rigorous and fair evaluation process that clearly communicates the evaluation process and results to participants, industry, press and the public.
Achieving transparency requires a focus on three related principles: First, communication is critical and central to everything; second, to the maximum extent possible, everything should be public information; and finally, numeric scores and/or rankings by themselves are relatively meaningless and should always be accompanied by explanatory narratives.5 Whenever possible, issues and concerns should be resolved in favor of these principles.
Communicating the evaluation process begins with the establishment and publication in the solicitation of appropriate evaluation criteria with their relative weights. In a textbook world, criteria would be fashioned independently for each project. Realistically, though, public agencies rarely have the resources to develop new criteria for each new project and, over time, the evaluation criteria for any particular type of project have a tendency to become standardized. Nonetheless, the criteria and their relative weights should still be revisited and tweaked to reflect the particular needs of each individual project and incorporate lessons learned from prior projects. Criteria and their weights should be reflected accurately in the solicitation.
Similarly, if stated, subcriteria within each major criterion in the solicitation should be reviewed and revised, if necessary, to reflect the particular needs of the project being procured. Although not required by the APDM legislation, it is good practice to identify the relative weights of subcriteria as well, especially if one or more of the subcriteria may be significantly more important than others under that criterion. Absent any statement of the relative importance of subcriteria, competitors should be entitled to presume that subcriteria are approximately equal in importance and will be evaluated on that basis. An evaluation inconsistent with this expectation is likely to invite a protest.
You should also be aware that a variant of this principle may apply even where subcriteria are not listed or identified in the solicitation. The use of broad criteria such as “Qualifications of the Firm” or “Qualifications of the Project Manager” without subcriteria is simply an invitation for a firm to submit whatever it deems appropriate to address a criterion with the implication that it will all be considered more or less equally. If there is a particular type or category of information or a narrow subject matter that is substantially more important than anything else within that criterion, it should be disclosed to help get better proposals and avoid misleading potential respondents into submitting the wrong information. Not doing so runs the risk of a protest claiming that the protester would have provided that information if you had only told it what you wanted or, in many ways worse, leaving the perception that the agency is unwilling to disclose what it is really looking for so that it can “game” the system. This is not something that encourages firms to invest in responding to an agency's procurements and can be costly in the long run in terms of lost participation.
Building a high degree of transparency into an evaluation requires that evaluators provide explanatory narratives with their scores. This can be elicited by providing specific guidance to evaluators on the importance of providing meaningful comments and providing score sheets that demand categorized comments explaining the basis for the numeric score assigned by that evaluator. A sample from a recent Pima County procurement is attached as Exhibit 1 .
The three categories 6 of comments are defined as below:
Strength: A response or part of a response that goes beyond merely acceptable or adequate in relation to the stated requirement. More than meets requirements.
Weakness: A response or part of a response that suggests a minor flaw or misunderstanding in relation to the stated requirement; or a response that either should have contained additional detail or demonstrated more extensive experience or knowledge in relation to the stated requirement.
Deficiency: A response or part of a response that represents a material failure, omission or significant weakness that, if uncorrected, could threaten successful performance.
The major advantage of using these categories is that the hierarchy of comments provides a common structure to the way in which the evaluators assess proposals and promotes a more consistent approach to the evaluation. In addition, the accumulation of comments within categories provides a clear explanation of why any particular proposal received the score it did. Past experience using a single block for comments without categories was not favorable, with the most obvious drawback being that positive and negative references would be commingled and the underlying rationale for the evaluation score difficult to discern, particularly when the comments of multiple evaluators were combined. Dividing the comments into these categories provides much clearer communication of the rationale for the evaluation result.
Guidance provided to evaluators may take a variety of forms used either individually or in combination. Exhibit 2 is a generic narrative guideline for evaluating proposals and making comments; this guideline assumes a reasonably sophisticated and consistent level of subject matter expertise among evaluation panel members. This guideline is appropriate for any qualifications-based evaluation, but may require supplementation if the level of subject matter expertise among evaluators is uneven. In the latter case, a more detailed matrixed guideline may be necessary, such as that contained in Exhibit 3 from another recent Pima County procurement.
Developing matrixed guidelines can be resource-intensive, but is well worth the effort, particularly where an agency may have repetitive procurements of a similar nature. Using matrixed criteria helps ensure greater consistency not just within an individual procurement, but also across multiple procurements with different evaluation panels.
While matrixed criteria can be very helpful, care must be taken to ensure that they do not impinge unnecessarily on the essential qualitative component of the evaluation and reduce it to a mere quantitative exercise. For illustration, consider two engineers, A and B, each with 10 years work experience. Both do roadway design and both got their start designing right turn lanes. A is now a nationally recognized expert in the design of complex multi-level expressway and interstate intersections; B has added left turn lanes and bus stop cutouts to his repertoire. Although both nominally have 10 years experience, it should be obvious that there is a significant qualitative difference in experience between A and B that would be lost if we simply assigned the same number of points to everyone that had 10 years experience. While this is an obvious exaggeration for purposes of illustration, it does reflect a real world problem that must be avoided when developing matrixed guidelines.
At the conclusion of the initial evaluation team meeting, the evaluators are requested to evaluate proposals independently and return their completed score sheets and comments to the Contracts Officer and also reminded that their comments and scores will ultimately be public information. Upon receipt of the completed score sheets, the Contracts Officer tabulates the scores and prepares a consolidated matrix that integrates the scores and comments from all of the evaluators. Exhibit 4 is an evaluation matrix from a recent Pima County procurement. In the process of doing so, the Contracts Officer also informally assesses the consistency between the comments and scores of each individual evaluator. Apparent inconsistencies, nothing but favorable comments with a low score, for example, or vice-versa, trigger an inquiry to the evaluator and a request that they revise their comments and/or scores to be consistent with the underlying principle that the narrative should explain and be consistent with the score.
If the initial matrix is sufficiently consistent across all evaluators to reflect consensus, the Contracts Officer identifies the highest-ranking firms to invite to interviews and distributes the list and scoring results to the evaluators for confirmation. If the initial matrix does not reflect sufficient consistency to indicate consensus, then the Contracts Officer convenes an evaluation panel meeting at which the matrix is distributed to the panel and used as the basis for discussions to develop consensus.7 During this latter meeting, individual evaluators may adjust their scores based on the discussions – with appropriate comments explaining the adjustments – and the Contracts Officer retabulates the final scores and annotates the matrix to reflect the changes.
If interviews are going to be held, the agency must notify the firms to be interviewed and should include in the notice the evaluation criteria to be applied, if they differ from those stated in the original solicitation. Interviews may be evaluated following the same basic method in a simplified format. The evaluators are again provided with score sheets by criterion with space for comments on strengths, weaknesses, and deficiencies on which they note their comments and scores during the interviews. At the conclusion of the interviews, the panel members discuss the interviewees and their assessment of each. During this discussion, evaluators may add to or modify their comments and scores. At the conclusion of the discussion, the panel submits their score sheets to the Contracts Officer who tabulates the interview scores.
How an agency establishes the final scores that identify the final ranked list of competitors depends on how the agency has structured the procurement. If there are no interviews, then the ranked list is obviously based on the scores for the written statements of qualifications. Pima County typically regards the written statement of qualifications and the interview as being equal in weight and either sums or averages the scores from both evaluations to determine the final list. Other agencies discard the scores from the evaluation of written statements of qualifications and rely solely on interview scores to rank their final list. Yet other agencies may apply a hybrid of these two methods by assigning different weights to the written and interview scores. Any of these methods is acceptable—as are variations--so long as the relative weights of the written and oral submissions are disclosed in the solicitation.
The same basic evaluation method is also appropriate for use in evaluating proposals submitted in response to a request for proposals issued as a second-step in a qualifications-based procurement for either a design/build or job order contract, particularly if the second-step RFP process contemplates the conduct of discussions. In this latter circumstance, detailed evaluation comments are invaluable in identifying areas for discussion regarding each firm's proposal.
The evaluation process will end with either the selection of a ranked final list of competitors with whom negotiations may be conducted or with a winner, depending upon how the procurement is structured. At one or more points between issuing the solicitation and the final selection, competitors will have been eliminated from consideration.
In the interest of transparency and fairness, Pima County consistently notifies all competitors of the results at each point in the process where one or more firms may be eliminated from the competition, including the identification of firms selected for interviews. The form of the latter notice may differ somewhat – requiring individual “have/have not” been selected for interview notices – but it does free firms that were not selected for interviews to pursue other business. We also consistently notify all of the original competitors of the results, typically by faxing a one-page notice that identifies the firms on the ranked final list or the winner, as appropriate, to all of the original competitors.
Sending out notices that tell a recipient it was not selected and someone else was will almost invariably trigger debriefing requests. Debriefings commonly have two purposes—first, to explain to someone why their proposal was scored as it was in the current procurement and why they may not have been selected 8 and, secondly, to assist them to improve their responses in future procurements by identifying shortcomings that need to be addressed.
There is another unstated purpose for debriefings – opportunity might be a more accurate term – that may be far more important from an agency perspective. In the simplest terms, a debriefing is also an opportunity to sell the agency's process by not just explaining how the evaluation was done and how a firm's proposal was perceived, but by actually showing the firm how it was done and how the results of their evaluation were derived. Done well, the firm being debriefed may not leave happy, but they should at least depart with a greater understanding of the process that (hopefully) will at least foster a degree of respect and confidence in how the agency goes about its business.
Debriefings should be as open as possible. The first document Pima County typically provides in a debriefing is either the average score and compilation of the evaluation comments on the firm's proposal—taken from the matrix of evaluation comments--or the raw score sheets for the firm's proposal with the evaluators' comments intact, if we did not prepare a hardcopy matrix of comments. Anything else the firm requests is also made available, including other proposals, the comment matrix on all proposals, and even the raw score sheets and comments on all proposals, if asked. The identities of individual evaluators are masked by using numbers or letters in lieu of names on debriefing documents, although the key will also be provided if specifically requested.
All indications are that adopting this process has had favorable consequences. Certainly the anecdotal evidence to date suggests strongly that implementing a more open evaluation process has enhanced contractor and consultant confidence in the County's qualifications-based selections. Positive comments about the thoroughness of the evaluation process are common among the contractors and consultants that have served on County evaluation panels; the observation that a debriefing was the best a firm had ever received is frequent, if not universal. Moreover, an unanticipated benefit has been the ability to provide “electronic debriefings” for out-of-town firms and even some local firms by emailing them a copy of the evaluation comments on their proposal. In the vast majority of cases, this has been favorably received and satisfied the requester without a formal debriefing. All in all, the results have been very favorable.
This is not to suggest that the evaluation process we implemented is a panacea or that all agencies should conduct their business in the same way. It is, however, recognition that building and maintaining confidence in APDM procurement processes through transparency is essential to preserving the ability to attract the truly “most qualified” to public projects. This should be the goal of every public agency and I encourage all agencies to support this goal by pursuing greater transparency in their APDM procurements.
1 Mr. Carter currently manages the Design and Construction Division of the Pima County Procurement Department. He is a retired member of the DC Bar and was a contracts and procurement attorney focusing on defense acquisition issues at the United States General Accounting Office from 1975 until 1999. The views expressed in this article are solely his. He may be contacted by email at: John.Carter@Pima.Gov.
4 This process could not have been implemented without the patience, support and contributions of the Division's outstanding staff—Terri McMahon, Chris Barnhill, Jerome Rizzo, Melissa Hala'ufia, Connie Sanzo, Sue Aegerter, and Stephanie Steward--to whom the author extends his deepest gratitude.
5 Although this article is couched in terms of numeric scores, the same basic considerations apply to rankings as well. The procurement record should show not just what a firm's rank was, but why the firm was ranked as it was.
6 These particular three categories derive from the author's federal experience. An agency may call these by any names. The key is to provide evaluators an easy way to note strengths and degrees of weakness in a proposal without being so complex as to be counterproductive. In the author's experience, three categories is fairly typical and about optimal; more than four categories engenders more confusion than contribution.
7 As a practical matter, it would be unreasonable to expect all evaluators to be entirely consistent in their assessment of proposals. Different evaluators will invariably have different backgrounds and experiences that are reflected in their evaluations. What a mechanical engineer sees as a strength, for example, might be a structural engineer's weakness. These discussions allow evaluators to gain an appreciation of the perspectives of other evaluators and adjust their scores based on recognition of the differences.
8 This is important as a way to forestall protests. It should go without saying that the ability to explain the rationale for an evaluation to the satisfaction of a likely protester can stop a protest before it happens.