03 May 2016

For some time now academics and legal commentators have reported on the fact that some price scoring formulas commonly used by contracting authorities are flawed, with the potential of breaching the most important EU procurement principles – non-discrimination and equal treatment of economic operators, transparency and proportionality.

This is odd for several reasons. Firstly, because the procurement regime is predicated on producing transparent and fair results or outcomes. Secondly, because price, unlike quality, does appear to be a more straightforward criterion to evaluate, if only because price can be evaluated in accordance with an arithmetic formula – what could be simpler?

As I hope to demonstrate, however, evaluating prices in tenders can be arbitrary and lacking transparency, leading to discrimination in many instances.

An example

An Invitation to Tender (ITT) had a not-unusual split of quality 75%, price 25% and was seeking the most economically advantageous tender (MEAT), with no variant.

Bidder C quoted a Price of £65 per “unit” (the maximum permissible price per unit was £75). Bidder A’s bid is not known, but it is apparent that it did submit a price. Bidder B took a different approach. The evaluation of the three bids produced the following outcome:

So, Bidder B, the lowest in quality, despite a quality weighting of 75%, won the tender by margins of 16.46% and 23.13% respectively. Bidder C, the closest to Bidder B, would have had to score 72.08% in quality alone, out of a possible maximum of 75%, to have had any chance of winning the contract – I believe that to be and to have been an impossibility.

The price scoring formula

The formula used in my example was the “standard differential model”, otherwise known as a “relative scoring formula”. Most contracting authorities are familiar with this formula through using it, but they may not be aware of the full implications of its use and its unintended, and damaging, consequences. This formula is:

So, if Bidder Y bids £50, and this is the lowest price, then Bidder Y is given the highest score, i.e., in our example, 25%.

If Bidder Z bids £65, then its score measured against Y’s bid is:

In my example we know that Bidder B submitted a nil price. That is because Bidder C’s score for its £65 bid can only be calculated as follows:

Bidder A, by also submitting a price, as was requested by the contracting authority, suffered the same fate – it was allocated 0% out of 25%. One of the many issues was that neither the ITT nor the formula guarded against a bidder submitting a £0 bid – indeed, the contracting authority, by their own admission, did not expect it.

Even if Bidder A or C had bid £1, or indeed £0.01, they would still have scored 0%, despite the fact that the difference between £0 and £1 (or £0.01) is “wafer-thin”. The score differential was, and would always have been, 25% where the lowest bid was nil. In a tender based on MEAT and in the context of a clear emphasis placed on quality (reflected in the split of 75% quality, 25% price) this result is contradictory and absurd.

Applying the formula meant that the contracting authority ignored their own stated MEAT requirements and strong preference for quality in the ITT. They also ignored guidance issued by their national body.

Remarks

The Contracting authority was clearly in breach of procurement regulations in that they discriminated against Bidders A and C. And they did so because they failed to apply their formula to Bidder C by not appreciating that “the lowest price gets the highest score” is what the formula produces when applied to the lowest price. For example, if Bidder Alpha bids the lowest price of £10, Bidder Alpha obtains the highest score because:

However, had the contracting authority applied their formula to Bidder B, as they should have so as not to discriminate against Bidders A and C, the result would have been:

The outcome would have been bizarre (the three Bidders each scoring 0%), but it would certainly have been non-discriminatory and, on the face of it, transparent.

It would also have thrown up the absurdity of translating a price into a score expressed as a percentage. However, as the contracting authority chose to translate prices into percentages, it would have left the contracting authority contemplating awarding the contract to Bidder C on quality alone, but at a price of £65 per unit, despite the fact that it scored 0% on price!

Flaws identified with relative price scoring formulas

Academics and commentators have analysed some 28 different formulas in general use throughout Europe, 20 of which were relative scoring formulas, finding (as with the above examples) that relative scoring was flawed. Through this and other various investigations into relative price formulas, the following shortcomings have repeatedly been identified:

• In-built mathematical properties and permutations or variables that can lead to what is referred to as the “ranking paradox” – where a withdrawn tender may favour lower ranking bids rather than the second bid in the original evaluation.
  
  By way of analogy, a silver medallist may not automatically get gold if the gold medallist is eliminated (and bid rigging and collusion have been used to exploit the “ranking paradox” – by simply orchestrating withdrawals or forcing disqualifications).
  
  
• Non-transparency and discrimination compared with “absolute scoring formulas”.
  
  This is because relative scoring formulas, when assessing a particular bid, rely on information from all the submitted bids as a reference point – namely, the lowest, or, indeed, nil price, which tenderers know nothing about in advance of tendering.
  
  In effect, price in a bid is not evaluated “on its own merit” – rather, it is scored in relation to other bids. One of the main drawbacks of course is that tenderers are having to make assumptions about other competitors’ bids, which is hardly transparent and / or fair. It is also striking that transparency, a legal requirement contracting authorities must seek to adhere to, is obscured by the use of formulas that result in keeping economic operators in the dark.
  
  Again by way of analogy, it is like marking a student’s essay, not on its own merit, but by reference to someone else’s essay – no student is expected to write an essay by second guessing what another student is writing.
  
  Another remark I would make here is that the quality criteria of every tender are assessed on their own merit, not by comparison to other tenders – so it is again odd that price is not evaluated on the same basis.
  
  
• “Unevenness”, in that the formula may lead to the offers closer to the best price being penalised more heavily than the offers that are furthest away from the best price. In other words, the middle ranking bids tend to fare worse.

Additionally, the World Bank Organisation has been advised that, due to major shortcomings identified in relative scoring formulas, they should not promote the use of such formulas in their guidance to non-EU member states.

Clearly, where scoring formulas are flawed, then the related tendering exercises are also flawed. This may explain why Portugal banned the use of relative scoring formulas in 2008.

What about case law?

In a French case, (CAA Paris, 8 fevrier 2016, Communes de Lognes, no 15PA02953) the Appeal Judge did not hesitate to strike down a pricing methodology that attributed the maximum points to the best offer and zero points to the least competitive, noting that the formula (see below) was “irregular” in that it did not take account of the actual or real difference between the prices.

In this case a score of up to 60 was available for quality and, for price, the following relative scoring formula was used where a maximum score of 40 was available:

This formula scores a bid in relation to where it sits between the L (least expensive) and M (most expensive) bids – ensuring that the L bid will score full marks (40) and the M bid no marks (0). Only 2 bidders came forward and their scores were therefore 40 and 0 regardless of the actual prices quoted and regardless of whether the bids may have offered very similar, or wildly different, prices. The L bid won the tender but the court ruled that this was a result which, due to the failures of the formula used for price scoring, relied entirely on the price submission and ignored the MEAT principles – and the contracting authority was ordered to pay damages to the M bidder as a result.

The defence argued that the anomaly of the formula was due to the fact that there were only two bidders. The Appeal Judge, however, dismissed this argument as irrelevant and of no merit.

Another case that touches on the matters discussed here, is a Swedish case in the Gothenburg Administrative Court of Appeal (Case 5293-10). As a Swedish case this may not be binding on EU member states. However, it may be persuasive enough to use in a British court, or indeed to persuade contracting authorities in the EU and beyond to refrain as much as possible from using relative scoring formulas.

The facts of the case are not crucial to the debate. Suffice to say, however, that the issue was not the description of the formula in the ITT. The main issue rather was that the mathematical properties of the relative scoring formula produced an outcome that ran contrary to the principle of MEAT and to the well-established principle of equal treatment. The court therefore had no hesitation in ruling that the relative scoring formula used resulted in a breach of the Swedish Act (1997:1091) of Public Procurement (“SAPP”) (the equivalent of EU Procurement Regulations).

What the court did not do though is to rule that relative scoring formulas should be banned as being in breach of SAPP and European Treaty principles. However, we know that French authorities are now recommending avoiding:

“Des systèmes dans lesquels les écarts de note traduisent un rang de classement sans tenir compte des écarts réels …” (“Systems where the score differences translate into a ranking classification that ignores real (price) differences”. – Le prix dans les marchés publics. Guide et recommendations; avril 2013).

Absolute Scoring Formulas

We have seen that, often, relative price scoring formulas are non-transparent and / or unfair. By comparison, absolute price scoring formulas are transparent because they do not rely on other submitted bids as a reference point. Other advantages are that:

1. Bidders can calculate their score before submitting their bid.
   
   
2. Bidders may rule themselves out by simply not wanting to participate in what is very often a “race to the bottom”. This offers something positive for all those involved – fewer bids to evaluate for the contracting authorities, and economic operators not wasting time and resources submitting bids they know cannot win.

The downside, perhaps, is that contracting authorities need to have market price knowledge and awareness for what they require – but it may be a “price” worth paying so as not to be in breach of European Treaty principles.

And in Scotland?

Note now that contracting authorities in Scotland “may not use price only or cost only as the sole award criteria.” (Regulation 67 (1) (b) – The Public Contracts (Scotland) Regulations 2015).

Also, “A contracting authority must identify the most economically advantageous tender on the basis of the best price-quality ratio …,” (Regulation 67 (2) – The Public Contracts (Scotland) Regulations 2015). The interesting point to note is the reference to a “ratio”, i.e. P over Q (where P is the price and Q is the quality) – not P + Q. In addition, it is P over Q, not Q over P.

Again academics have written extensively on the matter of ratios, encouraging their use as being fairer and closer to contracting authorities’ often stated aim of achieving price and quality, i.e. value for money – the point being that P + Q can distort the sought-after aim of achieving a price and quality evaluation.

Lessons?

Setting aside all that is novel in the new Scottish regulations and the Procurement Reform (Scotland) Act 2014, contracting authorities need to reassess their procurement strategy – and perhaps contracting authorities should consider the following:

1. Ensure that the evaluation methodology produces predictable outcomes by testing the formula used.
2. Ascertain which formula is best suited to what is being procured.
3. Take steps to protect themselves against tenderers “buying” the work, or against “rogue” bidders, and also spot, and eliminate, “strategic bidding” (for example, offering a very low price on heavily weighted priced elements and submitting higher prices for lower weighted priced elements).
4. Do not use or recycle templates or “off the shelf” ITT and / or formulas randomly.
5. Ensure “consistency” between the text of the Notice, the ITT and the formula. 
6. Crucially – ensure that the model / formula satisfies the Treaty principles.

As for tenderers, our advice is to check which formulas are being used in any tender and ascertain whether or not they potentially breach European Treaty principles of transparency and equal treatment.

I also hope that economic operators will ask fundamental questions at the appropriate stage of the tendering exercise.

Ultimately, though, it is in all parties’ interests, whether contracting authorities or economic operators, to avoid the anomalous results thrown up by the above examples and case law.

If you have any questions in relation to procurement, and this topic in particular, please do not hesitate to call me on 0141 221 8012 or email me at pfa@bto.co.uk.

Look out for my next eUpdate in this series on “Price Scoring, Fair Work Practices…and the Living Wage.”

Sources and Acknowledgements

“I wish to thank Philipp Kiiver and Jakub Kodym for their assistance - and in particular acknowledge their contribution to the debate through their text book: “The Practice of Public Procurement — Tendering, Selection and Award 2014”. Intersentia - Antwerps. However, any errors are mine.

“La question de la régularité de la méthode proportionnelle d’évaluation du critère du prix.” Etude par Pascal MOREAU doctorant en droit pubic à l’université de Poitiers - capitaine de police chargé d’enquêtes en matière de corruption en marchés publics. (12, Décembre 2014, etude 11).

“The Art of identifying “The Most Economically Advantageous Tender” - The Use of Relative Evaluation Models in Public and Utilities Procurement. Joakim Laver and Olof Larsberger, Hannes Snellman Attorneys Ltd. (July 2011).

“Random Effects of Scoring Price in a Tender Evaluation.” Michael Bowsher QC (Barrister, QC).