Rating System Commission 

The Rating System Commission was set up at the 2019 AGM, with convenor Toby Manning of the UK.

Here are the  RATING COMMISSION'S Terms of Reference and Membership

The Rating Commission are making the following Recommendations:

The Rating System should show the true entry grade for all players

The Rating System floor should be moved from 20 kyu to 30 kyu.

The Rating System parameters should be amended.

Downward Rating Resets should be allowed.

The EGF should continue to monitor and improve the EGF Rating System, so the Commission should be made permanent

Please click on each recommendation for a more detailed explanation.

The Rating Commission recommend that this Report is accepted and implemented in full.

-------------------------------------------------------------------------------------------------

Implementation of changes to Rating Parameters.

We have recommended changes to the rating system parameters, to more accurately reflect the probability of winning against players of different strengths, and the rate at which European players improve.

This change could either:

  • be implemented from 2020 (today)
  • be backdated to 2000 (or some other date).

Backdating the proposed changes would be more accurate, and would help align European grades with those elsewhere in the world (pariticularly the USA and China/Korea/Japan), but would result in everyone's rating changing, in general increasing by up to a grade. For example the Convenor's grade would change from 2064 (0.6 dan) to 2150 (1.5 dan).

Implementing the changes from today would mean that nobody's grade changes, but it will be several years before they showed a significant effect.

The Commission considers that this change is a political decision, to be made by the EGF, rather than a technical decision, and we leave it to the AGM (or the EGF Executive) to determine the date of implementation.

The Rating Commisssion asks the AGM to advise on the date of implementation.

-------------------------------------------------------------------------------------------------------------------------

True Entry Grade

At the moment, if a player enters a tournament at a grade below 20 kyu, the Rating System wil show the entry grade as 20 kyu.

Here is an example. Eriksson, Michael entered the Tournament at 30 kyu.

The upper part of the screenshot on the left shows the file as submitted to the EGD. It shows Eriksson, Michael's grade as 30kyu.

The lower part shows an extract from the European Go Database. It shows his grade as 20 kyu. If this Report is accepted, this will change from "20 kyu" to "30 kyu"  and it will be clear to his opponents that he was not 20 kyu.

If the rating floor is moved to 30 kyu, as recommended, and someone enters a Tournament at 35 kyu, then the Wall List (lower part of screenshot) will show 35 kyu but the rating system calculations will assume 30 kyu.
.
This change only changes the display, it does not alter any of the database calculations.


Changing the Floor
At the moment the rating system does not recognise grades below 20 kyu (the floor), equivalent to a GoR of 100. Anyone who enters a Tournament at a grade below 20 kyu will be put into the rating system as 20 kyu, and it is not possible to go below 20 kyu, even by losing all their games.This is both the most important and the most controversial area considered by the Commission.

We accept that, as players get weaker, their grade becomes less precise; there is an increased level of randomness in their results. At 5 dan the probability of beating someone 1 grade better is around 20%; as grades get lower this probability increases until it becosmes close to 50% for 30 kyu players.

The Commission considered a number of possibilities, as follows:-

  • Retain the Status Quo (i.e. no change), keeping the floor at 20kyu.
  • Move the floor to 30 kyu
  • Move the floor to 40 kyu
  • Have no floor (although it may be necessary to have a floor of 99 kyu for technical reasons)
  • Have an entry floor at (say) 30 kyu, but allow people to go below it. This would mean that you could not enter a tournament below 30 kyu, but a 30 kyu who lost games could go below 30 kyu.

The arguments for "no change" are:

  • The "randomness" of results below 20 kyu will give players a false sense of precision
  • Having a floor give players below 20 kyu an incentive to improve
  • Players below 20 kyu will either rapidly improve (and get above 20kyu) or give up playing.

While there is a demand from many teachers of young go players for the rating system to be extended, this is not universal: some people who work a lot with children teaching go in schools do not think that providing ratings to all children is a good idea, and it is much better to motivate them by setting a goal for them saying that they will get a proper rating if they reach a certain level

The arguments for extending the floor below 20 kyu are :

  • The floor distorts the ratings of players close to the rating bottom, up to about 17 kyu.
  • The system discriminates against weaker players, particularly youngsters. It is wrong to say to players "you are too weak to go on the database". On the other hand, it has been argued that this gives players an incentive to improve "get up to 20 kyu and we will put you on the database".
  • The ratings of double digit players who are a bit above 20 kyu are inaccurate. They are likely to have played players with a nominal strength of 20 kyu, but whose actual strength may be anywhere below 20 kyu.

The Commission never reached a consensus concerning the floor. We are in agreement that the floor can be moved to 30 kyu without significant disruption to the system. Some Members wanted the floor to be extended below 30 kyu, but there was no unanimity.

Amending Rating System Parameters
 Assume Player A of rating GoR(A) plays player B with rating GoR(B).
Then Player A's GoR will change by an amount Delta(GoR(A))
where Delta (GoR(A)) is a function of (GoR(A), GoR(B), Win/Loss).

Normally the number of points gained by the winner slightly exceeds those lost by the loser, hence there is a net gain of points.This reflects the fact that players generally get stronger with time.

Various parameters, referred to as epsilon, con, etc. are used in the formula to calculate Delta(GoR).

These parameters have remained unchanged since the Database inception 20 years ago, with the exception of the introducction of the parameter " epsilon" in 2003 or 2004..

There is some evidence that these parameters have resulted in "rating deflation", whereby the value of rating points reduces with time. So the rating system shows that someone who was 2 dan in 2000 would become 1 dan in 2020, even if their playing strength had not changed. In particular European Grades appear to be out-of line with grades in the USA and Japan/Korea/China, as well as with on-line grades.

We now have a lot more data (from 20 years of results). This has been examined in detail and we are proposing some amendments.

The reasons for amending these parameters are:.

  • To correct rating deflation (change and increase the bonus per game)
  • To fit winrate expectations to observed winrates (modify the parameter "a")
  • improve the mathematical model of rating updates (switch from Elo model to Bradley-Terry model)
  • decrease rating volatility of low ratings (decrease the parameter "con" and remove the loss limit as it becomes mostly redundant as a result).

We have analysed all this data and we recommend a change to the parameters. Our interim recommendations are as follows:

  • correct deflation: bonus = 0.0017 * (2900 - rating) / (1 + exp[(rating - 2150) / 200]) - 0.3)
  • rating’ = rating + con * (Sa - Se) + bonus
  • winrate fitting: a = (3200 - rating) / 6
  • Use the Bradley-Terry model: Se = 1 / (1 + exp[B(rating2) - B(rating1)])
    where B = -6 * ln(3200 - rating), which is the integral of 1/a ]
  • Volatility reduction: con = 0.13 * a

However, the Commission wants to do a little more work in this area, so the implementation (if the recommendations are accepted by the EGF) may be slightly different. In particular the Commission may simplify the equation for the deflation bonus if an acceptable approximation can be found.

The Commission would like to pay tribute to Dave de Vos for undertaking the analysis which led to these conclusions.. 

Permitting Downward Rating Resets
Sometimes a player is in the system with a rating that is clearly incorrect. This might be due to an administrative error, rapid improvement, simple entry to a tournament at the wrong grade (particularly the case with players from outside Europe), or someone who has not played for several years and has returned to the game but with a lower level of skill. 

The system does allow "Upward rating resets" where by a player's rating can be artificially increased: this is normally used for rapid improvers.The process is not automatic,but requires manual intervention.. However, the current system does not permit downward resets.

We believe that manually operated downward resets should be permitted.

Permanent Commission
We believe that the Commissiont should be made permanent, although the Terms of Reference will need to change. It will be responsible for the following tasks:

  • Agreeing the final mathematical equations to implement the changes to the rating equations
  • Implementing the changes agreed by the EGF AGM
  • Monitoring the effect of these changes
  • Proposing frurther improvements and enhancement to the Database
  • The Commission may wish to negotiate a budget with the EGF Executive.

Commission Membership
On 28 August 2019 the convenor sent out an appeal for Members to serve on the Commission. The response was a bit disappointing, only 4 countries responding. As a result  the Commission was set up with the following members:-

  • Toby Manning UK (Convenor)
  • Dave de Vos NL
  • Ales Cieply CZ
  • Michael Silcher B
  • Wilhelm Buehler D

The Commission had valuable input from Geoff Kaniuk (UK).

The Commission Members were all acting as Experts, who had experience, knowledge or an interest in ratings. They were not acting as representatives of their respective Governing Bodies, and were not necessarily appointed by their Governing Body. 

With the exception of a discussion betweeen Toby, Dave and Geoff at the Lodon Open Go Congress, the commission met entirely on-line (we exchanged over 250 emails); it had been hoped to have a face-to-face meeting during 2020 but Coronavirus meant that this was not possible.