日本語での概要：PGRU はパネリスト方式になります。これは MPGR と同様の方式で、パネリストが上位100名の順位リストを提出し、その平均を取ってランキングにするということです。PGRU は上位100名に拡張され、依然6ヶ月の結果を反映して、6ヶ月に1度発表します。パネリストは各国からバランスのよく集めることに全力を尽くします。これは日本を含みます。世界中のスマブラSP競技シーンに十分な知識がある方で ballot（PGRUの草案を提出する権利）を受け取るパネリストの選考に志願される方は、こちらのフォームを提出してください。
Hi, I'm PracticalTAS, lead statistician for PGstats and team lead for MPGR. Some of you also know me as the "algorithm guy" for PGRU.
This was originally going to be an explanation and justification about the PGR algorithm and its design choices. However, we at PGstats have decided to step back and take a hard look at our fundamental decision first: the choice to make the PGR algorithm-based. At the time the decision was originally made, Smash 4’s competitive scene was still developing and PGstats felt that an algorithm provided the structure and consistency that the scene needed while maintaining impartiality.
One of the limitations of pre-existing algorithms is that some are completely win-based, and others are completely placement-based, but none take both into account. From others' research and my own testing, I came to the conclusion that we must take both into account if we want to rank players accurately, as neither half of the data gives a complete picture of the season on its own. Thus, the only potentially viable algorithms are ones which consider both wins and double elimination placements; and when I started working on a Smash rankings algorithm over 4 years ago, none existed.
The goal of using an algorithm was to give everyone a fair shake, regardless of location, stream time, or social media reach. It unearths hidden bosses and finds players who can’t travel, while still rewarding people who can travel (so as to encourage tournament attendance). We tried to make an algorithm that would benefit people no matter how they wanted to play. We just wanted them to be able to focus on playing.
The decision to target a list of 50 players was made the year before I joined PGstats, but it turned out to be quite convenient—I don’t think my algorithm is able to differentiate between players too far beyond the top 50 with only 6 months of data, so this remained a reasonable cut-off point.
We also decided not to publish the PGRU algorithm, which has been a controversial decision we've maintained for years now. There were two main reasons for this. The first is Goodhart’s Law, which states, "When a measure becomes a target, it ceases to be a good measure." In other words, though the algorithm is designed to find the best players, a player with knowledge of the inner workings of the algorithm could in theory use that knowledge to game the algorithm into thinking the player is better than they are and rewarding him with a higher rank than they deserve; even now, players have already started to weigh options and choose whether or not to attend PGR-ranked events based on whether they believe their rank will be helped by attending.
The second reason is that with a public algorithm, anyone could input the data and publish the list ahead of time, ruining our reveal. The PGR’s only revenue comes from our publishing deal with Red Bull (so please check out our articles on their site, that helps more than you know!), and we split that revenue with graphics guys, writers, photographers, other volunteers, and a lawyer. None of this happens if the reveal isn’t really a reveal. There’s also a reason why PG is the only esports organization building Smash rankings: the return on investment is very small, even before taking into account the amount of work required. In fact, it took years before the PGR started to show any return on investment at all.
I still strongly feel that the algorithm has served the community well in the years since it was introduced. However, our system has always had some flaws. The biggest is that the requirements for maintaining a sustainable algorithm (it is private, and it updates each season to correct flaws) mean that a player base which takes its rankings seriously is constantly chasing a moving target. Players observe a ranking, attempt to determine what the algorithm values, and alter their behavior to try to improve their spot; then, once the next ranking comes out with changes to the algorithm, the cycle repeats and players are left in a constant state of not knowing what matters.
Up until the Fall 2019 PGRU, we felt this was a necessary evil to maintain an accurate, impartial (which is not to be confused with “unbiased”, algorithms are often biased) ranking system. However, I now expect this issue to only become worse in subsequent seasons, meaning the status quo is not sustainable in the long term. For example, I would expect to see more and more reverse-engineering of the algorithm in attempts to game the system. Therefore, though we all stand by the results of the Fall 2019 PGRU, the PGstats team has made the decision together to change our process going forward.
Beginning season 3 (which started on December 15, 2019), PGRU will become a panel-based system, using the same ballot process as MPGR. We will make every effort to region-balance the panelists; we will need the community’s help for this, as Ultimate’s competitive scene is far more international than Melee’s.
Another flaw in the system is the length of the list. In Smash 4, a top 50 was reasonable given the scene’s size. However, with Ultimate comes a new wave of players, and the scene has grown massively. The appetite for a longer list is there, but given the amount of data we have, the algorithm is limited in its ability to differentiate between players beyond top 50. Ultimate also has plenty of international talent that has difficulty traveling across the world, and many of these players can’t get their share of the spotlight when the list only extends to 50.
With the switch to a panel, however, another door has opened. PGRU will also be expanded to top 100, while still releasing every 6 months and still covering 6-month seasons. The switch to a panel removes this limitation of the algorithm and allows us to extend the list.
My position on the PGstats team is unchanged, including maintaining the Tournament Tier System and running MPGR. Also, panelist applications are open now; please click here to apply.
===
The Tournament Tier System will remain for PGRU Season 3 (Spring 2020), and will be re-evaluated for PGRU Season 4 (Fall 2020) once a top 100 has been created. We are also implementing six changes to the TTS beginning in Season 3. Here are the TTS "Patch Notes".
- Splitting the international bonus by region
We introduced an international bonus to counteract the tendency for the TTS to create a positive feedback loop: players in America have the most PGR events, they place highly on the PGR, and the events they subsequently go to have an easier time becoming PGR.
We're now splitting the globe into three regions: 1) the contiguous US, which as in past seasons will require 200 entrants for PGR status and 1200 for S Tier; 2) Japan, which as in past seasons will require 160 entrants for PGR status and 960 for S Tier; and 3) International, which is all other territories and will require 120 entrants for PGR status and 720 for S Tier. In all cases these numbers are translated from TTS score, so events can achieve TTS tiers with the equivalent PGR Player Score.
The international bonus was originally applied to Japan to provide them with the opportunity to have higher-ranked PGR events despite their limited venue sizes. As their events are still limited in size with few exceptions, Japan will continue to benefit from the international bonus for PGRU Season 3. The international bonus will also be re-evaluated at the end of Season 3, including whether Japan should still benefit.
- Area 51 players worth TTS points
In the past, the hard cap of the TTS was 4800 points, equivalent to having all 50 of the top 50 in attendance. This restriction has already been relaxed with the international bonus, and will now be relaxed further.
Area 51 players will be worth 64 PGR Player points, the same as players ranked 31-50.
- A Mid-Season TTS Update
On top of the Area 51 players counting for the TTS, we will also evaluate players 3 months into the season for likely additions to the Season 3 PGR. We will be nominating up to 5 “Mid-Season Rising Stars” for inclusion at that same 64 PGR Point level.
More info is to come on this, so make sure to follow @PGstats on Twitter where we make our official announcements.
- A one-DQ buffer for S Tiers
If an event has achieved S Tier via PGR Player Score, but falls below the S Tier threshold by one player due to DQs on the weekend of the event, the event keeps the S Tier label. In other words, if the event would have been S Tier if one PGR player that registered hadn't dropped out, then the event remains S Tier. This only applies to S Tier events, and one player that drops out must be worth as many or more points than the gap between the event's final score and the S Tier threshold.
The reason why we haven't done something like this in the past is that it opens the TTS up to abuse, so I just want to make this next part explicitly clear: if we ever catch an event organizer trying to take advantage of this rule to fraudulently achieve S Tier (for instance, by signing up a player that has no intention of going to an event, then benefitting from this rule when falsely claiming that they couldn't make it because something came up on the weekend of the event), we will impose sanctions up to banning the organizer's events from the PGR.
- Checking free events for DQs
Normally we don't check for DQs in general entrants because that would encourage TOs to misreport DQs as 0-2s to keep their PGR score from falling. Additionally, it's generally much more difficult to get an accurate DQ count than it is to get an accurate entrant count.
However, for some events the DQ count is a very significant fraction of the total entrant count, so we are instituting a rule to target these events. Any event with free entry (such as conventions) will be checked for DQs and have its PGR score revised based on total attendees that competed. In the absence of an accurate DQ count, this does mean we'll have to estimate based on incomplete data.
- An end-of-year offseason
We are actively investigating how and when to implement a PGR offseason at the end of the year for Season 4 (Fall 2020). We are in contact with tournament organizers who are expected to run events in December and early January, and we’d like to minimize the effect an offseason will have on their tournaments. At this point in time, the summer months are too packed with events to be able to implement an offseason in the summer at the end of Season 3, but we will also investigate this possibility going forward.
===
I want to thank all of you again for your support. This change marks a new era in the PGR, and I hope you all come along with us for the ride.
Until next time,
Andrew “PracticalTAS” Nestico