129. Analysing NetEase's "EnMatch" Research from 2024
Following in the footsteps of Activision (2015) and Electronic Arts (2016) research into unfair matchmakers designed to covertly boost engagement, NetEase did their own research in 2024. My analysis.
I didn’t find a link to the research, there is a link to download the research. You can search for that yourself. Here is the title of the paper:
So, is the research good? Are their arguments sound/valid? Is it ethical (obviously no)? Will it be banned in the EU next year (quick answer, 100%)? Can it be improved (you will have to wait for my next paper on Asymmetrical Matchmakers for that answer) or made ethical? There is good and bad here, and most of the bad was because this team trusted/adopted the EA conclusions. I didn’t detect anything inherently malicious coming from this team. They seem to actually want to make better products. Sure that may be a low bar to set, but in this industry that’s noteworthy.
The paper starts with this claim, which is foundational to their research:
The argument that anything useful was proven in the 2017 EA research was debunked in my detailed analysis of the research they cite. You can read that analysis here. This already puts this newer research on rocky standing.
Their use of the same definition of “churn” from the 2017 research, which I demonstrated is not churn, but in fact “fatigue”, further compromises this research. No actual churn was ever evaluated in either study.
The idea that the 0.3% reduction in fatigue over 18 games “shows that fair games are not sufficient to ensure player engagement” is a bizarre statement. This also indicates that this team failed to identify the various failings of the 2017 research, including the use of Cherry Picking to try to artificially create a result that met the requirements of their corporate sponsor. That’s hard for me to imagine, but this team may have suffered from some conflicts of interest, especially if they wanted to maintain a good relationship with EA.
What the research does provide that is an improvement on the 2017 published data is how moving from a 1 vs. 1 pvp environment to a k vs. k scenario affects things. Here “k” is used by the authors to describe a team of players working cooperatively.
Moving secretly from “fair” games to “unfair” games using AI, with the purpose of increasing “engagement” is what is certainly going to get Marvel Rivals banned in the EU next year, and subject to a 7% global gross revenue fine under the Artificial Intelligence Act.
In 2024 the company made a revenue of $23.52 Billion an increase over the revenue in the year 2023 that were of $21.43 Billion. So NetEase has more than twice the revenue of Electronic Arts. A 7% fine would be $1.65B USD. I could render AAA non-competitive with a budget like that.
This is where social involvement is addressed, which I think has a lot of potential in regards to surveying user perception. Again this comes with various potential AIA violations if it is kept secret, like it is currently.
“Collaboration Engagement” (aka my Power Rotation from 2018)
The team here defined “Diversity Teams” as a team with an above normal skilled player mixed with two below normal skilled players. This is exactly the scenario I was describing in my 2018 paper on How to Make Healthy Games. That paper even came with screenshots from World of Tanks where I explained this concept which I called “Power Rotation”. I also explained the science of how it works. I was not cited in the NetEase paper so it is possible they didn’t know that this research/design was already done by prior teams.
The NetEase paper provides data showing that their “Diversity” teams demonstrated double digit improvements in chat frequency, upvote rate, and decrease in downvote rate. I wasn’t at Wargaming long enough to gather this sort of quantitative data so this is very exciting to see this validated.
Long Term Engagement
Their work in this section of the paper is tainted for all the reasons the 2017 EA paper was, as they essentially just accepted and used the methods in that paper without questioning the methodology at all. Thus I will not critique it here other than to direct you to the previous paper where I do. The only difference reported was that fatigue (“churn”) rates were much higher in the NetEase study, presumably done using alpha test data from Marvel Rivals. I would attribute the higher fatigue rates in Marvel Rivals to the developer attempts to highly (over) stimulate the players, and their use of this unfair matchmaker. This does seem to corelate with the high actual churn rates for players in the game.
Tier Gap
The NetEase researchers set a “Tier Gap” maximum of 3 in their algorithm. Presumably these were performance tiers/ranks. In World of Tanks and World of Warships there is an actual Tier Gap maximum of 2, but this does not just relate to performance. The actual tanks and ships used by players have their own tiers which dictate their offensive and defensive characteristics. Thus the 2 tier gap in the Wargaming products is likely even greater in effect than what the NetEase team used in their Diversity teams.
Their Comparison Testing
Here they measured how many games were played under various matchmaking systems without taking a 30 minute or longer rest break. While it was not explicitly stated, I would expect that these were all done on the same game. If not, then these results would be void as a shorter or easier game would allow more plays before fatiguing out. As none of this information is provided, this leaves a cloud of doubt. This sort of information is always provided in proper research to allow for reproducibility. This in turn allows pier review. In an environment where peer review is not permitted, the study is not scientific, and the results are conjecture.
Assuming that the study was done properly on the same product, I would attribute the improvement of the EnMatch algorithm to it’s use of asymmetrical matchmaking. These sorts of matchmakers, which I described in detail in 2018, are superior to traditional skill based matchmakers.
My Conclusion
The research was valuable in demonstrating the advantages of asymmetrical matchmakers over traditional skill based MMs.
The research demonstrated that asymmetrical (“Diverse”) teams enjoy higher player satisfaction both internally and among peers.
The research was harmed by its unquestioning reliance on flawed prior research.
Their system seems to demonstrate higher performance than previous attempts at k vs k MMing using CO.
Perhaps the greatest weakness of this NetEase project is that it assumes that the weakness of prior Skill Based MMs (SBMMs) was due to their “fairness”. They explicitly state that their goal was to overcome this limitation by making unfair matches.
The cognitive leap that is missing here is that an MM can be both fair AND asymmetrical.
I’m not suggesting this is easy. I began working on that solution back in ~2012 to assist the End of Nations team. They rejected my offer of assistance so the solution was ultimately deployed to World of Warships in 2014. Convincing WG and Serb that my MM was both fair and asymmetrical took me 3 months and was the most contested part of creating World of Warships.
I see this team as thinking in a very binary way, which is typical for computer and math-based occupations where a statement is either true or false. Scientists have a more nuanced view of the subjects they study, which makes it possible to solve problems that might appear impossible under a binary world view, and which are still difficult for AI to resolve.
In my next paper on Asymmetrical Matchmakers, I will explain in more detail what an AsymMM is, how it works, why it works, and what was involved in solving this problem in 2014. Matchmakers are very important for revenue generation in multiplayer games, so I can understand why so much money is being thrown at this research. But the problems they are trying to solve were solved a long time ago. They just weren’t published until 2018, and then they probably were ignored because I described them as “Healthy”.
That probably also had to do with my having no financial incentive to share that technology, along with the chaos and secrecy at Wargaming resulting in no one in or out of WG knowing much about the tech I developed there. Also, the division created after I assisted regulators in protecting children in 2013. That division resulted in the interactive media industry not having access to a lot of tech they would/will later realize they needed.