"King of Glory players, in the future you may not even be able to beat the AI"

2019-12-24

This summer, Tencent announced that it has developed an artificial intelligence system that can defeat professional players in the game "King of Glory". Moreover, the AI system selects its lineup through blind selection, meaning that it does not have complete information about the game. Therefore, the significance of this system goes beyond winning the game and can be used to develop systems that can solve society's most challenging problems.


In a paper published by Tencent this week, various aspects of this AI system technology were revealed, with the authors describing it as "highly scalable". Compared to traditional board games and arcade games, real-time strategy games like "King of Glory" require highly complex skill control, and the game state is often filled with uncertainty (with approximately 10 * 600 possible states and 10 * 18000 possible skill combinations), making the overall objective more complex. The system not only needs to learn how to formulate strategies, attack and defend, but also needs to control the order of skill casting, induce and deceive opponents, while also dealing with minions and defense towers.Tencent's AI system architecture consists of four modules: reinforcement learning (RL), artificial intelligence servers, scheduling modules, and memory pools.


Artificial intelligence server - It runs on a single processor core, enabling it to learn how to interact with the game environment's AI models, adjust game strategies, and, because it has the ability to extract information from the game state, it predicts player behavior and forwards it to the chip. The chip then returns a command state and the corresponding reward value.As for the scheduling module, it is bundled with multiple AI servers on the same computer, and before sending information to the memory pool, it collects data samples composed of feedback, functions, skill probabilities, etc.


The memory pool (also a server) supports sampling of various lengths and data sampling based on time, and implements a circular queue structure that performs storage operations in an efficient manner.


Finally, reinforcement learning (a distributed training environment) accelerates policy updates through the aforementioned actions. Multiple reinforcement learners retrieve parallel data from the memory pool and communicate with them using shared memory. One mechanism (target attention) helps with the selection of enemy targets, while another (long short-term memory, LSTM) learns algorithms that have long-term dependencies, maximizing the damage of the AI system's heroes.


The system developed by Tencent researchers encodes image features and game state information, allowing each unit and enemy target data to be represented. The AI system cleverly combines rich experiential knowledge from human players to prevent the AI from entering physical "forbidden zones" (i.e., dangerous terrains) in the game.


In the experiment, researchers ran this AI framework on a total of 600,000 cores and 1,064 graphics cards (a mixture of Nvidia Tesla P40 and Nvidia V100) and processed features containing hidden unit attributes and game information. Training a hero requires 48 graphics cards and 18,000 processor cores, with a speed of approximately 80,000 samples per card per second. Collective training is conducted every day, and the system accumulates the equivalent of 500 years of experience from human players.The authors point out that the AI's Elo score comes from a system that calculates the relative skill levels of players in zero-sum games (referring to the participants in the game, under strict competition, one party's gain inevitably means the other party's loss, and the sum of the gains and losses of the parties involved is always "zero", with no possibility of cooperation between the two parties.). According to the researchers, it becomes relatively stable within 80 hours and starts to defeat the top 1% of human players in just 30 hours.


The system executes an action through the AI model every 133 milliseconds (or approximately the response time of top amateur players). Five professional players - "QGhappy.Hurt", "WE.762", "TS.NuanYang", "QGhappy.Fly", and "eStarPro.Cat" - were invited to participate in the battle. The researchers noted that although eStarPro.Cat performed well in mage-type heroes, the AI still killed it 5 times per game, while being killed only 1.33 times on average. In public matches, its win rate was 99.81% out of 2,100 games, with five out of eight AI-controlled heroes achieving a 100% win rate.


Tencent researchers stated that they plan to open-source their framework and algorithms in the near future to promote research on complex games like "King of Glory".

RECENT AI NEWS

RECENT AI TOOLS