The researchers claim that Sima 2 can perform a range of complex tasks within virtual worlds, figure out how to solve certain challenges on its own, and communicate with its users. It can also improve itself by tackling tough tasks more than once and learning through trial and error.
“Agents have been a driving force behind research in games,” Joe Marino, a research scientist at Google DeepMind, said at a press conference this week. He notes that even a simple action in a game, like lighting a lantern, can involve multiple steps: “It’s a really complex task that you need to solve to progress.”
The ultimate goal is to develop next-generation agents capable of following instructions and performing open-ended tasks in more complex environments than a web browser. In the long run, Google DeepMind wants to use such agents to drive real-world robots. Marino claimed that the skills Sema 2 has learned, such as navigating environments, using tools, and collaborating with humans to solve problems, are essential building blocks for future robot companions.
Unlike previous work on gaming agents such as AlphaZero, which defeated Go Grandmaster in 2016, or Alphaster, which Beats 99.8% of ranked human competitive players In the 2019 video game StarCraft 2, the idea behind SIMA is to train an agent to play an open-ended game without any preset goals. Instead, the agent learns to follow instructions given by people.
Humans control Sima 2 through text chat, talking out loud, or drawing on the game screen. An agent takes the pixels of a video game frame by frame and tells it what steps it needs to take to perform its tasks.
Like its predecessor, Sima 2 was trained to play eight commercial video games on footage of humans, including No Man’s Sky and Goat Simulator 3, as well as three virtual worlds developed by the company. The agent learned to match keyboard and mouse inputs with actions.
The researchers claim that while Gemini is bound to Gemini, Sima 2 is far better at following instructions (asking questions and providing updates) and figuring out on its own how to perform some complex tasks.
Google DeepMind tested the agent inside an environment it had never seen before. In one set of experiments, the researchers asked Gene3, the latest version of the firm’s global model, to create environments from scratch and dropped Sema2 into them. They found that the agent was able to navigate there and follow instructions there.