Tools powered by artificial intelligence and machine learning can also be used in animation and visual effects. Nvidia senior solutions architect Rick Grandy noted that the benefit of such tools is that artists don’t have to replicate their own work. That includes deep learning used for realistic character motion created in real-time via game engines and AI, as well as a phase-functioned neural network for character control, whereby the network can be trained by motion capture or animation.

Grandy broke down how it works. “We’re dealing with 3D skeletal data that is animated or motion captured,” he said. “You want to capture all the ranges of motion required. Then we label the motions because we want the tools to know the difference between a walk, run, and swim, for example.” The system also handles transitioning between types of motion.

“Then we fit the data,” he continued. “We generate many different height fields that can fit a given set of character positions.”

The next step, said Grandy, is “doing the math and training the networks.” “This is building the car, so to speak,” he said. “Next comes the runtime inferencing part, which is driving the car. As the character moves through the environment, we look at terrain, velocity, rotation of joints.” Controlled in real time via a video game engine, this software becomes “an exciting way to previsualize … or even for final animation.”

“You can see the nuances of the character’s change in position over the terrain,” he said. “This also has the ability to scale to 500 characters in real-time.”

Currently, however, this solution requires manual input of all the data, something that is being addressed with the development of DeepLoco, a dynamic locomotion skills tool. “What if you captured a 30-degree angle walk with a character but not the 15-degre angle,” said Grandy. “With DeepLoco, you put the character in an environment and give it goals — teach it to walk the way we learned to walk. We can teach animations to walk on certain terrains and to avoid obstacles. The results are pretty impressive.”

Still, the characters and their motion look more like a robot than a human. “We’re looking at ways to combine the two [solutions] into one succinct trainable model,” said Grandy. “We can marry the two techniques, and apply that to multiple environments.”

In response to a question as to how filmmakers can use the tool, Grandy responded that Nvidia is looking to nudge software vendors in this direction and will offer SDKs. “We may also take a direct approach with major game engines,” he added, reporting that the company already has the inference running for the Unreal engine.