September 6, 2016
“Morgan,” a horror film released by 20th Century Fox, has the distinction of being the first movie whose trailer was created with artificial intelligence. The studio partnered with IBM Research to develop what they’re calling a “cognitive movie trailer” largely created by AI. The IBM team faced several challenges, not the least of which is that the nature of horror is subjective. The team had to teach the system to be able to identify scary moments, and then make a trailer that audiences would find compelling.
In an IBM blog, the team described how they used “machine learning techniques and experimental Watson APIs” to train the system to understand what was “scary.” First, they input 100 horror movie trailers, segmenting out each scene from the trailers into “moments.”
Next, each scene was analyzed and identified for people, objects and scenery and “tagged with an emotion from a broad bank of 24 different emotions and labels from across 22,000 scene categories, such as eerie, frightening and loving.”
The audio was also analyzed for tone of voice, musical score and ambient sounds “to understand the sentiments associated with each of those scenes.” Finally, each scene’s composition – including shot location, framing and lighting – was analyzed to “categorize the types of locations and shots that traditionally make up suspense/horror movie trailers.”
Each area was analyzed separately, “and in combination with each other using statistical approaches.”
The IBM team then fed the full-length “Morgan” into the system, which went through the movie and identified “10 moments that would be the best candidates for a trailer,” totaling six minutes.
“Neither our team, or the Fox team, knew exactly what we were looking for before we started the process,” the team wrote. “We didn’t know which [scenes] the system would pick to create a complete trailer.” Then a resident IBM filmmaker edited the chosen moments together into a trailer, including “the addition of black title cards, the musical overlay and the order of moments in the trailer.”
As Engadget notes, “trailers typically take 10 days to a month to complete, but Watson’s involvement shortened the process to 24 hours.” The IBM Research team sums up that, “the combination of machine intelligence and human expertise is a powerful one.”
“This research investigation is simply the first of many into what we hope will be a promising area of machine and human creativity,” they said. “We’re excited about pushing the possibilities of how AI can augment the expertise and creativity of individuals.”