Open AI's Sora is just too good that it might've been trained in Unreal 5

OpenAI took the world by storm after launching Sora, its latest text-to-video model that looks terrifying yet mind-blowing at the same time. The model, which will soon make its way to an OpenAI product, is an AI wonder that could turn any word prompt into a 60-second video at a realism level like never before.

But how does Sora exactly work? That’s still a big, tricky question because it’s not a question of “can we do it,” but “should we do it.” OpenAI said that Sora is taught to use a network that makes videos simpler. This network takes regular videos as its starting point and makes them smaller, keeping only the most important parts. Then, Sora learns from these simplified videos to create new ones. 

Additionally, a part of the network is also trained to turn these simplified videos back into full-sized ones that people can watch. You can even take an existing still image and make a video out of it.

And it’s a “data-driven physics engine,” which isn’t remotely similar to DALL-E. Take NVIDIA’s Lead of AI Agents, Jim Fan, who predicts that Sora might have been even “trained on lots of syntehtic data using Unreal Engine 5” due to its complexity. 

You may know Unreal Engine 5 as a “game engine” which recent video games were mostly built on. But it’s not just games anymore, Unreal Engine 5 unlocks cutting-edge 3D content across industries, giving power to creators with unprecedented freedom and detail.

And that’s not a baseless prediction at all. He further explains that Sora is a “simulation” program that learns complex rendering, natural physics, long-term planning, and connecting concepts to real-world meanings, all through processes involving noise reduction and mathematical adjustments based on gradients.

While impressive, the model has its downsides. OpenAI acknowledges that, despite its capabilities, the model hasn’t mastered certain body movements, particularly hands. Additionally, it isn’t perfect at replicating the natural physics of intricate scenes.

What are your thoughts on OpenAI’s new Sora model? Let us know in the comments!