OpenAI Sora vs RunwayML: Which Creates Better Videos?

We don’t know if dear Santa is coming to bear gifts for you all this holiday season, but OpenAI surely is in the mood to come up with new surprises. On Day 3 of 12 Days of OpenAI, the team launched its high-end accelerated video generation model: Sora Turbo. In this article, we will learn how to use Sora and then compare it’s capabilities with those of Runway. Let’s see who wins in the OpenAI Sora vs RunwayML ultimate face-off!

Features of OpenAI’s Sora

Let’s begin by understanding the capabilities of OpenAI’s Sora Turbo model. Its key features include:

• Generating Videos from Text: Create videos from simple text prompts.
• Extending Existing Videos: Continue or modify existing videos.
• Animating Images: Bring still images to life with animation.
• Handling Complex Scenes: Maintain continuity across multiple frames.
• Scaling and Adapting: Generate videos in various formats and lengths.
• Transforming Videos: Modify existing videos based on text prompts.

Now before we jump into tis comparison with RunwayLM, let’s learn how to access these two GenAI models.

Setting-up an OpenAI Sora Account

Sora can be accessed via the new website – sora.com. You can use your ChatGPT plans to get access to the model:

• ChatGPT Plus Account: This plan gives you upto 50 videos at 480p and fewer videos at 720p each month. 
• OpenAI Pro Account: This offers unlimited generations in slow queue mode, plus 500 faster generations per month. This tier seems to be focused on offering more flexibility, with the option to process tasks faster for a certain number of generations, while other tasks might be queued and processed more slowly.

We will be accessing Sora through our ChatGPT Plus Account.

Also Read: I Tested OpenAI Sora and Here’s What I Discovered

Note: The service won’t be available to the UK and EU at launch, which might be due to legal, regulatory, or data privacy considerations (such as GDPR). This limitation could be lifted in the future as they expand to more regions.

Setting Up a RunwayML Account

Runway can be accessed by clicking on Get Started at runwayml.com. You can Sign In with your google account. Click on Try it now and this will lead you to the dashboard.

The free version of Runway comes with 125 credits which translates to five video generations. You can access their Gen-2 model to generate images with just text and the Gen-3 Alpha Turbo model which can be used to generate videos from reference images.

OpenAI Sora vs RunwayML: Output-based Comparison

Now let’s compare the capabilities of OpenAI’s Sora and RunwayML. In this section, we’ll be testing out 3 different tasks on both models and comparing the quality of their results. Let’s begin!

Task 1: Video Generation with Prompt

We’ll start with generating videos using a prompt and no reference images in both models and compare the output. Let’s begin with Sora. We will use the following prompt:

Prompt: “Create a video of white dog playing with kitten” 

And as you can see, Sora has produced a great video with a Kitten and dog playing. Also, it is admirable how well it has filled the details in the background without any directions in the prompt. But, the only problem is that the cat rendered is imperfect. The cat has 2 heads with the second head popping from its rear end. So we can say that Sora is not producing the best possible output.

Now, let’s test Runway with the same prompt. We will be using the Gen-2 model for this.

Well, even though it has produced the dog and the cat well, they seem to be cuddling and not playing with each other. It has missed the main element of the prompt altogether.

Observation:

Both Sora and Runway exhibit limitations in fulfilling the prompt. Sora generates detailed backgrounds but produces flawed outputs, like a two-headed cat. Runway delivers realistic animals but misses the action element, portraying cuddling instead of play. Both models need improvement in accuracy and context alignment to achieve better results.

Verdict:

Sora ❌ | RunwayML ❌

Task 2: Video Generation with a Reference Image Using a Short Prompt

Now let’s see how they perform with a reference image and a short prompt.

Input Image:

Prompt: “Cinematic shot of the dog flying towards the camera. The dog is moving his neck to look.”

Output by Sora:

Well, Sora didn’t soar my expectations with its response! All it has done is add some minor expressions on the dog’s face without actually adding any movements to his legs or neck as was mentioned in the prompt.

Output by Runway:

On the other hand, Runway is able to generate a video better than I expected! The dog’s face has more expressions and its body is moving as it skims through the sky. In fact, we get to see the impact of the wind as its cape floats through the sky. Although the neck movement of the dog is limited even in this response, overall this outcome is pretty exciting!

Observation:

So it is pretty clear from the results that Runway is way ahead of Sora when it comes to generating outputs with shorter and crisper prompts. In fact, it’s pretty fast compared to Sora too.

Verdict:

Sora ❌ | RunwayML ✅

Task 3: Video Generation with an Image Using a Long Prompt

Now, let’s put both the image generation models to test using a long prompt.

Input Image:

Prompt:  “Create a 5-second time-lapse video capturing a serene sunset. The scene should feature the sun slowly descending below the horizon, casting vibrant hues of orange, pink, and purple across the sky. Clouds drift lazily, reflecting the changing colors, and the transition from day to dusk should be smooth. Include a silhouette of a tranquil landscape in the foreground, such as a beach with gentle waves, mountains, or a peaceful meadow with trees swaying lightly in the breeze. Ensure the video feels calming and natural, emphasizing the beauty of nature’s transition.”

Output by Sora:

Sora has produced a very impressive result with this elaborate prompt. We have got a proper time-lapse this time with the sun going down, and the frame darkening with the only imperfection being the swans and clouds not moving as per the time-lapse.

Output by Runway:

Runway’s output is not close to anything we have prompted. There is no sunset happening, and no change in the lighting as compared to Sora’s output, suggesting that it has completely missed the time-lapse.

Observation:

With a detailed prompt, Sora can produce a video that is quite detailed and mesmerizing. Although it does take longer to generate the video, its results are far better than what we got from Runway. Here, patience is worth the wait!

Verdict:

Sora ✅ | RunwayML ✅

Also Read: OpenAI Sora vs AWS Nova: Which is Better for Video Creation?

OpenAI Sora vs RunwayML: Features-based Comparison

Now let’s compare the different features available in both the video generation models.

Storyboard

One key difference between the features is the availability of the Storyboard feature in Sora. For more complex video projects, Storyboard mode allows you to direct the video creation along a timeline. This provides:

• Control Over Sequence: You can define the order of elements (text, images, and videos).
• Advanced Editing: It allows for more precise adjustments and sequencing of scenes.
• Multimedia Integration: You can combine images, text, and video clips to create a narrative or complex visual story.

In the above image, there are 2 prompts for the same video with the first prompt generating the video for the first 2 seconds and the second frame generating the video for the rest of the 3 seconds. To be honest, it’s a pretty handy tool for aspiring filmmakers/AI filmmakers to generate high-quality videos without relying on other people and figure out how their imaginations will pan out on the big screen. A unique feature to have, only if the output quality is good.

Meanwhile, Runway does not have this feature as of now.

Verdict:

Sora ✅ | RunwayML ❌

Remix

Remix in Sora is an innovative feature that lets users alter specific elements in existing videos effortlessly. It allows creators to replace objects, characters, or backgrounds while maintaining the scene’s overall consistency and visual quality. For instance, you can swap a person in a video with an animal or adjust key components without recreating the entire video. This feature offers flexibility and precision, making it a powerful tool for storytellers and video creators to refine their content dynamically and creatively.

Though the Runway does not have anything explicitly called Remix, it has a feature where it lets you upload the video and then change the desired element. But the performance is not at par. I had a video of a man walking towards the burning house.

When I asked Runway:

“Replace the man with a tiger walking towards the burning house.“

So, with respect to the Remix feature, Sora seems to have the upper hand.

Verdict:

Sora ✅ | RunwayML ❌

Camera Angles

Runway’s Gen-3 Alpha Turbo has advanced camera control features that empower creators to direct AI-generated videos with cinematic precision. Users can manipulate six distinct camera movements:

• Horizontal (X-axis): Move the camera left or right.
• Vertical (Y-axis): Shift the camera up or down.
• Pan: Rotate the camera horizontally from a fixed point.
• Tilt: Adjust the camera’s vertical angle.
• Zoom: Change the camera’s distance to the subject.
• Roll: Rotate the camera around its lens axis.

Each movement is adjustable on a scale from -10 to 10, determining the intensity and direction. Combining these controls with descriptive text prompts enhances scene interpretation, allowing for dynamic and intentional storytelling.

Meanwhile Sora does not have these features. You have to explicitly mention these things in the prompt making the requirement heavy.

Verdict:

Sora ❌ | RunwayML ✅

Result: Sora vs Runway

Conclusion

In conclusion, OpenAI Sora and RunwayML both bring innovative capabilities to the evolving field of AI-driven video generation. While Runway excels at delivering precise results with short prompts and offers robust camera control features, Sora shines in its unique features like Storyboard and Remix, enabling creative storytelling and dynamic editing. Though Sora’s outputs may currently require more detailed prompts to achieve optimal results, its potential for complex and adaptive video generation suggests a promising future. As both tools continue to evolve, creators can look forward to more powerful and accessible tools that redefine how we craft visual narratives. Exciting times ahead!

Frequently Asked Questions

My name is Abhiraj. I am currently a manager for the Instruction Design team at Analytics Vidhya. My interests include badminton, voracious reading, and meeting new people. On a daily basis I love learning new things and spreading my knowledge.