This AI Tool Changes Animation Forever

A new era of animation is emerging with the help of artificial intelligence.

Some of our favorite anime TV series like Naruto, Dragon Ball Z, and Demon Slayer were all hand-drawn and took hundreds of hours of human labor to complete.

Today, it’s now as easy as feeding a couple of images into a tool, describing how the video should look, waiting for a few minutes, and boom! You have a unique anime sequence.

In this article, I’ll show you a free-to-use AI tool called ToonCrafter that has been getting a lot of attention in the AI community recently.

What is ToonCrafter?

ToonCrafter is a brand-new AI framework designed to bridge the gap between traditional cartoon animation and modern AI technology. It leverages pre-trained video diffusion models to create visually convincing and natural-looking animations.

This tool is perfect for those who want to create anime sequences without extensive drawing skills or manual labor. Imagine having two input sketches, and the AI predicts and fills in the video frames in between.

How does it work?

ToonCrafter’s architecture is built upon the open-sourced DynamiCrafter interpolation model, a state-of-the-art image-to-video generative diffusion model that demonstrates robust motion understanding for live-action interpolation but falls short when applied to cartoon animations.

The framework incorporates three key improvements for generative cartoon interpolation:

1. Toon Rectification Learning Strategy: This carefully designed strategy fine-tunes the spatial-related context understanding and content generation layers of the underlying image-conditioned video generation model on collected cartoon data, effectively adapting live-action motion priors to the cartoon animation domain.
2. Dual-Reference-Based 3D Decoder: This novel decoder compensates for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. It injects detailed information from input images into the generated frame latents using a cross-attention mechanism in shallow decoding layers and residual learning in deeper layers, considering computational cost burdens.
3. Frame-Independent Sketch Encoder: This flexible sketch encoder empowers users with interactive control over the interpolation results, enabling the creation or modification of interpolation results with temporally sparse or dense motion structure guidance.

If you want to learn more about the details of how it works, check out this whitepaper on ToonCrafter.

Try it yourself

There are two ways you can try this animation tool:

1. HuggingFace
2. Run on your local PC (Windows)

Feel free to try running the tool on your local machine if you have a powerful GPU. In this article, I’ll show you how to use ToonCrafter via HuggingFace.

Head over to the ToonCrafter page on HuggingFace. The dashboard looks like this:

This setup requires you to upload two images to the two drop zones. Here are two image examples:

Next, adjust the parameters and set the text prompt. In this example, I used the prompt below:

an anime scene

Click on the “Generate” button and wait for the AI to complete the process in the background. The final video is a 512x320 file.

It looks pretty cool! The transition between frames is very smooth.

Here are a few more examples:

Cartoon Sketch Interpolation

The AI tool is also capable of interpolating sketches. Take a look at these two input images:

This is the final result:

According to the researchers, this is more than challenging due to its extremely sparse structure without color and textures. Nonetheless, the ToonCrafter can still produce decent results.

Imagine the potential for comic book artists or storyboard creators.

Reference-based Sketch Colorization

ToonCrafter can also support reference-based sketch colorization by providing 1 or 2 reference images and per-frame sketches.

Coloring your sketches has never been this easy. ToonCrafter is taking it further by coloring frames on videos!

Run ToonCrafter Locally

I won’t be going into the details of running ToonCrafter locally. You can check the details on the GitHub page.

Be aware that the model size is 10.5 GB so make sure you have that extra bandwidth and local disk space.

Running ToonCrafter locally gives you more control and potentially faster processing times, especially if you have a powerful computer. This option is great for those who want to experiment with the tool more extensively or use it for larger projects.

Final Thoughts

In the last couple of months, I’ve seen a handful of AI tools that can produce anime videos from text descriptions. However, none of them can generate a level of smoothness between frames and expressive faces from subjects such as ToonCrafter.

With regard to the ethical implications of using such a tool, I am aware of the risks and concerns surrounding the use of AI in generating any form of media. If you are thinking about using ToonCrafter to generate NSFW or any harmful video, please do not do it.

This article is not sponsored by the developers of that tool, I am just genuinely impressed by the technology. Plus, it’s completely free to use, so go ahead and be amazed too.

‍