Adding Animated Overlays To Videos On Android

I was really excited when it turned out my new Android project was related to the video processing because the task was very interesting and I thought I knew some possible solutions. But in some time after starting the development, I realized Android OS did not (and does not) have instruments for that kind of manipulations. There was no simple solution and I had to solve the task and besides do it quickly. After lots of googling and reviewing possible options and existing similar apps and examples, I found only a few tools which seemed reasonable at that point. They were several libraries allowing some video manipulations but I did not know much about their functionality at that point.

Tha final list of the candidates looked like this:

  1. JCodec
  2. Java Mp4Parser
  3. FFmpeg

I tried each library and settled upon FFmpeg because it allows lots of basic video operations including even frame breakdown and in this way really stands out from JCodec and Java Mp4Parser, which are slower and have poorer functionality. Another advantage of FFmpeg framework is that it is supported for many processor architectures and in this way we can use it with NDK almost on all Android devices. In case you do not have much experience with NDK or you have not worked with it at all you can use FFmpeg-android-java library which allows using FFmpeg without having to work with NDK.

To add this library just add this code to your Gradle file:

compile 'com.writingminds:FFmpegAndroid:0.3.2’

Note: minSdkVersion should be 16 or higher!

For initialization before using please use the following piece of code:

FFmpeg ffmpeg = FFmpeg.getInstance(context);

try {
  ffmpeg.loadBinary(new LoadBinaryResponseHandler() {

    @Override
    public void onStart() {}

    @Override
    public void onFailure() {}

    @Override
    public void onSuccess() {}

    @Override
    public void onFinish() {}
  });
} catch (FFmpegNotSupportedException e) {
  // Handle if FFmpeg is not supported by device
}

This command does the following things:

  • Loads/Copies binary to device according to architecture
  • Updates binary if it is using old FFmpeg version
  • Provides callbacks through FFmpegLoadBinaryResponseHandler interface

To combine FFMPEG framework with this library you need to use this code:

FFmpeg ffmpeg = FFmpeg.getInstance(context);

try {
  ffmpeg.execute(cmd, new ExecuteBinaryResponseHandler() {

    @Override
    public void onStart() {}

    @Override
    public void onProgress(String message) {}

    @Override
    public void onFailure(String message) {}

    @Override
    public void onSuccess(String message) {}

    @Override
    public void onFinish() {}
  });
} catch (FFmpegCommandAlreadyRunningException e) {
  // Handle if FFmpeg is already running
}

where cmd is a String (array of Strings) and each item of the massive is a separate part (word) of the usual console command for FFmpeg.

For example, the console command looks like this:

ffmpeg -i input.avi -b:v 64k -bufsize 64k output.avi

And cmd creation:

String[] cmd = {"-i", "input.avi", "-b:v", "64k", "-bufsize", "64k", "output.avi"};

The initial goal was to find out how to add animated images to the video so my first thought was to divide the video into frames and add a particular animation moment to each of them. However, after dashing off this plan I realized even sooner that the idea was absolutely unviable — for a 30 fps video of 3 sec the processing lasted for about 5-10 minutes (even on high-class devices!). So after that idea failed I went on googling and in the FFmpeg documentation, I found the functionality which allowed adding to video a static image (a sort of a watermark). After that, it struck me one should never forget the golden RTFM rule!

After looking into the new functionality I found out there is an opportunity to place the overlay in some particular coordinates in the video at each particular moment of time. Putting it simply you can set x=F(t) and y=G(t), where х and у are coordinates and F(t) and G(t) are general time functions. The command for simple adding looks like:

String strFilter = "[1:v]scale=h=-1:w=" + widthOverlay + "[overlay_scaled],"
  + "[0:v][overlay_scaled]overlay=eval=init:x=W*" + xPositionPercent
  + ":y=H*" + yPositionPercent;

String[] сmd = new String[] {
  "-i",
  strPathSrcVideo,
  "-itsoffset",
  String.valueOf(timeSecStartAnimation),
  "-i",
  strPathOverlay,
  "-filter_complex",
  strFilter,
  "-preset",
  "ultrafast",
  "-g",
  "120",
  strPathDstVideo
};

Where:

  • widthOverlay is the final width of the overlay just as it will be in the video
  • xPositionPercent is the left margin in the percent from the video width
  • yPositionPercent is the top margin in the percent from the video height
  • strPathSrcVideo is the full way to the source video
  • timeSecStartAnimation is the time when the overlay appears on the video
  • strPathOverlay is the complete way to the source overlay
  • strPathDstVideo is the way for saving the result

You could pay attention to the lines preset and ultrafast, here we are defining the global settings for the framework. From their names you can see we have chosen one of the fastest options because our main target is the speed of the video processing.

The basic functionalities of my animations were quite simple — they were moving, scaling, changing transparency but the trouble was that I could set the time function only for the position while the dynamic scaling and changing transparency were absent at all. It became a serious challenge at the point when we decided to create animations for several dozens of frames and then create the video from these frames with the FFmpeg and add them to the final video in the same way. To speed up the processing we decided to set the resolution of the recorded video to 800х480 which is quite enough for smartphones and social networks. We also decided to diminish the overlays size in order to create animated videos faster.

Conclusion

It is the truth universally acknowledged that audio and video processing is a really difficult task for Android developers because of the absence of such instruments as AVFoundation for iOS in Android SDK. By the way, in iOS this task is solved much easier. To evaluate the level of optimization and improvements compared to my first solution you can just look at the productivity. Initially, the processing for a 3 sec video took about 10 minutes while at the end of the development the processing for the 15 sec video was from 30 to 60 sec. The final tests were made for the “worst” option with the maximum video length and the overlay size, which means than in other cases it takes even less time. If your task is to convert, cut or manipulate with media files in similar ways, it means FFmpeg will be a good solution.

And to see the results of how I implemented everything you can review it in Pezz app on Google Play!

Vitaliy Obideiko Android Developer