Introduction
Artificial Intelligence to ease the replacement of elements in pictures and videos is of great importance in many situations. (like video editing, video segmentation, video enhancement, media watermarking, etc.)
These operations have been considered for a long time as tedious and complicated. However, in recent years, many works have shown how to employ artificial intelligence to handle these tasks.
We made the choice not to enter the details of how these techniques are working, as we are focusing on demystifying how AI is currently used to help us transform images the way we want more efficiently without needing technical editing skills. However, we are providing links to interesting articles for each of these use-cases in case you want to learn more!
🎨Note: This article is part of our series on content creation. We hope to provide you with a better idea about the usages of AI for content creators, on a wide range of domains. Don't hesitate to check our other articles!
Background removal and image relighting
In this first example, AI is helping in the removal of the background in pictures. It is a common task when it is needed to replace the backdrop in an image or use an element of the picture somewhere else.
For a long time, this operation had to be performed manually. It required knowledge and experience to use an image-manipulation software to manually detour the whole face pixel by pixel to extract it from the picture.
Thanks to Artificial Intelligence, it is now possible to automatically extract the foreground of a picture within a few seconds and with promising results! No more tedious manual labor is needed. Upload your image, and an AI will automatically detect which objects to extract and detour them with near perfection! Content creators can apply the same process to videos.
Background removal of videos
It is common to use a green screen to put actors in another context in the cinematographic industry. Actors are performing in front of a vast green background. Then, we can easily change the green pixels of the videos for a new environment.
However, it is now possible to perform such background replacement on videos that are already in context, without using green screens. The substitution is done with machine learning-based algorithms that will learn how to separate the people and objects of interest from what appears to be the background
View example here
The process is similar to the one used to remove the background in pictures, however, we can mention two important points:
- The algorithm can benefit from the fact the video has multiple frames to better understand where is the foreground, which helps the algorithm to produce better results by having access to more information.
- The algorithm has to maintain a temporal coherence throughout the video, which means that it has to consider multiple neighboring images from the video to provide good results. Otherwise, the result may look like some parts of the foreground are blinking, as they are sometimes detected as part of the foreground and sometimes not.
Now, we have to cover one more problem that may appear, where we can also use artificial intelligence to help with it. Extracting yourself from an image to put it on a different background is fantastic, especially when you can do it with a single click in a split second. But what happens if the lighting is very different on the image where you want to put yourself in?
Well, it results in very unrealistic-looking images. You are simply taking your picture in a specific lighting environment and pasting it in a different scene, expecting it to look natural. Professionals could manually adjust the lighting, shadows, and highlights to make you look like you were there. But to make it available to anyone, artificial intelligence can also help. This task is called image relighting.
Image relighting
Google recently shared an exciting research paper achieving excellent results. It can automatically adapt the lighting of human foregrounds to new scenes. This relighting is challenging because the algorithm has to understand how the foreground reacts to the surrounding light and define a coherent lighting scheme with the new background.
Such understanding of how the foreground reacts to surrounding light is complex for a computer that only has the information of 2-dimensional images instead of the actual 3D world. We humans can easily imagine the depth of different objects based on our visual perception and memories. That is what we train the algorithms to achieve: replicate how humans understand a scene by understanding depth, the different items, lighting sources, etc.
We are excited to see and participate in more of such advances in the future!
We've now seen the extraction of objects from images or videos and adapted the lighting to a new background. But what if you would like to remove the friend that photobombed your profile picture where you look incredible? Artificial intelligence can also help with this with a process called image inpainting.
Image inpainting
Image inpainting is a process similar to background removal. Still, instead of extracting a foreground to do something else with it, it removes an element from a picture and fills the missing part. You would need to manually extract this part and create a plausible replacement with the appropriate textures or objects with traditional techniques. You could also manually draw something that would seem realistic for the replacement.
Fortunately, we do not need to go through such a tedious process over pictures anymore. Artificial Intelligence allows us to automate this process as well! The algorithm will understand what's happening in the image and create a plausible replacement for the element we want to remove, considering specific textures, borders, and objects to make the new picture as realistic as possible.
Stalinpainting, showing that inpainting is by no mean a new concept
A similar application of image inpainting is to "extend the canvas" to make a picture bigger. Indeed, it uses the same process of understanding the image and adding information that would fit well with it.
Image inpainting can be helpful for anyone to change the aspect ratio of images intelligently. This operation can also remove undesired objects from your pictures. Of course, this algorithm is just like the image upscaling one that we also covered on our blog. It has the same underlying problem: it cannot know what was really behind an object you removed. Thus, the reconstruction will be a simple (but good-looking) guess and no more than that. Unfortunately, we cannot dive into a picture and check behind objects as of now. But this might come much faster than we think!
Conclusion
These were some of the most exciting applications of AI we wanted to share, covering image and video editing. Still, many more are out there, and many more are to come.
We invite you to read our other articles about content creation. They are highly related to this one using similar algorithms with different media types. All with the common goal of helping creators produce their vision more efficiently.
.jpg)
