AI & ART - Thanos Snap: photobomb begone!
I don't have to convince you that our world is becoming more and more visually oriented, where photos and cameras are gaining in importance. Ever since the smartphone has supplanted many a small camera and apps like Instagram have democratized photography, everyone has become an amateur photographer. With such photos, users like to strive for perfection. But what if an accidental passer-by, a difficult friend or a piece of garbage mar your shot? What if you could just erase those jammers with an AI model.
Are you the new Thanos?
You can clearly see what's going on in the GIF above. Disturbing objects simply disappear like snow in the sun. Ideal, therefore, when you have just taken that one holiday snapshot, but there is still one disruptive device to be seen.
But what AI technology is at work here? Is this form of inpainting completely new? Where can we use this technology?
An easy job for Photoshop
Avid users of photo editing software like Photoshop will know the clone stamp. It is a tool to clone pixels from one place to another place in the photo. If you do this very carefully and if you use the surrounding pixels of the disturbing object, you can manually remove it. The main drawback of this approach is, of course, that you have to do this manually, so the workload is on the user. A well trained AI model can help the user!
The content aware brush is such a handy tool that means you don't have to clone all pixels yourself, so manually, from one place on the photo to another place on the photo. This brush within programs such as Photoshop itself looks at which pixels are eligible to be used in the inpainting. The results are not always as convincing as if you were to do it manually, but it is a lot faster!
A new trick for Google Pixel
In October 2021, Google launched its new smartphones, the sixth member of the Pixel line. These are notable because they contain a SoC (system-on-a-chip) designed by Google itself. Google used to buy mobile chips from other manufacturers. With their own design, they bring their Tensor computing cores to mobile phones. These cores are specialized for processing AI applications and algorithms! In the past, this all had to be done on the CPU or central computing unit of a smartphone. This required too much computing power for the average processor.
One of their new AI applications is called the magic eraser and you guessed it, you can erase unwanted objects from a photo with it. So where before you had to edit the photo with a special program on a computer with a decent processor, you can now do it with AI on your smartphone!
How does this work?
The AI model we use in this project uses so-called 'large mask inpainting with Fourier Convolutions'. A whole mouth full. But how does this actually work?
Default inpainting: This shape attempts to erase unwanted objects by looking at the immediately surrounding pixels. This approach does not look at the bigger picture.
Large mask inpainting: Here, in the initial phase of the approach of the model, a larger analysis field is used. This large mask therefore analyzes much more at a glance of the photo in order to adjust the unwanted parts. This approach succeeds better in including (predictable) textures of, for example, walls or fences in its solution.
In the classroom!
Questions, feedback?
Questions or feedback? Hit the botton down below and I’ll get back to you.