Over the past few years, FB has been advancing its image recognition capabilities, which could soon support a whole new consideration for marketers. Now Facebook has released a new research paper outlining their latest image recognition improvement, including advanced search capabilities based on objects.
Here is how Facebook’s image search is evolving and the potential opportunities that will provide.
Last April, Facebook showed the release of automatic alternative text – or automatic alt text – for images posted to FB. Automatic alt text uses object recognition technology to generate a description of a photo, processing every through FB’s artificial intelligence engine to establish image content.
The system is the culmination of years of work – back in November 2015, Facebook showcased the progress they would made with their image recognition AI, with their system capable to distinguish between objects in a photo 30% faster and using 10x less training data, than previous industry benchmarks.
The level of detail the system translate may not seem like much of a return for years of effort, but the number of work involved in teaching a computer to “see” what is in an image is massive. This system needs to be trained on huge datasets with relevant descriptions of every image in order to ascertain the key elements & determine what each object is.
Yet even with all that training, there’re still limitations in what image recognition can do – for instance, in the image below, you could see how the system still stumbles when objects are ‘on top’ of each other, recognizing 2 people as one.
These types of more finite recognition abilities are much harder to train, which gives you some idea of the challenges FB’s researchers face in working to build an image recognition system. Things that a human can do easily aren’t simple for a system based on code and inputs.
But they’re getting much better and the applications for such advancements likely go beyond what you might think.
Sight & Insight
In their latest research paper, FB has outlined their advances in image recognition.
First, they have added new descriptions of actions within images, as opposed to just objects:
“Until recently, those captions described only the objects in the photo. Today we are announcing that we have added a set of 12 actions, so image descriptions will include things like “people walking,” “people playing instruments,” “people dancing,” “people riding horses,” and more.”
Again, it may not seem ground breaking and it may not have a huge impact on non-visually impaired users, but the improvement is big. The fact that Facebook’s system is moving onto actions underlines just how smart their systems are getting, and the amount of work involved to get to this stage is staggering – though FB does have a huge training model to work with, with somewhere in the vicinity of two billion photos being uploaded to Facebook, Instagram, Messenger & WhatsApp every day.
In addition to this, Facebook has announced a new search capability based on image recognition.
“…we have built a search system that leverages image understanding to sort through this vast amount of information & surface the most relevant photos quickly & easily. In other words, in a search for “black shirt photo,” the system can “see” whether there’s a black shirt in the photo & search based on that, even if the photo was not tagged with that information.”
But as much as these individual applications are interesting and will no doubt provide benefit, it is the extended application of Facebook’s image recognition AI that is far more compelling.
Right now, Facebook is using their image recognition AI for two purposes:
- To identify & remove objectionable content. As noted in this post from Ars Technica:
“In the past, users tagged the images as objectionable, and that info was funneled to the Protect & Care team. Images confirmed objectionable were canceled by a team member. Then machine learning models were built to identify & delete these images. In 2015, the ML models examined & eliminated more of these images than people did. Now, the Protect & Care group independently creates new classifiers to identify new types of objectionable material & retrain the models to automatically respond to it.”
- To build the “Memories” montages you see in your News Feed – those image packages are put together by FB’s AI based on the content you are more likely to engage and interact with.
And again, those are interesting, but further than that, it is not hard to imagine that FB’s advanced image recognition AI will soon provide additional chances for marketers, with a whole new data set to work with.
For instance, using this Google Chrome extension, you can see what Facebook’s AI in any photo posted to the network.
Now imagine this – let’s say you are a café looking to achieve new audiences in a certain region. A handy new search parameter might be the ability to locate FB users who regularly post images like this.
As you can see, Facebook’s AI has correctly identified both “coffee” & “coffee cup”. Such capacity would not only enable you to pinpoint users who are interested in coffee (based on the images they post, not what they write), but it would mean you could target people who post pictures of their coffee, which would also increase your chances of developed exposure via user-generated content.
Reach out to these people with relevant coupons or offers, and there is a good possibility that they will not only take you up on the offer, but that they will also let their friends know by posting images of your business.
In this post, Facebook’s AI has identified an ultrasound image. And while you would expect most announcements like this would contain words like “baby” or “expecting” in the text, the additional qualifier may make it easier to locate potential audiences for baby / maternity products.
Facebook has highlighted the potential benefits of such data in their own research. Last August, FB conducted a study of more than 160,000 people in the United States who have shared photos of cats / dogs (or both) to gain further insight into the differences between every group.
It is light-hearted research, for sure, but again, it underlines the potential of the image recognition technology to add an additional qualifier, another way to further clarify & refine your audiences to target the exact right people with the exact true messaging.
The next level of image recognition, of course, is video, which FB is also developing.
That capacity could further enhance Facebook’s offerings – as noted by Ars Technica:
“AI inference can rank the Live video streams, personalizing the streams for individual user’s newsfeeds & removing the latency of video publishing & distribution. The personalization of real-time reality video can be very compelling, again increasing the time that users spend in the FB app.”
It may not seem like much – searching for specific images you have taken is not a game-changing addition to your FB experience. But do not limit your perspective to what you can see now.
There is far more to Facebook’s image recognition tech than what you see on the surface.