Photo to Paragraph: Unveiling Facebook’s Stylistic Text Conversion AI

The concept of deepfakes scares a lot of people. Most of the time, when these AI-powered projects gain widespread attention, it’s because they were used unethically.

From identity theft to fake news, we’ve seen all the bad side of this tech. But Facebook’s one company that wants to use it for future innovations instead.

Disclaimer: This post includes affiliate links

If you click on a link and make a purchase, I may receive a commission at no extra cost to you.

What Is TextStyleBrush?

Researchers over at Facebook have introduced their new self-supervised AI model, “TextStyleBrush” onNewsroom . Using just one image of a single word, it can edit and replace the text while copying the font or visual style that it was written in.

The tool works on both handwritten text and text in real-word scenes. It’s an impressive feat, because—as Facebook itself notes—the AI has to understand “unlimited” text styles.

Not only is there all sorts of different kinds of typography and calligraphy out there, but there’s so many stylistic details that need to be taken into consideration. What if someone writes on a slant or a curve? Or the surface that the text is on makes it hard to read? What about background clutter or image noise?

Facebook explained the TextStyleBrush model works similar to the way style brush tools work in word processors, but for text aesthetics in images:

It surpasses state-of-the-art accuracy in both automated tests and user studies for any type of text. (…) We take a more holistic training approach and disentangle the content of a text image from all aspects of its appearance of the entire word box. The representation of the overall appearance can then be applied as one-shot-transfer without retraining on the novel source style samples.

If you want a more technical breakdown of TextStyleBrush’s functions, you can read the full-length article on Facebook’sAI blog .

The Google Translate mobile app has a function similar to Facebook’s new AI in that it also replaces text, but it aims to solve a different problem. Just like its name would suggest, it instead translates text into a language of your choosing.

Faces Aren’t the Only Thing That Can Be Deepfaked

We’re hearing about this research because Facebook hopes that by publishing it, there will be even more research and discussion on deepfake text attacks.

The tech world frequently talks about how concerning deepfake faces are, but doesn’t discuss often enough how it’s possible now to also use tech to create convincing fakes of handwriting, signage, etc.

If AI researchers and practitioners can get ahead of bad actors, Facebook says, then it’ll be easier to detect when deepfakes are used maliciously, and to build systems to combat them.