Cleaning up the Collateral Damage
The market developing around combating abusive usage of generative AI
It’s been a while since my last post. Hand up, that’s on me. Turns out building a startup leaves minimal time for writing.
However, I see writing as a forcing function for clearer thinking, and selfishly I do this for me, not for you, so I’m requiring myself take a moment to collect and process my thoughts on the landscape I find myself operating in so that I might help my own decision making and communication going forward.
As Paul Grahm says:
If writing down your ideas always makes them more precise and more complete, then no one who hasn't written about a topic has fully formed ideas about it. And someone who never writes has no fully formed ideas about anything nontrivial.
It feels to them as if they do, especially if they're not in the habit of critically examining their own thinking. Ideas can feel complete. It's only when you try to put them into words that you discover they're not. So if you never subject your ideas to that test, you'll not only never have fully formed ideas, but also never realize it.
With that being said, a piece exploring the market developing around combatting deepfakes is long overdue as I find myself having lots of opinions about the space, yet I haven’t taken the time to put many of my thoughts into writing.
Therefore I cannot in good conscious consider my opinions to be fully thought out or complete without first going through this refinement exercise.
Ready to figure out what I believe? Me too. Let’s go.
What is a deepfake?
Over the past couple of years I’ve observed an interesting trend.
When talking about something created using generative AI, people collectively refer to the outputs as “AI audio”, or “AI video”, or “AI text”, or something similar.
This is until its intent is deemed malicious. Then it’s called a “deepfake”.
The end result of this is that the word “deepfake” is imprecise and can refer to all kinds of malicious outputs of generative AI.
The defining characteristic of a deepfake is its harmful intent. Other common characteristics are the non-consensual aspect of the content generation and distribution, as well as the intention to deceive a target audience - even if that audience is yourself (as can be the case with deepfake sexual imagery).
This is a large part of why my eyes to roll into the back of my skull whenever I hear someone say “deepfakes are bad” or “we need to combat deepfakes”, or “deepfakes are a problem” or some other similar sounding statement.
Of course they are bad. The very word itself indicates badness.
However each of these statements oversimplifies what is fundamentally a much more nuanced and messy reality: The only differentiator between standard generative AI outputs and deepfakes is the intent.
The very imprecision of the term “deepfake” hinders our ability to address the actual issues aggravated by harmful usage of generative AI.
This same premise is part of what gets me so fired up about the concept of “deepfake detection”.
To my knowledge, there is not a vendor out there that is actually selling “deepfake detection”. They are all selling “AI detection”. Their models are trained to identify synthetic content, but not the key indicator that differentiates a deepfake from regular AI audio, video, or text - the intent.
I won’t excuse myself from this either. My own startup marketed “AI detection” as “deepfake detection” for the longest time. Until we came to the realization that we needed to understand the context around the thing we were analyzing in order to truly assess if it was a deepfake or not and how that in turn impacted the perception of the person being targeted.
I’m going down a bit of a rabbit hole here, but this nuance is important to set the stage for my analysis of the new market being created around combating abusive usage of generative AI.
Disclaimer
Let’s also not pretend that this analysis is objective.
I have a stake in this market. And I’ve already influenced your perspective with my commentary above, as it differs greatly from the opinions held by many of the other players in this space.
But since this analysis is me attempting to refine my own thinking, I’m not going to slow down or qualify for the sake of objectivity.
Onwards.
The different players
To your disappointment I’m sure, I’m not going to name names.
If this were an objective analysis by an unbiased 3rd party, that would be appropriate and expected. However, that’s not me. I’m fully invested in this space, fully biased, and fully aware that it’s not productive to call out peers, competitors, and prospective partners by name.
Instead, I’m going to map out the predominant strategies and bets being placed by the various players in this space. I’ll let you to determine who belongs where.
Finally, an important factor I want to highlight before we begin is that while organizations in all of these categories are active in combatting abusive usage of AI, they are all playing different games.
For someone on the outside, it’s important to understand the approach and strategy of organizations in each category in order to make sense of how helpful they might actually be.
Let’s get into it.
The “deepfake detection” (AI detection) companies
Companies in this group focus on helping their customers differentiate AI generated content from non-AI generated content.
These companies tend to see broad applications for their technology, with use cases spanning across industries and problem sets. They tend to highlight the accuracy of their detection algorithms, and their ability to work across audio, image, and text.
Core Problem Addressed:
Humans can no longer identify AI generated content.
Value Prop:
Quickly identify AI generated content wherever it appears.
Technology:
AI analysis platforms and real-time AI detection for audio, video, and images.
Results Output:
AI probability score in calls or in their analysis platform.
Best for:
Understanding the likelihood of something being AI generated.
The “prove what’s real” companies
Companies in this group take an inverse approach to the deepfake detection companies. Instead of trying to identify what’s fake, they focus on proving what’s real.
Like the deepfake detection companies, they also tend to see broad applications for their technology, with use cases spanning across industries and problem sets.
They tend to highlight the accuracy of their verification and credentialing methods and usually tend to focus on a specific medium or two between audio, video, or text.
Core Problem Addressed:
Humans can no longer differentiate real from fake.
Value Prop:
Verify the legitimacy of something in order to trust it - don’t trust anything else.
Technology:
Watermarking, biometrics, 2FA, content credentials and provenance records.
Results Output:
Provenance records, trust scores and alerting for low trust content.
Best for:
Evaluating if content or communications comes from a known source.
The “narrow use case” companies
Unlike the previous two categories, companies in this group focus on addressing narrowly defined problems that exist outside of AI and deepfakes, but are adapting to how abusive usage of AI is impacting their defined problem set.
Since the specific problems addressed by companies in this group vary widely, I’m going to use my own startup DeepTrust as an example of a company in this category.
Other examples include companies focused on IP and brand protection, call security, email security, copyright protection, digital forensics, disinformation protection, trust and safety, KYC, employee training - you get the point.
DeepTrust Example
Core Problem Addressed:
Social engineering, phishing, and fraud in live voice and video calls.
Value Prop:
Help employees identify and address security risks in live calls across voice and video platforms.
Technology:
Call security across platforms that includes deepfake detection, social engineering detection, and real-time security training.
Results Output:
Just-in-time security training for employees directly in live calls with security alerting and organizational risk analysis for security teams.
Best for:
Helping employees identify and action on risks in live calls.
The regulators and NGO’s
The final group is comprised of those involved in shaping the macro landscape around AI development and usage, but who are not commercially motivated.
Core Problem Addressed:
AI development and usage is outpacing regulation.
Value Prop:
Update legislation and guidelines to ensure fair and ethical AI development, adoption, and usage.
Technology:
Bills, research, memos, news, thought pieces.
Results Output:
Public attention, research, legislation, and law enforcement.
Best for:
Shaping the macro landscape around AI development, usage, and distribution.
My 2 pennies
Hey, you’re still here - you must be curious right?
While my opinions will likely evolve over time, here’s what I think right now.
The regulators and NGO’s will do their thing.
It will probably be well intentioned, it will also almost certainly be wrong. As long as they continue to view deepfakes as separate and distinct from generative AI more broadly, they will be ineffective.
If their goal is to reduce abusive usage of AI, penalties should be increased for crimes committed where AI was involved. But the language used in legislation needs to specify “AI” usage, not “deepfake” usage. Otherwise enforcement will devolve into a court battle over the semantics of what constitutes a deepfake and how a deepfake is different from other generative AI outputs.
A crime should be viewed as a crime whether or not AI was used. AI usage shouldn’t be criminalized, but criminal use of AI should be punished harder.
In the commercial arena, the “deepfake detection companies” and the “prove what’s real” companies will continue battling over who has the better answer to navigating this new world where we can’t trust our own eyes and ears anymore to differentiate between real and fake.
They will both continue trying to sell across use cases and across industries until they ultimately come to terms with the fact that their offerings are partial solutions to many problems and not comprehensive solutions to anything.
HOWEVER, once they acknowledge this, they will then realize that their true customer base is the “narrow use case companies”, NOT the end customers that the “narrow use case companies” serve.
All of the “narrow use case companies”, mine included, need to have deepfake detection and/or some form of verification as part of a much more complex and comprehensive solution to the very specific problem they are solving for their customer base.
They can either build it in house or buy it. My guess is many will end up buying it over time as it will not be a differentiated part of their offering and as a result not worth the time, energy, and money required to keep improving and maintaining in-house in the long term.
For the “narrow use case companies”, massive value will be accrued in solving each of the specific problems that are being greatly aggravated as a result of malicious use of generative AI.
There will be winners across IP and brand protection, call security, email security, copyright protection, digital forensics, disinformation protection, trust and safety, KYC, employee training - the list goes on.
Closing thoughts
Generative AI tools are an extreme accelerant. Businesses are doing more with less. People are doing more with less. Knowledge is becoming commoditized.
Rocket fuel has been given to everyone for achieving their goals - for better and for worse.
With this, the quantity, scale and severity of problems all across society are rapidly increasing.
As a result, there is a massive market developing for cleaning up the unintended consequences of this new technology.
Build, buy, and invest accordingly. But know the games being played around you.
I hope you enjoyed this blog! Please subscribe and share with your friends. New subscribers help assure me that people care and let me know that I should keep putting these out.
A little about me…
I'm a co-founder at DeepTrust where we are helping employees identify and address security risks in live calls across platforms with deepfake detection, social engineering detection, and real-time security training.
If you’re interested in learning more, or have any additional questions, I’d love to chat! Feel free to shoot me an email at noah@deeptrust.ai.
Love this breakdown. Terminology matters!