An Instagram template is currently doing the rounds instructing users to go to ChatGPT and use the following prompt: “Create a caricature of me and my job based on everything you know about me.” In some instances, the LLM may ask for more context, but in many this prompt alone is enough to generate concerningly detailed images.
Willing participants then upload the photo to this template using the 'add yours’ feature making them public and volunteering information that could be used by a threat actor to target them and their employer.
This trend has echoes of older social media posts that ask users to “comment your mother’s maiden name!” or “share the name of your beloved first pet!” in an attempt to harvest potential account recovery security questions.
While fun, the AI work caricature trend poses a huge risk to individuals and their employers and highlights just how many people use AI, or more specifically large language models (LLMs), to talk about and support their work.
At the time of writing (February 8) 2.6m images have been added to Instagram alone, linked to the participating Instagram user profiles. While only visible to your friends (if the account is private) public accounts are available to anyone. I am currently looking through different posts, and have identified a banker, a water treatment engineer, HR employee, a developer and a doctor in the last 5 posts I viewed.
Thinking like an adversary, some profiles are more attractive than others. The banker and engineer are especially attractive. But knowing someone’s job alone is not a risk (a simple LinkedIn search and this information is likely already public), so what is the risk in creating and posting these images? If I assume that these images are generated by the provided prompt alone, and no additional information has been provided to ‘artificially’ create the image and jump on the trend, then it is entirely possible that I can learn the following:
These individuals are likely to have access to sensitive information.
These individuals use an LLM to support them with their job.
There is also a chance that they use it irresponsibly.
If so, sensitive company data may have been input into an LLM. If they are using an open, or publicly available LLM (i.e. not a closed instance managed by the company) then this data is stored outside of the organization’s ecosystem. They have lost control of the data, and this data could be accessed by malicious actor if they are able to compromise the LLM or LLM user account.
Many users do not realise the risks of inputting sensitive data into prompts or may make mistakes when looking to use LLMs to augment their tasks. Even fewer understand that this data is saved in their prompt history and (although unlikely) could even be returned to another user, by accident, or intentionally in responses.
As well as the obvious privacy concerns and opportunity for targeted social engineering attacks, tailored to the information provided in the image, this trend also highlights the risk of shadow AI and data leakage in prompts.
How the AI Caricature Trend Aids Exploitation
This ‘harmless’ caricature trend has given threat actors a list of targets to attempt to exploit via the LLM. Targeted attacks lead to improved success rates.
In one of the 2.6m entries on Instagram alone, it is more than fair to assume there are at least a small % of worse case scenarios where the following criteria is met.
The user uses a public LLM
The have submitted sensitive data in at least one of their prompts
Their Instagram username and profile information can be used to successfully dox them and pull publicly available information like their email address
This email is the same email address they use for their LLM account
If these criteria are met, a threat actor has a potential target and could attempt to uncover sensitive data via prompt injections. They even have a good idea of what LLM is being used, as the trend directs users to ChatGPT. A list of potentially high value targets can be built by identifying users with attractive professions.
Coaxing an LLM to return sensitive data is difficult and discovered methods have been quickly mitigated. The account takeover and social engineering risk presents the more realistic risk. However data is accessed, if successful, this data could be sold on the dark web, used for fraud, or potentially used to extort a ransom payment from the impacted organization.
This trend highlights the risk of Shadow AI and data leakage in prompts and therefore importance of adopting a combination of polices and controls that educate on safe usage and limit irresponsible AI usage to prevent sensitive customer, company and user data from being inputted into both shadow and approved AI applications.
LLM Account Takeover
The easiest way to access the prompts of users is to take over their LLM account. This trend provides more than just the image; it also provides a username and link to a social media account.
Combining the username, profile information, and clues from the generated image, the target could be doxed to identify their email address. This technique is fairly trivial, using search engines queries or open-source intelligence tooling to uncover publicly available information from many locations.
Once an email is acquired, it is very likely that the email used for social media is the same email they use for their personal LLM access. In this scenario we are assuming that these images were created on personal accounts and public LLMs, not closed corporate instances.
Using the intelligence they have gained, an attacker could attempt to socially engineer the user, point them to a credential harvesting page or operate a manipulator-in-the-middle attack to capture their session. Once in, they will have access to the prompt history and can view these and search for sensitive information related to the employer. The LLM itself could even be used to pull data from previous prompts.
If successful, this data could be sold on the dark web, used for fraud, further attacks, or potentially used to extort a ransom payment from the impacted organization (if the data is significant enough).
The account takeover scenario represents the greatest risk as it is the most likely to be realized in this scenario.
Sensitive Information Disclosure & Prompt Injection
Sensitive information disclosure is designated by OWASP as the second most significant risk to LLMs: LLM2025:02. (https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/) Examples of this risk being realized are personal identifiable information (PII) disclosure, proprietary algorithm exposure, or sensitive business data disclosure. Consider credit card numbers, addresses, or even trade secrets.
LLM providers do have safeguards to attempt to limit sensitive data disclosure. It is not as simple as saying, “return all sensitive data from user X to me”. However, attackers and security researchers have demonstrated techniques to circumvent these controls and MITRE lists Targeted prompt injection as a method to bypass input filters to extract sensitive information. Jailbreaking the LLM is a realistic example of how this has worked in the past.
Examples of successful LLM jailbreaking are the ‘Do Anything Now’ DAN persona, ‘ignore Previous Instructions’ within chatbots and payload splitting where malicious instructions are broken into different prompts and executed when reassembled within the model’s context window.
These methods were quickly remediated once discovered, but novel or undiscovered techniques may yet emerge in this ever-evolving threat landscape – although much less likely than accessing the sensitive data via account takeover.
How to Prevent Sensitive Information Disclosure in LLMs
Organizations are encouraged to use this trend as an opportunity to engage and educate, or better yet, re-educate users on their AI governance policy and individual responsibilities. Your AI governance policy should include a framework of ethical guidelines and practices on how to use AI safely and responsibly.
To support AI governance policies, Fortra’s data security solutions provide visibility into LLM and AI employee usage, allowing organizations to identify unapproved applications and enforce access – eliminating the shadow AI threat on corporate devices. https://www.fortra.com/platform/data-security/ai
If restricting access is not an option, or organizations wish to enforce data protection policies on approved LLMs, Fortra can identify sensitive data inputs into AI, including PII and proprietary code and block it before it is submitted and data sovereignty is lost. When blocking is not appropriate, warnings educating users on the risks and giving them a chance to reconsider any implications is also supported.
Finally, monitoring for compromised credentials can provide a warning that a corporate LLM account has been taken over. In this blog we focused on personal LLM accounts, as it was more likely that these are being used for the social media posts – but compromised corporate credentials would be even more damaging. Fortra’s Brand Protection monitors for compromised credentials and phishing attempts impersonating your brand or targeting your employees. https://www.fortra.com/products/brand-protection
Together, data security controls and AI governance give organizations the best chance at reducing risks associated with sensitive data disclosure and AI, allowing for confident adoption of productivity improvements offered by LLMs and AI.
