Poisoned AI answers: Why AI chatbot links can be dangerous

AI chatbots can be useful for finding shops that sell exactly what you’re looking for, websites that have the information you need, or software to fix an issue, but the links they give you aren’t always safe. A team of Threat Researchers at Gen (the company behind Norton) led by Jakub Křoustek investigated how scam websites and malware are reaching people through chatbot answers. Their findings show that a helpful-sounding recommendation can lead to a fake store, a malicious download, or a phishing page.

Read on for an exploration of how AI answer poisoning works, or read the full technical analysis of poisoned AI answers on the Gen blog.

Four ways scam links can reach chatbot answers

Our Threat Researchers identified four major ways scam links could infiltrate an AI chatbot’s responses: models learn from bad data, chatbots recommend fake or spoofed websites, hackers compromise legitimate websites that AI chatbots then cite, and shared AI chats give instructions to download malware. This research focuses on the last three ways.

• Models learn from poisoned data: This is known as AI model poisoning. Bad actors manipulate content so an AI model absorbs false information or unsafe recommendations during its training process. When someone later asks a related question, the chatbot may regurgitate that information back.
• Fake websites impersonate real ones: Cybercriminals create spoofed domains that resemble official sites of legitimate brands, products, or software. As chatbots crawl the internet to serve you relevant links, they may find one of these sites online and mistake it for the real deal. They then recommend it in a response.
• Chatbots cite hacked websites: Cybercriminals can hack legitimate websites and redirect their human visitors to a malicious website. Since it’s often possible for websites to differentiate between human visitors and AI crawlers, chatbots may see the legit version of the site, while users get redirected to a fake.
• Shared AI chats spread malware: Cybercriminals create shared conversations on popular AI platforms that contain malicious installation instructions. They can promote links to those conversations through search results, ads, social media, or messages, leading people to open the chat and follow instructions that download malware.

To fool AI chatbots, some cybercriminals use a technique called cloaking, which is when a website shows different content to different types of visitors. For example, the website might check whether it’s being visited by an AI chatbot or security tool, rather than a person. It may then show the chatbot a normal webpage that looks safe, while sending someone who clicks the link to a scam or malware download.

What our Threat Researchers’ investigation found

Over a 90-day period, Gen’s cybersecurity products, which include Norton 360, blocked more than 260,000 threat-related events associated with AI chat services and protected more than 121,000 users from this type of threat.

That works out to one web threat linked to AI chats blocked every 30 seconds. And the problem is growing fast: the number of people Gen products protect from dangerous AI links each day is more than 50% higher than it was three months ago.

Here are a few of the documented cases of threats associated with AI chatbots that our Threat Researchers uncovered.

Lookalike software

Someone asking ChatGPT for the official website of the image-upscaling app Upscayl (Upscayl.org) was directed to a similar-looking fake domain. The download contained an installer that delivered information-stealing malware.

Fake stores

ChatGPT recommended a fraudulent shop impersonating EcoFlow, a well-known maker of portable power stations. The site copied the brand’s look but had a recently registered domain and no physical address, privacy policy, or terms.

Investment and phishing pages

One chatbot referral led to an “AI trading platform” called BitGPT Pro with no verifiable company details, built to collect personal information.

Other links led to fake Netflix and Microsoft login pages that sent credentials straight to cybercriminals.

Compromised legitimate sites

A real citation from the Asociación Nacional de Periodistas (ANP), a respected press association in Peru, led to a website that attackers had hacked. The site showed human and AI visitors different versions of the content, depending on who was visiting. AI visitors got the legitimate site, while humans were redirected to a scam page.

Malware spread by AI chats

In an earlier investigation of cybercriminals abusing Claude's own website, Gen researchers found cybercriminals using Claude’s shared-chat feature to host fake Claude installation instructions. They promoted links to the chats through Google Ads, and users who followed the instructions ran a command that downloaded malware.

Scammers have always gamed search results

AI answer poisoning is an evolution of a cyberthreat that has been around for a long time. For years, criminals have used a tactic called SEO poisoning. SEO (search engine optimization) refers to a set of techniques that help ensure specific websites appear as close as possible to the top of a search engine results page for specific queries. It’s a legitimate practice employed around the world.

SEO poisoning is when bad actors abuse these methods. It could involve building pages designed to rank highly in Google search results for popular queries, such as “download free photo editor” or “cheap hiking boots,” so that fraudulent stores, malicious downloads, and phishing sites appear near the top of the answer page. The result looks like any other search hit, and people click because the search engine put it there.

AEO poisoning as an emerging threat

As users turn to chatbots for answers alongside (or instead of) traditional search engines, scammers are meeting them there. AEO poisoning — an evolved form of SEO poisoning — exploits the techniques behind answer engine optimization (AEO), which makes a website more likely to appear in answers and recommendations from AI assistants, much as SEO aims to improve search rankings. Legitimate businesses use AEO to make their content easier for AI systems to find and use, but criminals can use the same methods to get fraudulent or malicious websites recommended or slip manipulated information into a chatbot’s answer.

Poisoned AI chatbot answers could potentially be more harmful than a fraudulent search result. A search engine shows you many links and leaves the judgment to you. A chatbot gives you one confident answer in a friendly voice, and many people accept it without the skepticism they would bring to a list of search results. This trust can be exploited by criminals.

Red flags to watch for in AI chatbot answers

You may not realize a link is fake until you’re already on the page, so look for these warning signs:

• The address is slightly off: Look for extra words, misspellings, unusual endings like “.shop” or “.io” on a brand that doesn’t use them, or numbers swapped for letters.
• The business has little verifiable information: A lack of a physical address, phone number, return policy, or privacy policy is a strong warning sign that you might be on a scam website.
• Contact details don’t match: Multiple unrelated phone numbers or social accounts with few followers or negative reviews should raise suspicion.
• Payment options are limited: A store that accepts only one payment method, such as wire transfer, or requires registration deserves caution.
• The offer is too good: Unusually low prices or guaranteed high returns on investments are common bait.
• You’re asked to run a command: Any page that tells you to paste something into PowerShell or Terminal, or to “verify” you’re human by running code, is a major red flag.
• Login screens appear unexpectedly: A surprise login prompt, especially one that follows a link from a chatbot, should be treated with suspicion.

How to defend against poisoned AI answers

When using AI chatbots or agents, it’s best to take extra precautions to help keep yourself, your identity, and your accounts safer.

• Verify before you click: Check if a website is safe before clicking. When a chatbot gives you a website link, search for the company or product yourself and compare the address with the result from the official source. Better yet, go to the brand’s known website by typing the address directly.
• Check the developer’s official domain before downloading software: A familiar name in the address bar isn’t enough, since lookalike sites are designed to fool people who skim.
• Research unfamiliar stores: Look for contact details, a physical address, clear policies, and independent reviews. Confirm the store’s relationship to any brand it claims to sell.
• Be skeptical of investment and crypto platforms: If you can’t find verifiable company information, leadership information, or product details, don’t share personal information or money.
• Never run commands from a website to fix a problem or prove you’re human: A site that asks you to paste code into PowerShell or a terminal to fix a problem or prove you’re human is a major red flag, even if other parts of the site look legitimate.
• Use AI-powered tools to check suspicious links and offers: Advanced scam protection tools, like Norton Genie, can help you assess whether a website, message, or offer looks like a scam before you share information or pay. If something feels off, run it past trusted Cyber Safety software before you continue.

Remember, just because a chatbot recommends a website doesn’t mean it’s safe. Criminals don’t need to hack the AI chatbot itself. They use planted or compromised web pages, invented lookalike web addresses, or malicious lures hosted on the chatbot platform itself to deliver scams through answers that feel trustworthy.

Checking the link address, researching unfamiliar businesses, watching for red flags, and keeping protection enabled will help you benefit from AI assistants without handing your information to criminals.

FAQs

What is AI poisoning?

AI poisoning, also called AI model poisoning or data poisoning, is when criminals manipulate the information used to train an AI model. This can cause the model to absorb false or harmful information and later produce unsafe answers or recommendations.

Is SEO poisoning the same thing as what’s happening with chatbots?

They’re closely related. SEO poisoning manipulates search rankings so that malicious pages appear near the top of results for popular queries. The chatbot version works similarly, but criminals are using Answer Engine Optimization (AEO) techniques to infiltrate the answers an AI chatbot provides.

Did criminals hack ChatGPT or other AI chatbots?

Not in the cases our Threat Researchers described. Criminals are exploiting how chatbots find and recommend information, such as by planting or compromising web pages, creating lookalike domains, or hosting lures on chatbot platforms. The AI systems themselves were not broken into in these instances.

What is cloaking?

Cloaking is when a website identifies who is visiting based on factors like location, browser, or device, then decides what to show. Security scanners and AI assistants may see a harmless page, while a regular visitor gets redirected to a scam.

How can I tell if a website is fake?

Watch for warning signs such as a web address that is slightly off, a business with little verifiable information (like a missing physical address, privacy policy, or terms), contact details that don’t match, limited payment options, offers that seem too good, requests to run commands, and unexpected login screens. Investment sites with unusually attractive returns but no verifiable company information also deserve skepticism.

What should I do if a chatbot recommends a software download?

Find the developer’s official website yourself, either by searching for it or by typing the address you already know, and download the software only from there. Don’t rely on the address the chatbot gave you, even if the name looks familiar.

Are AI agents at risk too?

Yes. AI agents that browse websites, shop, or download software can be targeted, too. This risk was highlighted during Black October 2026 — so named for a run of cybersecurity developments — when researchers demonstrated how malicious instructions hidden in an email could hijack AI assistants.