Cybercriminals no longer need to write their own phishing messages. A new pilot study suggests that generative AI can produce personalized spear phishing texts that are at least as persuasive as those written by trained humans, and that people have almost no ability to tell the difference by intuition.

The study, conducted at Brigham Young University, involved 25 volunteers who were asked to sort a set of text messages printed on cards. The messages were personalized for each participant using information from a survey: job, workplace, hobbies, city, and a recent social media post. Twelve messages were created per person. Half were written by GPT-4 from a simple prompt template. The other half were written by undergraduate students in a deception course, who had received phishing instruction and had their work screened by a review team that included two cybersecurity professors. Roughly a third of the student submissions were discarded for being incomplete or unusable.

The volunteers were not told which messages came from the AI. They were asked to sort all twelve from most to least likely to click, and then draw a line above which they would have clicked. GPT-4 messages crossed that click line 28% of the time. Student-written messages crossed it 21.3% of the time. The difference was 6.7 percentage points, but the confidence interval was extremely wide, ranging from 2.9 points in favor of the students to 16.3 points in favor of the model. With only 25 participants, the study cannot definitively say that GPT-4 outperformed the human writers. It also cannot say they were equal.

Why the Small Difference Matters

The important point is not the exact numbers. It is the comparison in cost and effort. The AI messages came from one short prompt built from a survey and generated in a single run. The human messages came from students who had been trained and whose work was filtered by a professional review process. A simple automated system landed in the same neighborhood as that carefully managed human pipeline, and the AI can be run again and again for the price of an API call.

That asymmetry changes the risk picture for organizations. Attackers can now automate the creation of highly individualized messages to thousands of targets at once. Spear phishing, once a time-intensive activity reserved for high-value targets, can be scaled to practically anyone with a digital footprint.

Work-Themed Messages Were the Most Dangerous

The strongest finding in the study was the theme of the message. Every participant received three types of messages: one based on their job, one on a hobby, and one on a recent social media post. Job-related messages crossed the click line 38% of the time. Hobby-themed messages crossed it 19% of the time. Social media messages crossed it 17% of the time. This was the only comparison that survived statistical adjustment, making it the clearest takeaway from the research.

Work-related context creates urgency and trust. A message that mentions a fraud alert, a coworker, or an internal process looks like something the recipient would expect to receive. The study included one banker who stopped sorting when they saw a message that looked exactly like a real fraud alert used in their workplace.

But personalization also introduced obvious failures. One participant pointed out that a message referenced a coworker who did not exist. Another noted that the named person on a message was not part of their university department. A dance-related message went to the bottom of the pile because the participant had quit dancing. Wrong details can be worse than no details because they give the target a reason to be suspicious.

People Could Not Detect AI Messages

After sorting the messages, participants were told that some messages had been written by an AI and were asked to mark them. Across 300 judgments, they were correct 52% of the time, barely better than chance, which is 50%. They had many theories but none worked. Some said AI writes too formally. Others said AI writes too generically. Perfect grammar was often seen as a machine tell, while typos were seen as proof of a human. One participant wondered why there were so many exclamation points. Twelve of the 25 admitted they were mostly guessing.

The emoji finding illustrates how unreliable these instincts were. Emojis were actually a strong signal: 66% of GPT-4 messages contained them, while only 2% of the human messages did. But only five participants mentioned emojis at all, and their interpretations were split. Two thought emojis indicated AI, while three thought emojis indicated a human, reasoning that a bot would not use them.

Software Detected What People Could Not

The researchers also tested a machine learning approach. They converted all 300 messages into embeddings and trained a classifier to distinguish AI-written text from human-written text. Under the strictest test conditions, which included standardizing URLs, removing emojis, flattening case, digits and punctuation, and matching message lengths, the classifier reached 88.7% balanced accuracy. It was tested only on messages from participants the model had never seen during training, so it was not simply memorizing targets.

People could not find the pattern, but a logistic regression found it easily. This suggests that there are linguistic fingerprints in AI-generated text, at least in this controlled setting.

However, this should not be treated as a practical detector. The classifier was trained and tested on one message set, from one model, with one prompt design, against one group of student writers. It has no known ability to generalize to other models, prompts, or attacker styles. Research cited in the paper also shows that paraphrasing AI text with a detection system in the loop can significantly degrade the performance of these classifiers.

Limitations and Open Questions

There are other important limitations. The messages were printed on cards, so there were no phone notifications, sender numbers, links, or timestamps. Participants reported what they thought they would click, which is a common proxy in phishing research but still not the same as observing actual clicks. The human comparison was novice students, not professional social engineers. To detect a difference the size of the one observed with confidence, the study would need about 100 completed targets instead of 25.

The exact GPT-4 snapshot and API logs were not recorded, so the generation run cannot be perfectly reproduced. The messages themselves survive and the analysis can be reproduced, but the exact setup is incomplete. This is a pilot study with a small sample, and the findings should be read with that in mind.

Practical Advice for Staying Safe

The practical advice at the end of the study is simple and does not depend on the uncertainties. Check the sender, the channel, the link, and the request against what you would expect to receive. Do not try to decide whether the message sounds like a robot. That is the one capability the study shows people do not have.

Organizations should treat AI-generated phishing as a default threat rather than an exotic one. Security awareness training should move away from teaching people to spot AI patterns, since the evidence suggests intuition is not up to the task. Instead, the focus should be on verifying requests through trusted channels, using multi-factor authentication, and reporting suspicious messages quickly.

The broader implication is that the phishing landscape is shifting. Attackers who can generate personalized messages at scale will create new demands on email security, endpoint detection, and user behavior. Human instinct, the study suggests, will not be the defense that saves the day.

Source: Help Net Security News