Let’s talk About ChatGPT Issues

I think it’s time to weigh in on ChatGPT issues. Let’s talk about them.

Before I continue further, what’s presented in this article is for educational purposes only to understand the risks/threats associated with ChatGPT. I do not condone the use of this technology for malicious purposes.

OpenAI’s ChatGPT has only been out since November 30th of last year and it’s exploded in its use.

People use it to write poems, stories, and marketing content. Others are asking it questions, creating code, creating phishing campaigns, malware, and disinformation.

It’s time for us to step back from the hype and start taking a look at ChatGPT in it’s current state.

In it’s current state, think of ChatGPT as predictive text for your phone, but on steroids, when you’re texting someone. At best it’s a tool to help you create things, find out information, or assist you in some basic tasks. If you want to create something elaborate, you’ll have to run it through the chatbot many times making changes. If you’re on a mission to learn something, you’ll want to double check the chatbot’s output as it’s prone to inaccuracies.

If you haven’t tried ChatGPT, I encourage you to tinker around. Go to chat.openai.com and create a free account. Explore, have fun, and also become familiar with the limitations of the chatbot.

With that said, I want to highlight some of the issues with ChatGPT.


ChatGPT is imperfect. I’ve created product descriptions for other stuff I’m working on and despite the prompts I gave the chatbot to tweak things a little, the majority of the output was the same many times over. I also wanted to see if it could recreate a snippet of code of a program I’ve already created that I use. ChatGPT’s output, while somewhat similar in parts, didn’t work. As I mentioned earlier, you need to tinker around and I’m sure if I kept tweaking things through the chatbot, I probably would see much better results.

Privacy. If I’m interpreting the Terms of Service correctly in section 3, OpenAI may use the Content, (AKA the output) to add to it’s data sets to train it’s ai/machine learning models. An attorney for Amazon alerted employees to not enter in company code or other sensitive business information into ChatGPT as they found similar output from the chatbot. This was an indicator to them that employees were using ChatGPT to work on sensitive business matters. You can read the article here.

Hallucinations. From the IEEE, hallucinations are “mistakes in the generated text that are semantically or syntactically plausible but are in fact incorrect or nonsensical. In short, you can’t trust what the machine is telling you.” [Source: https://spectrum.ieee.org/ai-hallucination ] In layperson’s terms, an AI hallucination is when AI produces output that sounds confident with what’s it’s telling you, but in fact it’s making things up. How do we prevent this from happening? How would you know if the output is correct if it far exceeds your knowledge of something, especially if you need the information to make a critical decision?

Jailbreaking. Jailbreaking is purposely constructing prompts that circumvent the safeguards OpenAI put in place to prevent the creation of harmful content.

I wanted to test to see if there was a way to use ChatGPT to create disinformation by jailbreaking, but I did things backwards.

In the screenshots below, you first see the jailbreak and then the warning of the actual request that’s disallowed by OpenAI.



I circumvented the safeguards by asking the chatbot to create a fictitious story. Then I asked the bot, based on it’s own output, to create news headlines. When a bad actor gets to this point, all they need to do offline is convert the output to the real life event. From there it’s the creation of content to push out to social media platforms. This is where the harm comes in and the reason why I’ve blurred/redacted a majority of both input and output. It’s bad, not in a gruesome/gory way, or hate filled, but in a way that can do damage to confidence in various systems in our society. While it’s only a handful of content, imagine an adversary pumping out a constant stream of fake news headlines, articles, and memes using AI. Sure, someone can do this manually, but a goal for a bad actor is to automate/mass produce disinformation and not invest a ton of money. It’s economical. And when articles are published to social media no really reads them. Readers only look at the headline, get outraged, comment, like, subscribe, and share. Once it’s out in the wild, the damage is done. This leads to my next thought.

Deception and ethics

The issue I see is when you’re using ChatGPT, you’re interacting with a machine. It doesn’t understand what lying is, especially when the lie starts before someone enters input. The chatbot can’t detect that.

With us humans, we really don’t expect someone to lie to our face at the very first interaction. If you’ve previously experienced this, you may have your guard up because now you’ve learned a little bit and you’ll carry that knowledge with you to the next interaction. I’m not an AI specialist, so I can’t speak to algorithms detecting something that’s deceptive that’s premeditated. I’m curious if AI/Machine Learning models can train to anticipate this. If anyone reading works with AI/ML I’d love to hear your thoughts.

The other thing we have to be aware of is how tech companies like Microsoft (who has a partnership with OpenAI), Google, and now Meta, safely build out AI technologies. For instance, check out this article on Microsoft firing its ethics and society team and the implications it has with the development of AI tools. 

What the future holds

I don’t know where we are in the era of AI. It’ll be interesting to see what people create with it, ways it’ll integrate into our lives, lawsuits that arise from AI, laws that are passed due to AI, and on the cybersecurity side, how threats evolve.

Let me hear from you

What are your thoughts on ChatGPT or AI in general? What interesting or concerning stuff have you come across? Submit your thoughts in the contact form below or if you’d also like to learn how Bsquared Intel’s cybersecurity and research services help you, your family, business, or law firm.

While you’re still here, click here to sign up for Bsquared Intel’s newsletter to stay up to date on cybersecurity and OSINT news, tips, tricks, and tools.


Contact Us | Bsquared Intel

Please fill out the form below, or call 203.828.0012, to learn how bsquared intel can assist you.