A reliable way of detecting AI content?

[Dani]

As the question states, is there a reliable way of detecting AI content? I vaguely recall OpenAI announcing something a long time ago that they were going to release something that says whether content was generated via ChatGPT, or am I misremembering?

[Reverend Jim]

I recall reading, "If the text is unusually coherent or consistent, it might raise suspicion."

I'm afraid that does not speak well of our educational system. And it's not very useful in trying to automate detection.

As a side note, I read recently about some researchers using a "needle in a haystack" test on an AI. They buried an anomalous question about pizza in with a bunch of physics questions. The AI pointed out the anomaly along with the question, "Was this inserted to test me?"

[Dani]

Well, I think what they mean is that the writing does not have a soul behind it. ;)

[Dani]

What was fast?

[Reverend Jim]

Your response to my last post.

[Dani]

I’ve been working on DaniWeb all day.

[jkon]

There are some apps that claim to detect AI Generated content and have paid APIs (e.g. GPTZero , Originality) , I haven't use them except of testing , I didn't had any good reason and furthermore the landscape is changing so fast that all the statistical patterns they might detect need better and better adjustment to work decently.

We still can detect as humans most of generated by AI texts but as things move forward that will become tougher but still feasible (even when we will have artificial super intelligence (ASI) the way we think as humans will always be different). I understand the need in forums like Daniweb to automatically detect AI generated posts and in these cases maybe one of the detectors APIs will work for you.

In EU we have the AI Act , that when implemented will require when ever you use AI (in images or texts) to inform the final user (this goes far from watermarking the content or the image with some unnoticeable by real humans technique). If this will implemented as it sounds will be one more ridiculous thing that medium and small EU tech businesses will have to do , while the bigger ones (having many lawyers) will declare that this doesn't apply to them (as have done with GDPR). Of course minimizing innovation and competition in favor of big companies.

I have heard of some proposals to include watermarking techniques in AI generated texts but I don't know anyone that implements them yet. In images they do it (e.g. DALL-E does it , and probably Midjourney will also soon). I am not sure how well watermarking techniques in AI generated texts are going to work because the text that we see even today is in most cases edited by a human , and it will always be rather easy to create a bot editor (especially with use of opensource AI)

[Dani]

Thank you for your reply. Unfortunately, I've used quite a handful of those AI detection APIs to no avail. They also often catch false positives, which is perhaps even more so detrimental to my use case.

[Reverend Jim]

I wonder how this will apply to individuals who use AI to generate, then post content? It's even more problematic in the US. Take this scenario...

If I have a party in my house, and while at that party, some people use the opportunity to sell illegal drugs, I can be charged since the illegal activity is taking place in my house.

Likewise, if a newspaper publishes lies, even those made by freelance journalists or people submitting letters to the editor, as long as that material is published in the newspaprer, the publisher can be held liabel.

But a special exemption was made in the 1990s (under Bill Clinton) which exempted internet hosted agencies. Under that exemption, Facebook, Twitter, etc. are not responsible for libelous content.

Here at Daniweb we moderate posts and remove those posts which are blatantly false, or even in some cases not false but overly nasty in tone. We do this to maintain a certain standard but apparently there is no legal requirement to do so.

[jkon]

How will US law enforcement react (or not) into AI usage by users in internet platforms we will all going to find soon (in this election cycle). Not only doctored AI images and videos for political gains , but also if you have a sheer number of posts from AI users that seemingly use different language style and have different audiences supporting a political party or opinion , even if the words aren't offensive they create a false narrative affecting the democratic process.

In my country that maybe happens in few years , but when you ask now anything a LLM in my native language you probably doing it to have a big lough with the apparent ridiculous translation of the answer from English that happens (not really) behind the scenes.

As for the EU AI Act I could hope that the implementation of it will target interacting with AI without knowing it and news / opinion articles. That would be great , but as I have seen with GDPR the implementation is always broad and vague enough to target the medium / small companies leaving the big ones almost unaffected. We will see ...

[Reverend Jim]

I realize that moderating Facebook posts is an exponentially greater task than moderating Daniweb, but if Facebook is not doing its job now, what happens once the election cycle heats up and untold thousands of AI posts created by bots start flooding the ecosphere?

[Dani]

Woot! Woot!

https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text

[Dani]

... And, oh well:

As of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy. We are working to incorporate feedback and are currently researching more effective provenance techniques for text, and have made a commitment to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated.

[fixwhatsapp]

As for me, Copyleaks offers a simple-to-use AI-content detector on their website.