AI Detection - Phase 1: sample collection

[fresh_42]

TL;DR Summary

Two AI detectors give frequently opposite results on the same input. I need your help to test them and collect data for statistical tests.

I know two programs that claim to be able to detect whether a text has been written by a machine or by a human.

A (ZeroGPT): https://www.zerogpt.com/
B (OpenAI): https://openai-openai-detector.hf.space/

Character Count: https://www.lettercount.com/

If you have time and examples, please test them and report the results. OpenAI also lists the number of checked tokens, so you should add this information, and of course, whether your sample was artificial or not.

Thanks, folks. A little experiment in statistics. And I guess that's only the start of such experiments around the world.

I tested two text passages from two of my insight articles and A and B detected 100% human, so no discrepancies here. That cannot be said from all text passages published at PF.

 

[fresh_42]

Here is what an optimal test report would look like:

CC: number of characters (source text is human / AI)
A: AI percentage by ZeroGPT
B: AI percentage by GPT-2

CC: 657 (100% human / 0% AI)
A: 24.53% AI
B: 0.11% AI

CC: 1638 (100% human / 0% AI)
A: 0% AI
B: 0.98% AI

 

[jack action]

From the intro of an Insight I wrote:

CC: 621 (100% human / 0% AI)
A: 100% AI (104 words)
B: 0.04% AI (113 tokens)

From another Insight of mine:

CC: 3087 (100% human / 0% AI)
A: 6.64% AI (532 words)
B: 0.02% AI (510 tokens from 624)

 

[fresh_42]

CC: 279 (96% AI, 4% hunman)
A: 0% AI (47 words)
B: 0.06% AI (60 tokens)

As the first raindrops descend from the heavens, tapping lightly on windows and leaves alike, a symphony of nature begins. The earth awakens to the rhythmic dance of droplets, filling the air with the soothing scent of petrichor and the gentle melody of raindrops kissing the ground.

As the first raindrops descend from the heavens tapping lightly on windows and leaves alike a symfony of nature begins. The earth awakens to the rhythmic dance of droplets, and fills the air with the soothing scent of petrichor, the gentle melody of raindrops kissing the ground.

Test result on the original text:

CC: 279 (100% AI, extract from the answer to "How would I start an essay about rain?")
A: 0% AI (47 words)
B: 3.47% AI (60 tokens)

 

[kuruman]

CC: 2000 (100% human)
A: 18.53% AI
B: 0.02% AI

CC: 1491 (100% AI, generated answer by ChatGPT)
A: 44.91% AI
B: 92.89% AI
The question was "How can a mission to Mars be justified when people are starving on Earth?"

I also picked a science related article of unknown (to me) composition.
Source text: https://www.theguardian.com/science/2023/nov/20/red-wine-headaches-phenolic-flavonoids-scientists
CC: 3776 (see link above)
A: 20% AI
B: 0.03% AI