The Battle of the Little Big Chat at Texas A&M

[jedishrfu]

An inept Texas A&M prof flunks students because he said Chat GPT has told him the students let Chat write their essays. Now some cant graduate because of the snafu.

https://www.rollingstone.com/cultur...ssor-flunks-students-false-claims-1234736601/

 

[berkeman]

Dr. Jared Mumm, a campus rodeo instructor who also teaches agricultural classes,

 

[gleem]

There are a number of so-called ChatGPT detectors that look for characteristics of ChatGPT generated text. They seem good for native language speakers but poor for non-native speakers.

https://techxplore.com/news/2023-05-gpt-detectors-solution-ai-problem.html

While the detectors were "near-perfect" in evaluating essays written by U.S.-born eighth-graders, they classified more than half of TOEFL essays (61.22%) written by non-native English students as AI-generated (TOEFL is an acronym for the Test of English as a Foreign Language).

It gets worse. According to the study, all seven AI detectors unanimously identified 18 of the 91 TOEFL student essays (19%) as AI-generated and a remarkable 89 of the 91 TOEFL essays (97%) were flagged by at least one of the detectors.

 

[PeroK]

What we need is an AI system to assess whether other AI systems can accurately identify AI generated essays!

 

[gleem]

GPTZero apparently does.

https://www.forbes.com/sites/rashis...-launches-new-browser-plugin/?sh=784772b63f7b

GPTZero scans text using its own large language model, which is an “ensemble” of other open-sourced AI models and is trained on both human and AI-generated text, including news articles and questions and answers. By learning from existing generative AI models, the tool calculates and predicts the probability of words in an AI-generated sentence. GPTZero also analyzes patterns in writing using syntax and sentence length to identify machine-generated text. It does a good job of that, though Tian concedes there is always room for improvement. Since chatbot outputs are dependent on the data and prompts given to them, the tool might occasionally flag human content as AI-written and vice-versa. That’s why GPTZero uses both machine learning and human labor to recreate content from different AI programs, ensuring it will evolve to spot AI-generated text with increasing accuracy.

 

[jedishrfu]

In the end though, we will have a terrible mess of text generated by AI combined with human generated writing in the training sets that will be indistinguishable from one another. It may be the end of having a unique voice in your writing.

Alternatively, the AI developers will need to restrict the training data to be written before some seminal start date for when AI text generation first appeared (2020-2021 or earlier) to prevent the mixing of the two. The consequences may affect to creation of new words and the evolution of language may get stunted unless we can label writing that is definitely written by a human which will be hard knowing humans cheat until caught. (aka the recent chess scandal involved Magnus Carlson accusing another player of using AI assists in a game repeatedly)

In my case, Chat changed my writing to a more readable but blander version of what I wrote and I could only sigh because the Chat version was better for what I needed.

 

[Grelbr42]

So if I understand what ChatGPT does, it goes on the net and samples a huge amount of web sites. Then it does some sort of synthesis.

Surmise that a student under pressure does a Google search for web sites on his topic. And suppose he even does "the right thing" by reading these web sites, thinking about them, and then writing his essay on his own. His essay is going to have significant elements of style and significant aspects of content that come through simply because he read those web pages. This would be true if his sources were textbooks or journal articles or whatever. He is going to be influenced by the stuff he read just before he writes his essay. Britannica or WikiPedia or the class text.

In other words, the student is, supposing he is doing "the right thing," the student is doing much what ChatGPT is purported to be doing.

So, wouldn't there almost by definition, be some overlap in the results?

I have read a ton of books by a certain select set of authors. I recognize that I have absorbed certain of their stylistic peculiarities. For example, I have read everything I can find by Robert Heinlein. He has a tendency to start three or four sentences in a row with "but." I have frequently found myself doing that and tried to correct out of it. But I still do it a lot, particularly when I'm in a hurry. (See?) If an AI found me doing that it might well flag me for plagiarism. Or at least, undue influence without a citation.

If essays are part of the mark for a class, I would be pretty nervous about getting dinged for using an AI to create them. And nervous about being able to defend myself from the charge. Maybe take-home essays are going to be deprecated in favor of "in class" essays?

 

[russ_watters]

In my case, Chat changed my writing to a more readable but blander version of what I wrote and I could only sigh because the Chat version was better for what I needed.

One of the reasons I haven't tried it yet is I'm very particular about my writing and don't think I'd like what it says. But maybe I'd have a similar experience...

 

[russ_watters]

So if I understand what ChatGPT does, it goes on the net and samples a huge amount of web sites. Then it does some sort of synthesis.

Not exactly; it doesn't (in the current incarnation) access the actual sources, but uses a pre-constructed model that analyzed them. It's writing isnt paraphrased from the sources, but predicted from the prompt and the model.
That's it's big flaw/source of inaccuracy.

A mentor who will remain nameless is writing(?) an insight article to explain in more detail.

In other words, the student is, supposing he is doing "the right thing," the student is doing much what ChatGPT is purported to be doing.

So, wouldn't there almost by definition, be some overlap in the results?

By that logic a class full of humans should turn in 30 identical essays too.

If essays are part of the mark for a class, I would be pretty nervous about getting dinged for using an AI to create them. And nervous about being able to defend myself from the charge.

Ethics should not be a matter of fear of getting caught, but yes, in college anyway the consequences are severe - often zero tolerance expulsion. In high school you just fail - they can't expel you.