Referenews

If you haven’t yet tried ChatGPT, OpenAI’s new artificial-intelligence chatbot, it will blow your mind. Tell the bot to write you anything—an email apologizing to your boss, an article about the world’s richest hamster, a “Seinfeld” script set in 2022—and it spits out text you’d think was written by a human. Knowledge of the topic, proper punctuation, varied sentence structure, clear organization. It’s all there.

You can also tell it to write a 500-word essay about “The Great Gatsby” or the Spanish Inquisition. So I did what any masochistic tech journalist would: I pulled a “Billy Madison” and went back to school. I wanted to test the capabilities—and limits—of a technological marvel that stands poised to disrupt how every student in the world is tested and how every teacher grades.

At first, I thought I’d return to the halls and pimples of middle school. But when I sent a ChatGPT-generated essay to a seventh-grade writing teacher, she told me she could easily spot the fake. The writing and vocabulary were too advanced.

So off to 12th-grade AP Lit I went.

Michael Diamond,

an English teacher at High Tech High School in Secaucus, N.J., welcomed me—and my AI stand-in. He had already tried out ChatGPT with his own essay assignments.

So did I get an A? Not exactly.

Test 1: Turning In the Assignment 

Here’s a short version of Mr. Diamond’s assignment:

“In a 500- to 1,000-word essay, compose an argument that attempts to situate ‘Ferris Bueller’s Day Off’ as an existentialist text. Use specific evidence from the class materials, and make explicit comparisons or connections between characters, setting and/or themes in both ‘Ferris Bueller’ and ‘The Metamorphosis’ by Franz Kafka.”

The classic 1986 John Hughes movie? No problem. I grew up singing “Twist and Shout” into a hair brush and pretending the couch was floating along the Chicago streets. But Franz Kafka’s novella about a man who wakes up as a bug? I swatted that away almost immediately.

I pasted the assignment into chat.openai.com, hit enter and watched the bot type out 400 words before giving me a “network error.” Great, I’m an hour from deadline and my AI ghostwriter was napping.

An OpenAI spokeswoman said the system has been struggling with demand and the company has been working to scale it up. 

Finally, it worked. I pasted the 800-word essay into a document, asked ChatGPT how to format a high-school AP paper (double spacing, 12-point Times New Roman font, indented paragraphs), put my name on top and emailed it to Mr. Diamond. I added a note:

“I am writing to apologize for the lateness of my essay. I know that you have specific expectations for deadlines and I am sorry that I did not meet them.”

Of course, the note was by ChatGPT. Mr. Diamond wrote back within minutes:

“Dear Joanna, I wanted to let you know that I received your assignment and appreciate you taking the time to complete it. However, it was submitted after the due date, and as a result, it will be marked as late.” 

Of course, he also used ChatGPT. 

Test 2: Writing the Essay 

I was impressed with my essay. It drew parallels between Kafka’s Gregor Samsa and Ferris Bueller. The writing was well organized, but without a whiff of robotic precision. (You can read the full essay here.)

As you’ll see in my video, Mr. Diamond was less impressed. While he praised my piece for quickly getting to the thesis, the opening paragraph had a factual error. I cited Ferris, speaking at the beginning of the movie, saying he’s “not going to sit on [his] ass as the events that affect [him] unfold to determine the course of [his] life.” But that quote is from Ferris’s sidekick, Cameron, and it’s spoken at the film’s end, moments before the famous Ferrari fall.

Mr. Diamond spotted other errors. My paper said Ferris is reserved and rarely seen next to his peers. (Again, that’s Cameron.) It said “The Metamorphosis” was set in a suburban setting. (It’s in an unnamed city.)

I got three out of six on the assignment, which according to the AP rubric, is in the B- to C range. While that’s a passing grade, the work certainly didn’t meet my standards.

“The overall quality of your writing puts you in the lower 30th percentile of the class,” Mr. Diamond told me. “You may have the mind to get there, but it’s the skills that you need to work on.” He said my writing was “wooden” and “lacked verve and voice.” (I might give my real editors very, very many reasons to complain—these aren’t among them!)

When I asked him if he would have suspected this was written by AI, he said he didn’t think so. Even though he knows his students’ writing styles, he often plows through 60 or more essays. One like this—efficient, decently structured, gets to the point—might not set off any alarms.

Mr. Diamond couldn’t put an essay of mine through Google’s Classroom plagiarism checker because I wasn’t a registered student. When I put it through Grammarly, a writing tool that helps improve grammar and checks for plagiarism, only a few common phrases were flagged as suspicious. It really is an original text—just one written by a robot.

Google Classroom and Turnitin, a company that offers plagiarism detection tools to schools, use AI to compare a student’s work with their earlier assignments. Eric Wang, Turnitin’s vice president of AI, said that could help teachers identify new ChatGPT cheaters. He also told me that his company is able to detect AI-generated text based on cues that are imperceptible to humans, and that it will add an AI writing detection feature in 2023.

An OpenAI spokeswoman said the ChatGPT maker is also exploring and researching ways to make it easier to spot AI writing.

Test 3: Participating in Group Discussion 

The final test: See if ChatGPT would allow me to keep up in a group discussion without actually having done the reading. In this case, it was Denis Johnson’s short story “Car Crash While Hitchhiking,” from the collection “Jesus’ Son.”

While my fellow students immediately jumped into a conversation about the story’s characters, ChatGPT left me hanging:

“I don’t have any information about a book or movie called ‘Car Crash While Hitchhiking.’” 

When I searched for the book title, the bot gave me some minimally useful information, but got a big part wrong: the main character’s name. Finally, a human student gave me a clear synopsis.

Overall, Mr. Diamond gave me and ChatGPT a C. Even OpenAI’s Chief Executive

Sam Altman

says it’s not reliable for anything important right now and needs work on its “robustness and truthfulness.” But the accuracy and the data will get better fast, numerous AI experts told me. When that day comes, we’ll have the writing equivalent of a scientific calculator.

Still, it’s unlikely to replace the sometimes grueling, sometimes fun task of putting words on paper. “The winning combo is going to be this artful interaction of AI and humans,”

James Lester,

a computer-science professor at North Carolina State University who focuses on AI and education, told me.

Some of my new high-school friends told me they use AI tools such as Grammarly to improve their punctuation and word choice. And Mr. Diamond is already thinking about how to work ChatGPT into his curriculum. Plus, I used ChatGPT to help generate some ideas for lines in this column. 

There’s just one thing I keep wondering: Could ChatGPT have helped Ferris have an even more successful day off? (ChatGPT says yes.)

—Sign up here for Tech Things With Joanna Stern, a new weekly newsletter. Everything is now a tech thing. Columnist Joanna Stern is your guide, giving analysis and answering your questions about our always-connected world.

Write to Joanna Stern at joanna.stern@wsj.com