Is ChatGPT better than a professional writer? Ralph Grayden puts it to the test to find out.
There’s been a lot of talk about AI replacing journalists, editors and writers. But how accurate is it?
I decided to put ChatGPT to the test to determine whether it could defeat a professional writer in a one-on-one battle of words.
Introducing the professional writer: me
That’s right. I’ll be the one who takes on artificial intelligence for this challenge.
I’m not claiming to be a literary giant like Steven King, Margaret Atwood, or Michael Ondaatje. But I have been a professional writer for over two decades, and I’ve written in almost every style – from advertising copy to journalism and from technical writing to novel writing.
I’d like to think over that time I’ve learned something about my craft.
On the other hand, our AI challenger, ChatGPT, has comparatively little writing experience. It’s only been around since November 2022. But it does know everything: well, everything up to September 2021 (which is when its knowledge cuts out – at least without an add-on like this one).
That said, with the right prepping, I’ve learned that you can get ChatGPT to write well – which I think this experiment will show. So I’m not just going to ask it to write something and leave it at that. Instead, I’ll use some of the same commands and prompts I’ve been teaching in our Write Better, Faster webinar.
I’ll also be working with ChatGPT to edit its own writing, using it to refine its own words and get a good result.
Introducing the judge: Google Bard
Judging the quality of writing can be a difficult thing. To some extent, it’s subjective. I thought the best thing to do would be to take human emotion out of the assessment process and hand that job over to an AI judge, Google Bard.
As I’ve written, Google Bard isn’t quite as accomplished at writing as ChatGPT. Even it admits as much. But it’s impartial and dispassionate, and it should be able to provide at least some rational analysis of whose work is best.
Now, let’s kick off the contest…
Round 1. Using ChatGPT to write an article
The first round is article writing, where I get ChatGPT to write part of a simple factual article and then compare it with my non-AI work.
This is the kind of writing I’ve been experimenting with most when it comes to AI. I’ve found ChatGPT can excel at it, so long as you give it the right prompts and you’re prepared to put effort into the editing process.
For the basis of the piece, I used an excerpt from an explainer article I’d written about ‘negative equity’.
To get ChatGPT to compete, I asked it to explain negative equity using simple language that someone with a year six reading level could understand. I also asked it to include an example of negative equity, and I gave it a few editing prompts to tidy things up and remove the jargon it used.
Here are its version and mine, one after the next. (I won’t tell you which is which until the end of the article).
Example 1
|
What is negative equity? Equity is essentially the percentage of a home’s value that you own. Negative equity happens when you owe more on your home loan than your property is worth. Say you buy a home for $500,000 and take out a mortgage for $475,000. If the value of your home falls by 10%, your property would be worth $450,000. In this instance, if you still owed $475,000, your equity would be -$25,000, or roughly -5.6%. The opposite of negative equity is positive equity. This happens when you owe less than the value of the home. For instance, if you owed just $300,000 on the same home worth $450,000, your equity would be $150,000, or 33.3%. |
Example 2
|
What is negative equity? Negative equity is when the value of something you own, like a home, is lower than the amount you still owe. It happens when the value of the property drops, and you end up owing more on your loan than the property is worth. For example, let’s say you bought a house for $500,000 and took out a mortgage for $475,000. If the value of the house goes down to $400,000, you would be in negative equity. Even though you still owe $475,000 on the mortgage, the house is worth less than what you owe. Negative equity can be a problem because it makes it hard to refinance your property or sell it without losing money. It’s important to keep an eye on the value of your property and how much you still owe to avoid getting into negative equity. |
Is ChatGPT better than a human at writing articles? Google Bard’s verdict.
I asked ChatGPT to analyse both my and ChatGPT’s articles based on clarity, style and technique. Here’s what it said.
Test 2. Using ChatGPT to write poetry
There’s been a lot of debate about whether ChatGPT is creative; most people seem to agree that it’s not. Ask it to write poetry, and it will usually give you hackneyed phrases and twee rhymes that belong more in a year five English lesson than an anthology.
But this isn’t just any kind of poetry we’re going to be competing over; it’s a haiku.
That’s because my experience so far has revealed that ChatGPT is a master haiku writer. Over the past few months, it has given me some fantastic verses that any poet would be proud of.
Before I put it to work, I asked ChatGPT why it was so good at haikus compared with other forms of poetry.
“Since haikus have a specific structure and style, it may be easier for an AI like me to generate coherent and aesthetically pleasing haikus compared to other forms of poetry that require more complex and nuanced expression. The limited syllable count and focused subject matter of haikus allow for a more concise and structured approach, which can result in a more successful outcome.”
OK, fair enough. (And note, ChatGPT’s answer should also provide an important insight into how to use it for best effect).
To put ChatGPT to the test, I asked it to write a haiku about the scrub turkey that keeps coming into my backyard to eat the bugs in our compost. I told it that the turkey keeps destroying the plants we’ve spent hundreds of dollars on, but there’s nothing we can do about it. We spray it with water, but it won’t stop returning.
Example 1
| Unwanted guest comes, Scrub turkey feasts on compost, Sprayed, yet undeterred. |
Example 2
| Turkey reappears Compost strewn, plants destroyed, more Dollars down the drain. |
Is ChatGPT better than a human at writing poetry? Google Bard’s verdict.
This time, I asked Google Bard to rate both my and ChatGPT’s efforts based on their simplicity and technique, as well as the vividness of the imagery. I also asked it to assess a fourth important category in poetry writing: whether it contained a certain “je ne sais quoi”.
Here’s what it said.
Test 3. Using ChatGPT to write a novel
Moving along to the third and final test, this one is all about novel writing.
Back in 2012, I published my first novel. Writing it was a long and arduous process, and the rewards didn’t match my effort. (You can read about my attempt to get ChatGPT to write a second novel here.)
I’m using the very first part of the novel as the final basis of my contest with ChatGPT.
Taking the same themes introduced at the start of my book, I asked ChatGPT to write an opening about a disaffected 30-year-old male lawyer who has all the trappings of a successful life but is deeply unsatisfied. I also told it that he had an apartment in Bondi (a fashionable Sydney beachside suburb) and that his wife was beautiful and her name was Sarah. As always, I didn’t accept what it gave me the first time but got it to edit its work based on my prompts.
See if you can tell which one is human and which is AI-generated.
Example 1
|
I stood by my window, fixating on the labyrinth of buildings and people below. The fading sunlight called the city workers home, but I had plenty of work left. I was thirty, a lawyer. The pinnacle of success, they said. A career to envy, a life to aspire to. But as I sat in my office, surrounded by mahogany and leather, I couldn’t help but feel a gnawing dissatisfaction. The weight of expectations pressed upon me, suffocating my spirit. I should have felt happy. I had a mortgage on an apartment in Bondi and the love of a beautiful woman, Sarah. But beneath the surface of this seemingly perfect life, something tugged at my core. |
Example 2
|
I would like to think that at the age of thirty, I knew a thing or two about myself. I knew, for instance, I was a lawyer – a senior associate – at Australia’s most prestigious firm. I knew I had a career path, a European car and a very healthy salary. I knew I had a mortgage on a two-bedroom apartment in Bondi, Sydney’s most fashionable and most expensive beachside suburb. And I knew I had the love of Sarah, a gorgeous woman who, despite having the choice of any man in the world, had chosen to marry me. What’s more, I knew that the future would only get better. A few more years of solid work almost certainly guaranteed a partnership with the firm. And that, in turn, guaranteed a very comfortable life indeed. It meant the chance to settle down to a bigger house in a more traditional suburb. It meant an annual overseas holiday. It meant an in-ground swimming pool and dinner party conversation about real estate and rugby and school fees. It meant ski lodge membership, a permanent seat at the Sydney Cricket Ground, weekend wine courses and a Southern Highlands hobby farm. In short, it meant the chance to give my as-yet unconceived children exactly the same upbringing my parents had given me. And that knowledge was depressing. |
Is ChatGTP better than a human at writing a novel? Google Bard’s verdict
Again, I asked Google Bard to rate both mine and ChatGPT’s novel writing efforts based on four factors: technique, narrative and whether or not the writing was captivating. I also asked for an overall score. Here’s what it gave me.
So who is better: a professional writer or ChatGPT?
Well, that’s that for the contest. Three different styles of writing that put ChatGPT to the test and pitted it against a human counterpart. So who was the better writer: me or artificial intelligence?
Now, drum roll, please….
The good news is that, according to Google Bard, I won. The bad news is that it wasn’t by much: I took out the article writing and the novel writing; ChatGPT penned the winning haiku. Yes, artificial intelligence delivered a poem rated four out of a possible five for ‘je ne sais quoi’, according to Google Bard! (I warned you it was good.)
Still, this experiment tells us several things about ChatGPT’s strengths and how we should expect to use it.
1. ChatGPT can write well
Don’t believe the people who tell you ChatGPT can’t write. It can. But, also, don’t believe those people who claim that it’s already better than us.
To get ChatGPT to write well, you must put in the time and effort to guide it there. For each of these samples, I didn’t just ask it to write something and leave it at that. I walked it through the same writing process professional writers use. That’s what got these results – understanding and applying the writing process to AI.
2. ChatGPT’s novel writing leaves a lot to be desired
ChatGPT’s first efforts at writing a novel were poor, and it took a lot of prompting and editing to get anything halfway decent from it. Perhaps because this is the most free-form and personal style of writing?
If you’re going to use ChatGPT to write a novel, well, good luck. Be prepared to work hard.
3. ChatGPT is at its strongest when it has strict parameters.
Look back at the answer ChatGPT itself gave me when I asked it why it was so good at writing haikus. It told me that it excelled when creating something with a specific structure and style rather than trying to complete more nuanced work.
4. But ChatGPT can still produce a certain ‘je ne sais quoi’.
ChatGPT thrashed me in the haiku writing task. Sure, it wasn’t my finest work, but ChatGPT’s writing was great. Play around with it yourself. Give it strict parameters and see what it’s capable of.
5. ChatGPT is not going to replace highly skilled writers just yet.
Given the answer above, the more structured the writing task, the better ChatGPT is at producing it.
6. Every writer should be experimenting with ChatGPT.
Writers are naturally curious, and how could any writer not be curious about how ChatGPT could help them in their work?
Whether you write professionally or just as part of your daily job, you can’t (and shouldn’t) ignore it. Work out how to use it to your advantage to produce better work more quickly.
Want more?
If you’d like to find out how you can incorporate ChatGPT into your organisation’s writing, get in touch.
