ChatGPT has taken the world by storm. The platform became a to-go tool for drafting paper outlines, writing “concise” conclusions, or putting together reading lists for master’s theses. After spending years building Bibcitation – a citation generator for creating automatic citations and bibliographies – we started wondering: What if AI could create better citations, too?
Zach Beekman, an IU junior studying Entrepreneurship, has never used ChatGPT for citations, instead opting for reference managers. According to Zach, while he has “seen his peers use ChatGPT for citation and even research,” he chose not to “because ChatGPT can create false evidence and cite false evidence”.
Our team decided to put ChatGPT to the test to see whether AI could compete with reference managers such as Bibcitation in generating citations and bibliographies. The verdict? Keep reading to see why we think that ChatGPT still has quite a few things to learn.
3 things ChatGPT got wrong.
Citing a conference paper in APA
Unfortunately, ChatGPT could not correctly cite a conference paper in APA – even when we gave the chatbot all the required information, such as the title or DOI. For our first test, we chose a 2015 proceedings article "Big data, big data quality problem" presented at the IEEE International Conference. With Bibcitation, all we had to do was type the conference paper in the search bar and choose a citation style to get an accurate reference.
With ChatGPT, things weren’t so breezy. Since OpenAI cannot fetch internet data or access databases, it could not retrieve any specific details from the DOI we provided. Instead, the chatbot returned a generic citation placeholder with the following information:
“AuthorLastName, AuthorInitials. (Year). Title of paper. In Proceedings of the conference title (pp. page numbers). Publisher. DOI”
However, a quick glance at the official APA guidelines revealed that conference proceedings published in a journal required a completely different citation format than that given by ChatGPT.
Citing a book in MLA
For our next test, we decided to compare citations for Stephen King’s “Carrie”. Once again, with Bibcitation’s automatic search feature, using a few keywords was enough to generate a citation for our chosen book edition and add it to the bibliography:
Now we put ChatGPT to the same task. To level the playing field, we asked OpenAI to generate a citation for every book edition available, so we’d have plenty to choose from. However, because ChatGPT did not have real-time access to internet databases or catalogs, it could only provide basic information on formatting citations for different book editions in MLA style:
“Below, I'm providing citations for a hypothetical first edition, second edition, and an eBook edition. Please note you would need to fill in the correct publisher and year of publication for each edition. First edition: King, Stephen. Carrie. 1st ed., Doubleday, 1974.”
So far, ChatGPT’s performance was not what we expected, so we decided to take the chatbot for a final test.
Generating a bibliography in Chicago style
For our last test, we wanted to see whether ChatGPT could act as a bibliography generator. Since our platform creates a citation list automatically, copying and exporting a bibliography with Bibcitation takes only a few seconds. How would OpenAI fare at this task?
Our prompt included a detailed description of each source, with a DOI or a web link whenever available. At first, the bibliography looked correct – until we looked closely at each reference.
First, we noticed our AI-generated bibliography was missing a title and was not alphabetized. OpenAI also did not italicize the necessary parts of the citations or include the hanging indent. But most importantly, the formatting of each citation was simply wrong, from the misplaced publication years to the BBC reference missing an article author. If this bibliography was submitted as a part of a real academic paper, it would be flagged as incorrect (or worse!). As OpenAI failed each of our tests, we have to caution IU students from using ChatGPT as a citation generator, or at least double-check each citation before submitting your bibliography.
2 things ChatGPT got right (sort of).
To give OpenAI a chance at redemption, we now asked the chatbot to provide basic information about certain aspects of the MLA and IEEE citation formats.
How to format a Works Cited in MLA?
Even though ChatGPT’s response did include instructions on how to title the Works Cited page, double-space the entries, and add a hanging indent, it completely missed some crucial pieces of information, such as the guidelines on italicization, margins, and page numbers.
How do I format an in-text citation in IEEE?
OpenAI correctly indicated that in IEEE style, in-text references should be numbered and placed in square brackets [ ]. It also gave helpful examples on how to format direct quotes, or what to do if the sources are cited in consecutive order: “Several recent studies [1]-[3] explain this”.
Bottom line
While ChatGPT can serve as a great aide during the writing process, we found that its algorithm is not advanced enough to generate citations on par with reference managers. With automatic citation formatting and add-ons for Chrome, Google Docs, and Microsoft Word, Bibcitation will generate accurate bibliographies in MLA, APA, and Chicago in one click. Or choose from 9 thousand available styles, including those specific to your university, to create your references!
Follow this link to learn more about Bibcitation: https://www.bibcitation.com/
