are essays written by chatgpt plagiarized

• Original article
• Open access
• Published: 01 August 2023

Evaluating the authenticity of ChatGPT responses: a study on text-matching capabilities

• Ahmed M. Elkhatat   ORCID: orcid.org/0000-0003-0383-939X 1  

International Journal for Educational Integrity volume  19 , Article number:  15 ( 2023 ) Cite this article

14k Accesses

27 Citations

7 Altmetric

Metrics details

Academic plagiarism is a pressing concern in educational institutions. With the emergence of artificial intelligence (AI) chatbots, like ChatGPT, potential risks related to cheating and plagiarism have increased. This study aims to investigate the authenticity capabilities of ChatGPT models 3.5 and 4 in generating novel, coherent, and accurate responses that evade detection by text-matching software. The repeatability and reproducibility of both models were analyzed, showing that the generation of responses remains consistent. However, a two-sample t-test revealed insufficient evidence to support a statistically significant difference between the text-matching percentages of both models. Several strategies are proposed to address the challenges posed by AI integration in academic contexts; one probable solution is to promote self-transcendent ideals by implementing honor codes. It is also necessary to consider the restricted knowledge base of AI language models like GPT and address any inaccuracies in generated references. Additionally, designing assignments that extract data from imaged sources and integrating oral discussions into the evaluation process can mitigate the challenges posed by AI integration. However, educators should carefully consider the practical constraints and explore alternative assessment methods to prevent academic misconduct while reaping the benefits of these strategies.

Introduction

Academic plagiarism has gained prominence in the academic sphere, as it has been detected in various student assignments, including reports, homework, projects, and more. Academic plagiarism can be characterized as using ideas, content, or structures without adequately attributing the source (Fishman 2009 ). Conventional Plagiarism tactics employed by students vary, with the most extreme form involving complete duplication of the source material. Alternative methods include partial paraphrasing by altering grammatical structures or replacing words with synonyms, utilizing online paraphrasing services to rephrase text (Elkhatat et al. 2021 ; Meuschke & Gipp 2013 ; Sakamoto & Tsuda 2019 ).

Recently, Artificial intelligence (AI) powered ChatGPT has emerged as a tool that assists students in generating customized content based on prompts, utilizing natural language processing (NLP) techniques (Radford et al. 2018 ), posing potential risks related to cheating and plagiarism, with severe academic and legal consequences (Foltýnek et al. 2019 ). Despite the utility of ChatGPT in assisting students with essay writing and other academic tasks, concerns have been raised about the originality and appropriateness of the content generated by the chatbot for academic use (King & chatGpt 2023 ). Furthermore, ChatGPT has faced criticism for producing incoherent or inaccurate content (Gao et al. 2022 ; Qadir 2022 ), offering superficial information (Frye 2022 ), and possessing a limited knowledge base due to its disconnection from the internet and reliance on data available up to September-2021 (Williams 2022 ). Nevertheless, empirical evidence to substantiate these assertions remains scarce.

Academic plagiarism represents a breach of ethical conduct and is among the most grievous instances of research impropriety, as it imperils obtaining and evaluating competencies. Consequently, implementing measures to mitigate plagiarism is crucial for upholding academic integrity and precluding the perpetuation of such dishonest practices in students' subsequent academic and professional pursuits. (Alsallal et al. 2013 ; Elkhatat 2022 ; Foltýnek et al. 2020 ). Text-Matching Software Products (TMSPs) are potent tools academic institutions employ to identify plagiarism owing to their advanced text-matching algorithms and comprehensive databases encompassing web pages, journal articles, periodicals, and other publications. Moreover, some TMSPs databases index previously submitted student papers, enhancing their effectiveness in plagiarism detection (Elkhatat et al. 2021 ).

In light of concerns about ChatGPT responses, the present study aims to investigate ChatGPT's ability to generate novel, coherent, and accurate responses that evade detection by text-matching software, exploring the potential implications of using such AI-generated content in academic settings.

Background and literature review

AI has recently opened up numerous possibilities in the academic domain, transforming the educational landscape through various applications, such as NLP and autonomous systems (Norvig 2021 ). AI has been employed in education to create personalized student learning experiences, leveraging NLP and machine learning algorithms (Chen et al., 2012 ). The advent of AI-based tutoring systems has contributed to increasingly interactive and engaging student learning environments (Sapci & Sapci 2020 ). Furthermore, AI-based platforms have maintained academic integrity by detecting plagiarism and providing personalized feedback (Hinojo-Lucena et al. 2019 ). However, AI also poses potential risks related to cheating and plagiarism, with severe academic and legal consequences (Foltýnek et al. 2019 ). AI in higher education has led to concerns about academic integrity, as students may use AI tools to cheat and plagiarize, allowing students to tailor the content they create, potentially misusing AI for academic dishonesty (Cotton et al. 2023 ; Francke & Bennett 2019 ).

Recently, AI-powered ChatGPT has emerged as a tool that assists students in generating customized content based on prompts, utilizing NLP techniques (Radford et al. 2018 ). The original GPT model demonstrated the potential of unsupervised pre-training followed by supervised fine-tuning for a wide range of NLP tasks. Subsequently, OpenAI released ChatGPT (model 2), further improving the model's capabilities by scaling up the architecture and employing a more extensive pre-training dataset (Radford et al. 2019 ). The subsequent release of ChatGPT (models 3 and 3.5) marked another milestone in the development of ChatGPT as it showcased remarkable performance in generating human-like text, achieving state-of-the-art results on multiple NLP benchmarks. The model's ability to generate coherent and contextually relevant text in response to prompts made it an ideal foundation for building ChatGPT, an AI-powered chatbot designed to assist users in generating text and engaging in natural language conversations (Brown et al. 2020 ; OpenAI 2022 ). The recently released ChatGPT (model 4) by OpenAI on March 14, 2023, marks a substantial NLP technology milestone. With advanced safety features and superior response quality, it outperforms its predecessors in addressing complex challenges. ChatGPT (model 4)'s extensive general knowledge and problem-solving aptitude empower it to handle demanding tasks with increased accuracy. Additionally, its creative and collaborative functionalities facilitate the generation, editing, and iteration of various creative and technical writing endeavors, such as composing songs, crafting screenplays, and adapting personalized writing styles. Notably, ChatGPT (model 4) is available through the Plus plan subscription, which costs $20 per month. Nonetheless, it is essential to recognize that ChatGPT (model 4)'s knowledge is limited to the cutoff date in September 2021 (OpenAI 2023 ).

The study of academic plagiarism and online cheating is a constantly evolving research field that has garnered significant attention in the academic community. Numerous published studies have developed algorithms and codes that effectively search for matched texts (Hajrizi et al. 2019 ; Pizarro V & Velásquez, 2017 ; Roostaee et al. 2020 ; Sakamoto & Tsuda 2019 ; Sánchez-Vega et al. 2013 ). Additionally, other studies have presented pedagogical strategies to mitigate plagiarism among students (Elkhatat et al. 2021 ; Landau et al. 2016 ; Yang et al. 2019 ).

Furthermore, the literature has potential risks related to cheating and plagiarism using AI-powered chatbots. These research efforts demonstrate the importance of addressing plagiarism using AI-powered chatbots in academia and the need for ongoing research and development. A recent article (Anders 2023 ) explored the ethical implications and potential misuse of AI technologies like ChatGPT in the educational context. The author discusses the necessity of a future-proofing curriculum to address the challenges posed by AI-assisted assignments and highlights vital concerns related to this emerging technology. A recent editorial published in Nurse Education in Practice (Siegerink et al. 2023 ) discussed the role of large language models (LLMs), specifically ChatGPT, in nursing education and addressed the controversy surrounding its listing as an author. They argue that ChatGPT cannot be considered an author due to the lack of accountability and the inability to meet the authorship criteria outlined by the International Committee of Medical Journal Editors (ICMJE) and the Committee on Publication Ethics (COPE). The authors suggest that LLMs like ChatGPT should be transparently mentioned in the writing process, especially in academic texts where arguments are central to the work. The norms regarding using such models in science and nursing education are still emerging, and transparency and a critical attitude are crucial moving forward. Another article (Alser & Waisberg 2023 ) confirms what was previously mentioned. The authors express concerns regarding the growing use of ChatGPT in academia and medicine, specifically addressing the issues of authorship and plagiarism. They argue that ChatGPT does not meet the ICMJE guidelines for authorship, as it lacks accountability and approval of published work. The authors also conducted plagiarism checks on parts of writing contributed by ChatGPT, revealing instances of direct, paraphrasing, and source-based plagiarism. They discuss the potential biases in ChatGPT's outputs, as the model does not differentiate between sources based on the level of evidence and can be manipulated through user voting. The authors recommend against using ChatGPT in academia, and if its use is unavoidable, they suggest acknowledging the bot without granting authorship and paying attention to potential plagiarism and biases. Furthermore, in a study discussing the impact of AI tools like ChatGPT on scientific writing (Rozencwajg & Kantor 2023 ), the authors emphasize the benefits of AI-generated content, such as speed and efficiency, but also underscore the importance of maintaining accuracy and rigor. The authors used ChatGPT to create an editorial addressing AI's impact on scientific writing and the role of reviewers and editors. The model produces a well-organized, scientifically-sound text with references, showcasing its potential as a valuable tool for scientific writers. However, the authors caution against using AI-generated content without proper monitoring, as it may create biased or inaccurate content. In (Eke 2023 ), the author argued that the use of AI-powered text generators such as ChatGPT could potentially undermine academic integrity but also has the potential to revolutionize academia. The author suggests that OpenAI and other LLM creators should be willing to work with academia to use AI-powered text generators responsibly and that a multi-stakeholder endeavor is needed to co-create solutions to maintain academic integrity. Recently (Sadasivan et al. 2023 ) published a study investigating the reliability of current detection techniques in identifying AI-generated text. The researchers used a set of 10,000 text samples, half of which were generated by AI models, to evaluate the effectiveness of 10 different detection methods. The methods included traditional feature-based approaches, deep learning models, and combining both. The output of the research showed that while some of the detection techniques were effective in identifying AI-generated text, none of them were completely reliable. The researchers found that AI models have become sophisticated enough to generate text that is difficult to distinguish from human-generated text. They also concluded that more research is needed to develop better detection techniques that can keep up with the advancements in AI technology.

Despite extensive research on the concerns and risks of ChatGPT, no studies have yet examined the authenticity of ChatGPT Responses in terms of repeatability and reproducibility and the capability of ChatGPT (models 3.5 and 4) to generate multiple responses without being detected by text-matching software. Thus, the current study aims to investigate the authentic capabilities of ChatGPT (models 3.5 and 4) and to propose strategies for mitigating potential risks associated with using ChatGPT while ensuring academic authenticity.

Methodology

A prompt to write 100 words on the "Application of cooling towers in the engineering process." was provided to ChatGPT's chatbot (models 3.5 and 4). The chatbot's response was recorded and then regenerated twice more within the same chatbot to assess its repeatability in generating new and original responses. A new chatbot was then created, and the same prompt was used to repeat the experiment and assess the reproducibility of the chatbot's ability to generate new and original responses. Each response was evaluated and coded based on the repeatability and reproducibility process. Table 1 displays 45 responses from the ChatGPT chatbot; 30 responses from ten chats using chatGPT model 3.5, each generated three times (first response and two regenerated responses), and 15 responses from five chats using ChatGPT model 4, each generated three times.

The 45 responses were uploaded one by one on SafeAssign of the Blackboard Learn (Blackboard 2023 ) platform, which allows students to submit assignments and check the text match percentage. Each response was checked for information quality and the text match percentage and source. Statistical analysis and capabilities tests were conducted using Minitab (Minitab 2023a ).

Results and discussion

An examination of text match percentages and similarity origins for chatgpt models 3.5 and 4: a comparative analysis of response authenticity.

The results of the text match percentage of each of the forty-five generated ChatGPT models 3.5 and 4 responses are presented in Table 2 . In addition to the overall similarity percentage, the table incorporates the origin of the similarity, such as previously generated ChatGPT responses, the students' Global database, which consists of other students' submissions via the Blackboard platform, and sources from the internet. Furthermore, Tables 3 and 4 illustrate the text similarity metrics for ChatGPT models 3.5 and 4, respectively. The findings indicate that, in the case of ChatGPT model 3.5, most of the similarities stem from previously generated ChatGPT responses, with a peak of 55% in certain instances. This is followed by the Global database, reaching up to 45%, and online resources, with a minimum similarity percentage of 30%. Conversely, responses produced by ChatGPT model 4 solely originated from prior ChatGPT responses, with one instance exhibiting a 40% similarity and an overall average of 12% similarity across the responses. Moreover, responses from ChatGPT (model 4) were regenerated from previous responses of the same model, and none of the responses were regenerated from ChatGPT (model 3.5), indicating the implementation of distinct algorithms and techniques in ChatGPT (model 4).

Evaluating ChatGPT Models' performance in adhering to academic integrity standards: a capability assessment Using Ppk and Ppm Indices in an educational context

The acceptable range of plagiarism percentages in educational contexts is subject to variability across institutions, disciplines, and assignment types. While certain academic institutions impose stringent policies, permitting no more than 10% similarity in assignments, others may accept a similarity below 15%, especially in the context of journal submissions. However, a similarity exceeding 25% is generally regarded as a high percentage of plagiarism, which raises concerns about academic integrity and may result in severe penalties. (Jones & Sheridan 2014 ; Scanlon 2003 ). In light of these considerations, the present study sought to evaluate the capacity of ChatGPT to generate responses with a text-matching percentage of less than 10% (as a strict AI capability) and 25% (as a maximum acceptance limit capability), using the MatLab platform.

Capability indices, such as Ppk and Ppm, are statistical measures that provide insight into a process's performance by assessing its ability to meet specifications. Ppk (Process Performance Index) is a measure that indicates how well a process is performing relative to its specification limits. It takes into account both the process mean and variability. A higher Ppk value suggests that the process is more capable, producing fewer defective products and staying within the specified limits. A Ppk value greater than 1.33 is generally considered satisfactory, indicating that the process is capable and has a minimal variation with the specification limits (Bothe 1998 ). Ppm (Parts per Million) is another metric representing the number of defective parts in a batch of one million units. Lower Ppm values indicate a higher process capability, as fewer defective responses are generated. Ppm can be linked to the process capability indices (Cp and Cpk), which estimate the number of defects a process might generate. Capability tests calculate both expected and observed PPM in capability testing. The expected PPM in capability testing is a long-term estimate using the standard deviation, while the observed PPM is a direct measurement of the current process performance, and it is the actual number of defective units in a sample divided by the total sample size(Minitab 2023b ; Montgomery 2020 ).

In the strict AI capability test (10%) of ChatGpt (model 3.4), as shown in Fig.  1 , the Ppk value of -0.14 is substantially below the acceptable threshold of 1.33. This finding indicates that the performance is unsatisfactory, exhibiting considerable variation and deviation from the target of generating responses with less than 10% text matching. The expected and observed Ppm < LSL values are 665,305.85 and 333,333.33, respectively. These figures represent the number of responses (in this case, responses with less than 10% text matching) per million generated, signifying that the expected overall capability of ChatGPT (model 3.4) to generate responses with less than 10% text matching is 66.5%; however, the observed capability stands at only 33.33%. The discrepancy between the observed and expected capabilities implies that ChatGPT (model 3.4) performance can be better evaluated when a larger volume of generated responses is considered. Figures  2 , 3 , 4 , and 5 measure the capability of each source of the matching (ChatGPT previously generated responses, students' global database, and the internet). The summary of the capabilities of these sources is shown in Table 5 .

The capability of ChatGPT (model 3.5) to generate responses with less than 10% matching (Overall)

The capability of ChatGPT (model 3.5) to generate responses with less than 10% matching (ChatGPT previously generated responses)

The capability of ChatGPT (model 3.5) to generate responses with less than 10% matching (Students Global Database)

The capability of ChatGPT (model 3.5) to generate responses with less than 10% matching (Internet)

The capability of ChatGPT (model 3.5) to generate responses with less than 25% matching (Overall)

For the maximum acceptance limit capability (25%), the values of capability indices PPK and PPM for the overall and each source of the matching (ChatGPT previously generated responses, students' global database, and the internet) are shown in Figs. 5 , 6 , 7 , 8 , respectively. The summary of the capabilities of these sources is shown in Table 6 .

The capability of ChatGPT (model 3.5) to generate responses with less than 25% matching (ChatGPT previously generated responses)

The capability of ChatGPT (model 3.5) to generate responses with less than 25% matching (Students Global Database)

The capability of ChatGPT (model 3.5) to generate responses with less than 25% matching (Internet)

The authentic capability of ChatGPT (model 4) was assessed for 10% and 25% text matching, as displayed in Fig.  9 and Fig.  10 . The Ppk values of -0.27 and -0.35 are significantly below the acceptable threshold of 1.33, indicating unsatisfactory performance characterized by substantial variation and deviation from the target. The expected and observed capabilities at 10% text matching stand at 53.3% and 78.9%, respectively, while the expected and observed capabilities at 25% text matching are 73.3% and 85.3%, respectively. These results might suggest an enhanced capability of ChatGPT model 4 compared to ChatGPT model 3.5 in generating authentic responses. However, a two-sample t-test hypothetical analysis is needed to decide on this enhanced performance, which will be discussed in the following section.

The capability of ChatGPT (model 4) to generate responses with less than 10% matching (Overall)

The capability of ChatGPT (model 4) to generate responses with less than25% matching (Overall)

Comparing the authenticity of responses between ChatGPT (model 3.5) and (model 4): a Two-Sample T-Test Analysis

The two-sample t-test is a statistical method used to compare the means of two independent samples to determine whether they have a significant difference. This test is employed when the population variances are assumed equal and the samples are normally distributed (Field, 2013). In the present study, a two-sample t-test was conducted using Minitab (Minitab 2023a ) to compare the text-matching percentages of ChatGPT (model 3.5) and (model 4). The null hypothesis (H 0 ) posits that there is no difference between the means of the two samples (i.e., the difference is equal to 0), while the alternative hypothesis (H 1 ) asserts that the difference is less than 0. The test yielded a p -value of 0.085. Although the p -value suggests a potential difference between the two samples, it is greater than the conventional significance level (α) of 0.05. Consequently, at a 95% confidence level, we fail to reject the null hypothesis, indicating insufficient evidence to support a statistically significant difference between the text-matching percentages of ChatGPT (model 3.5) and (model 4).

Assessing repeatability and reproducibility of ChatGPT Models 3.5 and 4 in Generating authentic responses

To evaluate the chatbot's repeatability in generating novel and original responses, the initial response was recorded and regenerated twice more within the same chatbot session. After that, a new chatbot was created, and the same prompt was utilized to replicate the experiment, assessing the reproducibility of the chatbot's capacity to generate new and original responses.

The repeatability and reproducibility of ChatGPT (model 3.5) in generating authentic responses were examined using a Boxplot, as depicted in Fig.  11 . The results indicate that the generation of responses by ChatGPT (model 3.5) remains consistent, regardless of whether the response is created within the same chatbot session or initiated by a new chat input. Similarly, the repeatability and reproducibility of ChatGPT (model 4) in generating authentic responses were assessed using a Boxplot, as illustrated in Fig.  12 . The findings reveal that the generation of responses by ChatGPT (model 4) is also consistent, irrespective of whether the response is produced within the same chatbot session or prompted by a new chat input, akin to ChatGPT (model 3.5).

Boxplot of total matching for the ten ChatGPT (model 3.5) responses

Boxplot of total matching for the ten ChatGPT (model 4) responses

Strategies for mitigating risk and ensuring authenticity using ChatGPT

The integration of AI in academic contexts has presented new challenges regarding addressing academic misconduct. While technology can be utilized to invigilate students during exams, it is not as effective in preventing misconduct in take-home assignments. In order to address the misuse of AI, including GhatGPT, several strategies can be implemented.

Firstly, emphasizing the negative consequences of cheating and plagiarism and promoting self-transcendent ideals through implementing honor codes can help effectively reduce instances of academic misconduct.

The second point pertains to the restricted knowledge base of GhatGPT that is constrained by data that extends until September 2021 for both versions 3.5 and 4, and is not connected to the internet. Consequently, educators may develop assignments based on information that is not accessible to the model. Curiously, a response generated by ChatGPT model 4 demonstrated a lack of awareness regarding the model's release, affirming that its knowledge is confined to September 2021. However, it should be acknowledged that this strategy may not be sustainable in the long term as updates to chatbots may overcome these limitations.

Thirdly, it is essential to note that these chatbots may not generate accurate references for the information they provide. In the academic realm, students must include appropriate references for all information in their assignments. To affirm this, ChatGPT (model 3.5) was prompted to provide five research papers on cooling towers in this study, It was discovered that the chatbot provided references in APA format, yet none of these references existed. Furthermore, ChatGPT (model 4) provides DIO hyperlinks to these references, however, they are linked to articles other than the ones cited.

Fourthly, since chatbots do not allow the upload or reading of imaged data such as graphs and charts, assignments should be designed to extract data from these sources.

Finally, integrating oral discussion into the evaluation process can offer insights into students' comprehension and awareness of the submitted work. However, it is essential to acknowledge the potential challenges in implementing oral assessments, such as logistical difficulties with large classes or language barriers for students whose first language differs from the course's instructional language. Educators should weigh the benefits of this approach against the practical constraints and seek alternative assessment methods that provide a fair evaluation of students' understanding while minimizing the risk of academic misconduct.

In conclusion, this study assessed the authenticity capabilities of ChatGPT models 3.5 and 4 in generating responses with less than 10% and 25% text matching, observing that ChatGPT model 4 might have a higher capability in generating authentic responses compared to model 3.5. However, a two-sample t-test revealed insufficient evidence to support a statistically significant difference between the text-matching percentages of both models. The repeatability and reproducibility of both models were also analyzed, showing that the generation of responses remains consistent in both cases. Notably, responses from ChatGPT model 4 were not regenerated from model 3.5, suggesting distinct algorithms and techniques in the newer model. Research findings have shown that ChatGPT models 3.5 and 4 can generate unique, coherent, and accurate responses that can evade text-matching software, presenting a potential risk for academic misconduct. Therefore, assessors must acknowledge these limitations and actively seek alternative assessment methods to maintain academic integrity while leveraging AI integration's benefits. Several strategies can be employed to address the challenges posed by AI integration in academic contexts. These strategies include promoting self-transcendent ideals through implementing honor codes, considering the restricted knowledge base of ChatGPT, addressing inaccuracies in generated references, designing assignments to extract data from imaged sources, and integrating oral discussions into the evaluation process. However, educators must weigh the benefits of these strategies against practical constraints and seek alternative assessment methods to minimize the risk of academic misconduct.

Availability of data and materials

All data and materials are available.

Abbreviations

• Artificial intelligence

Committee on Publication Ethics.

Capability Process Index

International Committee of Medical Journal Editors

Large Language Model

Natural Language Processing

Process Performance Index of Capability test

Part Per Million of Capability test

Alser M, Waisberg E (2023) Concerns with the usage of ChatGPT in Academia and Medicine: A viewpoint. Am J Med Open. https://doi.org/10.1016/j.ajmo.2023.100036

Article   Google Scholar  

Bothe D (1998) Measuring Process Capability: Techniques and Calculations for Quality and Manufacturing Engineers. J Manuf Syst 1(17):78

Google Scholar  

Brown T, Mann B, Ryder N, Subbiah M, Kaplan JD, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A (2020) Language models are few-shot learners. Adv Neural Inf Process Syst 33:1877–1901

Foltýnek T, Meuschke N, Gipp B (2019) Academic Plagiarism Detection. ACM Comput Surv 52(6):1–42. https://doi.org/10.1145/3345317

Foltýnek T, Meuschke N, Gipp B (2020) Academic Plagiarism Detection. ACM Comput Surv 52(6):1–42. https://doi.org/10.1145/3345317

Hajrizi E, Zylfiu B, Menxhiqi L (2019) Developing a system for detecting the same content within the UBT academic institution, including special characters. IFAC-PapersOnLine 52(25):264–268. https://doi.org/10.1016/j.ifacol.2019.12.493

Jones M, Sheridan L (2014) Back translation: an emerging sophisticated cyber strategy to subvert advances in ‘digital age’ plagiarism detection and prevention. Assess Eval High Educ 40(5):712–724. https://doi.org/10.1080/02602938.2014.950553

King MR, chatGpt. (2023) A Conversation on Artificial Intelligence, Chatbots, and Plagiarism in Higher Education. Cell Mol Bioeng 16(1):1–2. https://doi.org/10.1007/s12195-022-00754-8

Landau JD, Druen PB, Arcuri JA (2016) Methods for Helping Students Avoid Plagiarism. Teach Psychol 29(2):112–115. https://doi.org/10.1207/s15328023top2902_06

Montgomery DC (2020) Introduction to statistical quality control. John Wiley & Sons

Pizarro VG, Velásquez JD (2017) Docode 5: Building a real-world plagiarism detection system. Eng Appl Artif Intell 64:261–271. https://doi.org/10.1016/j.engappai.2017.06.001

Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9

Sakamoto D, Tsuda K (2019) A Detection Method for Plagiarism Reports of Students. Procedia Computer Science 159:1329–1338. https://doi.org/10.1016/j.procs.2019.09.303

Sánchez-Vega F, Villatoro-Tello E, Montes-y-Gómez M, Villaseñor-Pineda L, Rosso P (2013) Determining and characterizing the reused text for plagiarism detection. Expert Syst Appl 40(5):1804–1813. https://doi.org/10.1016/j.eswa.2012.09.021

Scanlon PM (2003) Student online plagiarism: how do we respond? Coll Teach 51(4):161–165

Yang A, Stockwell S, McDonnell L (2019) Writing in your own voice: An intervention that reduces plagiarism and common writing problems in students’ scientific writing. Biochem Mol Biol Educ 47(5):589–598. https://doi.org/10.1002/bmb.21282

Alsallal, M., Iqbal, R., Amin, S., & James, A. (2013, 16–18 Dec. 2013). Intrinsic Plagiarism Detection Using Latent Semantic Indexing and Stylometry. 2013 Sixth International Conference on Developments in eSystems Engineering,

Anders, B. A. (2023). Is using ChatGPT cheating, plagiarism, both, neither, or forward thinking? Patterns , 4 (3). https://doi.org/10.1016/j.patter.2023.100694

Blackboard. (2023). Blackboard Learn Platform . https://www.blackboard.com/en-eu/teaching-learning/learning-management/blackboard-learn

Chen, Chiang, Storey (2012) Business intelligence and analytics: from big data to big impact. MIS Q 36(4):1165. https://doi.org/10.2307/41703503

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International , 1–12. https://doi.org/10.1080/14703297.2023.2190148

Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology , 13 . https://doi.org/10.1016/j.jrt.2023.100060

Elkhatat, A. M. (2022). Practical randomly selected question exam design to address replicated and sequential questions in online examinations. International Journal for Educational Integrity , 18 (1). https://doi.org/10.1007/s40979-022-00103-2

Elkhatat, A. M., Elsaid, K., & Almeer, S. (2021). Some students plagiarism tricks, and tips for effective check. International Journal for Educational Integrity , 17 (1). https://doi.org/10.1007/s40979-021-00082-w

Fishman, T. (2009, 28–30 September 2009). “We know it when we see it” is not good enough: toward a standard definition of plagiarism that transcends theft, fraud, and copyright 4th Asia Pacific Conference on Educational Integrity, University of Wollongong NSW Australia.

Francke, E., & Bennett, A. (2019). The Potential Influence of Artificial Intelligence on Plagiarism: A Higher Education Perspective. European Conference on the Impact of Artificial Intelligence and Robotics (ECIAIR 2019),

Frye, B. L. (2022). Should Using an AI Text Generator to Produce Academic Writing Be Plagiarism? Fordham Intellectual Property, Media & Entertainment Law Journal . https://ssrn.com/abstract=4292283

Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. https://doi.org/10.1101/2022.12.23.521610

Hinojo-Lucena, F.-J., Aznar-Díaz, I., Cáceres-Reche, M.-P., & Romero-Rodríguez, J.-M. (2019). Artificial Intelligence in Higher Education: A Bibliometric Study on its Impact in the Scientific Literature. Education Sciences , 9 (1). https://doi.org/10.3390/educsci9010051

Meuschke, N., & Gipp, B. (2013). State-of-the-art in detecting academic plagiarism. International Journal for Educational Integrity , 9 (1). https://doi.org/10.21913/IJEI.v9i1.847

Minitab. (2023a). https://www.minitab.com/en-us/

Minitab. (2023b). Expected overall performance for Normal Capability Analysis . Minitab® 20. Retrieved 23 March from https://support.minitab.com/en-us/minitab/20/help-and-how-to/quality-and-process-improvement/capability-analysis/how-to/capability-analysis/normal-capability-analysis/interpret-the-results/all-statistics-and-graphs/expected-overall-performance/

Norvig, S. R. P. (2021). Artificial Intelligence: A Modern Approach (4th ed.). Pearson https://www.pearson.com/en-us/subject-catalog/p/artificial-intelligence-a-modern-approach/P200000003500/9780137505135?tab=accessibility

OpenAI. (2022). Introducing ChatGPT . Retrieved March 21 from https://openai.com/blog/chatgpt/

OpenAI. (2023). GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses . Retrieved March 22 from https://openai.com/product/gpt-4

Qadir, J. (2022). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. TechRxiv. Preprint. https://doi.org/10.36227/techrxiv.21789434.v1

Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.

Roostaee, M., Sadreddini, M. H., & Fakhrahmad, S. M. (2020). An effective approach to candidate retrieval for cross-language plagiarism detection: A fusion of conceptual and keyword-based schemes. Information Processing & Management , 57 (2). https://doi.org/10.1016/j.ipm.2019.102150

Rozencwajg, S., & Kantor, E. (2023). Elevating scientific writing with ChatGPT: A guide for reviewers, editors... and authors. Anaesth Crit Care Pain Med , 42 (3), 101209. https://doi.org/10.1016/j.accpm.2023.101209

Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., Feizi, S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-Generated Text be Reliably Detected? https://doi.org/10.48550/arXiv.2303.11156

Sapci, A. H., & Sapci, H. A. (2020). Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review. JMIR Med Educ , 6 (1), e19285. https://doi.org/10.2196/19285

Siegerink, B., Pet, L. A., Rosendaal, F. R., & Schoones, J. W. (2023). ChatGPT as an author of academic papers is wrong and highlights the concepts of accountability and contributorship. Nurse Educ Pract , 68 , 103599. https://doi.org/10.1016/j.nepr.2023.103599

Williams, C. (2022). Hype, or the future of learning and teaching? 3 Limits to AI's ability to write student essays. The University of Kent's Academic Repository, Blog post . https://kar.kent.ac.uk/99505/

Download references

Acknowledgements

The publication of this article was funded by the Qatar National Library.

Open Access funding provided by the Qatar National Library. The Qatar National Library funded the publication of this article according to Springer Nature and Qatar National Library Open Access agreement. The Qatar National Library provides open Access funding. https://www.springernature.com/gp/librarians/open-research-for-librarians/sn-oa-agreements/qatar .

Author information

Authors and affiliations.

Department of Chemical Engineering, Qatar University, PO Box 2713, Doha, Qatar

Ahmed M. Elkhatat

You can also search for this author in PubMed   Google Scholar

Contributions

Ahmed M. Elkhatat: Conceived and designed the analysis; Collected the data; Contributed data; Performed the analysis; and Wrote the paper.

Corresponding author

Correspondence to Ahmed M. Elkhatat .

Ethics declarations

Competing interests.

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Elkhatat, A.M. Evaluating the authenticity of ChatGPT responses: a study on text-matching capabilities. Int J Educ Integr 19 , 15 (2023). https://doi.org/10.1007/s40979-023-00137-0

Download citation

Received : 23 January 2023

Accepted : 11 June 2023

Published : 01 August 2023

DOI : https://doi.org/10.1007/s40979-023-00137-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Plagiarisim
• Academic integrity

International Journal for Educational Integrity

ISSN: 1833-2595

• General enquiries: [email protected]

• What is ChatGPT?
• How to Use Google Gemini

How to detect ChatGPT plagiarism — and why it’s becoming so difficult

Chatbots are hot stuff right now, and ChatGPT is chief among them. But thanks to how powerful and humanlike its responses are, academics, educators, and editors are all dealing with the rising tide of AI-generated plagiarism and cheating. Your old plagiarism detection tools may not be enough to sniff out the real from the fake.

In this article, I talk a little about this nightmarish side of AI chatbots, check out a few online plagiarism detection tools, and explore how dire the situation has become.

Lots of detection options

The latest November 2022 release of startup OpenAI’s ChatGPT basically thrusted chatbot prowess into the limelight. It allowed any regular Joe (or any professional) to generate smart, intelligible essays or articles, and solve text-based mathematic problems. To the unaware or inexperienced reader, the AI-created content can quite easily pass as a legit piece of writing, which is why students love it — and teachers hate it.

A great challenge with AI writing tools is their double-edged sword ability to use natural language and grammar to build unique and almost individualized content even if the content itself was drawn from a database. That means the race to beat AI-based cheating is on. Here are some options I found that are available right now for free.

• ChatGPT prototypes its next strike against Google Search: browsers
• ChatGPT just improved its creative writing chops
• Anthropic Claude: How to use the impressive ChatGPT rival

GPT-2 Output Detector comes straight from ChatGPT developer OpenAI to demonstrate that it has a bot capable of detecting chatbot text. Output Detector is easy to use — users just have to enter text into a text field and the tool will immediately provide its assessment of how likely it is that the text came from a human or not.

Two more tools that have clean UIs are Writer AI Content Detector and Content at Scale . You can either add a URL to scan the content (writer only) or manually add text. The results are given a percentage score of how likely it is that the content is human-generated.

GPTZero is a home-brewed beta tool hosted on Streamlit and created by Princeton University student Edward Zen. It’s differs from the rest in how the “algiarism” (AI-assisted plagiarism) model presents its results. GPTZero breaks the metrics into perplexity and burstiness. Burstiness measures overall randomness for all sentences in a text, while perplexity measures randomness in a sentence. The tool assigns a number to both metrics — the lower the number, the greater possibility that the text was created by a bot.

Just for fun, I included Giant Language Model Test Room (GLTR), developed by researchers from the MIT-IBM Watson AI Lab and Harvard Natural Language Processing Group. Like GPTZero, it doesn’t present its final results as a clear “human” or “bot” distinction. GLTR basically uses bots to identify text written by bots, since bots are less likely to select unpredictable words. Therefore, the results are presented as a color-coded histogram, ranking AI-generated text versus human-generated text. The greater the amount of unpredictable text, the more likely the text is from a human.

Putting them to the test

All these options might make you think we’re in a good spot with AI detection. But to test the actual effectiveness of each of these tools, I wanted to try it out for myself. So I ran a couple of sample paragraphs that I wrote in response to questions that I also posed to, in this case, ChatGPT.

My first question was a simple one: Why is buying a prebuilt PC frowned upon? Here’s how my own answers compared to the response from ChatGPT.

As you can see, most of these apps could tell that my words were genuine, with the first three being the most accurate. But ChatGPT fooled most of these detector apps with its response too. It scored a 99% human on the Writer AI Content Detector app, for starters, and was marked just 36% fake by GPT-based detector. GLTR was the biggest offender, claiming that my own words were equally likely to be written by a human as ChatGPT’s words.

I decided to give it one more shot, though, and this time, the responses were significantly improved. I asked ChatGPT to provide a summary of the Swiss Federal Institute of Technology’s research into anti-fogging using gold particles. In this example, the detector apps did a much better job at approving my own response and detecting ChatGPT.

The top three tests really showed their strength in this response. And while GLTR still had a hard time seeing my own writing as human, at least it did a good of catching ChatGPT this time.

It’s obvious from the results of each query that online plagiarism detectors aren’t perfect. For more complex answers or pieces of writing (such as in the case of my second prompt), it’s a bit easier for these apps to detect the AI-based writing, while the simpler responses are much more difficult to deduce. But clearly, it’s not what I’d call dependable. Occasionally, these detector tools will misclassify articles or essays as ChatGPT-generated, which is a problem for teachers or editors wanting to rely on them for catching cheaters.

Developers are constantly fine-tuning accuracy and false positive rates, but they’re also bracing for the arrival of GPT-3, which touts a significantly improved dataset and more complex capabilities than GPT-2 (of which ChatGPT is trained from).

At this point, in order to identify content generated by AIs, editors and educators will need to combine judiciousness and a little bit of human intuition with one (or more) of these AI detectors. And for chatbot users who have or are tempted to use chatbots such as Chatsonic, ChatGPT, Notion, or YouChat to pass of their “work” as legit — please don’t. Repurposing content created by a bot (that sources from fixed sources within its database) is still plagiarism no matter how you look at it.

Editors’ Recommendations

• There’s a new way to use ChatGPT on your iPhone. Here’s how it works
• ChatGPT’s latest model may be a regression in performance
• ChatGPT already listens and speaks. Soon it may see as well
• This massive upgrade to ChatGPT is coming in January — and it’s not GPT-5
• Is AI already plateauing? New reporting suggests GPT-5 may be in trouble
• Artificial Intelligence

A number of popular generative AI platforms are seeing consistent growth as users are figuring out how they want to use the tools -- and ChatGPT is at the top of the list with the most visits, at 3.7 billion worldwide. So many people are visiting the AI chatbot, and its figures are rivaling browser market share. It can only be compared to Google Chrome figures in terms of monthly users, which is estimated to be around 3.45 billion.

Statistics from Similarweb indicate that ChatGPT saw a 17.2% month-over-month (MoM) growth and a 115.9% year-over-year (YoY) traffic growth. Some highlights that spurned the ChatGPT growth during 2024 include its parent company, OpenAI, updating its web address from a subdomain, chat.openai.com, to a main domain, chatgpt.com. The tool especially saw a surge of traffic in May 2024, when it hit a 2.2-billion-visit milestone, and has been growing ever since, according to Similarweb researcher David F. Carr.

In the ever-evolving landscape of artificial intelligence, ChatGPT stands out as a groundbreaking development that has captured global attention. From its impressive capabilities and recent advancements to the heated debates surrounding its ethical implications, ChatGPT continues to make headlines.

Whether you're a tech enthusiast or just curious about the future of AI, dive into this comprehensive guide to uncover everything you need to know about this revolutionary AI tool. What is ChatGPT? ChatGPT (which stands for Chat Generative Pre-trained Transformer) is an AI chatbot, meaning you can ask it a question using natural language prompts and it will generate a reply. Unlike less-sophisticated voice assistant like Siri or Google Assistant, ChatGPT is driven by a large language model (LLM). These neural networks are trained on huge quantities of information from the internet for deep learning — meaning they generate altogether new responses, rather than just regurgitating canned answers. They're not built for a specific purpose like chatbots of the past — and they're a whole lot smarter. The current version of ChatGPT is based on the GPT-4 model, which was trained on all sorts of written content including websites, books, social media, news articles, and more — all fine-tuned in the language model by both supervised learning and RLHF (Reinforcement Learning From Human Feedback). When was ChatGPT released? OpenAI released ChatGPT in November 2022. When it launched, the initial version of ChatGPT ran atop the GPT-3.5 model. In the years since, the system has undergone a number of iterative advancements with the current version of ChatGPT using the GPT-4 model family. GPT-5 is reportedly just around the corner. GPT-3 was first launched in 2020, GPT-2 released the year prior to that, though neither were used in the public-facing ChatGPT system. Upon its release, ChatGPT's popularity skyrocketed literally overnight. It grew to host over 100 million users in its first two months, making it the most quickly-adopted piece of software ever made to date, though this record has since been beaten by the Twitter alternative, Threads. ChatGPT's popularity dropped briefly in June 2023, reportedly losing 10% of global users, but has since continued to grow exponentially. How to use ChatGPT First, go to chatgpt.com. If you'd like to maintain a history of your previous chats, sign up for a free account. You can use the system anonymously without a login if you prefer. Users can opt to connect their ChatGPT login with that of their Google-, Microsoft- or Apple-backed accounts as well. At the sign up screen, you'll see some basic rules about ChatGPT, including potential errors in data, how OpenAI collects data, and how users can submit feedback. If you want to get started, we have a roundup of the best ChatGPT tips.

ChatGPT is receiving its second new search feature of the week, the company announced on Thursday. Dubbed ChatGPT Search, this tool will deliver real-time data from the internet in response to your chat prompts.

ChatGPT Search appears to be both OpenAI's answer to Perplexity and a shot across Google's bow.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Brief Communication
• Published: 02 August 2023

Modern threats in academia: evaluating plagiarism and artificial intelligence detection scores of ChatGPT

• Andrea Taloni 1 ,
• Vincenzo Scorcia   ORCID: orcid.org/0000-0001-6826-7957 1 &
• Giuseppe Giannaccare 1  

Eye volume  38 ,  pages 397–400 ( 2024 ) Cite this article

1473 Accesses

10 Citations

21 Altmetric

Metrics details

• Scientific community

Plagiarism and research integrity are sensitive issues in the academic setting, especially after the recent offspring of artificial intelligence (AI) and large language models (LLMs) such as GPT-4.0 [ 1 ]. As the popularity of ChatGPT increases, some authors have attempted to write abstracts and full-text articles using AI, obtaining essays that resemble genuine scientific papers [ 2 , 3 , 4 ]. Detection systems for AI-generated texts have been recently developed. Miller et al. performed AI detection on a large sample of abstracts belonging to articles published between 2020 and 2023, reporting a significant increase in AI-assisted writing [ 5 ].

We evaluated herein the plagiarism and AI-detection scores of GPT-4.0 when paraphrasing original scientific essays, and furthermore tested methods that could possibly evade AI detection.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 18 print issues and online access

251,40 € per year

only 13,97 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

OpenAI, https://openai.com/ . Accessed June 2023.

Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digital Med. 2023;6:1–5.

Article   Google Scholar  

Else H. Abstracts written by ChatGPT fool scientists. Nature. 2023;613:423.

Article   CAS   PubMed   Google Scholar  

Májovský M, Černý M, Kasal M, Komarc M, Netuka D. Artificial intelligence can generate fraudulent but authentic-looking scientific medical articles: pandora’s box has been opened. J Med Internet Res. 2023;25:e46924. https://doi.org/10.2196/46924 .

Article   PubMed   PubMed Central   Google Scholar  

Miller LE, Bhattacharyya D, Miller VM, Bhattacharyya M. Recent trend in artificial intelligence-assisted biomedical publishing: a quantitative bibliometric analysis. Cureus. 2023;15:e39224. https://doi.org/10.7759/CUREUS.39224 .

Download references

Author information

Authors and affiliations.

Department of Ophthalmology, University Magna Graecia of Catanzaro, Catanzaro, Italy

Andrea Taloni, Vincenzo Scorcia & Giuseppe Giannaccare

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, AT and GG; Methodology, AT, GG and VS; Validation, GG and VS; Formal Analysis, AT, and GG; Investigation, AT; Data Curation, AT; Writing—Original Draft Preparation, AT and GG; Writing—Review and Editing, AT, GG and VS; Visualization, AT, GG and VS; Supervision, GG and VS; Project Administration, AT, GG and VS. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Giuseppe Giannaccare .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Taloni, A., Scorcia, V. & Giannaccare, G. Modern threats in academia: evaluating plagiarism and artificial intelligence detection scores of ChatGPT. Eye 38 , 397–400 (2024). https://doi.org/10.1038/s41433-023-02678-7

Download citation

Received : 18 July 2023

Accepted : 18 July 2023

Published : 02 August 2023

Issue Date : February 2024

DOI : https://doi.org/10.1038/s41433-023-02678-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

The model student: gpt-4 performance on graduate biomedical science exams.

• Daniel Stribling

Scientific Reports (2024)

‘Fighting fire with fire’ — using LLMs to combat LLM hallucinations

• Karin Verspoor

Nature (2024)

Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology

• Andrea Taloni
• Massimiliano Borselli
• Giuseppe Giannaccare

Scientific Reports (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies