This library is a discriminator of AI-generated text. It is designed to detect whether a given text was generated by an AI model or written by a human. The library is based on a pre-trained model and can be used to identify AI-generated text in a variety of applications.
The motivation for this project derives of the necessity to succesfully identify AI generated text in educational environments and other situations where an individual is meant to the work by themselves, however masking the work they deliver by leveraging AI tools to do the heavylifting for them. Making it unsure whether the indivudual understood the task in the first place, and moreover, them being unsure about the quality of their solutions.
Note
This motivation text, written by hand, shows that it is still possible to
trick any LLM detector if the writing is done in a way that uses as many
uncommon terms as possible. Even when the detector is highly precise as
later seen.
As a starting point, we selected the datasets provided by the Kaggle Competition LLM - Detect AI Generated Text. When examing the data, we saw that they where only human generated texts. To address this problem, we consulted the best solutions of the competition and selected datasets with both AI and Human generated content. The following where selected:
- ArguGPT: A corpus of argumentative essays written by 7 different GPT models.
- DRCAT_V3: Another dataset for a AI generated text detection competition.
- Essay Forum Essays: A collection of human essays published on EssayForum, a forum to get help by native english speakers.
- Ivypanda Essays: A group of essays of several topics from Ivypanda, a web of academic support that has many essay examples.
- Extension for this competition: A gathering between different datasets of AI and human content used for this competition by a user.
This datasets where used for training the model. For post evaluation purposes we have used PERSUADE 2.0 and 1.0, a state-of-the-art corpus.
As the different datasets origins differed from each other, their format did as well. So to process them and normalize them we decided to create a global configuration class, in which to define their structure and any extra processing needed.
For all datasets we trimmed them down to two columns:
- text
- generated
Then, for the ivypanda dataset, which shows to have much larger text than other dataset in number of characters (roughly 1000x on average), we trimmed down significantly the lenght of its text to aroud 300 characters, and selected 50.000 random instances to keep an the training dataset balanced.
After merging all datasets, we end up with the following distribution:
| generated | count |
|---|---|
| i8 | u32 |
| ======== | ===== |
| 1 | 74790 |
| 0 | 75616 |
As is, we cannot use the text directly to train any sort of model. We need to make a transformation on the nature of the data. For this we decided to use the following:
- Tokenizer: To split text into chunk, representing words or partial words in the same fashion as model LMs and LLMs do during training and when generating text. For this task we decided to use bert base uncased, a light language model trained on vasts ammounts of texts and largely used to train larger models.
- Lemmatizer: To remove differences between words of the same nature and meaning, only differing in small gramatical features (dogs and dog)
- Stopwords removal: To remove words used continuosly by both LLMs and humans which do not add any value as they only serve the purpose of connecting words. Doing this allows us to reduce the dimensionality of the dataset.
Once all of this has been applied to the dataset, we are ready to vectorize the data and represent each instance of the dataset as a vector in a n-dimmensional space. To do this we apply the tf-idf algorithm to assign a value to each term that appears in our set of documents, and weight their importance based on the number of appearances it makes trhoughout the dataset.
All this results in a sparse matrix of roughly 20,000 terms and around 150,000 documents.
Once all the preprocessing is done, we split our dataset 80/20 for train and test, maintaining a balanced split of each class, then train a Logistic Regressor using scikit-learn, and observe its performance.
-----------------------------------
Classification Metrics:
Accuracy: 0.99035968353168
Precision: 0.9948993288590604
Recall: 0.9857693842266259
F1 Score: 0.9903133141826441
-----------------------------------
Confusion Matrix
[[14968 76]
[ 214 14824]]
-----------------------------------
Regression Metrics
Mean Absolute Error: 0.009640316468319926
Mean Squared Error: 0.009640316468319926
R2 Score: 0.9614387325926672
-----------------------------------
As we saw good results, we wanted to inspect how the model responded to increasing ammounts of data and see if the training was performing as expected or we saw either over or underfitting as data grew in size. For this we incrementally perform cross-validation training of a regressor over increments of 10% on the full dataset. This results in the following graph, which shows that the model was behaving well to increasing data, showing that with more, clean data its performace would appear to only grow without risking overfitting in training data.
Regarding the evaluation, we decided to use a different dataset, with text of different nature. As mentioned previously this is a state of the art corpus that contains, for a single prompt, text generated by both humans and AI, challenging any biased data selection that makes a hard distinction between styles of AI-generated and human-generated text.
Validation against a 50/50 split of AI-generated and human-generated text in a dataset containing roughly 75,000 instances results in the following results:
-----------------------------------
Classification Metrics:
Accuracy: 0.5565205230403468
Precision: 0.9975320829220138
Recall: 0.1130850795960048
F1 Score: 0.2031410981279055
-----------------------------------
Confusion Matrix
[[35752 10]
[31701 4042]]
-----------------------------------
Regression Metrics
Mean Absolute Error: 0.4434794769596532
Mean Squared Error: 0.4434794769596532
R2 Score: -0.7739180330857851
-----------------------------------
As shown by the results, our model seems to overlook many AI generated text, but it does show to be extremely precise with over 99% precision. Meaning that when text is flagged as AI-generated by WordJudge, it is most likely AI-generated. Although as shown in the Motivation section, we can still trick it if we know how a LLM writes text.
Upon completing the development of a robust and accurate model, we implemented a Text-Based User Interface (TUI) to facilitate user interaction. This TUI allows users to input text and receive predictions on whether the content is AI-generated or human-written.
Below are some captures of the application in action:
To run the application use
uv run wordjudgeTo contribue. First install uv.
Then run, in order:
uv venv
uv sync
uv tool install mypy
uv tool install pre-commit
uv tool install ruff