Machine-Generated Text Detection
CMU 11-711 Course: Advanced NLP
The Challenge—
With the rapid advancement of large language models (LLMs), distinguishing between human-written and machine-generated text has become increasingly critical. From combating misinformation to ensuring academic integrity, the ability to identify machine-generated text is essential for maintaining trust and accountability in various domains such as education, journalism, and social media. Traditional methods often struggle with diverse text generators and multilingual data, highlighting the need for robust, scalable solutions.
Some methods for detecting machine-generated text (MGT)
Approach—
This project addresses these challenges by developing the Multi-Scale Feature Fusion Network (MSFFN), a framework designed to detect machine-generated text with high accuracy and efficiency. Leveraging a unique combination of CNN-based architecture and feature fusion techniques, the MSFFN outperforms traditional transformer models despite being significantly smaller and computationally lighter.
Key highlights of this approach include:
Multi-Scale Feature Fusion: By combining low, medium, and high-level text features extracted at different layers of the network, the MSFFN achieves nuanced classification.
Efficiency and Scalability: The core MSFFN model has just 1.1 million parameters, making it 100x smaller than transformer-based baselines while delivering better performance.
Ensemble Learning: The MSFFN-Ensemble enhances accuracy further by training specialized models on domain- and generator-specific data, combining their outputs for robust classification.
Dataset—
We used the M4 dataset (Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection).
Domains: Arxiv, Wikipedia, Reddit, wikiHow, PeerRead, Baike, True and Fake News
Generators: Humans, ChatGPT, Davinci, Cohere, Dolly-v2, Bloomz
Languages: English, Chinese, Russian, Urdu, Indonesian, Arabic, Bulgarian
Experiments—
We use pre-trained transformers RoBERTa for monolingual (English) and XLM-R for multilingual (all languages).
Train detectors on all-except-some generators and test the unseen generators
Train detectors on all-except-some domains and test the unseen domains
Train detectors on multiple languages and evaluate them on unseen languages
Method—
The model is an extremely small CNN with 2 million parameters
The multiscale features extracted from each layer of the CNN are concatenated and passed to a classifier
The features at various scales help us extract features at various levels of semantic granularity
Results—
The MSFFN model:
Achieved a 6.5% accuracy improvement over transformer baselines, with a parameter count nearly two orders of magnitude smaller.
The ensemble variant, MSFFN-Ensemble, reached an 86.82% accuracy, highlighting the strength of domain-specific learning.
Demonstrated exceptional generalization to unseen text generators, with up to 94.6% accuracy on challenging test data.
GitHub: https://github.com/thebharathsk/machine_gen_text_detection
Future Directions—
Support for multilingual datasets, enabling broader applicability across global contexts.
Integration with real-time systems for immediate detection of machine-generated text.
Enhanced generalization through advanced data augmentation and contrastive learning techniques.
Acknowledgments—
This project was a collaborative effort between me and two other students at Carnegie Mellon University. Special thanks to Ashwin Pillay (MSMT, CMU) and Bharath Somayajula (MSCV, Robotics Institute, CMU).
