Skip to content

This is a Python script designed to prepare novel data into training data for LLMs. It realigns the text and overwrites the original files with the processed content.

Notifications You must be signed in to change notification settings

AkiraChisaka/novel-data-processor-for-sakura-python

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

48 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Novel Data Preparer

Description

This is a Python script designed to prepare novel data into training data for LLMs. It realigns the text and overwrites the original files with the processed content.

Usage

To use the application, run the main.py script with the paths to the files you want to process as arguments:

python main.py path_to_jp_file path_to_cn_file

Testing

To run the tests for the application, use the pytest command.

About

This is a Python script designed to prepare novel data into training data for LLMs. It realigns the text and overwrites the original files with the processed content.

Resources

Stars

Watchers

Forks

Languages