📓
NOTE: This repository is intended for learning and research purposes. It is also worth nothing that this is a WIP and will continue to be updated.
The purpose of this project is to provide a way for any organization to leverage GPT inside their organization without having to send data externally. We also want to help democratize these foundation models so that everyone can use them in a clear transparent way without relying on a black box or a pay wall as a barrier.
In this repository you will learn how to create your own expert system by fine-tuning and retraining a powerful open source foundation model GPT-NeoX. We will walk through the complete end to end process from the experimentation in notebooks to productionisation and finally deployment as an API or Gradio application that can be used by anyone internally in a secure fashion for any number of applications.
-
Download and install the latest version of Anaconda (https://docs.anaconda.com/anaconda/install/index.html)
-
Create your venv from the environment.yml file
conda env create -f environment.yaml
-
Activate the environment
conda activate expert-system-gpt
-
Move into the main pipeline directory
cd expert-system-gpt-pipeline
-
Install the dependencies
pip install -r src/requirements.txt
-
Setup your IDE for Kedro projects
- PyCharm or IntelliJ (https://kedro.readthedocs.io/en/stable/development/set_up_pycharm.html)
- Visual Studio Code (https://kedro.readthedocs.io/en/stable/development/set_up_vscode.html)
-
Continue the next steps in this README
- HuggingFace https://huggingface.co/
- EleutherAI https://www.eleuther.ai/
- Kedro https://kedro.org/
- BentoML https://www.bentoml.com/