Skip to content

Latest commit

 

History

History
113 lines (88 loc) · 5.28 KB

llm_finetunes.md

File metadata and controls

113 lines (88 loc) · 5.28 KB

master document for LLM Finetune ideas

labels: training, experimentaiton

feel like maybe I'll be more inspired here if I put the ideas in one place.

I also like that PR/FAQ thing I saw the other day where someone was writing reviews for SD finetunes they hadn't yet trained or something like that.

anyway, I was inspired to make this now for the following idea:

  • document-to-mindmap
    • summarize a document into a graphical summary saved as a text file. graphical summary document GSD.
    • needs to be parameterizable as a text format so we can just use an LLM to generate these.
    • construct a procedure for growing this representation that pushes in "missing" notes

--

Sources for codegen finetuning data

Python

  • Django
  • FastAPI
  • huggingface/transformers
  • huggingface/diffusers
  • numpy
  • scipy
  • pandas
  • sklearn
  • pytorch
  • pytorch-lightning

Go

  • Gin
  • terraform

k8s

  • kubernetes
  • knative
  • prometheus
  • etcd
  • helm
  • argoCD
  • docker
  • singularity
  • slurm

Creative writing (planning) assistant

Finetune an adapter to interpret specialized context signals

e.g. could recursively utilize a coarse-to-fine strategy to develop multi-resolution outline structures, and use different focused sections of outline as context signal provided to an adaptor module.

this is probably overkill and in-context learning probably accomplishes most of what I want here. but yeah, could finetune a LoRA or something.

hierarchical document structure

with each chunk and level of depth of a document, associate with it a "scratchpad". this can be a single file, or multiple files. should represent considerata relevant to that "local" context. represent doucment as a nested file tree. each level of the tree contains a "content" directory for the actual content, and a "scratch" directory for context documents. as content gets authored, scratch content in adjacent directories can be updated.

document can be accompanied by a log (git log?) documenting change history, and each scratch space can be accompanied by its own respective "to do" list of action items for work that needs to be developed locally, content to be developed, etc.

auto-wiki

"you are generating content for ___ textbook in the form of wiki articles. highlight topics and phrases that merit their own articles as hyperlinks, using the following markup syntax: this is an [[inline link]] to another [[article]]. the current article is on the topic: '{article topic}. Begin."

spider to push out more content