README.md

ling is a golang toolkit for natural language processing

Implementation references

Similar NLP tools

• Stanford CoreNLP Java
• spaCy Python
• lingo Golang

Multilingual text toknization

• Unicode Standard Annex #29
• blevesearch segment

Text normalization

• Text normalization in Go

Lemmatization

词干提取（stemming）和词形还原（lemmatization）

• Stemming and lemmatization
• Lemmatization Lists_{Datasets by MBM}
• The UniMorph Project
• 中文繁简转换
  • gocc Golang version OpenCC
  • OpenCC
  • Chinese-Character Jian<=>Fan converting library in Go
  • Traditional and Simplified Chinese Conversion in Go
  • Han unification

Tagging

• Regex tagger
  • commonregex, a collection of common regular expressions for Go.
  • xurls, a Go package of regex for urls.

Natural language Detection

getlang is much slower than franco

• getlang
• franco
• test scripts
  • franco: Duration: 5.12s, 26.93%
  • getlang: Duration: 11.58s, 59.54%