Skip to content

Commit

Permalink
Merge pull request #107 from svera/v4
Browse files Browse the repository at this point in the history
V4
  • Loading branch information
svera authored Oct 12, 2024
2 parents 4e36efb + cb52225 commit 412c4c6
Show file tree
Hide file tree
Showing 88 changed files with 935 additions and 693 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
#branches: [ main ]

jobs:

Expand All @@ -23,4 +23,4 @@ jobs:

- name: Test
run: go test -v ./... --cover

9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A personal documents server, Coreander indexes the documents (EPUBs and PDFs wit
* Fast search engine powered by [Bleve](https://github.com/blevesearch/bleve), with support for documents in multiple languages.
* Search by author, title and even document series ([Calibre's](https://calibre-ebook.com/) `series` meta supported)
* Improved search for documents with metadata in english, spanish, french, italian, german and portuguese, including genre and singular/plural forms of words in the results among others.
* Estimated reading time calculation.
* Estimated reading time calculation.
* High-performance web server powered by [Fiber](https://github.com/gofiber/fiber).
* Lightweight, responsive web interface based on [Bootstrap](https://getbootstrap.com/).
* Web interface available in english, spanish and french, more languages can be easily added.
Expand All @@ -18,6 +18,7 @@ A personal documents server, Coreander indexes the documents (EPUBs and PDFs wit
* Read indexed epubs and PDFs from Coreander's interface thanks to [foliate-js](https://github.com/johnfactotum/foliate-js).
* Restrictable access only to registered users.
* Upload documents through the web interface.
* Download as kepub (epub for Kobo devices) converted on the fly thanks to [Kepubify](https://github.com/pgaskin/kepubify).

## Installation

Expand Down Expand Up @@ -55,8 +56,7 @@ Coreander requires a `LIB_PATH` environment variable to be set, which tells the

On first run, Coreander will index the documents in your library, creating a database with those entries located at `$home/coreander/index`. Depending on your system's performance and the size of your library this may take a while. Also, the database can grow fairly big, so make sure you have enough free space on disk.

Every time is run, the application check for new entries, reindexing the whole library. You can
avoid this behaviour by setting the environment variable `SKIP_INDEXING` to `true`.
Every time is run, the application scans the library folder only for documents not yet indexed and adds them to the index. You can force to index all documents wether they were previously indexed or not by setting the environment variable `FORCE_INDEXING` to `true`.

Even if the application is still indexing entries, you can access its web interface right away. Just open a web browser and go to `localhost:3000` (replace `localhost` with the hostname / IP address of the machine where the server is running if you want to access it from another system). It is possible to change the listening port just executing the application with the `PORT` environment variable (e. g. `PORT=4000 coreander`)

Expand Down Expand Up @@ -97,7 +97,7 @@ On first run, Coreander creates an admin user with the following credentials:
* `PORT`: Port number in which the webserver listens for requests. Defaults to 3000.
* `BATCH_SIZE`: Number of documents persisted by the indexer in one write operation. Defaults to 100.
* `COVER_MAX_WIDTH`: Maximum horizontal size for documents cover thumbnails in pixels. Defaults to 600.
* `SKIP_INDEXING`: Whether to bypass the indexing process or not.
* `FORCE_INDEXING`: Whether to force indexing already indexed documents or not. Defaults to false.
* `SMTP_SERVER`: Address of the send mail server.
* `SMTP_PORT`: Port number of the send mail server. Defaults to 587.
* `SMTP_USER`: User to authenticate against the SMTP server.
Expand All @@ -109,5 +109,4 @@ On first run, Coreander creates an admin user with the following credentials:
* `SESSION_TIMEOUT`: Specifies the maximum time a user session may last, in hours. Floating-point values are allowed. Defaults to 24 hours.
* `RECOVERY_TIMEOUT`: Specifies the maximum time a user recovery link may last, in hours. Floating-point values are allowed. Defaults to 2 hours.
* `UPLOAD_DOCUMENT_MAX_SIZE`: Maximum document size allowed to be uploaded to the library, in megabytes. Set this to 0 to unlimit upload size. Defaults to 20 megabytes.
* `HOSTNAME`: **Deprecated, use FQDN instead**.
* `FQDN`: Domain name of the server. If Coreander is listening to a non-standard HTTP / HTTPS port, include it using a colon (e. g. example.com:3000). Defaults to `localhost`.
6 changes: 2 additions & 4 deletions config.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ package main
type Config struct {
// LibPath holds the absolute path to the folder containing the documents
LibPath string `env:"LIB_PATH" env-required:"true"`
// Deprecated. Use FQDN instead
Hostname string `env:"HOSTNAME" env-default:"localhost"`
// FQDN stores the domain name of the server. If the server is listening on a non-standard HTTP / HTTPS port, include it using a colon (e. g. :3000)
FQDN string `env:"FQDN" env-default:"localhost"`
// Port defines the port number in which the webserver listens for requests
Expand All @@ -14,8 +12,8 @@ type Config struct {
BatchSize int `env:"BATCH_SIZE" env-default:"100"`
// CoverMaxWidth sets the maximum horizontal size for documents cover thumbnails in pixels
CoverMaxWidth int `env:"COVER_MAX_WIDTH" env-default:"600"`
// SkipIndexing signals whether to bypass the indexing process or not
SkipIndexing bool `env:"SKIP_INDEXING" env-default:"false"`
// ForceIndexing signals whether to force indexing already indexed documents or not
ForceIndexing bool `env:"FORCE_INDEXING" env-default:"false"`
// SmtpServer points to the address of the send mail server
SmtpServer string `env:"SMTP_SERVER"`
// SmtpPort defines the port in which the mail server listens for requests
Expand Down
2 changes: 1 addition & 1 deletion filewatcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
package main

import (
"github.com/svera/coreander/v3/internal/index"
"github.com/svera/coreander/v4/internal/index"
)

func fileWatcher(idx *index.BleveIndexer, libPath string) {
Expand Down
4 changes: 2 additions & 2 deletions filewatcher_linux.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"log"

"github.com/rjeczalik/notify"
"github.com/svera/coreander/v3/internal/index"
"github.com/svera/coreander/v4/internal/index"
)

func fileWatcher(idx *index.BleveIndexer, libPath string) {
Expand All @@ -20,7 +20,7 @@ func fileWatcher(idx *index.BleveIndexer, libPath string) {
select {
case ei := <-c:
if ei.Event() == notify.InCloseWrite || ei.Event() == notify.InMovedFrom {
if err := idx.AddFile(ei.Path()); err != nil {
if _, err := idx.AddFile(ei.Path()); err != nil {
log.Printf("Error indexing new file: %s\n", ei.Path())
}
}
Expand Down
16 changes: 14 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
module github.com/svera/coreander/v3
module github.com/svera/coreander/v4

go 1.21
go 1.21.0

toolchain go1.21.1

require (
github.com/blevesearch/bleve/v2 v2.4.0
Expand All @@ -27,24 +29,33 @@ require (

require (
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/beevik/etree v1.4.1 // indirect
github.com/blevesearch/go-faiss v1.0.13 // indirect
github.com/blevesearch/zapx/v16 v16.0.12 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/geek1011/kepubify v2.3.2+incompatible // indirect
github.com/glebarez/go-sqlite v1.22.0 // indirect
github.com/gofiber/template v1.8.3 // indirect
github.com/gofiber/utils v1.1.0 // indirect
github.com/gosimple/unidecode v1.0.1 // indirect
github.com/hhrutter/lzw v1.0.0 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.5 // indirect
github.com/kr/smartypants v0.1.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/mattn/go-zglob v0.0.5 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
github.com/pgaskin/kepubify v2.3.2+incompatible // indirect
github.com/pgaskin/kepubify/_/go116-zip.go117 v0.0.0-20210611152744-2d89b3182523 // indirect
github.com/pgaskin/kepubify/_/html v0.0.0-20211223234002-6ee2cc632cdc // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tinylib/msgp v1.1.9 // indirect
golang.org/x/sync v0.7.0 // indirect
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
gopkg.in/check.v1 v1.0.0-20200902074654-038fdea0a05b // indirect
modernc.org/libc v1.49.3 // indirect
Expand Down Expand Up @@ -90,6 +101,7 @@ require (
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/mschoch/smat v0.2.0 // indirect
github.com/pgaskin/kepubify/v4 v4.0.4
github.com/pirmd/epub v0.3.0
github.com/rivo/uniseg v0.4.7 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
Expand Down
24 changes: 24 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ github.com/antzucaro/matchr v0.0.0-20210222213004-b04723ef80f0 h1:R/qAiUxFT3mNgQ
github.com/antzucaro/matchr v0.0.0-20210222213004-b04723ef80f0/go.mod h1:v3ZDlfVAL1OrkKHbGSFFK60k0/7hruHPDq2XMs9Gu6U=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/bamiaux/rez v0.0.0-20170731184118-29f4463c688b/go.mod h1:obBQGGIFbbv9KWg92Qu9UHeD94JXmHD1jovY/z6I3O8=
github.com/beevik/etree v1.1.0/go.mod h1:r8Aw8JqVegEf0w2fDnATrX9VpkMcyFeM0FhwO62wh+A=
github.com/beevik/etree v1.4.1 h1:PmQJDDYahBGNKDcpdX8uPy1xRCwoCGVUiW669MEirVI=
github.com/beevik/etree v1.4.1/go.mod h1:gPNJNaBGVZ9AwsidazFZyygnd+0pAU38N4D+WemwKNs=
github.com/bits-and-blooms/bitset v1.12.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/bits-and-blooms/bitset v1.13.0 h1:bAQ9OPNFYbGHV6Nez0tmNI0RiEu7/hxlYJRUA0wFAVE=
github.com/bits-and-blooms/bitset v1.13.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
Expand Down Expand Up @@ -67,6 +71,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/flotzilla/pdf_parser v0.1.96 h1:SlgvO7NZqFzhBO+o6X1u7rUYjhv+81V3dYQF+LTfGOE=
github.com/flotzilla/pdf_parser v0.1.96/go.mod h1:/CPB1OWEeFqRbtnFWXgArmOnA3u7smVHxr5dFy4U6Nk=
github.com/geek1011/kepubify v2.3.2+incompatible h1:G1dAwpTpSHN79/bOqQ64SjkllYPhpp25kUkJ95WNFnA=
github.com/geek1011/kepubify v2.3.2+incompatible/go.mod h1:xMWLgn5FQSh6oNcq//MwSwPf2WgyQq6T+fY9nSn9+Cs=
github.com/glebarez/go-sqlite v1.22.0 h1:uAcMJhaA6r3LHMTFgP0SifzgXg46yJkgxqyuyec+ruQ=
github.com/glebarez/go-sqlite v1.22.0/go.mod h1:PlBIdHe0+aUEFn+r2/uthrWq4FxbzugL0L8Li6yQJbc=
github.com/glebarez/sqlite v1.11.0 h1:wSG0irqzP6VurnMEpFGer5Li19RpIRi2qvQz++w0GMw=
Expand Down Expand Up @@ -104,6 +110,7 @@ github.com/gosimple/slug v1.14.0 h1:RtTL/71mJNDfpUbCOmnf/XFkzKRtD6wL6Uy+3akm4Es=
github.com/gosimple/slug v1.14.0/go.mod h1:UiRaFH+GEilHstLUmcBgWcI42viBN7mAb818JrYOeFQ=
github.com/gosimple/unidecode v1.0.1 h1:hZzFTMMqSswvf0LBJZCZgThIZrpDHFXux9KeGmn6T/o=
github.com/gosimple/unidecode v1.0.1/go.mod h1:CP0Cr1Y1kogOtx0bJblKzsVWrqYaqfNOnHzpgWw4Awc=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hhrutter/lzw v1.0.0 h1:laL89Llp86W3rRs83LvKbwYRx6INE8gDn0XNb1oXtm0=
github.com/hhrutter/lzw v1.0.0/go.mod h1:2HC6DJSn/n6iAZfgM3Pg+cP1KxeWc3ezG8bBqW5+WEo=
github.com/hhrutter/tiff v1.0.1 h1:MIus8caHU5U6823gx7C6jrfoEvfSTGtEFRiM8/LOzC0=
Expand All @@ -122,6 +129,8 @@ github.com/klauspost/compress v1.16.3/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQs
github.com/klauspost/compress v1.17.8 h1:YcnTYrq7MikUT7k0Yb5eceMmALQPYBW/Xltxn0NAMnU=
github.com/klauspost/compress v1.17.8/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/smartypants v0.1.0 h1:Sn8hn5XrY+uXrxSWUdcr621Gfpk11mOGGVs4XX06kEw=
github.com/kr/smartypants v0.1.0/go.mod h1:EcTX9ge+SWNaGwbQvHwNICsMGavh98FLUqyOWFr+j9c=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
Expand All @@ -138,6 +147,9 @@ github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh
github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-runewidth v0.0.15 h1:UNAjwbU9l54TA3KzvqLGxwWjHmMgBUVhBiTjelZgg3U=
github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.14.12/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
github.com/mattn/go-zglob v0.0.5 h1:LKgpZXHg94zoBPDbb6aeiCs4SiQSS8otal4JnzkIvMc=
github.com/mattn/go-zglob v0.0.5/go.mod h1:MxxjyoXXnMxfIpxTK2GAkw1w8glPsQILx3N5wrKakiY=
github.com/microcosm-cc/bluemonday v1.0.26 h1:xbqSvqzQMeEHCqMi64VAs4d8uy6Mequs3rQ0k/Khz58=
github.com/microcosm-cc/bluemonday v1.0.26/go.mod h1:JyzOCs9gkyQyjs+6h10UEVSe02CGwkhd72Xdqh78TWs=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
Expand All @@ -153,6 +165,15 @@ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWb
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
github.com/pdfcpu/pdfcpu v0.7.0 h1:cd7/z7hAyyDuzdciKfNZyQ3TYreJza2DsuPdIHYURcA=
github.com/pdfcpu/pdfcpu v0.7.0/go.mod h1:kmpD0rk8YnZj0l3qSeGBlAB+XszHUgNv//ORH/E7EYo=
github.com/pgaskin/kepubify v2.3.2+incompatible h1:0wWWxop5T5O+p6tlZ59saC9hZYtkrXaKWLiTXmUNHHg=
github.com/pgaskin/kepubify v2.3.2+incompatible/go.mod h1:vQR3SJUwNyKStXpUPsVcCjBZtjZ1TgYtgKb8jHMyEDg=
github.com/pgaskin/kepubify/_/go116-zip.go117 v0.0.0-20210611152744-2d89b3182523 h1:pYGj3rKTy+TDs5Z707kT+ztjoIDCy76lc2UPkZocAFM=
github.com/pgaskin/kepubify/_/go116-zip.go117 v0.0.0-20210611152744-2d89b3182523/go.mod h1:FNMbV/TSSnhqyzjq8jsS+VD0o/gwpuCH0dh8G1uQ/fw=
github.com/pgaskin/kepubify/_/html v0.0.0-20211223234002-6ee2cc632cdc h1:mJk4TIXTO+JmxgHJ5iyil42PLQJWkyaKB/qNcjJU6h4=
github.com/pgaskin/kepubify/_/html v0.0.0-20211223234002-6ee2cc632cdc/go.mod h1:fxzoIpMFAReNKunZ+ttVbf3hNVrJGtrSZMI4olZizbs=
github.com/pgaskin/kepubify/v4 v4.0.4 h1:8ePyepo4eRNSmeDs5MdJLJtit+zxmK36wILmGcvpccU=
github.com/pgaskin/kepubify/v4 v4.0.4/go.mod h1:wzUdFNYW2uZh2xfHDuzNRRUO4WqV+y99UBxVd3rBTus=
github.com/pgaskin/koboutils/v2 v2.1.2-0.20220306004009-a07e72ebae42/go.mod h1:wTzkDIlsxmUyfwfspGcm0Ap+HOxSUYV0S8kMYrf+0gM=
github.com/philhofer/fwd v1.1.1/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
github.com/philhofer/fwd v1.1.2 h1:bnDivRJ1EWPjUIRXV5KfORO897HTbpFAQddBdE8t7Gw=
github.com/philhofer/fwd v1.1.2/go.mod h1:qkPdfjR2SIEbspLqpe1tO4n5yICnr2DY7mqEx2tUTP0=
Expand Down Expand Up @@ -181,6 +202,8 @@ github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8=
github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
Expand Down Expand Up @@ -235,6 +258,7 @@ golang.org/x/net v0.24.0 h1:1PcaxkF854Fu3+lvBIx5SYn9wRlBzzcnHZSiaFFAb0w=
golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
Expand Down
14 changes: 7 additions & 7 deletions internal/index/bleve.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,12 @@ import (
"github.com/blevesearch/bleve/v2/analysis/tokenizer/unicode"
"github.com/blevesearch/bleve/v2/mapping"
"github.com/spf13/afero"
"github.com/svera/coreander/v3/internal/metadata"
"github.com/svera/coreander/v4/internal/metadata"
)

// Version identifies the mapping used for indexing. Any changes in the mapping requires an increase
// of version, to signal that a new index needs to be created.
const Version = "v2"
const Version = "v3"

// Metadata fields
var (
Expand All @@ -33,11 +33,11 @@ var (
)

var noStopWordsFilters = map[string][]string{
es.AnalyzerName: {es.NormalizeName, lowercase.Name, es.LightStemmerName},
en.AnalyzerName: {en.PossessiveName, lowercase.Name, porter.Name},
de.AnalyzerName: {de.NormalizeName, lowercase.Name, de.LightStemmerName},
fr.AnalyzerName: {fr.ElisionName, lowercase.Name, fr.LightStemmerName},
it.AnalyzerName: {it.ElisionName, lowercase.Name, it.LightStemmerName},
es.AnalyzerName: {lowercase.Name, es.NormalizeName, es.LightStemmerName},
en.AnalyzerName: {lowercase.Name, en.PossessiveName, porter.Name},
de.AnalyzerName: {lowercase.Name, de.NormalizeName, de.LightStemmerName},
fr.AnalyzerName: {lowercase.Name, fr.ElisionName, fr.LightStemmerName},
it.AnalyzerName: {lowercase.Name, it.ElisionName, it.LightStemmerName},
pt.AnalyzerName: {lowercase.Name, pt.LightStemmerName},
}

Expand Down
4 changes: 2 additions & 2 deletions internal/index/bleve_read.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"github.com/blevesearch/bleve/v2/search/query"
"github.com/gosimple/slug"
"github.com/spf13/afero"
"github.com/svera/coreander/v3/internal/metadata"
"github.com/svera/coreander/v3/internal/result"
"github.com/svera/coreander/v4/internal/metadata"
"github.com/svera/coreander/v4/internal/result"
)

func (b *BleveIndexer) IndexingProgress() (Progress, error) {
Expand Down
39 changes: 34 additions & 5 deletions internal/index/bleve_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ import (

"github.com/blevesearch/bleve/v2"
"github.com/spf13/afero"
"github.com/svera/coreander/v3/internal/index"
"github.com/svera/coreander/v3/internal/metadata"
"github.com/svera/coreander/v3/internal/result"
"github.com/svera/coreander/v3/internal/webserver/model"
"github.com/svera/coreander/v4/internal/index"
"github.com/svera/coreander/v4/internal/metadata"
"github.com/svera/coreander/v4/internal/result"
"github.com/svera/coreander/v4/internal/webserver/model"
)

func TestIndexAndSearch(t *testing.T) {
Expand All @@ -35,7 +35,7 @@ func TestIndexAndSearch(t *testing.T) {
appFS.MkdirAll("lib", 0755)
afero.WriteFile(appFS, tcase.filename, []byte(""), 0644)

err = idx.AddLibrary(1)
err = idx.AddLibrary(1, true)
if err != nil {
t.Errorf("Error indexing: %s", err.Error())
}
Expand Down Expand Up @@ -290,5 +290,34 @@ func testCases() []testCase {
},
),
},
{
"Test spanish stemmer returning accented word while using unaccented word in search",
"lib/book9.epub",
metadata.Metadata{
Title: "Últimos días en Colditz",
Authors: []string{"Patrick R. Reid"},
Description: "Just test metadata",
Language: "es",
Subjects: []string{"History", "WWII"},
},
"ultimos",
result.NewPaginated[[]index.Document](
model.ResultsPerPage,
1,
1,
[]index.Document{
{
ID: "book9.epub",
Slug: "patrick-r-reid-ultimos-dias-en-colditz",
Metadata: metadata.Metadata{
Title: "Últimos días en Colditz",
Authors: []string{"Patrick R. Reid"},
Description: "Just test metadata",
Subjects: []string{"History", "WWII"},
},
},
},
),
},
}
}
Loading

0 comments on commit 412c4c6

Please sign in to comment.