Releases: apache/lucenenet
v4.8.0-beta00016
This release contains several important bug fixes and performance enhancements.
Breaking Index Changes
There are 2 breaking changes that may affect some users when reading indexes that were created from version 4.8.0-beta00015 and all prior 4.8.0 beta versions (not including 3.0.3).
- A bug was fixed in the generation of segment file names. This only affects users with more than 10 segments in their index.
Lucene.Net.Documents.DateTools
has been modified to return milliseconds since Unix epoch (that is, since Jan 1, 1970 at 00:00:00 UTC) by default to match Java Lucene. This only affects users who explicitly useLucene.Net.Documents.DateTools
in their application and store the result (in .NET ticks) in their index.
If you are affected by either of the above issues, it is recommended to regenerate your indexes during upgrading. However, if that is not feasible, we have provided the following workarounds.
- If you have a large index with more than 10 segments, see #576 for details on how to enable legacy segment name support.
- If you are storing the result of
Lucene.Net.Documents.DateTools.StringToTime(string)
orLucene.Net.Documents.DateTools.Round(long)
(along
) in your index, you are storing .NET ticks. There are now optional parametersinputRepresentation
andoutputRepresentation
on these methods to specify whether thelong
value represents .NET ticks, .NET ticks as milliseconds, or millisenonds since the Unix epoch. To exactly match version 4.8.0-beta00015 and prior (including prior major versions):Lucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation)
should specifyNumericRepresentation.TICKS
foroutputRepresentation
.Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation)
should specifyNumericRepresentation.TICKS_AS_MILLISECONDS
forinputRepresentation
andNumericRepresentation.TICKS
foroutputRepresentation
.
.NET Framework Recommendations
It is recommended that all .NET Framework users migrate as soon as possible.
- In cases where
Lucene.Net.Support.WeakDictionary<TKey, TValue>
was used in .NET Framework and .NET Standard 2.0 due to missing APIs, but there is now a better solution usingPrism.Core
's weak events in combination withConditionalWeakTable<TKey, TValue>
, which means memory management is handled entirely by the GC inLucene.Net.Index.IndexReader
,Lucene.Net.Search.FieldCacheImpl
,Lucene.Net.Search.CachingWrappingFilter
andLucene.Net.Facet.Taxonomy.CachedOrdinalsReader
. See #613. - All known issues with loss of floating-point precision on .NET Framework x86 have been fixed.
Change Log
Breaking Changes
- #547 -
Lucene.Net.Util.StringHelper.GOOD_FAST_HASH_SEED
: converted from a static field to a property and marked obsolete. Added a new propertyGoodFastHashSeed
. RemovedSystemProperties
call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability. - #547 -
Lucene.Net.TestFramework
: AddedLuceneSetUpFixtureBuilder
class to load either a subclass or our default instance ofLuceneTestFrameworkInitializer
. Also addedLuceneTestCase.SetUpFixture
to control initialization ofLuceneTestFrameworkInitializer
so it is only called on setup and teardown for the assembly. AddedInitialize()
method toLuceneTestFrameworkInitializer
that must be used when setting factories during testing. - #547 -
Lucene.Net.TestFramework.Util.LuceneTestCase
: DeprecatedGetClassType()
method and addedTestType
property - #547 -
Lucene.Net.TestFramework.Util.AbstractBeforeAfterRule``: Removed
LuceneTestCaseparameter from
Before()and
After()` methods. - #551 - Changed constructors of
Lucene.Net.Util.NumberFormat
andLucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat
to acceptIFormatProvider
rather thanCultureInfo
and changedLucene.Net.Util.NumberFormat.Culture
property toLucene.Net.Util.NumberFormat.FormatProvider
. - #554 -
Lucene.Net.Misc
: MadeDocFreqComparer
andTotalTermFreqComparer
into static singletons, only accessible by theDefault
property. - #428, #429, #570 -
Lucene.Net.Search.FieldComparer
: Redesigned implementation to use reference types for numerics (from J2N) to avoid boxing. - #570 -
Lucene.Net.Search.FieldCache.IParser
: Renamed method fromTermsEnum()
toGetTermsEnum()
to match other APIs - #570 -
Lucene.Net.Queries
:ObjectVal()
returns aJ2N.Numerics.Number
-derived type rather than a value type cast to object. Direct casts toint
,long
,double
,single
, etc. will no longer work without first casting to theJ2N.Numerics.Number
-derived type. Alternatively, use the correspondingConvert.ToXXX()
method for the type you wish to retrieve from the object. - #574 -
Lucene.Net.Suggest.Fst.FSTCompletionLookup/WFSTCompletionLookup
: ChangedGet()
to returnlong?
instead ofobject
to eliminate boxing/unboxing - #574 -
Lucene.Net.Index.MergePolicy::FindForcedMerges()
: Removed unnecessary nullable fromFindForcedMerges()
and allMergePolicy
subclasses - #574 -
Lucene.Net.Replicator
: Changed callback signature fromFunc<bool?>
toAction
, since the return value had no semantic meaning - #575 -
Lucene.Net.Index.DocValuesFieldUpdates
: Refactored so the subclasses will handle getting the values fromDocValuesFieldUpdatesIterator
orDocValuesUpdate
via a cast rather than boxing the value. Also marked internal (as well as all members ofBufferedUpdates
), since this was not supposed to be part of the public API. - #573, #576 - Changed segment file names to match Lucene 4.8.0 and Lucene.NET 3.x
- #577 -
Lucene.Net.Index.SegmentInfos
: ChangedInfo()
method to an indexer (.NET Convention) - #580 -
Lucene.Net.Documents.DateTools
- AddedNumericRepresentation
enum to allow converting to/from long in the following formats:-
Unix Epoch (default): Milliseconds since Jan 1, 1970 12:00:00 AM UTC.
-
Ticks: The raw ticks from
DateTime
orDateTimeOffset
. -
Ticks as Milliseconds: This is for compatibility with prior versions of Lucene.NET (3.0.3 and 4.8.0-beta00001 - 4.8.0-beta00015). The conversion done on input values is
time * TimeSpan.TicksPerMillisecond
and the conversion to output values istime / TimeSpan.TicksPerMillisecond
.The
long
return value fromLucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation)
has been changed fromNumericRepresentation.TICKS
toNumericRepresentation.UNIX_TIME_MILLISECONDS
by default.The
long
input parameter provided toLucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation)
has been changed fromNumericRepresentation.TICKS_AS_MILLISECONDS
toNumericRepresentation.UNIX_TIME_MILLISECONDS
by default.The
long
return value fromLucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation)
has changed fromNumericRepresentation.TICKS
toNumericRepresentation.UNIX_TIME_MILLISECONDS
by default.
-
- #580 -
Lucene.Net.Documents.DateTools
- De-nestedResolution
enum and renamedDateResolution
. - #580 -
Lucene.Net.QueryParsers.Flexible.Standard
: Changed numeric nodes to accept and returnJ2N.Numerics.Number
-derived types instead ofobject
. - #581 - SWEEP:
Lucene.Net.Util.Fst
: Changed API to useJ2N.Numerics.Int64
instead oflong?
for generic closing type as it was designed to use reference equality comparison. - #581 - SWEEP:
Lucene.Net.Util.Fst
: Added class constraints to each generic FST type and reverted to reference equality comparisons. - #581, #279 -
Lucene.Net.Util.Fst.Int32sRefFSTEnum
: AddedMoveNext()
method and markedNext()
method obsolete. This change had already been done to BytesRefFSTEnum, which made them inconsistent. - #583 -
Lucene.Net.QueryParsers.Flexible
: Removed unnecessary nullable value types fromConfigurationKeys
and configuration setters/getters inStandardQueryParser
. AddedAbstractQueryConfig.TryGetValue()
method to allow retrieving value types so they can be defaulted properly. - #583 -
Lucene.Net.Queries.Function.ValueSources.EnumFieldSource::ctor()
- changedenumIntToStringMap
to acceptIDictionary<int, string>
instead ofIDictionary<int?, string>
(removed unnecessary nullable) - #587 -
Lucene.Net.TestFramework.Store.MockDirectoryWrapper
: RenamedAssertNoUnreferencedFilesOnClose
toAssertNoUnreferencedFilesOnDispose
- #619 -
Lucene.Net.Spatial
: Upgraded to new Spatial4n NuGet package that unifies the types from Spatial4n.Core and Spatial4n.Core.NTS - #619 -
Lucene.Net.Spatial.Prefix.Tree.Cell
: Renamedm_outerInstance
>m_spatialPrefixTree
and constructor parameterouterInstance
>spatialPrefixTree
- #619 -
Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter.BaseTermsEnumTransverser
: renamedm_outerInstance
>m_filter
, constructor parameterouterInstance
>filter
- #619 -
Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter
: De-nestedBaseTermsEnumTraverser
class - #619 -
Lucene.Net.Spatial.Prefix.Tree.GeohashPrefixTree.Factory
: de-nested and renamedGeohashPrefixTreeFactory
- #619 -
Lucene.Net.Spatial.Prefix.Tree.QuadPrefixTree.Factory
: de-nested and renamedQuadPrefixTreeFactory
- #619 -
Lucene.Net.Spatial.Prefix.AbstractVisitingPrefixTreeFilter
: De-nestedVisitorTemplate
class and changed protected fieldm_prefixGridScanLevel
to a public property namedPrefixGridScanLevel
. - #619 -
Lucene.Net.Spatial.Query
: RenamedUnsupportedSpatialOperation
>UnsupportedSpatialOperationException
to match .NET conventions
Bugs
v4.8.0-beta00015
This release contains important bug fixes, performance enhancements, concurrency improvements, and improved debugging support (full stack traces, consistent exception types, attributes for debug view, and structurally formattable lists).
Much of the exception handling has been changed so it is recommended to test thoroughly, especially if your application relies on catching exceptions from Lucene.NET for control flow. The full extent of the exception handling changes are not documented here, but can be viewed at https://github.com/apache/lucenenet/pull/476/files.
Known Issues
Lucene.Net.Index.IndexWriter::Dispose()
: UsingThread.Interrupt()
to shutdown background threads in .NET is problematic becauseSystem.Threading.ThreadInterruptedException
could be thrown on anylock
statement with contention on it. This includeslock
statements on code that we depend on or custom components that are engaged during aCommit()
(such as a customDirectory
implementation). These exceptions may causeCommit()
to fail in unexpected ways duringIndexWriter.Dispose()
. While this affected all prior releases to a much larger degree, this release provides a partial solution usingUninterruptableMonitor.Enter()
to ensure these exceptions are ignored and theThread.Interrupt()
state restored, which greatly reduces the chance aCommit()
could be broken or a deadlock can occur. This problem will not affect applications that do not callThread.Interrupt()
to shut down a thread. It is recommended never to useThread.Interrupt()
in conjunction withIndexWriter
,ConcurrentMergeScheduler
, orControlledRealTimeReopenThread
.
Change Log
Breaking Changes
- #455 -
lucene-cli
: Changed exit codes to well-defined constants to make testing simpler - #407 - Moved all Document extensions to the
Lucene.Net.Documents.Extensions
namespace and added tests forDocumentExtensions
inLucene.Net.Tests._J-S
,Lucene.Net.Tests.ICU
andLucene.Net.Tests.Facet
. Added guard clauses and updated documentation of Document extension methods and some related fields. - #474 -
Lucene.Net.TestFramework.Util.TestUtil
: Renamed method parameters from abbreviations to whole words to follow .NET API conventions and improved documentation. - #475 -
Lucene.Net.Grouping
: Refactored and improvedGroupingSearch
Search API and addedGroupByField()
andGroupByFunction()
methods. - #479 - Moved
Lucene.Net.Join
types toLucene.Net.Search.Join
namespace - Marked public exception constructors that were meant only for testing internal (affects only .NET Framework)
- #446, #476 - Redesigned exception handling to ensure that exception behavior is the same as in Lucene and so we consistently throw the closest .NET equivalent exception across all of the projects.
- #480 - Changed
Cardinality()
methods toCardinality
property. Added obsoleteCardinality()
extension methods to the namespace of each of the pertinent types for backward compatibility.Lucene.Net.Index.RandomAccessOrds
Lucene.Net.Util.FixedBitSet
Lucene.Net.Util.Int64BitSet
Lucene.Net.Util.OpenBitSet
Lucene.Net.Util.PForDeltaDocIdSet
Lucene.Net.Util.WAH8DocIdSet
- #481 -
Lucene.Net.Index.Term
: ChangedText()
method intoText
property. Added an obsoleteText()
extension method toLucene.Net.Index
namespace for backward compatibility. - #482 -
Lucene.Net.BinaryDocValuesField
: ChangedfType
static field toTYPE
(as it was in Lucene) and added obsoletefType
field for backward compatibility. - #483 - Changed all
GetFilePointer()
methods into properties namedPosition
to matchFileStream
. Types affected:Lucene.Net.Store.IndexInput
(and subclasses),Lucene.Net.Store.IndexOutput
(and subclasses). Added obsolete extension methods for each type inLucene.Net.Store
namespace for backward compatibility. - #484 -
Lucene.Net.QueryParser
: Factored outNLS
/IMessage
/Message
support and changed exceptions to use string messages so end users can elect whether or not to use .NET localization, as is possible with any other .NET exception type. - #484 -
Lucene.Net.QueryParsers.Flexible.Messages
: Removed entire namespace, as we have refactored to use .NET localization rather than NLS - #484 -
Lucene.Net.Util
: RemovedBundleResourceManagerFactory
andIResourceManagerFactory
, as these were only to support NLS. The new approach to localizing messages can be achieved by registeringQueryParserMessages.SetResourceProvider(SomeResource.ResourceManager, SomeOtherResource.ResourceManager)
at application startup using anyResourceManager
instance or designer-generated resource'sResourceManager
property. - #497, #507 - Factored out
Lucene.Net.Support.Time
in favor ofJ2N.Time
. Replaced all calls (exceptLucene.Net.Tests.Search.TestDateFilter
) that wereEnvironment.TickCount
andTime.CurrentTimeMilliseconds()
to useTime.NanoTime() / Time.MillisecondsPerNanosecond
for more accurate results. This may break some concurrent applications that are synchronizing with Lucene.NET components usingEnvironment.TickCount
. - #504 -
Lucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer
: Implemented singleton pattern so the class can only be used via theDefault
property. - #502 -
Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode
: AddedRemoveChildren()
method from Lucene 8.8.1 to fix brokenRemoveFromParent()
method behavior (applies patch LUCENE-5805). This requires existingIQueryNode
implementations to implementRemoveChildren()
andTryGetTag()
. - #502 -
Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode
: AddedTryGetTag()
method to simplify looking up a tag by name. - #528 -
Lucene.Net.Analysis.Stempel.Egothor.Stemmer.MultiTrie
: Changed protectedm_tries
field fromList<Trie>
toIList<Trie>
- #528 -
Lucene.Net.Search.BooleanQuery
: Changed protectedm_weights
field fromList<Weight>
toIList<Weight>
- #528 -
Lucene.Net.Search.DisjunctionMaxQuery
: Changed protectedm_weights
field fromList<Weight>
toIList<Weight>
Bugs
- #461 -
Lucene.Net.Grouping.GroupingSearch::GroupByFieldOrFunction<TGroupValue>()
: Fixed casting bug ofallGroupsCollector.Groups
by changing the cast toICollection
instead ofIList
. - #453, #455 - lucene-cli: Made
appsettings.json
file optional. This was causing a fatalFileNotFoundException
after installing lucene-cli without adding anappsettings.json
file. - #464 -
Lucene.Net.Codecs.SimpleText.SimpleTextStoredFieldsWriter
+Lucene.Net.Codecs.SimpleText.SimpleTextTermVectorsWriter
: FixedAbort()
methods to correctly swallow any exceptions thrown byDispose()
to match the behavior of Lucene 4.8.0. - #394, #467 -
Lucene.Net
NuGet does not compile under Visual Studio 2017. DowngradedLucene.Net.CodeAnalysis.CSharp
andLucene.Net.CodeAnalysis.VisualBasic
from .NET Standard 2.0 to .NET Standard 1.3 to fix. - #471 - Lucene.Net.Documents.FieldType: Corrected documentation to reflect the actual default of
IsTokenaized
astrue
andNumericType
asNumericType.NONE
, and to set toNumericType.NONE
(rather thannull
) if the field has no numeric type. - #476 -
Lucene.Net.Analysis.Common.Util.CharArraySet
: ThrowNotSupportedException
when the set is readonly, notInvalidOperationException
to match .NET collection behavior - #476 -
Lucene.Net.Codecs.Bloom.BloomFilteringPostingsFormat::FieldsConsumer()
: ThrowNotSupportedException
rather thanInvalidOperationException
- #476 -
Lucene.Net.Codecs.Lucene42.Lucene42DocValuesProducer::LoadNumeric()
: ThrowAssertionError
rather thanInvalidOperationException
- #476 -
Lucene.Net.Store.CompoundFileDirectory::ReadEntries()
: throwAssertionError
rather thanInvalidOperationException
- #476 -
Lucene.Net.Util.Packed.DirectPackedReader::Get()
: ThrowAssertionError
rather thanInvalidOperationException
- #476 -
Lucene.Net.Facet
: ThrowInvalidOperationException
rather thanThreadStateException
- #476 -
Lucene.Net.Grouping.BlockGroupingCollector
: ThrowNotSupportedException
rather thanInvalidOperationException
- #476 -
Lucene.Net.Tests.Index.TestUniqueTermCount
: ThrowNotSupportedException
rather thanInvalidOperationException
- #486 - Changed all references that were
float.MinValue
anddouble.MinValue
tofloat.Epsilon
anddouble.Epsilon
because those are the .NET equivalent constants toFloat.MIN_VALUE
andDouble.MIN_VALUE
in Java - #492, #497 -
Lucene.Net.Search.ControlledRealTimeReopenThread
- Fixed time calculation issue that was causing wait to happen for unusually long time periods. Lucene.Net.Tests.Search.TestMultiThreadTermVectors
: Removed stray[Test]
attribute that was causing extra overhead with no benefit- #509 -
Lucene.Net.Support.WeakDictionary
: ChangedWeakKey
to useWeakReference<T>
instead ofWeakReference
to avoid problems with garbage collection - #504 -
Lucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer
: Implemented singleton pattern so the class can only be used via theDefault
property. - #506, #509 -
Lucene.Net.Index.IndexReader
: UseConditionalWeakTable<TKey, TValue>
/WeakDictionary<TKey, TValue>
to ensure dead elements are pruned and garbage collected - #525 - Fixed
Lucene.Net.Index.TestIndexWriter::TestThreadInterruptDeadlock()
andLucene.Net.Index.TestIndexWriter::TestTwoThreadsInterruptDeadlock()
that were failing due to a difference in .NETThread.Interrupt()
behavior. In Java, anInterruptedException
is never thown fromsynchronized (this)
(the equivalent oflock (this)
), but .NET may throw `ThreadInterruptedExcepti...
v4.8.0-beta00014
This release contains bug fixes and minor performance improvements
Known Issues
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
Benchmarks (from #310)
Index Files
Click to expand
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
[Host] : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|
IndexFiles | 4.8.0-beta00005 | 905.6 ms | 131.82 ms | 197.30 ms | 43000.0000 | 8000.0000 | 7000.0000 | 220.99 MB |
IndexFiles | 4.8.0-beta00006 | 707.1 ms | 18.57 ms | 26.04 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.99 MB |
IndexFiles | 4.8.0-beta00007 | 712.2 ms | 16.45 ms | 23.06 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.04 MB |
IndexFiles | 4.8.0-beta00008 | 785.7 ms | 17.37 ms | 25.46 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.54 MB |
IndexFiles | 4.8.0-beta00009 | 824.9 ms | 32.86 ms | 48.17 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.34 MB |
IndexFiles | 4.8.0-beta00010 | 789.6 ms | 16.40 ms | 24.04 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.35 MB |
IndexFiles | 4.8.0-beta00011 | 805.4 ms | 21.26 ms | 31.82 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.37 MB |
IndexFiles | 4.8.0-beta00012 | 827.8 ms | 13.95 ms | 20.89 ms | 56000.0000 | 7000.0000 | 6000.0000 | 287.03 MB |
IndexFiles | 4.8.0-beta00013 | 793.6 ms | 13.63 ms | 19.55 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.22 MB |
IndexFiles | 4.8.0-beta00014 | 812.0 ms | 21.97 ms | 30.79 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.29 MB |
Search Files
Click to expand
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
[Host] : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
IterationCount=15 LaunchCount=2 WarmupCount=10
Method | Job | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
SearchFiles | 4.8.0-beta00005 | 421.1 ms | 111.47 ms | 163.38 ms | 326.3 ms | 18000.0000 | 1000.0000 | - | 82.11 MB |
SearchFiles | 4.8.0-beta00006 | 349.8 ms | 24.03 ms | 35.97 ms | 338.9 ms | 18000.0000 | 1000.0000 | - | 82.11 MB |
SearchFiles | 4.8.0-beta00007 | 333.6 ms | 17.36 ms | 25.98 ms | 336.8 ms | 18000.0000 | 1000.0000 | - | 81.9 MB |
SearchFiles | 4.8.0-beta00008 | 191.7 ms | 7.17 ms | 10.51 ms | 187.9 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
SearchFiles | 4.8.0-beta00009 | 186.6 ms | 8.56 ms | 12.55 ms | 184.0 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
SearchFiles | 4.8.0-beta00010 | 182.2 ms | 6.69 ms | 9.16 ms | 181.6 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
SearchFiles | 4.8.0-beta00011 | 208.9 ms | 17.73 ms | 26.54 ms | 207.9 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
SearchFiles | 4.8.0-beta00012 | 192.3 ms | 10.99 ms | 16.46 ms | 187.8 ms | 18000.0000 | 1000.0000 | - | 81.11 MB |
SearchFiles | 4.8.0-beta00013 | 177.4 ms | 7.74 ms | 11.59 ms | 175.1 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
SearchFiles | 4.8.0-beta00014 | 172.7 ms | 5.93 ms | 8.88 ms | 168.9 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
Change Log
Breaking Changes
- #424 - Deprecated
TaskMergeScheduler
, a merge scheduler that was added to support .NET Standard 1.x - #424 -
Lucene.Net.TestFramework
: Removed the publicLuceneTestCase.ConcurrentMergeSchedulerFactories
class
Bugs
- #405, #415 -
Lucene.Net.Index.DocTermOrds
: Fixed issue with enumerator (OrdWrappedTermsEnum
) incorrectly returningtrue
when the value isnull
. - #427 -
Lucene.Net.Analysis.Common
: FixedTestRollingCharBuffer::Test()
to prevent out of memory exceptions when running withVerbose
enabled - #434, #418 - Hunspell affixes' file parsing corrupts some affixes' conditions
- #434, #419 -
HunspellStemFilter
does not work with zero affix - #439 -
Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader
: Fixed synchronization issue between adding new items to the cache and readingRamBytesUsed
method - #439, #417, #319 -
Lucene.Net.Spatial.Util.ShapeFieldCacheProvider
: Fixed atomicity issue with loading the cache by usingLazy<T>
. - #441 -
Lucene.Net.TestFramework.Support.Confguration.TestConfigurationFactory
: UseLazy<T>
to ensure theconfigurationCache.GetOrAdd()
factory is atomic. - #441 -
Lucene.Net.TestFramework.Search.ShardSearchingTestBase: Fixed possible
KeyNotFoundExceptionwhen getting the value from
collectionStatisticsCache` - #441, #417, #319 -
Lucene.Net.Spatial.Prefix.PrefixTreeFactory
: UseLazy<T>
inConcurrentDictionary
to make thevalueFactory
atomic. - #443 -
Lucene.Net.Benchmark.ByTask.Feeds.SpatialDocMaker: Since
Dictionary<TKey, TValue>.this[key]is not marked virtual in .NET, subclassing
Dictionary<string, string>is not a valid approach. So we implement
IDictionary<string, string>` instead. - #416 - CLI Documentation issue - environment variable token not replaced.
- #450 -
Lucene.Net.Facet
- Reverted locking in to the state it was in Lucene 4.8.1, however we are still making use ofReaderWriterLockSlim
to improve read performance of caches. Also, removed the 1 second lock timeout fromCl2oTaxonomyWriterCache
.
Improvements
- #269 - Added
[AwaitsFix]
attribute to known failing tests - #391 - Improved plugins in DocFx when generating API docs
- #392 - Enabled GitHub Actions to Run Tests on Pull Request
- #395 - Improved performance of build pipeline by publishing the whole solution in one step instead of one project at a time
- #395 - Fixed dependency NuGet package version conflicts
- #395 - Added crash and hang detection to the test runs
- #395 - Upgraded to the latest
dotnet
CLI commandsdotnet build
anddotnet test
rather thandotnet msbuild
anddotnet vstest
- #41...
v4.8.0-beta00013
This release contains important bug fixes and performance enhancements.
Known Issues
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Benchmarks (from #310)
Index Files
Click to expand
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
[Host] : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|
IndexFiles | 4.8.0-beta00005 | 628.1 ms | 8.41 ms | 12.05 ms | 43000.0000 | 8000.0000 | 7000.0000 | 220.82 MB |
IndexFiles | 4.8.0-beta00006 | 628.3 ms | 13.19 ms | 19.33 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.67 MB |
IndexFiles | 4.8.0-beta00007 | 617.2 ms | 8.44 ms | 11.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.73 MB |
IndexFiles | 4.8.0-beta00008 | 620.6 ms | 5.62 ms | 8.41 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.06 MB |
IndexFiles | 4.8.0-beta00009 | 632.8 ms | 12.57 ms | 18.43 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.95 MB |
IndexFiles | 4.8.0-beta00010 | 862.3 ms | 51.13 ms | 74.95 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.22 MB |
IndexFiles | 4.8.0-beta00011 | 636.5 ms | 11.06 ms | 15.87 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09 MB |
IndexFiles | 4.8.0-beta00012 | 668.8 ms | 14.78 ms | 21.66 ms | 56000.0000 | 7000.0000 | 6000.0000 | 286.63 MB |
IndexFiles | 4.8.0-beta00013 | 626.7 ms | 7.78 ms | 10.91 ms | 43000.0000 | 8000.0000 | 7000.0000 | 219.8 MB |
Search Files
Click to expand
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
[Host] : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
IterationCount=15 LaunchCount=2 WarmupCount=10
Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|
SearchFiles | 4.8.0-beta00005 | 274.8 ms | 7.01 ms | 10.28 ms | 18000.0000 | 1000.0000 | - | 82.12 MB |
SearchFiles | 4.8.0-beta00006 | 283.4 ms | 7.78 ms | 11.64 ms | 18000.0000 | 1000.0000 | - | 82.13 MB |
SearchFiles | 4.8.0-beta00007 | 291.5 ms | 8.91 ms | 13.33 ms | 18000.0000 | 1000.0000 | - | 81.9 MB |
SearchFiles | 4.8.0-beta00008 | 162.3 ms | 5.50 ms | 8.23 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
SearchFiles | 4.8.0-beta00009 | 165.6 ms | 2.61 ms | 3.90 ms | 17000.0000 | - | - | 80.13 MB |
SearchFiles | 4.8.0-beta00010 | 159.4 ms | 2.84 ms | 4.17 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
SearchFiles | 4.8.0-beta00011 | 160.8 ms | 1.93 ms | 2.77 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
SearchFiles | 4.8.0-beta00012 | 169.2 ms | 6.48 ms | 9.49 ms | 18000.0000 | 1000.0000 | - | 81.11 MB |
SearchFiles | 4.8.0-beta00013 | 161.6 ms | 3.28 ms | 4.80 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
Change Log
Breaking Changes
Lucene.Net.Search.FieldCache
: Added interfaceICreationPlaceholder
and changedCreationPlaceholder
class toCreationPlaceHolder<TValue>
.
Bugs
- #356 -
Lucene.Net.Store.NativeFSLockFactory
: Modified options to allow read access on non-Windows operating systems. This caused the copy constructor ofRAMDirectory
to throw "The process cannot access the file 'file path' because it is being used by another process" excpetions. - #296 -
Lucene.Net.Util.Automaton.State
: RemovedEquals()
implementation; it was intended to use reference equality as a unique key. This caused randomIndexOperationException
s to occur when usingFuzzyTermsEnum
/FuzzyQuery
. - #387 - Fixed formatting in
ArgumentException
message for all analyzer factories so it will display the dictionary contents - #387 - Lucene.Net.Util.ExceptionExtensions.GetSuppressedAsList(): Use
J2N.Collections.Generic.List<T>
so the call toToString()
will automatically list the exception messages - #387 -
Lucene.Net.TestFramework.Analysis.MockTokenizer
: Pass theAttributeFactory
argument that is provided as per the documentation comment. Note this bug exists in Lucene 4.8.0, also. - #387 -
Lucene.Net.Analysis.Common.Tartarus.Snowball.Among
: FixedMethodObject
property to return private field instead of itself - #387 -
Lucene.Net.Document.CompressionTools
: Pass the offset and length to the underlyingMemoryStream
- #388 - Downgraded minimum required
Microsoft.Extensions.Configuration
version to 2.0.0 on .NET Standard 2.0 and 2.1
Improvements
- Updated code examples on website home page
- Show cross-OS examples of building
Directory
paths - Demonstrate where to put
using
statements - Removed LinqPad's
Dump()
method and replaced withConsole.WriteLine()
for clarity - Fixed syntax error in initialization example of
MultiPhraseQuery
- Show cross-OS examples of building
- Upgraded NuGet dependency J2N to 2.0.0-beta-0010
- Upgraded NuGet dependency ICU4N to 60.1.0-alpha.353
- Upgraded NuGet dependency Morfologik.Stemming to 2.1.7-beta-0001
- #344 - PERFORMANCE:
Lucene.Net.Search.FieldCacheImpl
: Removed unnecessary dictionary lookup - #352 - Added Azure DevOps tests for x86 on all platforms
- #348 - PERFORMANCE: Reduced
FieldCacheImpl
casting/boxing - #355 - Setup nightly build (https://dev.azure.com/lucene-net/Lucene.NET/_build?definitionId=4)
- PERFORMANCE:
Lucene.Net.Util.Automaton.SortedInt32Set
: Removed unnecessaryIEquatable<T>
implementations and convertedFrozenInt32Set
into a struct. - PERFORMANCE:
Lucene.Net.Util.Bits
: Removed unnecessaryGetHashCode()
method fromMatchAllBits
andMatchNoBits
(didn't exist in Lucene) Lucene.Net.Util.Counter
: Changed Get() to Value property and added implicit operator.- #361 - Make
CreateDirectory()
method virtual so that derived classes can provide their ownDirectory
implementation, allowing for benchmarking of customDirectory
providers (e.q LiteDB) - #346, #383 - PERFORMANCE: Change del...
v4.8.0-beta00012
This release contains important bug fixes and performance enhancements.
Known Issues
-
After installation, when upgrading from versions of Lucene.Net 4.8.0-beta00009 or higher may require a restart of all instances of Visual Studio in order to reload the code analysis analyzer.
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Breaking Changes
Lucene.Net.Facet
: RenamedLRUHashMap
>LruDictionary
. Changed all members to be virtual to allow users to provide their own LRU cache.Lucene.Net.Facet.FacetsConfig
: RemovedProcessSSDVFacetFields
from public API (as was done in Lucene), avoid lock (this)Lucene.Net.Facet.TaxonomyReader
: ChangedDoClose()
toDispose(bool)
and implemented proper dispose pattern. Avoid lock (this).Lucene.Net.Facet.WriterCache
: RenamedNameInt32CacheLRU
>NameIntCacheLru
,NameHashInt32CacheLRU
>NameHashInt32CacheLru
. Refactored to utilize a generic type internally using composition to avoid boxing/unboxing without exposing the generic closing type publicly. Added publicINameInt32CacheLru
as a common interface betweenNameIntCacheLru
andNameHashInt32CacheLru
.Lucene.Net.Facet.Taxonomy.TaxonomyReader
: RestructuredChildrenIterator
intoChildrenEnumerator
Lucene.Net.Facet.Taxonomy.CategoryPath
: ChangedFullPathLength
from a method to a propertyLucene.Net.Facet.DrillSideways
: ChangedScoreSubDocsAtOnce
from a method to a propertyLucene.Net.Facet
: RefactoredOrdAndValue
into a generic struct that can be used in bothTopOrdAndSingleQueue
andTopOrdAndInt32Queue
. Added Insert method toUtil.PriorityQueue<T>
to allow adding value types without reading the previous value for reuse.Lucene.Net.Analysis.Common.Miscellaneous.CapitalizationFilter
: Changed default behavior to use invariant culture instead of the current thread's culture to match Lucene, which seems more natural when using filters inside of analyzers. This also fits more in line with how other filters are selected.- #279 -
Lucene.Net.Analysis.Compound.Hyphenation.TernaryTree
: RenamedIterator
>Enumerator
,Keys()
>GetEnumerator()
- #279 -
Lucene.Net.Benchmarks.ByTask.Feeds.DirContentSource
: RenamedIterator
>Enumerator
Bugs
- #269 - Removed cast from
NGramTokenAnymousInnerClassHelper::IsTokenChar(int)
that was causing surrogate pairs to fail in theTestUTF8FullRange()
tests ofNGramTokenizerTest
andEdgeNGramTokenizerTest
- Fixed potential issue with
ArgumentException
s being thrown fromchar.ConvertToUtf32(string, int)
by reverting back toCodePointAt()
method inTestCharTokenizers.TestCrossPlaneNomalization()
. Lucene.Net.QueryParser.Surround.Query.ComposedQuery::MakeLuceneSubQueriesField()
: Added missing using block on enumerator- #296 - Fixed surrogate pair and culture-sensitivity issues with many analyzers.
Lucene.Net.Analysis.Common
: Fixed classes that were originally using invariant culture to do so again. J2N'sCharacter
class default is to use the current culture, which had changed from the priorCharacter
class from Lucene.Net.Support that used invariant culture. FixesTestICUFoldingFilter::TestRandomStrings()
.- Lucene.Net.ICU: Fixed ThaiWordBreaker to account for surrogate pairs. Also added locking to help with thread safety. Note that the class is still not completely thread-safe, but this patch fixes the behavior.
Lucene.Net.Spatial.Util.ShapeFieldCache
: Removed unnecessary array allocationLucene.Net.TestFramework
: FixedLineFileDocs
to read byte by byte the same way that Lucene does, except using aBufferedStream
to improve performance.Lucene.Net.TestFramework
: FixedNightlyAttribute
,WeeklyAttribute
,AwaitsFixAttribute
, andSlowAttribute
so they work at the class levelLucene.Net.Analysis.Icu.Segmentation.ICUTokenizer
: Corrected call toICU4N.UChar.IsWhiteSpace()
rather thanSystem.Char.IsWhiteSpace()
, which may return different results.Lucene.Net.TestFramework.Search.SearchEquivalenceTestBase
: Fixed exception when usingOpenBitSet.FastGet()
instead ofOpenBitSet.Get()
, since the size of the bit set is unknown.Lucene.Net.Index.DocumentsWriterFlushControl
: Fixed issue due to misbehaving locking on Monitor.TryEnter(), the code was restructured to disallow any thread that doesn't have a lock intoInternalTryCheckoutForFlush()
so the threads do note compete for a lock.- #274 -
Lucene.Net.Facet
: Fixed null reference exception inDrillSidewaysScorer
from patch in Lucene 4.10.4 https://issues.apache.org/jira/browse/LUCENE-6001 Lucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache
: Fixed locking onDispose()
method and made it safe to call dispose multiple times- Reviewed and added asserts that existed in Lucene and were missing in Lucene.NET. Effectively, this meant we were missing several test conditions that have now been put into place.
Lucene.Net.ICU
: Added locking toICUTokenizer
to only allow a single thread to manipulate theBreakIterator
at a time. This is a temporary fix to get the tests to pass until a solution is found for makingBreakIterator
threadsafe.- #332 -
Lucene.Net.Replicator
: Fixed an issue inIndexInputStream
that meant the read method could return a number larger than what was passed in for read count or what the buffer could hold, it should instead return the total number of bytes that was read into the buffer, which logically can't be bigger than the buffer it self. Lucene.Net.Tests.Index.TestIndexWithThreads::TestRollbackAndCommitWithThreads()
: Must catch and ignoreAssertionException
, as was done in LuceneLucene.Net.Search.TopScoreDocCollector
: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestSearchAfter::TestQueries()
,TestTopDocsMerge::TestSort_1()
,TestTopDocsMerge::TestSort_2()
.Lucene.Net.Sandbox.Queries.SlowFuzzyTermsEnum
: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestTokenLengthOpt()
.Lucene.Net.Search.FuzzyTermsEnum
: Disabled optimizations forAccept()
method on .NET Framework because of float comparison failures on x86 in Release mode. FixesTestTokenLengthOpt()
.- Fixed several references to
J2N.BitConversion
that were calling the overload that normalizes NaN when they should have been calling the raw bit conversion instead (as was done in Lucene). - #323 -
Lucene.Net.Configuration
: Removed theIConfigurationRoot
interface from theConfigurationRoot
class when targeting a version ofMicrosoft.Extensions.Configuration
less than 2.0. This will allow the end user to upgradeMicrosoft.Extensions.Configuration
seamlessly to versions 2.0 or higher. - #286 -
Lucene.Net.CodeAnalysis
: Separated CSharp and VisualBasic into different assemblies to prevent cross-language dependency issues when using analyzers
Improvements
- #261 PERFORMANCE - Fixed
FSTTester
to delete while iterating forward instead of using.ElementAt()
to iterate in reverse, which takes about 3x longer - #261 PERFORMANCE -
Lucene.Net.Facet.Taxonomy.WriterCache.NameInt32CacheLRU
: Changed fromDictionary
toConcurrentDictionary
so we can delete items from the cache while forward iterating through it. - #261 PERFORMANCE -
Lucene.Net.Index.FieldInfos
: ChangedBuilder.FieldInfo()
method toTryGetFieldInfo()
to optimize check for value - Upgraded NuGet dependency J2N to 2.0.0-beta-0009
- Upgraded NuGet dependency ICU4N to 60.1.0-alpha.352
- Upgraded NuGet dependency Morfologik.Stemming to 2.1.6-beta-0007
- #261 PERFORMANCE - Use J2N's
ICollection<T>.ToArray()
extension method that usesICollection<T>.CopyTo()
, which takes precedence over the LINQIEnumerable<T>.ToArray()
extension method. Benchmarks show about a 1/3 increase in performance. - #261 PERFORMANCE -
Lucene.Net.Support.IO.FileSupport::CreateTempFile()
: Optimized the check for invalid characters Directory.Build.props
: Disabled warnings for features that require .NET Standard 2.1- #261 PERFORMANCE - Eliminated several calls to
FirstOrDefault()
,LastOrDefault()
,Skip()
,First()
, andLast()
Lucene.Net.Support.ListExtensions
: Factored out BinarySearch in favor of implement...
v4.8.0-beta00011
This release contains a critical patch for .NET Framework users that use
Microsoft.Extensions.Configuration
higher than version 1.1.2. See #311 for details.
This release contains impactful performance enhancements.
Known Issues
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Breaking Changes
- PERFORMANCE:
Lucene.Net.Analysis.Compound
: Changed protected m_tokens field fromLinkedList<T>
toQueue<T>
for better throughput
Bugs
- #311 -
Lucene.Net.Configuration
: RemovedIConfigurationBuilder
implementation that prevents .NET Framework users from being able to upgrade to a higher version than 1.1.2.
Improvements
- PERFORMANCE:
Lucene.Net.TestFramework
: RevertedBaseTermVectorsFormatTestCase
to use the original 5000 iterations instead of 500. RevertedTestUtil.RandomSimpleString(Random)
to default to a maximum string length of 10 instead of 20, which was slowing down several tests. - PERFORMANCE:
Lucene.Net.TestFramework
: RefactoredAssert
class to use custom comparisons for all members, since NUnit'sAssert
implementation uses very slow fluent expressions to do comparisons, which are not practical to use inside of tight loops. - PERFORMANCE: Replaced
LinkedList<T> with
Queue` where there is a performance advantage - PERFORMANCE: Reduced memory allocations of
CaseInsensitiveComparer
by using its singleton static property - PERFORMANCE: Lucene.Net.Util.RamUsageEstimator: Switched back to
System.Collections.Generic.Dictionary
because indexer of J2N's dictionary is slower - PERFORMANCE:
Lucene.Net.TestFramework.Util.Fst.FSTTester
: UseSystem.Collections.Generic.Dictionary
for better performance - PERFORMANCE: Switched all remaining tests from using
NUnit.Framework.Assert
toLucene.Net.TestFramework.Assert
, which are several orders of magnitude faster - PERFORMANCE:
Lucene.Net.Facet
: Optimized DirectoryTaxonomyReader by reducing locking, removing unnecessary casts, and usingLazyInitializer
for the taxonomy array initialization - PERFORMANCE:
Lucene.Net.Tests.Analysis.Common
: Changed HunspellStemmerTestBase
to use more optimized assert to compare arrays - SWEEP: Consolidated empty array creation code so it is more DRY
- Upgraded to C# LangVersion 8.0
Lucene.Net.Tests.Support.TestApiConsistency
: Added regex filter to exclude public fields from the scan
v4.8.0-beta00010
This release contains impactful performance enhancements.
Known Issues
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Breaking Changes
- PERFORMANCE:
Lucene.Net.Suggest
: Narrowed return type ofContexts
property fromIEnumerable<BytesRef>
toICollection<BytesRef>
to improve performance for certain operations Lucene.Net.TestFramework.Util.LuceneTestCase
: Refactored to correctly load static fields and converted them into static properties. The names were changed to match the conventions of .NET properties.- #261 PERFORMANCE: Changed to use
BitSet
instead ofBitArray
so we don't have to resort to slow extension methods for certain operations
Bugs
Lucene.Net.Support.Collections::ToString()
: Fixed overloads to write "null" when the collection passed isnull
rather than throw an exception- #267 -
Lucene.Net.Codecs
: Fixed testing condition forBaseTermVectorsFormatTestCase
onTermVectorsReaders
by throwingInvalidOperationException
- #301 - After upgrading NuGet package dependencies for NUnit to 3.12.0, NUnit3TestAdapter to 3.16.1, and Microsoft.NET.Test.Sdk to 16.6.1, several false positives were noted due to invalid try-catch logic in tests
Improvements
- Added missing documentation table of contents links for lucene-cli, Lucene.Net.Demo, Lucene.Net.Queries, Lucene.Net.QueryParser, Lucene.Net.Replicator, Lucene.Net.Sandbox, Lucene.Net.Spatial, and Lucene.Net.Suggest.
- #261 PERFORMANCE: - Removed all calls to
Type.GetTypeInfo()
and call properties/methods on theType
instance directly. - #261 PERFORMANCE: -
Lucene.Net.Util.AttributeSource
: Create built-in attributes directly rather than usingActivator.CreateInstance()
- #261 PERFORMANCE: -
Lucene.Net.Util.AttributeSource
: Optimized creation of string to identify attribute type based on attribute interface name - #295 PERFORMANCE: -
Lucene.Net.TestFramework
: Added overloads ofAssert.AreEqual
for collections where J2N's aggressive mode can be switched off - #295 PERFORMANCE: -
Lucene.Net.TestFramework
: Replaced overloads fromNUnit.Framework.CollectionAssert
with optimized implementations using J2N comparers - #295 PERFORMANCE: -
Lucene.Net.TestFramework
: Compile expensive string concatenation out of the release build by usingSystem.Diagnostics.Debug.Assert
- #295 PERFORMANCE: -
Lucene.Net.Util.Automaton
: FixedState
class to initialize and trim more efficiently.
Fixes the performance ofLucene.Net.Util.Automaton.TestBasicOperations::TestEmptyLanguageConcatenate()
. - #295, #261: PERFORMANCE: - Reduced zero length array allocations
- #295, #261: PERFORMANCE: - Reduced zero length collection allocations
- #295, #261: PERFORMANCE: -
Lucene.Net.TestFramework.Search.RandomSimilarityProvider::ToString()
: UseStringBuilder
for better efficiency - #295, #261: PERFORMANCE: -
Lucene.Net.TestFramework.Util.LuceneTestCase
: Cache codecType and similarityName as strings so they don't have to be regenerated for each test - #295, #261: PERFORMANCE: -
Lucene.Net.TestFramework
: UseAssert.IsFalse()
rather thanAssert.That()
- #295, #261: PERFORMANCE: - Reduced calls to LINQ methods and expressions.
- #295, #261: PERFORMANCE: -
Lucene.Net.Index.Term
: Optimized equality checking - #295, #261: PERFORMANCE: -
Lucene.Net.Util
(BytesRef
+CharsRef
): ImplementedIEquatable<T>
- #295, #261: PERFORMANCE: -
Lucene.Net.Tests.Index.TestDocumentsWriterDeleteQueue
: Updated comparisons to reduce memory allocations - #295, #261: PERFORMANCE: -
Lucene.Net.TestFramework
: ChangedConcurrentMergeSchedulerFactories.Values
to only return theTaskMergeScheduler
rarely, since it is no longer a default setting and is slowing down tests - #295, #261: PERFORMANCE: - Added some aggressive inlining
Lucene.Net.TestFramework
: Added overloads ofAssert.Throws
to supply messagesLucene.Net.TestFramework
: Use theMicrosoft.Extensions.Configuration.EnvironmentVariables
provider instead of our custom one.Lucene.Net.Analysis.Tokenizer
: Allow enabling "asserts" for testingazure-pipelines.yml
: Changed to use pipeline caching instead of build caching for better performanceLucene.Net.Tests
(core) - Rearranged how test projects are split so parallel jobs can process them faster- Fixed up leading whitespace to always use spaces instead of tabs
- Updated NuGet package dependency for J2N to 2.0.0-beta-0008
Lucene.Net.Tests.Analysis.Phonetic.Language.Bm.PhoneticEnginePeformanceTest::Test()
: Changed to useStopwatch
for more accurate timingLucene.Net.Tests
: Changed private/internal member variables to consistently use camelCase instead of PascalCase- SWEEP: Changed all properties to use expression style syntax, reordered to put get before set, and changed all backing field names back to their original without the "_Renamed" suffix
- SWEEP: Removed fully-qualified exceptions and added using directives instead
- PERFORMANCE: Pre-compile and statically cache regular expressions
- PERFORMANCE:
Lucene.Net.Analysis
: Removed unnecessary allocations caused by callingToString()
rather than passing theICharTermAttribute
directly - PERFORMANCE:
Lucene.Net.Analysis.Common
: Slight optimization ofToUpper
andToLower
methods, which are used in several analyzers - SWEEP: Removed .NET Standard 1.x/.NET Core 1.x support from all project files
- SWEEP: Removed unused dependencies for .NET Framework
- PERFORMANCE:
Lucene.Net.Codecs.SimpleText
: Usingdecimal
is 30% faster than usingBigInteger
for addition and subtraction - PERFORMANCE: Applied
Slow
andNightly
attributes where applicable for better testing performance Lucene.Net.TestFramework
: Added logging of system properties to the output of tests- PERFORMANCE:
Lucene.Net.Util.ArrayUtil::GetNaturalComparer<T>()
: Use statically cached singleton instance of comparer - Upgraded NuGet dependency ICU4N to 60.1.0-alpha.351
New Features
Lucene.Net.TestFramework.RandomExtensions
: Added missing overload ofNextInt64(long)
to choose only max upper bound- Added a global Lucene.Net.Diagnostics.Debugging.AssertsEnabled static property that can be used to toggle "asserts" on and off in the release build, similar to how it works in Java. The setting can be injected by end users with the "assert" system property (which is a boolean).
Lucene.Net.TestFramework
: Completed implementation ofNightly
,Weekly
,AwaitsFix
andSlow
attributes
v4.8.0-beta00009
Known Issues
-
The
lucene-cli
tool requires anappsettings.json
file, but none was shipped. Upon runninglucene
on the command line, the following error will be presented:F:\Projects\lucenenet>lucene Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'. at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named
appsettings.json
to the location specified in the error message with opening and closing brackets will prevent the exception.appsettings.json
{ }
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
-
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Bugs
- #294 - Fixed the description for Lucene.NET 2.9.4 in the API documentation.
Improvements
- #260 - Changed culture-sensitivity of function-based queries to use the invariant culture when parsing/formatting values, to match the Lucene implementation.
- #266 - Added documentation to show example usage of injecting custom codecs, doc values formats, and postings formats with pure DI or using a dependency injection container. An demo project named Lucene.Net.Tests.TestFramework.DependencyInjection has also been provided as an example of using
Microsoft.Extensions.DependencyInjection
to provide custom codecs to the test environment so they can be used during testing of other components. - #295 - PERFORMANCE Fixed boxing issue with NUnit asserts when passing common primitive types. This significantly improves the performance of running tests.
- Upgraded to ICU4N 60.1.0-alpha.350 and refactored the collation features of
Lucene.Net.ICU
to utilize the newUCultureInfo
class. - #282 - Restructured API documentation so each assembly has its own "mini-site" to fix x-ref linking issue, which fixes the TOC and breadcrumbs.
- #282 - Documented the docs and website building procedures.
- #282 - Fixed links on the download page and added missing packages (https://lucenenet.apache.org/download/version-4.8.0-beta00009.html).
New Features
- #254 - Implementation of "System Properties" which allow users to supply configuration values to the test framework and lucene-cli using configuration files or environment variables.
v4.8.0-beta00008
This release contains performance improvements and bug fixes.
The
Lucene.Net.Support
namespace has been phased out, and most of the components that previously existed there have been refactored and moved to J2N, a library to fill in gaps in functionality between the JDK and .NET.
Known Issues
- J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on
Xamarin.Android
which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Change Log
Breaking Changes
-
#239 - Changed
Append()
overloads ofOpenStringBuilder
to use startIndex/count rather than start/end to match the .NET convention -
#241 - Made public fields into properties in
ArabicStemmer
-
Lucene.Net.Collections
: Factored out unmodifiable methods and related classes in favor of J2N'sAsReadOnly()
extension methods -
SWEEP:
Lucene.Net.Support
: Factored outEquatable
,EquatableList
, andEquatableSet
, and replaced with collections from J2N -
SWEEP:
Lucene.Net.Support
: Factored outCollections.Equals()
andCollections.GetHashCode()
-
Lucene.Net.Support.Collections
: Factored outSwap()
andShuffle()
in favor of J2N's implementation -
Lucene.Net.Support
: Factored outIdentityComparer
,IdentityHashMap
, andIdentityHashSet
and usedJ2N.Runtime.CompilerServices.IdentityEqualityComparer
in conjunction with standardDictionary
andHashSet
types -
Factored out
TreeSet
andTreeDictionary
fromLucene.Net.Support
in favor ofJ2N.Collections.Generic.SortedSet
andJ2N.Collections.Generic.SortedDictionary
-
Lucene.Net.Support.PriorityQueue
: Factored out in favor of J2N's implementation -
Lucene.Net.Support
: RefactoredConcurrentHashSet
intoConcurrentSet
- a wrapper class that can be used to synchronize any set object (ordered or not), similar to how it was done in Java. Changed ordered concurrent set types back to the original type from Lucene. -
Lucene.Net.Search.FieldComparer
: ReplacedCompareTo()
calls withJCG.Comparer<T>.Default.Compare()
, factoring outLucene.Net.Support.SignedZeroComparer
in the process. -
Factored out
Character
,ICharSequence
,StringBuilderCharSequenceWrapper
,StringBuilderExtensions
,StringCharSequenceWrapper
, and mostStringExtensions
methods in favor of J2N's implementation -
Changed semantics of
CharTermAttribute.Append()
overloads to act more like the .NETStringBuilder
class.- The 3rd parameter was changed from an exclusive end index to a count.
- A
null
parameter on a single parameter overload will be a no-op instead of appending the text "null".
- A
null
parameter on a 3 parameter overload will be a no-op if bothstartIndex
andcount
are 0, otherwise it will throw anArgumentNullException
.
-
Lucene.Net.Support
: Factored outGeneralKeyedCollection
andAttributeItem
in favor ofJ2N.Collections.Generic.LinkedDictionary
-
SWEEP: Upgraded to account for breaking changes (
AsCharSequence()
andBitOperation
) in J2N. -
Lucene.Net.Support.Collections
: Factored outImplementsGenericInterface()
in favor of J2N's implementation -
Lucene.Net.Support.Collections
: Factored outSingleton()
method and used collection initializers instead -
Lucene.Net.Support.Collections
: RemovedNewSetFromMap
method and relatedSetFromMap
class -
Lucene.Net.Support
: AddedConcurrentHashSet
class for internal use, and factored out the useConcurrentSet
throughout the project -
Removed
Lucene.Net.Support.ConcurrentHashMapWrapper
from the public API and renamedConcurrentDictionary
-
Marked
ExceptionToClassNameConventionAttribute
,ExceptionToNetNumericConventionAttribute
,ExceptionToNullableEnumConventionAttribute
, andWritableArrayAttribute
internal -
Factored out
StringExtensions
-
Factored out
DictionaryExtensions
-
Factored out
LurchTable
in favor of J2N's implementation -
Lucene.Net.Support.Threading
: DeletedThreadLock
,DisposableThreadLocalProfiler
-
Factored out
Lucene.Net.Support.AssemblyExtensions
in favor of J2N's implementation -
Lucene.Net.Support
: Marked all types internal -
Moved
Lucene.Net.Support.SystemConsole
toLucene.Net.Util
namespace -
Lucene.Net.Support
: Moved ExceptionExtensions toLucene.Net.Util
namespace -
Lucene.Net.Support.ListExtensions
: MovedAddRange
,Sort
,TimSort
, andIntroSort
extension methods toLucene.Net.Util.ListExtensions
. -
Lucene.Net.Support.NumberFormat
: Moved toLucene.Net.Util
namespace -
Lucene.Net.TestFramework.Support.JavaCompatibility.AbstractBeforeAfterRule
: Moved fromLucene.Net.Support
namespace toLucene.Net.Util
-
Lucene.Net.TestFramework.Support
: Changed namespace ofApiScanTestBase
,CultureInfoSupport
, andExceptionSerializationTestBase
toLucene.Net.Util
-
Lucene.Net.Util.NumberFormat
,Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat
: Changed protected locale field to private, made property namedCulture
, and changed constructors and methods to use "culture" rather than "locale" for parameter names -
Lucene.Net.Support
: MovedSystemProperties
class toLucene.Net.Util
namespace -
Lucene.Net.Benchmark.Support
: MovedEnglishNumberFormatExtensions
toLucene.Net.Util
namespace
Bugs
- #237 - Fix duplicate
FragNum
value onHighlight.TextFragment
- #235 - Fixed broken links on API documentation home page
Lucene.Net.Search.BooleanClause::Equals(BooleanClause)
: Fixed potential null reference exception whenQuery
is set to null- LUCENENET-645 - Added missing call to
FileStream::Flush()
inFSIndexOutput::Flush()
that was preventing persistence to disk from occuring at the necessary phase - SWEEP: Pass Random instance to all
Shuffle()
method calls to ensure the same psuedo-random sequence is used based on the seed Lucene.Net.Index.DocumentsWriterDeleteQueue::DeleteSlice()
: Removed extraDebug.Assert()
statement that wasn't in Lucene which causedLucene.Net.Index.TestIndexWriterUnicode::TestEmbeddedFFFF()
test to fail when running in debug mode
Improvements
- #234 - Moved markdown converter to the
src/docs
folder - #235 - Updated API documentation so each release exists on its own URL
- #235 - Updated documentation converter to be more robust when dealing with namespace matching
- #239 - Replaced all private
IComparer<T>
implementations with the built in .NETComparer.Create()
factory method - #240 - Implemented
IAppendable
onOpenStringBuilder
- PERFORMANCE - Removed unnecessary memory copy operations from various
IndexReader
dependencies - LUCENENET-640 - PERFORMANCE - Factored out
WeakIdentityMap
in favor of .NET'sConditionalWeakTable
- SWEEP: Factored out LinkedHashMap in favor of J2N's LinkedDictionary
- Lucene.Net.Tests.Search.TestFieldCache::Test(): Simplified expression with LINQ query
Lucene.Net.Highlighter
,Lucene.Net.Tests.Spatial
: Swapped inLinkedHashSet
from J2N, like in the original Lucene implementation- SWEEP: Factored out
Lucene.Net.Support.HashMap
in favor ofJ2N.Collections.Generic.Dictionary
- SWEEP: Factored out
System.Collections.Generic.SortedDictionary
in favor ofJ2N.Collections.Generic.SortedDictionary
Lucene.Net.Analysis.Kuromoji.Dict.UserDictionary
: Swapped out C5TreeDictionary
forJ2N.Collections.Generic.SortedDictionary
- SWEEP: Changed
System.Collections.Generic.SortedSet
toJ2N.Collections.Generic.SortedSet
- SWEEP: Factored out C5's
TreeSet
- SWEEP: Swapped out
System.Collections.Generic.HashSet
forJ2N.Collections.Generic.HashSet
- SWEEP: Factored out
Arrays.AsList()
, which was causing both additional operational complexity and unnecessary memory allocations Lucene.Net.Util.Automaton.State
: ImplementedIEquatable<State>
, changed enumerator to a struct- Removed
FEATURE_HASHSET_CAPACITY
, since J2N now has the full .NET Core 3.x implementation with a capacity constructor Lucene.Net.Util.Fst.ListOfOutputs::Merge()
: Streamlined so we don't have so many castsLucene.Net.Util.Fst
: UseJ2N.Collections.List<T>
for the closing type ofOutputs<T>
to ensure the outputs can be compared for structural equalityLucene.Net.TestFramework.Search.AssertingScorer
: Changed to useConditionalWeakTable
/WeakDictionary
- LUCENENET-610, LUCENENET-640 - Reduced locking in
FieldCache
and its dependent classes - Added cross-framework compatibility for nullable attributes
- Changed project files to automatically generate
InternalsVisibleTo
attributes based onItemGroup/InternalsVisibleTo
elements Lucene.Net.Support.DictionaryExtensions
: OptimizedPut()
method, added guard clauses toPut
andPutAll
Lucene.Net.Support.DictionaryExtensions
: Factored outLoad()
andStore()
methods in favor of J2N's implementation- LUCENENET-642 -
Lucene.Net.Analysis.TokenStream
: Removed Reflection code that is used to force the end user to makeTokenStream
subclasses or theirIncrementToken()
method sealed - LUCENENET-643 - PERFORMANCE -
Lucene.Net.Support.IO.FileStreamExtensions::Read()
: Moved toStreamExtensions
class and optimized to read bytes in bulk instead of one byte at a time Lucene.Net.Tests.Index
(TestBagOfPositions
+TestBagOfPostings
): Fixed performance issue due to convertingint
tostring
andstring
to `...
v4.8.0-beta00007
This release contains impactful performance improvements and bug fixes.
NOTE: The
Lucene.Net.Support
namespace in the coreLucene.Net
assembly is being phased out for external use. Much of what used to be in this namespace has been made into first-class components and moved to J2N, a library to fill in gaps in functionality between the JDK and .NET. Please do not add any new dependencies onLucene.Net.Support
in the coreLucene.Net
library. Also, please open a new JIRA ticket if you have a dependency on a component fromLucene.Net.Support
that is not available in J2N.
This release drops support for .NET Standard 1.6 and adds support for .NET Standard 2.1.
Change Log
Breaking Changes
Lucene.Net.Codecs
(inLucene.Net
library) - Changed the following methods to properties with the same name for API consistencyLucene.Net.Codecs.DefaultCodecFactory::AvailableServices()
Lucene.Net.Codecs.DefaultDocValuesFormatFactory::AvailableServices()
Lucene.Net.Codecs.DefaultPostingsFormatFactory::AvailableServices()
Lucene.Net.Codecs.Codec::AvailableCodecs()
Lucene.Net.Codecs.DocValuesFormat::AvailableDocValuesFormats()
Lucene.Net.Codecs.PostingsFormat::AvailablePostingsFormats()
Lucene.Net.Analysis.Kuromoji
- Changed the following methods to properties namedInstance
for API consistencyLucene.Net.Analysis.Kuromoji.Dict.CharacterDefinition::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.ConnectionCosts::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.TokenInfoDictionary::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.UnknownDictionary::GetInstance()
Lucene.Net.Support.Collections::AddAll()
- Factored out in favor ofISet<T>.UnionWith()
Lucene.Net.Support.IdentityComparer::ctor()
- Factored out in favor of newDefault
static propertyLucene.Net.Support.DictionaryExtensions::EntrySet()
- Factored out becauseIDictionary<TKey, TValue>
is already enumerable and copying its contents to another data structure for the purpose of enumeration is wasteful and unnecessaryLucene.Net.Support.SetExtensions::AddAll()
- Factored out in favor ofUnionWith()
,Lucene.Net.Support.DictionaryExtensions.PutAll()
, orAddRange()
depending on collection typeLucene.Net.Support.HashMap<TKey, TValue>
- Changed behavior of indexer property to throw aKeyNotFoundException
if the key doesn't exist. The exception can already be avoided by using theTryGetValue()
method.Lucene.Net.Support.Buffer
- Factored out in favor ofJ2N.IO.Buffer
Lucene.Net.Support.ByteBuffer
- Factored out in favor ofJ2N.IO.ByteBuffer
Lucene.Net.Support.LongBuffer
- Factored out in favor ofJ2N.IO.LongBuffer
Lucene.Net.Support.Number
- Factored out the following methods in favor of their counterparts in J2NSignum()
>J2N.MathExtensions::Signum()
SingleToInt32Bits()
>J2N.BitConversion::SingleToInt32Bits()
SingleToRawInt32Bits()
>J2N.BitConversion::SingleToRawInt32Bits()
Int64BitsToDouble()
>J2N.BitConversion::Int64BitsToDouble()
DoubleToInt64Bits()
>J2N.BitConversion::DoubleToInt64Bits()
DoubleToRawInt64Bits()
>J2N.BitConversion::DoubleToRawInt64Bits()
ToBinaryString()
>J2N.IntegralNumberExtensions::ToBinaryString()
URShift()
>J2N.Numerics.BitOperationExtensions::TripleShift()
ToString(long, radix)
>J2N.IntegralNumberExtensions::ToString(long, radix)
BitCount()
>J2N.Numerics.BitOperationExtensions::PopCount()
NumberOfLeadingZeros()
>J2N.IntegralNumberExtensions::LeadingZeroCount()
NumberOfTrailingZeros()
>J2N.IntegralNumberExtensions::TrailingZeroCount()
RotateLeft()
>J2N.IntegralNumberExtensions::RotateLeft()
RotateRight()
>J2N.IntegralNumberExtensions::RotateRight()
Lucene.Net.Support.Number
- Removed the following methodsFlipEndian()
IsNumber()
ToInt64()
Lucene.Net.Support.MathExtensions
- Factored out the following methods in favor of their counterparts in J2NToDegrees()
>J2N.MathExtensions::ToDegrees()
ToRadians()
>J2N.MathExtensions::ToRadians()
Lucene.Net.Support.AtomicBoolean
- Factored out in favor ofJ2N.Threading.Atomic.AtomicBoolean
Lucene.Net.Support.AtomicInt32
- Factored out in favor ofJ2N.Threading.Atomic.AtomicInt32
Lucene.Net.Support.AtomicInt64
- Factored out in favor ofJ2N.Threading.Atomic.AtomicInt64
Lucene.Net.Support.AtomicObject
- Factored out in favor ofJ2N.Threading.Atomic.AtomicReference
Lucene.Net.Support.AtomicReferenceArray
- Factored out in favor ofJ2N.Threading.Atomic.AtomicReferenceArray
Lucene.Net.Support.Threading.ThreadClass
- Factored out in favor ofJ2N.Threading.ThreadJob
Lucene.Net.Support.Character
- Factored out the following methods in favor of their counterparts in J2NDigit()
ForDigit()
Lucene.Net.Support.StringTokenizer
- Factored out in favor ofJ2N.Text.StringTokenizer
Lucene.Net.Support.CultureContext
- Factored out in favor ofJ2N.Globalization.CultureContext
Lucene.Net.Support.IndexWriterConfigExtensions
- Moved toLucene.Net.Index.Extensions
namespaceLucene.Net.Documents.IndexableFieldExtensions
- Moved toLucene.Net.Documents.Extensions
namespaceLucene.Net.Documents.DocumentExtensions
- Moved toLucene.Net.Documents.Extensions
namespaceLucene.Net.Support.IResourceManagerFactory
- Moved toLucene.Net.Util
namespaceLucene.Net.Support.BundleResourceMangerFactory
Moved toLucene.Net.Util
namespace
Bugs
- LUCENENET-615 - PerFieldAnalyzerWrapper and PerFieldReuseStrategy must support null keys.
- LUCENENET-617 - Reverted changes introduced in pull #222 that caused deadlock in
Lucene.Net.Tests.Replicator.IndexAndTaxonomyReplicationClientTest::TestConsistencyOnExceptions()
. Lucene.Net.Tests.Analysis.Common/Analysis/Util/TestCharArrayMap_TestCharArrayMap()
- Was failing in Turkish, lowercasing must be done in the invariant culture to match LuceneLucene.Net.Tests.Util.Fst.TestFSTs::TestPrimaryKeys()
- Fixed sorting issue that was causing the test to fail with negative values- SWEEP: Corrected number-to-string and string-to-number conversions to use the invariant culture, as was the case in Lucene
- LUCENENET-622 -
Lucene.Net.Tests.Util.TestVersionComparer::TestVersions()
- Fixed version number conversion to use the invariant culture so the test consistently passes - LUCENENET-621 -
Lucene.Net.Tests.Search.TestSearchAfter::TestQueries()
- Fixed number-to-string conversion to use the invariant culture so the test consistently passes - `Lucene.Net.Tests.Benchmark.ByTask.Tasks.WriteLineDocTaskTest::TestMultiThreaded(): Added lock to synchronize write files. Also refactored test to assist future debugging efforts.
Lucene.Net.Analysis.Common
- Fixed to use invariant culture converting numbers and uppercasing/lowercasingLucene.Net.Benchmark
- Fixed to use invariant culture converting numbers and uppercasing/lowercasingLucene.Net.Facet
- Fixed to use invariant culture for number conversionLucene.Net.Misc
- Fixed to use invariant culture for number conversionLucene.Net.Suggest
- Fixed to use invariant culture for number conversionLucene.Net.Grouping
- Fixed to use invariant culture for number conversionLucene.Net.Spatial
- Fixed to use invariant culture for number conversionLucene.Net.Tests.Replicator
- Fixed issue with incorrect error message due to missingDispose()
callLucene.Net.Tests.Replicator.LocalReplicatorTest::TestObtainMissingFile()
- Added missing catch block forDirectoryNotFoundException
Lucene.Net.Support.Codecs
- Fixed initialization locking between parallel tasks. Also added guard clauses and inlined variable declarations.- #154 - Fix OpenBitSet.Union and .Xor methods.
- LUCENENET-618 - CRITICAL: Fixed
Lucene.Net.Store.NativeFSLockFactory
to be thread-safe on non-Windows operating systems - Fixed broken OS detection on .NET Framework
- LUCENENET-602 - Fixed generic stucts of LurchTable to function on Xamarin.iOS without throwing exceptions
Lucene.Net.TestFramework.Index.ThreadedIndexingAndSearchingTestCase
-WeakDictionary<TKey, TValue>
must be wrapped to make it thread safeLucene.Net.Codecs.Lucene45.Lucene45DocValuesConsumer::MISSING_ORD
- Correct value should be-1L
to match Lucene
Improvements
- azure-pipelines.yml: Added job to generate documentation
- Improved automated documentation converter accuracy
- Moved
Lucene.Net.Util.VirtualMethod
to its original location in theLucene.Net
assembly - Updated usage examples on home page of web site
- Changed categorization on API documentation to list by assembly rather than namespace
Lucene.Net.Analysis.Phonetic
- Create culture during static initialization to improve performanceLucene.Net.Index.CheckIndex
- Added guard clause to ensuredir
is not nullLucene.Net.Support.IO.FileSupport::CreateTempFile()
- Simplified error handling to use a when clause to only catch the relevant exceptions (improving thread safety)- SWEEP: CA1810 - Avoid static constructors (see [#224 (comment)](https://github.com/apache/luce...