Add Isolated Replicas feature to download index updates from remote backend

[aprudhomme]

Adds the isolated replicas feature. When enabled, the replica downloads index updates directly from the remote backend (S3). This feature is only usable if the index is started without a connection to the primary.

Config

isolatedReplicaConfig:
  enabled: true
  pollingIntervalSeconds: 30

• enabled: if the feature is enabled, default: false
• pollingIntervalSeconds: how long to wait before checking for a new index version, default: 120

CopyJobManager

The CopyJobManager interface was added to extract the gRPC specific logic out of the NrtReplicaNode. The existing logic was moved into the GrpcCopyJobManager and the RemoteCopyJobManager was created for copying from the remote backend. Methods:

• start: called at the end of NrtReplicaNode start
• newCopyJob: create copy job for use by Lucene nrt replication system
• finishNRTCopy: called after a copy job finishes

NrtDataManager

This class now manages starting a file download from the remote backend. It also keeps track of the currently active nrt point and point timestamp. When a copy job finishes successfully, the point state and timestamp are updated.

InputStreamDataInput

The base Lucene file copy api uses the DataInput interface. The InputStreamDataInput wraps the file InputStream so it can be used directly.

RemoteBackend

The remote backend is now able to download an index file directly, presenting it as an InputStream. Getting the point state from the backend now also returns the timestamp associated with that point.

CopyOneFile

This is a Lucene class that we are redefining and shadowing with the classpath. Since the definition was specific to gRPC, I changed this class into an interface. The previous implementation is now GrpcCopyOneFile. A new implementation has been added called StreamCopyOneFile. This is closer to the stock implementation and hopefully we can unhack this code at some point.

Note: many of the tests were generated by AI