Add a Time To Live (TTL) to the inventory cache (#605)

Normally, the digest of the plugin inventory is checked every time the
DB needs to be read.  Although this is faster than downloading the DB
each time, there is still between a 2.5s to 4.5s delay in checking the
digest. This makes every `plugin search` and `plugin group search`
command slower.  It also makes some `plugin install --group` commands
much slower when all plugins are available in the cache, since the
installation of each plugin in the group causes a digest check, even
if the plugin binary is already in the cache.

This commit provides a type of Time To Live for the DB. This means that
when within the TTL, the digest is not checked and the DB is considered
valid. The time the digest was last checked is stored as the
modification time (mtime) of the digest file. So, whenever the DB needs
to be read, if the TTL has not expired since the last time the digest
was verified, the DB is directly read from cache; if the TTL has
expired, the digest is checked and the DB downloaded if required.

On a "plugin source update" or "plugin source init" the TTL is ignored
and the digest automatically checked.  This is important as either of
these commands usually modify the URI of the plugin discovery and
therefore invalidates the DB.

Note that for any discovery source added through the
TANZU_CLI_ADDITIONAL_PLUGIN_DISCOVERY_IMAGES_TEST_ONLY variable, the
digest is checked every time (this TTL feature does not apply); this is
because there is no way to know if current cache was downloaded from
the same URIs as what is currently in the variable.  This is different
than for "plugin source update/init" because these two commands can
actively force a cache refresh but changing the
TANZU_CLI_ADDITIONAL_PLUGIN_DISCOVERY_IMAGES_TEST_ONLY variable cannot
do that.

The value of the cache TTL is of 30 minutes.  This means that it can
take a CLI up to 30 minutes to notice the publication of new plugins in
the central repository.  If for some reason a user wants to force a
refresh immediately, they can simply do `tanzu plugin source init` (or
`tanzu plugin source update default -u ...` if the discovery source is
not the default central repository). The TTL value can be overriden
using the environment variable TANZU_CLI_PLUGIN_DB_DIGEST_TTL_SECONDS.

* Store URI in the main plugin inventory digest file

To allow the CLI to know if a plugin inventory cache has become invalid
because it represents a different URI, we now store the OCI image URI
inside the main digest file.  Whenever the TTL is checked to know if
the digest must be checked, the URI is also checked; if the URI has
changed, the digest must be checked.

Signed-off-by: Marc Khouzam <[email protected]>

docs/dev/centralized_plugin_discovery.md

@@ -34,6 +34,22 @@ The CLI then uses SQLite queries to obtain the desired information about one
 or more plugins. Using that information, the CLI can either list plugins to the
 user, or proceed to install plugins.
 
+### Plugin Inventory Cache
+
+Pulling the plugin inventory OCI image takes a noticeable amount of time for the
+CLI.  Therefore, to keep the CLI quick and responsive, the plugin inventory DB is
+cached under `$HOME/.cache/tanzu/plugin_inventory/default` and the CLI uses this
+cache whenever it needs to query the DB.  This cache has a time-to-live of 30 minutes,
+which means that a CLI could at most require 30 minutes to become aware of new
+plugins published to the central repository of plugins.
+
+If for some reason a user wants to force an immediate refresh of the plugin
+inventory cache, they can run the `tanzu plugin source init` command.
+To refresh the plugin inventory DB, the CLI first compares the digest of the
+remote OCI image with the digest stored in the cache; if the digests match,
+the DB need not be downloaded and is considered to have been refreshed, which
+resets the TTL.
+
 ### Plugin Groups
 
 Plugin groups define a list of plugin/version combinations that are applicable

pkg/command/discovery_source.go

@@ -80,6 +80,7 @@ func newUpdateDiscoverySourceCmd() *cobra.Command {
 	var updateDiscoverySourceCmd = &cobra.Command{
 		Use:   "update SOURCE_NAME --uri <URI>",
 		Short: "Update a discovery source configuration",
+		Long:  "Update a discovery source configuration and refresh the plugin inventory local cache",
 		// We already include the only flag in the use text,
 		// we therefore don't show '[flags]' in the usage text.
 		DisableFlagsInUseLine: true,
@@ -101,6 +102,18 @@ func newUpdateDiscoverySourceCmd() *cobra.Command {
 				return err
 			}
 
+			// Check the discovery source *before* we save it in the configuration
+			// file. This way, if the discovery source is invalid, we don't save it.
+			// NOTE: We cannot first save and then revert the change if the discovery
+			// source is invalid because it is possible that the check of the discovery
+			// will fail with a call to log.Fatal(), which will exit the program before
+			// we can revert the change; this happens when the discovery source is
+			// not properly signed.
+			err = checkDiscoverySource(newDiscoverySource)
+			if err != nil {
+				return err
+			}
+
 			err = configlib.SetCLIDiscoverySource(newDiscoverySource)
 			if err != nil {
 				return err
@@ -155,6 +168,7 @@ func newInitDiscoverySourceCmd() *cobra.Command {
 	var initDiscoverySourceCmd = &cobra.Command{
 		Use:   "init",
 		Short: "Initialize the discovery source to its default value",
+		Long:  "Initialize the discovery source to its default value and refresh the plugin inventory local cache",
 		Args:  cobra.MaximumNArgs(0),
 		// There are no flags
 		DisableFlagsInUseLine: true,
@@ -165,7 +179,10 @@ func newInitDiscoverySourceCmd() *cobra.Command {
 				return err
 			}
 
-			// Refresh the inventory DB
+			// Refresh the inventory DB as the URI may have changed.
+			// It is also useful to refresh the DB even if the URI has not changed;
+			// this way, a user can force a refresh of the DB by running this command
+			// without waiting for the TTL to expire.
 			if discoverySource, err := configlib.GetCLIDiscoverySource(config.DefaultStandaloneDiscoveryName); err == nil {
 				// Ignore any failures since the real operation
 				// the user is trying to do is set the config
@@ -192,14 +209,18 @@ func createDiscoverySource(dsName, uri string) (configtypes.PluginDiscovery, err
 			Name:  dsName,
 			Image: uri,
 		}}
-	err := checkDiscoverySource(pluginDiscoverySource)
-	return pluginDiscoverySource, err
+	return pluginDiscoverySource, nil
 }
 
 // checkDiscoverySource attempts to access the content of the discovery to
-// confirm it is valid
+// confirm it is valid; this implies refreshing the DB.
 func checkDiscoverySource(source configtypes.PluginDiscovery) error {
-	discObject, err := discovery.CreateDiscoveryFromV1alpha1(source)
+	// If the URI has changed, the cache will be refreshed automatically.  However, if the URI has not changed,
+	// normally the TTL would be respected and the cache would not be refreshed.  However, we choose to pass
+	// the WithForceRefresh() option to ensure we refresh the DB no matter if the TTL has expired or not.
+	// This provides a way for the user to force a refresh of the DB by running "tanzu plugin source init/update"
+	// without waiting for the TTL to expire.
+	discObject, err := discovery.CreateDiscoveryFromV1alpha1(source, discovery.WithForceRefresh())
 	if err != nil {
 		return err
 	}

pkg/command/discovery_source_test.go

@@ -10,12 +10,14 @@ import (
 	"path/filepath"
 	"strings"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 
 	"github.com/vmware-tanzu/tanzu-cli/pkg/common"
 	"github.com/vmware-tanzu/tanzu-cli/pkg/config"
 	"github.com/vmware-tanzu/tanzu-cli/pkg/constants"
+	"github.com/vmware-tanzu/tanzu-cli/pkg/plugininventory"
 	configlib "github.com/vmware-tanzu/tanzu-plugin-runtime/config"
 	configtypes "github.com/vmware-tanzu/tanzu-plugin-runtime/config/types"
 	"github.com/vmware-tanzu/tanzu-plugin-runtime/log"
@@ -29,13 +31,69 @@ func Test_createDiscoverySource(t *testing.T) {
 	assert.NotNil(err)
 	assert.Equal(err.Error(), "discovery source name cannot be empty")
 
-	// With an invalid image
+	// With an invalid image, no error are thrown, as there is no verification in createDiscoverySource()
 	pd, err := createDiscoverySource("fake-oci-discovery-name", "test.registry.com/test-image:v1.0.0")
-	assert.NotNil(err)
-	assert.Contains(err.Error(), "unable to fetch the inventory of discovery 'fake-oci-discovery-name' for plugins")
+	assert.Nil(err)
 	assert.NotNil(pd.OCI)
 	assert.Equal(pd.OCI.Name, "fake-oci-discovery-name")
 	assert.Equal(pd.OCI.Image, "test.registry.com/test-image:v1.0.0")
+
+	// With a valid image
+	pd, err = createDiscoverySource(config.DefaultStandaloneDiscoveryName, constants.TanzuCLIDefaultCentralPluginDiscoveryImage)
+	assert.Nil(err)
+	assert.NotNil(pd.OCI)
+	assert.Equal(pd.OCI.Name, config.DefaultStandaloneDiscoveryName)
+	assert.Equal(pd.OCI.Image, constants.TanzuCLIDefaultCentralPluginDiscoveryImage)
+}
+
+// test that checkDiscoverySource() will download the DB and digest file
+// and that the digest file is updated even though the TTL has not expired.
+func Test_checkDiscoverySource(t *testing.T) {
+	assert := assert.New(t)
+
+	dir, err := os.MkdirTemp("", "test-source")
+	assert.Nil(err)
+	defer os.RemoveAll(dir)
+
+	common.DefaultCacheDir = dir
+
+	// Setup a valid image
+	pd, err := createDiscoverySource(config.DefaultStandaloneDiscoveryName, constants.TanzuCLIDefaultCentralPluginDiscoveryImage)
+	assert.Nil(err)
+	assert.NotNil(pd.OCI)
+	assert.Equal(pd.OCI.Name, config.DefaultStandaloneDiscoveryName)
+	assert.Equal(pd.OCI.Image, constants.TanzuCLIDefaultCentralPluginDiscoveryImage)
+
+	// Test that when we check the discovery image, the DB gets filled
+	err = checkDiscoverySource(pd)
+	assert.Nil(err)
+
+	// Check that the digest file was immediately created
+	pluginDataDir := filepath.Join(common.DefaultCacheDir, common.PluginInventoryDirName, config.DefaultStandaloneDiscoveryName)
+	matches, _ := filepath.Glob(filepath.Join(pluginDataDir, "digest.*"))
+	assert.Equal(1, len(matches))
+
+	// Get the timestamp of the digest file
+	digestFile := matches[0]
+	originalDigestFileStat, err := os.Stat(digestFile)
+	assert.Nil(err)
+
+	// Check that the DB was downloaded
+	dbFile := filepath.Join(pluginDataDir, plugininventory.SQliteDBFileName)
+	_, err = os.Stat(dbFile)
+	assert.Nil(err)
+
+	// check the discovery source again and make sure the digest file is immediately updated
+	// even though the TTL has not expired.
+	// sleep for 1 seconds to make sure the timestamp of the digest file is different
+	time.Sleep(1 * time.Second)
+	err = checkDiscoverySource(pd)
+	assert.Nil(err)
+	newDigestFileStat, err := os.Stat(digestFile)
+	assert.Nil(err)
+
+	// Check that the digest file was updated
+	assert.True(newDigestFileStat.ModTime().After(originalDigestFileStat.ModTime()))
 }
 
 // Test_createAndListDiscoverySources test 'tanzu plugin source list' when TANZU_CLI_ADDITIONAL_PLUGIN_DISCOVERY_IMAGES_TEST_ONLY has set test only discovery sources
@@ -294,6 +352,16 @@ func Test_updateDiscoverySources(t *testing.T) {
 				if spec.expectedFailure {
 					// Check we got the correct error
 					assert.Contains(err.Error(), spec.expected)
+
+					// Check the original discovery source was not updated
+					discoverySources, err := configlib.GetCLIDiscoverySources()
+					assert.Nil(err)
+					assert.Equal(1, len(discoverySources))
+
+					ds := discoverySources[0]
+					assert.NotNil(ds.OCI)
+					assert.Equal(config.DefaultStandaloneDiscoveryName, ds.OCI.Name)
+					assert.Equal("test/uri", ds.OCI.Image)
 				} else {
 					got, err := io.ReadAll(b)
 					assert.Nil(err)

pkg/constants/env_variables.go

@@ -42,4 +42,7 @@ const (
 
 	// Control the different ActiveHelp options
 	ConfigVariableActiveHelp = "TANZU_ACTIVE_HELP"
+
+	// Change the default value of the plugin inventory cache TTL
+	ConfigVariablePluginDBCacheTTL = "TANZU_CLI_PLUGIN_DB_CACHE_TTL_SECONDS"
 )

pkg/discovery/interface.go

@@ -41,24 +41,37 @@ type GroupDiscovery interface {
 // DiscoveryOpts used to customize the plugin discovery process or mechanism
 type DiscoveryOpts struct {
 	UseLocalCacheOnly       bool // UseLocalCacheOnly used to pull the plugin data from the cache
+	ForceRefresh            bool // ForceRefresh used to force a refresh of the plugin data
 	PluginDiscoveryCriteria *PluginDiscoveryCriteria
 	GroupDiscoveryCriteria  *GroupDiscoveryCriteria
 }
 
 type DiscoveryOptions func(options *DiscoveryOpts)
 
+// WithUseLocalCacheOnly used to get the plugin inventory data without first refreshing the cache
+// even if the cache's TTL has expired
 func WithUseLocalCacheOnly() DiscoveryOptions {
 	return func(o *DiscoveryOpts) {
 		o.UseLocalCacheOnly = true
 	}
 }
 
+// WithForceRefresh used to force a refresh of the plugin inventory data
+// even when the cache's TTL has not expired
+func WithForceRefresh() DiscoveryOptions {
+	return func(o *DiscoveryOpts) {
+		o.ForceRefresh = true
+	}
+}
+
+// WithPluginDiscoveryCriteria used to specify the plugin discovery criteria
 func WithPluginDiscoveryCriteria(criteria *PluginDiscoveryCriteria) DiscoveryOptions {
 	return func(o *DiscoveryOpts) {
 		o.PluginDiscoveryCriteria = criteria
 	}
 }
 
+// WithGroupDiscoveryCriteria used to specify the group discovery criteria
 func WithGroupDiscoveryCriteria(criteria *GroupDiscoveryCriteria) DiscoveryOptions {
 	return func(o *DiscoveryOpts) {
 		o.GroupDiscoveryCriteria = criteria

pkg/discovery/oci.go

@@ -28,6 +28,8 @@ func NewOCIDiscovery(name, image string, options ...DiscoveryOptions) Discovery
 	if useCacheOnlyForTesting, _ := strconv.ParseBool(os.Getenv("TEST_TANZU_CLI_USE_DB_CACHE_ONLY")); useCacheOnlyForTesting {
 		discovery.useLocalCacheOnly = true
 	}
+	discovery.forceRefresh = opts.ForceRefresh
+
 	return discovery
 }
 
@@ -46,6 +48,7 @@ func NewOCIGroupDiscovery(name, image string, options ...DiscoveryOptions) Group
 	if useCacheOnlyForTesting, _ := strconv.ParseBool(os.Getenv("TEST_TANZU_CLI_USE_DB_CACHE_ONLY")); useCacheOnlyForTesting {
 		discovery.useLocalCacheOnly = true
 	}
+	discovery.forceRefresh = opts.ForceRefresh
 
 	return discovery
 }