MVP: Cost attribution #10269

ying-jeanne · 2024-12-17T21:00:08Z

What this PR does

This is the follow up of #9733,

The PR intent to export extra attributed metrics in distributor and ingester, in order to get sample received, sample discarded and active_series attributed by cost attribution label.

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.

docs/sources/mimir/configure/configuration-parameters/index.md

pkg/costattribution/manager.go

pkg/ingester/activeseries/active_series.go

colega · 2024-12-18T13:13:30Z

pkg/ingester/activeseries/active_series.go

@@ -502,6 +525,18 @@ func (s *seriesStripe) remove(ref storage.SeriesRef) {
 	}

 	s.active--
+	if s.cat != nil {
+		if idx == nil {


Same here, we should assume this isn't nil. Just skipping the removal will break the numbers forever.

vendor and update in commit 4706bde

colega · 2024-12-18T13:14:05Z

pkg/ingester/activeseries/active_series_test.go

Please update the active series tracker tests with the costattribution.Tracker, otherwise the new code isn't tested.

addressed in 17b64a9

colega · 2024-12-18T13:20:41Z

pkg/ingester/ingester.go

+		idx, err := db.Head().Index()
+		if err != nil {
+			level.Warn(i.logger).Log("msg", "failed to get the index of the TSDB head", "user", userID, "err", err)
+			idx = nil
+		}


As commented previously, we should never proceed without an index.

If you check the implementation of db.Head().Index() it never returns an error. We have three options here:

Skip tenants if they don't have index: this is the least effort one.

Panic if err is not nil, this is ugly

Update mimir-prometheus to add a MustIndex() IndexReader method that does not return an error, and use that one.

PR in mimir-prometheus grafana/mimir-prometheus#811

vendor and update in commit 4706bde

colega · 2024-12-18T13:21:35Z

pkg/mimir/modules.go

+	if t.Cfg.CostAttributionRegistryPath != "" {
+		reg := prometheus.NewRegistry()
+		var err error
+		t.CostAttributionManager, err = costattribution.NewManager(3*time.Minute, time.Minute, t.Cfg.CostAttributionEvictionInterval, util_log.Logger, t.Overrides, reg)


I think these values should not be hardcoded.

removed unused parameter b27e379

tacole02

Thanks for updating the docs! I left a few suggestions.

docs/sources/mimir/configure/configuration-parameters/index.md

colega · 2024-12-20T09:30:10Z

pkg/costattribution/tracker.go

+	}
+}
+
+func (t *Tracker) shouldDelete(deadline int64) bool {


I would rename this method to "recovered from overflow". The fact that it should be deleted is a decision of the Manager.

addressed in 1ab1f00

colega · 2024-12-20T09:30:53Z

pkg/costattribution/tracker.go

+}
+
+func (t *Tracker) shouldDelete(deadline int64) bool {
+	if t.cooldownUntil != nil && t.cooldownUntil.Load() < deadline {


You can't check whether t.cooldownUntil without holding the mutex here.

addressed in fe8a1e5

pkg/costattribution/manager.go

colega · 2024-12-24T11:40:26Z

pkg/costattribution/manager.go

+	newTrackedLabels := m.limits.CostAttributionLabels(userID)
+
+	// sort the labels to ensure the order is consistent
+	sort.Slice(newTrackedLabels, func(i, j int) bool {


Is it safe to modify the slice returned by m.limits.CostAttributionLabels(userID)?

modify the copy instead, addressed in 888d8b0

colega · 2024-12-27T12:02:39Z

go.mod

-replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20241219104229-b50052711673
+replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20241224134504-460b7be5bce8


You should not update mimir-prometheus here. It should be updated in main, and then you merge main into your branch.

colega · 2024-12-27T12:03:14Z

pkg/costattribution/manager.go

+	return m.purgeInactiveAttributionsUntil(time.Now().Add(-m.inactiveTimeout).Unix())
+}
+
+func (m *Manager) EnabledForUser(userID string) bool {


I think this doesn't need to be exported.

colega · 2024-12-27T12:04:43Z

pkg/costattribution/manager.go

+	if tracker, exists = m.trackersByUserID[userID]; exists {
+		return tracker
+	}
+	tracker = newTracker(userID, m.limits.CostAttributionLabels(userID), m.limits.MaxCostAttributionCardinalityPerUser(userID), m.limits.CostAttributionCooldown(userID), m.logger)


Nit, but I would collect all the information needed to build a tracker (all the m.limits.* calls before we take m.mtx.Lock, to spend less time locked.

pkg/costattribution/manager.go

colega · 2024-12-27T12:46:21Z

pkg/costattribution/manager.go

+	lbls := m.limits.CostAttributionLabels(userID)
+
+	newTrackedLabels := make([]string, len(lbls))
+	copy(newTrackedLabels, lbls)


I wouldn't call these newTrackedLabels unless we know they're different from the previous ones. I'd just stick to lbls for the entire method.

Also, you can leverage slices.Clone to make it shorter.

Suggested change

lbls := m.limits.CostAttributionLabels(userID)

newTrackedLabels := make([]string, len(lbls))

copy(newTrackedLabels, lbls)

lbls := slices.Clone(m.limits.CostAttributionLabels(userID))

colega · 2024-12-27T13:03:04Z

pkg/costattribution/tracker.go

+func (t *Tracker) createNewObservation(key []byte, ts int64, activeSeriesIncrement, receivedSampleIncrement, discardedSampleIncrement float64, reason *string) {
+	t.observed[string(key)] = &observation{
+		lastUpdate:         atomic.NewInt64(ts),
+		activeSerie:        *atomic.NewFloat64(activeSeriesIncrement),
+		receivedSample:     *atomic.NewFloat64(receivedSampleIncrement),
+		discardedSample:    map[string]atomic.Float64{},
+		discardedSampleMtx: sync.Mutex{},
+	}
+	if discardedSampleIncrement > 0 && reason != nil {
+		t.observed[string(key)].discardedSampleMtx.Lock()
+		t.observed[string(key)].discardedSample[*reason] = *atomic.NewFloat64(discardedSampleIncrement)
+		t.observed[string(key)].discardedSampleMtx.Unlock()
+	}
+}


Since we're creating a new observation here, I would pass a key string here, allocated once.

I'm not sure if this method isn't allocating 4 strings here (not a big deal, since it's not a hot path, but still..)

colega · 2024-12-27T13:06:08Z

pkg/costattribution/tracker.go

+
+// updateObservations updates or creates a new observation in the 'observed' map.
+func (t *Tracker) updateObservations(key []byte, ts int64, activeSeriesIncrement, receivedSampleIncrement, discardedSampleIncrement float64, reason *string, createIfDoesNotExist bool) {
+	if o, known := t.observed[string(key)]; known && o.lastUpdate != nil {


How can o.lastUpdate be nil?

colega · 2024-12-27T13:07:13Z

pkg/costattribution/tracker.go

+	} else if len(t.observed) < t.maxCardinality*2 && createIfDoesNotExist {
+		// When createIfDoesNotExist is false, it means that the method is called from DecrementActiveSeries, when key doesn't exist we should ignore the call
+		// Otherwise create a new observation for the key
+		t.createNewObservation(key, ts, activeSeriesIncrement, receivedSampleIncrement, discardedSampleIncrement, reason)
+	}


I would move this to the beginning of the method to un-indent the most common path.
i.e.

o, known := t.observed[string(key)] if !known { if len(t.observed) < t.maxCardinality*2 && createIfDoesNotExist { // When createIfDoesNotExist is false, it means that the method is called from DecrementActiveSeries, when key doesn't exist we should ignore the call // Otherwise create a new observation for the key t.createNewObservation(key, ts, activeSeriesIncrement, receivedSampleIncrement, discardedSampleIncrement, reason) } return } // ...

colega · 2024-12-27T13:09:27Z

pkg/costattribution/tracker.go

+func (t *Tracker) IncrementActiveSeriesFailure() {
+	if t == nil {
+		return
+	}
+	t.totalFailedActiveSeries.Add(1)
+}


I'm still not convinced this belongs here.

What did fail here? costattribution.Tracker didn't fail. It's the caller (activeseries.ActiveSeries) who failed to retrieve labels for a series it should know.
So it's the caller who should register the failure.

costattribution.Tracker wasn't even called here, so it can't fail.

colega · 2024-12-27T13:10:47Z

pkg/distributor/distributor.go

 	for _, ts := range req.Timeseries {
 		receivedSamples += len(ts.TimeSeries.Samples) + len(ts.TimeSeries.Histograms)
 		receivedExemplars += len(ts.TimeSeries.Exemplars)
+		d.costAttributionMgr.Tracker(userID).IncrementReceivedSamples(ts.Labels, float64(receivedSamples), mtime.Now())


Suggested change

for _, ts := range req.Timeseries {

receivedSamples += len(ts.TimeSeries.Samples) + len(ts.TimeSeries.Histograms)

receivedExemplars += len(ts.TimeSeries.Exemplars)

d.costAttributionMgr.Tracker(userID).IncrementReceivedSamples(ts.Labels, float64(receivedSamples), mtime.Now())

cat := d.costAttributionMgr.Tracker(userID)

for _, ts := range req.Timeseries {

receivedSamples += len(ts.TimeSeries.Samples) + len(ts.TimeSeries.Histograms)

receivedExemplars += len(ts.TimeSeries.Exemplars)

cat.IncrementReceivedSamples(ts.Labels, float64(receivedSamples), mtime.Now())

Co-authored-by: Oleg Zaytsev <[email protected]>

Poc: cost attribution proposal 2

e315ebb

ying-jeanne force-pushed the final-cost-attribution branch from 5165a5b to 6f36b5f Compare December 17, 2024 21:07

ying-jeanne changed the title ~~Final cost attribution~~ MVP: Cost attribution Dec 17, 2024

ying-jeanne requested a review from colega December 17, 2024 21:10

ying-jeanne force-pushed the final-cost-attribution branch from 6f36b5f to 077a94a Compare December 17, 2024 22:01

refectory

f04c28f

ying-jeanne force-pushed the final-cost-attribution branch from 077a94a to f04c28f Compare December 17, 2024 22:08

ying-jeanne marked this pull request as ready for review December 17, 2024 22:13

ying-jeanne requested review from tacole02 and a team as code owners December 17, 2024 22:13

ying-jeanne requested a review from seizethedave December 18, 2024 10:20

colega reviewed Dec 18, 2024

View reviewed changes

tacole02 approved these changes Dec 18, 2024

View reviewed changes

ying-jeanne added 17 commits December 19, 2024 14:09

add experimental features in about-versioning.md

2c422d1

change const variable to private

d2eab6b

make timer service

1f39282

rename TrackerForUser to Tracker

9b4337d

use fine locking

1a523e1

add comments explain why we use unchecked collector

f10f787

rename deleteUserTracker to deleteTracker

cc0e939

rename cat in cost attribution package to t or tracker

c020be0

avoid get tracker twice

71e4666

refactor inactiveObservationsForUser

9dd101b

refactor shouldDelete function

7d4ea9a

rename calabels and calabelmap to labels and index

6754666

remove getter and setter of max cardinality and cooldown duration

fffc5b3

rename CompareLabels to hasSameLabels

2cf8c3e

remove the mapping logic since the slices are ordered

f994034

remove unnecessary tracker nil checking

b060c09

fix linting

e35a8d9

colega reviewed Dec 20, 2024

View reviewed changes

ying-jeanne added 3 commits December 24, 2024 12:02

remove time checking

8386503

add createIfDoesNotExist parameter

d8f1e9b

add more condition for trigger newTracker

b9efb94

colega reviewed Dec 24, 2024

View reviewed changes

pkg/costattribution/manager.go Outdated Show resolved Hide resolved

colega reviewed Dec 24, 2024

View reviewed changes

ying-jeanne added 6 commits December 24, 2024 13:34

remove the label adapter to labels call

a37e6de

remove useless function dum dum

211b3a2

make hardcoded increment value

f697e6f

rename + make cooldownuntil a normal int64 and lock with observedMtx

fe8a1e5

use build-in functon dum dum

8b5836f

modify the copy of calabels instead of directly the slice

888d8b0

ying-jeanne mentioned this pull request Dec 24, 2024

tsdb: Add MustIndex() function to Head grafana/mimir-prometheus#811

Open

ying-jeanne added 3 commits December 24, 2024 15:55

update mimir-prometheus

b15b487

Merge remote-tracking branch 'origin/r322' into final-cost-attribution

87209d6

vendor new mimir-prometheus

4706bde

ying-jeanne requested review from stevesg and grafanabot as code owners December 24, 2024 16:14

ying-jeanne added 7 commits December 24, 2024 17:15

rename function

1ab1f00

fix lint

8111b6c

add unittest in active series

17b64a9

copy slice instead

a191044

add test for discarded samples

2bb1845

change small map to slice since it is quicker

ddd507d

remove unused parameter

b27e379

colega reviewed Dec 27, 2024

View reviewed changes

ying-jeanne and others added 3 commits December 27, 2024 15:58

add new parameter

a79fac7

update config file

37901b7

Update pkg/costattribution/manager.go

f7115f4

Co-authored-by: Oleg Zaytsev <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MVP: Cost attribution #10269

MVP: Cost attribution #10269

ying-jeanne commented Dec 17, 2024 •

edited

Loading

colega Dec 18, 2024

ying-jeanne Dec 24, 2024

colega Dec 18, 2024

ying-jeanne Dec 26, 2024

colega Dec 18, 2024

ying-jeanne Dec 24, 2024

ying-jeanne Dec 24, 2024

colega Dec 18, 2024

ying-jeanne Dec 27, 2024

tacole02 left a comment

colega Dec 20, 2024

ying-jeanne Dec 24, 2024 •

edited

Loading

colega Dec 20, 2024

ying-jeanne Dec 24, 2024

colega Dec 24, 2024

ying-jeanne Dec 24, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

colega Dec 27, 2024

		replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20241219104229-b50052711673
		replace github.com/prometheus/prometheus => github.com/grafana/mimir-prometheus v0.0.0-20241224134504-460b7be5bce8

MVP: Cost attribution #10269

Are you sure you want to change the base?

MVP: Cost attribution #10269

Conversation

ying-jeanne commented Dec 17, 2024 • edited Loading

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tacole02 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ying-jeanne Dec 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ying-jeanne commented Dec 17, 2024 •

edited

Loading

ying-jeanne Dec 24, 2024 •

edited

Loading