Add Regression Indicators #186

drewrip · 2024-01-24T21:33:06Z

This sets up the notebooks performance test to create and use a RegressionIndicator from this matrix-benchmarking PR.

The idea is that we can create a number of different indicators from the base RegressionIndicator that provides a generic and flexible result that we can save and process upstream. I created a simple ZScoreIndicator to demonstrate this, it triggers when the new payload's KPI values are more than 3 standard deviations away from the mean value for that KPI among the LTS results. Hopefully we can use this to create a simple wrapper for datastax/Hunter.

There are still 2 work in progress issues to resolve with this PR:

Missing Settings: Some of the settings for an individual MatrixEntry are only present in the Matrix itself. In our testing payloads, an example would be version. I'm not sure why these settings aren't present in the MatrixEntry itself, but as currently written we can only restrict the settings that are present in the MatrixEntry.
Gathered Results: As discussed some of the LTS payloads have "gathered" results. I wasn't sure how to turn this off and couldn't flatten the Payloads since I couldn't deepcopy them.

openshift-ci · 2024-01-24T21:33:13Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kpouget for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kpouget · 2024-01-25T14:42:45Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py

+        zscore = regression.ZScoreIndicator(
+            entry,
+            lts_entries,
+            settings_filter={"image": "tensorflow:2023.1"},


can you try to make this dynamic?
common.Matrix.settings["image"] should contain all the available images

could you try to do the filtering outside of the regression.ZScoreIndicator function ? 🤔
I think it would be clearer if you could pass only the relevant lts_entries

Sure, I'll add the filtering outside first. It should be as simple as removing the settings_filter option. How do you mean dynamic though? As in, we want to run a regression analysis for each unique "image"?

kpouget · 2024-01-25T14:46:35Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py

+            entry,
+            lts_entries,
+            settings_filter={"image": "tensorflow:2023.1"},
+            combine_funcs={"notebook_performance_benchmark_time": np.mean}


can you use notebook_performance_benchmark_min_time, which already provides an "aggregated" value?
I think min() is more relevant than mean(), but that's open to discussion
if ever we chose to use mean, then we should expose it as a KPI instead of _min
(the idea is to expose as a KPI the value we're using for the regression analyses)

I should have explained this better. The combine_funcs={"notebook_performance_benchmark_time": np.mean} is just an optional way of handling KPIs that provide either a list[int] or list[float]. The combine_funcs option can be omitted and any KPIs that use a list of values will be skipped. All the other KPIs such as notebook_performance_benchmark_min_time are analyzed as normal.

kpouget · 2024-01-25T14:47:57Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py

-    for lts_entry in common.LTS_Matrix.all_records(): pass
+    lts_entries_raw = list(common.LTS_Matrix.all_records())
+    # Temporarily skip the gathered results
+    lts_entries = list(filter(lambda entry: type(entry.results) is not list, lts_entries_raw))


can you try to take the 1st value of the gathered entries?
in theory, all the entries in entry.results should contain the same result +/- noise, as they have been executed in the same conditions

Sure, I'll just do that. I previously tried flattening them, but I wasn't able to deepcopy MatrixEntry's

kpouget · 2024-01-25T14:49:52Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py


    logging.info(f"Received {common.LTS_Matrix.count_records()} historic LTS entries")
-    for lts_entry in common.LTS_Matrix.all_records(): pass
+    lts_entries_raw = list(common.LTS_Matrix.all_records())


you don't really need the list() here. Without it, lst_entries_row is a generator, that is processed (only once) at the line below
removing list() saves memory and processing time, especially if the list gets large

Fair point!

drewrip · 2024-01-26T17:31:29Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

kpouget · 2024-01-26T17:53:08Z

@drewrip something went wrong with build (matrixbenchmarking commit wasn't available)
try git commit --amend --no-edit and force push it, and try to launch the test again

drewrip · 2024-01-26T17:54:48Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

drewrip · 2024-01-26T17:57:32Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

drewrip · 2024-01-26T20:03:01Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

drewrip · 2024-01-26T20:33:57Z

results

Looks like the analysis took place as expected. No regressions were found but there were a number of issues regrading payload validation. Obviously we need to make sure the payload schemas are consistent, but I'm wondering why the matrix entry location was /tmp/matrix_benchmarking_results/expe/from_url.

kpouget · 2024-01-26T20:47:58Z

results

Looks like the analysis took place as expected. No regressions were found but there were a number of issues regrading payload validation. Obviously we need to make sure the payload schemas are consistent, but I'm wondering why the matrix entry location was /tmp/matrix_benchmarking_results/expe/from_url.

good :)
yes, we'll need to tidy up the entries on OpenSearch!

I'm wondering why the matrix entry location was /tmp/matrix_benchmarking_results/expe/from_url.
this location is where the entry is downloaded, when launching a "plotting" job, as you did. So this part is expected :)

We'll need to find a way to save the regression.json files, even when doing a replot. One way to do it is to include it in the LTS Documentation report, around this place: https://github.com/openshift-psap/topsail/blob/main/projects/notebooks/visualizations/rhods-notebooks-performance/plotting/lts_documentation.py#L73
if you do something like that:

html.Code(yaml.dump(entry.results.lts.regression_results), style={"white-space": "pre-wrap"})

it should appear in the HTML document

drewrip · 2024-01-26T21:39:00Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

drewrip · 2024-01-26T21:52:31Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/184/pull-ci-openshift-psap-topsail-main-rhoai-light/1750426742431944704/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.export=true
/var matbench.lts.opensearch.index=rhoai-notebooks-performance

kpouget · 2024-01-27T06:00:04Z

projects/notebooks/visualizations/rhods-notebooks-performance/plotting/lts_documentation.py

+    regression += [html.H2("regression analysis")]
+    regression += [html.Code(yaml.dump(entry.results.lts.regression_results, default_flow_style=False), style={"white-space": "pre-wrap"})]


Suggested change

regression += [html.H2("regression analysis")]

regression += [html.Code(yaml.dump(entry.results.lts.regression_results, default_flow_style=False), style={"white-space": "pre-wrap"})]

regression += [html.H1("regression analysis")]

regression += [html.Code(yaml.dump(lts.regression_results, default_flow_style=False), style={"white-space": "pre-wrap"})]

this adjusts the title size, to make it top level, and takes the regression_results field from lts, which is the focus of this file

kpouget · 2024-01-29T17:11:58Z

@drewrip , the regression results wasn't included (or was empty) in this document:
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/186/pull-ci-openshift-psap-topsail-main-rhoai-plot/1751000197686628352/artifacts/plot/test/artifacts/report_01_report:%20LTS%20Documentation.html

and we'll need to define a model defining the format of the regression_results field
I guess it should extend the KPI model, maybe the notebooks KPI (in the TOPSAIL part), so that it can be easily located in OpenSearch, with Grafana

kpouget · 2024-01-30T07:54:19Z

projects/notebooks/visualizations/rhods-notebooks-performance/models/lts.py

+    regression: NotebookPerformanceRegression

 NotebookPerformanceKPIs = matbench_models.getKPIsModel("NotebookPerformanceKPIs", __name__, kpi.KPIs, kpi.NotebookPerformanceKPI)
+NotebookPerformanceRegression = matbench_models.RegressionResult


this isn't necessary, if you don't modify matbench_models.RegressionResult, no need to rename it here

kpouget · 2024-01-30T07:57:18Z

projects/notebooks/visualizations/rhods-notebooks-performance/models/lts.py


 class Payload(matbench_models.ExclusiveModel):
    schema_name: matbench_models.create_schema_field("rhods-notebooks-perf") = Field(alias="$schema")
    metadata: Metadata
    results: Results
    kpis: Optional[NotebookPerformanceKPIs]
-    regression_results: Optional[Any]


can you let this here, in the Payload? in MatrixBenchmarking, overall, the results are workload specific, but the payload/metadata/kpis have a predefined structure (that can be extended, but the core is always the same) , so the regression results belong there. (I need to browse these entries to upload them as KPIs)

Gotcha, I'll move them under the Payload

kpouget · 2024-01-30T08:06:01Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py

+            )
+            zscore_ind = zscore.ZScoreIndicator(entry, controlled_lts_entries)
+            regression_results[check_setting] = zscore_ind.analyze()
+            number_of_failures += sum(map(lambda x: x["regression"]["status"], regression_results[check_setting]))


I think in Python, list comprehension are preferred to functional programming:

sum([x[["regression"]["status"]] for x in regression_results[check_setting]])

and would be nice to use the model here, instead of x[["regression"]["status"]
--> x.regression.status (the model or types.SimpleNamespace)

and, I'm not sure you can sum the statuses, if you have [-1, 1], you end up with 0 failures, no?

I need to start moving from functional to Pythonic with dict and list comprehensions, I'll convert that.

and, I'm not sure you can sum the statuses, if you have [-1, 1], you end up with 0 failures, no?

In the current way its implemented this is fine. The statuses are really just boolean flags that are set to 1 if there is some anomaly detected. A simple sum wouldn't be possible if we would like a more complex status, but more details regarding what the regression looks like, and why it is being reported is contained in the RegressionResult structure. If for some reason we want to roll up that info into a more complex "return code" status, then the sum likely wouldn't work but I suppose we'd want to discuss how we want to report regression.

kpouget · 2024-01-30T08:07:52Z

projects/notebooks/visualizations/rhods-notebooks-performance/analyze/__init__.py

    logging.info(f"Received {common.Matrix.count_records()} new entries")
+
+    number_of_failures = 0
+    settings_to_check = ["version", "repeat"]


I'm not use to understand exactly what's being checked here, ie how you build the reference data used for the regression checking. We'll need to clarify that, it's important to have that clear

also, I don't see clearly where you define the KPI being used for regression. Do you test all of them? 🤔
they won't necessarily have the same direction "of failure", eg a latency going higher is a failure, but a bandwidth going lower is a failure, so you need to define the regression properties for each of them individually, I guess

I'm not use to understand exactly what's being checked here, ie how you build the reference data used for the regression checking. We'll need to clarify that, it's important to have that clear

I can try to make that clear. Or we can add a note once we finalize it. At the moment it is considering all of the LTS datapoints with the same settings (minus the image version) to be the historical data. The settings_to_check is just the settings that we want to check for regression against. We iterate over them to perform the regression testing against each.

also, I don't see clearly where you define the KPI being used for regression. Do you test all of them? 🤔
they won't necessarily have the same direction "of failure", eg a latency going higher is a failure, but a bandwidth going lower is a failure, so you need to define the regression properties for each of them individually, I guess

Each indicator has an optional parameter called kpi. There you can specify a list of KPI's that you'd like to check. I suppose this difference in direction could be implemented in each RegressionIndicator we create. However, in the basic ZScoreIndicator there is no distinction made. There are two reasons in this simple instance:

I figured statistically significant variation in either direction was notable. Having a new image version run significantly faster seems like it would be suspicious? It wouldn't be wrong or bad necessarily but it would be good to make sure we know about them.

We also track the direction withing the RegressionResults, we can still recover that information if we'd like by filtering out the entries that exhibit regression in the correct direction

Having a new image version run significantly faster seems like it would be suspicious? It wouldn't be wrong or bad necessarily but it would be good to make sure we know about them.

this is linked with the must remain stable vs should remain stable distinction that I mentioned here and there.

If must remain stable, any positive or negative variation should be flagged. The binary is the same (the hardware also), so the performance shouldn't change significantly.

If should_remain_stable, the binary will change, like version bumps, so the new version can be faster, without need to raise a red flag for it. But it can be a flashy green flag :D

drewrip · 2024-02-05T22:02:48Z

/test rhoai-light kserve quick

drewrip · 2024-02-09T20:29:08Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance

drewrip · 2024-02-09T21:29:49Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance

drewrip · 2024-02-12T16:39:30Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T17:25:34Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T19:15:06Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T19:51:27Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T20:10:13Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T20:23:46Z

Regression workflow working now -> the results are appearing in the LTS Documentation report.

We needed another parse pass.

drewrip · 2024-02-12T21:04:41Z

/test rhoai-plot notebooks https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift-psap_topsail/195/pull-ci-openshift-psap-topsail-main-rhoai-light/1753099883524395008/artifacts/light/test/artifacts/002__sutest_notebooks__benchmark_performance/
/var matbench.workload=rhods-notebooks-performance
/var matbench.config_file=single_notebook_perf.yaml
/var matbench.lts.opensearch.index=regression-test.rhoai-notebooks-performance
/var matbench.lts.opensearch.upload=false

drewrip · 2024-02-12T21:18:33Z

Example of Regression results can be seen here

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 24, 2024

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 24, 2024

kpouget reviewed Jan 25, 2024

View reviewed changes

drewrip force-pushed the analyze branch from 1000c24 to ebc2f01 Compare January 25, 2024 17:22

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2024

drewrip changed the title ~~WIP Analyze Regression Indicators~~ Add Regression Indicators Jan 25, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 25, 2024

drewrip force-pushed the analyze branch from 3687b49 to 5a3c973 Compare January 26, 2024 17:27

drewrip force-pushed the analyze branch from 5a3c973 to 9433134 Compare January 26, 2024 17:56

kpouget reviewed Jan 27, 2024

View reviewed changes

drewrip force-pushed the analyze branch from 8488a4b to e993411 Compare January 29, 2024 19:21

kpouget reviewed Jan 30, 2024

View reviewed changes

drewrip force-pushed the analyze branch 4 times, most recently from 2f6c1a9 to b527dfa Compare February 1, 2024 15:27

drewrip added 13 commits February 9, 2024 15:28

calling RegressionIndicator in MatrixBench

fe194ca

filter outside of matrix-benchmarking

9c17d08

tolerate missing settings

65fcc19

show regression in LTS report

6970cda

add regression to results payload

d6bf4e5

restructure RegressionResult pydantic payload and parsing

133c566

regression results payload is list

20c419a

new field name for regression in lts_documentation report

7976dc0

temp fix in matbench for stale payloads

f5e51cc

handling gathering and regression model names

fce976a

[matbench] update

ae2fe96

parse env outside CI env

bae22c8

check for regression over ocp and rhoai version

26aa478

drewrip force-pushed the analyze branch from 7f30b40 to 26aa478 Compare February 9, 2024 20:28

remove test setting, change regression.json path and check present

a052ae8

drewrip force-pushed the analyze branch from 3617c95 to a388348 Compare February 12, 2024 19:50

fix regression analysis report dump in yaml

be46eee

drewrip force-pushed the analyze branch from a388348 to be46eee Compare February 12, 2024 20:10

smaller controlled settings set

71fc574

		regression += [html.H2("regression analysis")]
		regression += [html.Code(yaml.dump(entry.results.lts.regression_results, default_flow_style=False), style={"white-space": "pre-wrap"})]

Add Regression Indicators #186

Are you sure you want to change the base?

Add Regression Indicators #186

Conversation

drewrip commented Jan 24, 2024

openshift-ci bot commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewrip commented Jan 26, 2024

kpouget commented Jan 26, 2024

drewrip commented Jan 26, 2024

drewrip commented Jan 26, 2024

drewrip commented Jan 26, 2024

drewrip commented Jan 26, 2024

kpouget commented Jan 26, 2024

drewrip commented Jan 26, 2024

drewrip commented Jan 26, 2024

Choose a reason for hiding this comment

kpouget commented Jan 29, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewrip Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drewrip commented Feb 5, 2024

drewrip commented Feb 9, 2024

drewrip commented Feb 9, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip commented Feb 12, 2024

drewrip Jan 30, 2024 •

edited

Loading