Unable to retrieve logs from crashed containers in k8s

[dianabarsan]

Describe the issue
There is a problem with the observability stack:

1. in K8S, once a container is gone, you can no longer access its logs - not 100% true, as logs can be accessed for a short time if the pod still exists, by passing the --previous parameter. but that's not a reasonable solution to create an consistent ability to troubleshoot crashed containers.
2. It seems that Loki will, as well, delete logs from dead containers. People complaining here

Discovered while trying to get logs from a container that crashed on Feb 11th and there is no trace of it in logs in Grafana .

Describe the improvement you'd like
Troubleshooting and access to logs of crashed containers should be straightforward!

[Hareet]

Our observability stack is here