[Bug]: When running evaluation/swe_bench/scripts/eval/prep_eval.sh

[jatinganhotra]

Is there an existing issue for the same bug?

• I have checked the troubleshooting document at https://opendevin.github.io/OpenDevin/modules/usage/troubleshooting
• I have checked the existing issues.

Describe the bug

When running evaluation/swe_bench/scripts/eval/prep_eval.sh, we get the error manifest unknown for these django versions

→ sh evaluation/swe_bench/scripts/eval/prep_eval.sh
...
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:1.10
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:1.10 not found: manifest unknown: manifest unknown
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:1.11
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:1.11 not found: manifest unknown: manifest unknown
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:1.9
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:1.9 not found: manifest unknown: manifest unknown
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:2.0
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:2.0 not found: manifest unknown: manifest unknown
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:2.1
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:2.1 not found: manifest unknown: manifest unknown
Pulling Docker image: xingyaoww/swe-bench-django_django-testbed:2.2
Error response from daemon: manifest for xingyaoww/swe-bench-django_django-testbed:2.2 not found: manifest unknown: manifest unknown
...

Current OpenDevin version

Running locally with the latest commit

Installation and Configuration

- local setup as provided on README: https://github.com/OpenDevin/OpenDevin/tree/main/evaluation/swe_bench#evaluate-generated-patches
command run - `evaluation/swe_bench/scripts/eval/prep_eval.sh`

Model and Agent

No response

Operating System

No response

Reproduction Steps

No response

Logs, Errors, Screenshots, and Additional Context

No response

[SmartManoj]

Available versions
cc @xingyaoww

[xingyaoww]

We fork the environments from the original SWE-bench-docker repo, and this is actually an existing issue there: aorwall/SWE-bench-docker#19

These containers are not required for SWE-Bench-lite but may be required by the full SWE-Bench. We will try to see if we build these and update it in the coming weeks. In the meantime, if you just want to run SWE-Bench-lite, feel free to ignore this error.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[xingyaoww]

This issue should be addressed since we have already switched to SWE-Bench's official containerized evaluation in #2728.

[SmartManoj]

This issue should be addressed

Wrongly closed?