Uploaded files stay at the status of Queuing in 0.12.1 version

[qadeerikram-art]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.12.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Simply go to knowledgebase , upload file. It will stay in queue. One more thing when i delete that file i get error document not found.

✔️ Expected Behavior

File should be available.

❌ Actual Behavior

File stuck in queue.

[qadeerikram-art]

ada538b85b49 nginx:latest "sh -c 'cp /docker-e…" 10 minutes ago Up 10 minutes 0.0.0.0:4443->4443/tcp, :::4443->4443/tcp, 80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp docker-nginx-1
5462a2fc33b2 langgenius/dify-api:0.12.1 "/bin/bash /entrypoi…" 10 minutes ago Up 2 seconds 5001/tcp docker-api-1
50237fe5ae9c langgenius/dify-api:0.12.1 "/bin/bash /entrypoi…" 10 minutes ago Up 10 minutes 5001/tcp docker-worker-1
5b71f890307d postgres:15-alpine "docker-entrypoint.s…" 10 minutes ago Up 10 minutes (healthy) 5432/tcp docker-db-1
24c169fe85f1 redis:6-alpine "docker-entrypoint.s…" 10 minutes ago Up 10 minutes (healthy) 6379/tcp docker-redis-1
e855be7c93d1 semitechnologies/weaviate:1.19.0 "/bin/weaviate --hos…" 10 minutes ago Up 10 minutes docker-weaviate-1
a6a5df7c9c87 langgenius/dify-sandbox:0.2.10 "/main" 10 minutes ago Up 10 minutes (healthy) docker-sandbox-1
43055b2f1209 ubuntu/squid:latest "sh -c 'cp /docker-e…" 10 minutes ago Up 10 minutes 3128/tcp docker-ssrf_proxy-1
601df6135547 docker-web "/bin/sh ./entrypoin…" 10 minutes ago Up 10 minutes 3000/tcp docker-web-1

These are all docker services that are running.

[qadeerikram-art]

I have tried this updating this in env but still file is in queue.
LOG_LEVEL=INFO

Log file path

LOG_FILE=/app/logs/server.log

Log file max size, the unit is MB

LOG_FILE_MAX_SIZE=20

Log file max backup count

LOG_FILE_BACKUP_COUNT=5

Log dateformat

LOG_DATEFORMAT=%Y-%m-%d %H:%M:%S

Log Timezone

LOG_TZ=UTC

[dosubot]

There are several similar threads regarding the issue of files being stuck in the queue when uploaded to the knowledge base in Dify. Here are some relevant discussions and solutions:

1. Closed Issue: A similar issue was resolved by checking the rate limit for the Embedding model API key, ensuring the Celery worker service is running correctly, and verifying that all necessary services are up and running. You can start the Celery worker with docker compose up -d worker and check the logs for any errors. Additionally, ensure that services like api, worker, web, weaviate, db, redis, and nginx are all up ^[1].

2. Open Issue: Another suggestion is to check if the rate limit for your Embedding model API key has been reached, as this can cause documents to remain queued. Also, ensure that the Celery worker service is running correctly and review the logs for any specific error messages ^[2].

3. Closed Issue: Upgrading to the latest version of the software may resolve the issue, as there have been several fixes regarding the queue scheduler. Additionally, check for any pending segmentation or embedding processes, and ensure all dependencies and services like db and redis are running smoothly ^[3].

Given your current setup, ensure that the Celery worker service is running and check the logs for any errors. Also, verify that the rate limit for your Embedding model API key has not been exceeded. These steps should help address the issue of files being stuck in the queue.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu This is not the solution. Every time i post bug you just say to upgrade to latest version. My whole app is running on 0.11.0. I already did upgrade to 0.15.1 but then there are new issues . That is not creating chatbot workflow using yml, but in 0.11.0 it's working.
There is no rate limit error.
All workers are working , i have sent you the logs.

[dosubot]

This is a tough nut to crack, could really use your help on this, @takatost.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

One more thing on production server i am using 0.11.0 and it's working fine. But on dev server this is not working fine.

[qadeerikram-art]

@dosu Please update

[dosubot]

When upgrading from version 0.11.0 to 0.12.1, there are specific configuration changes that could affect file processing. The image upload feature has been replaced by a more comprehensive file upload feature. For Chatflow applications, the system automatically switches from image upload to file upload, requiring no manual updates. However, for Workflow applications, you need to manually migrate before the official deprecation of the sys.files variable. It's recommended to create a file-type variable in the "Start" node and reference this file variable in subsequent nodes instead of using the sys.files variable ^[1].

Since your production server is working fine with version 0.11.0, it might be worth checking if these changes have been correctly implemented on your dev server. Additionally, ensure that all services and dependencies are correctly configured and running, as outlined in the Celery worker service setup ^[2][3][4].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu
One thing i noticed in prod init api is getting called but this api is not getting called in dev.

[dosubot]

The issue you're experiencing with the 'init api' not being called in your development environment could be related to differences in configuration or setup between your production and development environments. Here are a few things to check:

1. Environment Configuration: Ensure that the environment variables and configuration files in your development setup match those in production. Pay special attention to any API-related configurations, such as endpoints and authentication keys.

2. Network and Firewall Settings: Verify that there are no network restrictions or firewall settings in your development environment that might be blocking the API call.

3. Service Dependencies: Check that all necessary services and dependencies are running correctly in your development environment. This includes ensuring that the Celery worker, database, and any other required services are operational.

4. Logs and Error Messages: Review the logs for any error messages or warnings that might indicate why the 'init api' call is not being triggered. This can provide clues about missing configurations or other issues.

5. Version Differences: Since you mentioned that version 0.11.0 works fine in production, ensure that any changes or updates in version 0.12.1 are correctly applied in your development setup.

By addressing these areas, you may be able to identify the cause of the issue and ensure that the 'init api' call is made as expected in your development environment.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu please ask someone else for help.

[dosubot]

A maintainer has already been looped in.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu it's been one day and i have not heard anything from you. Could you Please ask someone else to jump in.

[dosubot]

A maintainer has already been looped in.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu please mention him again or ask someone else.

[dosubot]

A maintainer has already been looped in.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@dosu please update

[dosubot]

A maintainer has already been looped in.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[qadeerikram-art]

@takatost