Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Flow cache consider only flow.dag.yaml file #3861

Open
lucasrcezimbra opened this issue Nov 19, 2024 · 0 comments
Open

[BUG] Flow cache consider only flow.dag.yaml file #3861

lucasrcezimbra opened this issue Nov 19, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@lucasrcezimbra
Copy link

Describe the bug
Flow considers only the flow file to create the hash used for caching: https://github.com/microsoft/promptflow/blob/promptflow_1.16.1/src/promptflow-devkit/promptflow/_sdk/entities/_flows/base.py#L166-L169.

If we have two flows with the exactly same flow.dag.yaml, but different tools, it may run the wrong flow.

How To Reproduce the bug
(see the test reproducing the issue in Additional context section below)

  1. Create a flow with at least one tool.
  2. Create a second flow with the exactly same flow.dag.yaml but different tools.
  3. Invoke both flows using Python API.

Current behavior: Only the first loaded flow runs twice.
Expected behavior: Two different flows are loaded and run.
Running Information:

  • Promptflow Package Version:
    {
      "promptflow": "1.16.1",
      "promptflow-core": "1.16.1",
      "promptflow-devkit": "1.16.1",
      "promptflow-tracing": "1.16.1"
    }
    
    Executable '/home/user/workspace/test-promptflow/.venv/bin/python'
    Python (Linux) 3.12.1 (main, Dec 12 2023, 10:34:48) [GCC 10.2.1 20210110]
    
  • Operating System: Debian GNU/Linux 11 (bullseye)
  • Python Version: 3.12.1

Additional context
Test reproducing the issue:

import tempfile
from pathlib import Path
from textwrap import dedent

import pytest

from promptflow.client import load_flow


@pytest.fixture
def flow_yaml_content():
    return dedent(
        """
        $schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json
        inputs:
          text:
            type: string
        outputs:
          text:
            type: string
            reference: ${transform}
        nodes:
          - name: transform
            type: python
            source:
              type: code
              path: transform.py
            inputs:
              text: ${inputs.text}"""
    )


@pytest.fixture
def lower_flow_dir(flow_yaml_content):
    with tempfile.TemporaryDirectory() as lower_flow_dir:
        lower_flow_dir = Path(lower_flow_dir)
        with open(lower_flow_dir / 'flow.dag.yaml', 'w') as f:
            f.write(flow_yaml_content)

        with open(lower_flow_dir / "transform.py", "w") as f:
            f.write(
                dedent(
                    """
                    from promptflow import tool

                    @tool
                    def lower(text):
                        return text.lower()"""
                )
            )

        yield lower_flow_dir


@pytest.fixture
def upper_flow_dir(flow_yaml_content):
    with tempfile.TemporaryDirectory() as upper_flow_dir:
        upper_flow_dir = Path(upper_flow_dir)
        with open(upper_flow_dir / 'flow.dag.yaml', 'w') as f:
            f.write(flow_yaml_content)

        with open(upper_flow_dir / "transform.py", "w") as f:
            f.write(
                dedent(
                    """
                    from promptflow import tool

                    @tool
                    def upper(text):
                        return text.upper()"""
                )
            )

        yield upper_flow_dir


def test_ignore_cache_when_tools_are_different(lower_flow_dir, upper_flow_dir):
    assert load_flow(lower_flow_dir).invoke({"text": "TEST"}).output == {"text": "test"}
    assert load_flow(upper_flow_dir).invoke({"text": "test"}).output == {"text": "TEST"}
___________________________ test_ignore_cache_when_tools_are_different ___________________________

lower_flow_dir = PosixPath('/tmp/tmpiizohoz4'), upper_flow_dir = PosixPath('/tmp/tmp1wl2g71f')

    def test_ignore_cache_when_tools_are_different(lower_flow_dir, upper_flow_dir):
        assert load_flow(lower_flow_dir).invoke({"text": "TEST"}).output == {"text": "test"}
>       assert load_flow(upper_flow_dir).invoke({"text": "test"}).output == {"text": "TEST"}
E       AssertionError: assert {'text': 'test'} == {'text': 'TEST'}
E
E         Differing items:
E         {'text': 'test'} != {'text': 'TEST'}
E         Use -v to get more diff

test_promptflow.py:79: AssertionError
@lucasrcezimbra lucasrcezimbra added the bug Something isn't working label Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant