Postgresql Agent #3001

yonitjio · 2024-06-22T15:00:42Z

Why are these changes needed?

This is an alternative approach to run SQL queries on PostgreSQL database without using function calling.
Mostly copy-pasted from coding package and user proxy agent.

Related issue number

Serve as an example for issue #28
Answer discussion #1558

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

yonitjio · 2024-06-22T15:02:55Z

@microsoft-github-policy-service agree

qingyun-wu · 2024-06-22T18:29:10Z

Nice PR! @mdfahad999, @MihirDavada, @ChristianWeyer, @tyler-suard-parker, @Namec999, @tytung2020, @walker-null-byte, @nengisuls for awareness and it would be great if you could help testing and reviewing this PR!

qingyun-wu · 2024-06-22T18:29:46Z

@yonitjio could you resolve the conflict in setup.py. Thanks!

gitguardian · 2024-06-23T02:24:49Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
11616921	Triggered	Generic High Entropy Secret	`d02303a`	notebook/agentchat_agentops.ipynb	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

yonitjio · 2024-06-24T10:37:27Z

What happened? Did I make a mistake? The gitguardian points to a file from a commit from another PR but why the name is main? I tried to sync from my fork but it seems it has no effect.

qingyun-wu · 2024-06-27T17:13:39Z

What happened? Did I make a mistake? The gitguardian points to a file from a commit from another PR but why the name is main? I tried to sync from my fork but it seems it has no effect.

Hi @yonitjio, sorry for the confusion. It is not caused by this PR and it is a false alarm. Please ignore it.

yonitjio · 2024-06-27T22:14:57Z

What happened? Did I make a mistake? The gitguardian points to a file from a commit from another PR but why the name is main? I tried to sync from my fork but it seems it has no effect.

Hi @yonitjio, sorry for the confusion. It is not caused by this PR and it is a false alarm. Please ignore it.

Thanks for clarifying.

Hk669 · 2024-07-05T04:23:47Z

autogen/agentchat/contrib/postgresql_agent.py

+
+        **This will execute SQL SELECT query statement on PostgreSQL server.**
+
+        The code blocks are executed or save in the order they are received.


Suggested change

The code blocks are executed or save in the order they are received.

The code blocks are executed or saved in the order they are received.

I'll remove it instead since the sql will only be executed, never saved:
The code blocks are executed in the order they are received.

Is that alright?

Hk669 · 2024-07-05T04:41:03Z

test/agentchat/contrib/test_postgresql_agent.py

+    skip or skip_openai,
+    reason="dependency is not installed OR requested to skip",
+)
+def test_postgresql_agent():


@yonitjio can you please check, why this test is skipping?

I believe it's because the skip_openai variable are set from contrib-test.yml.

test/agentchat/contrib/test_postgresql_agent.py

Hk669 · 2024-07-05T04:41:32Z

test/agentchat/contrib/test_postgresql_agent.py

+sys.path.append(os.path.join(os.path.dirname(__file__), "../.."))
+from conftest import skip_openai  # noqa: E402
+
+sys.path.append(os.path.join(os.path.dirname(__file__), ".."))


this line too.

Will do, but I'm going to need to change the following line too:

from from test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST # noqa: E402
to from agentchat.test_assistant_agent import KEY_LOC, OAI_CONFIG_LIST # noqa: E402

WaelKarkoub

@yonitjio thanks for the PR!

I was wondering if this design could also work for other sql databases as well, not just postgres. Have you considered using sqlalchemy as well?

yonitjio · 2024-07-05T14:32:38Z

@yonitjio thanks for the PR!

I was wondering if this design could also work for other sql databases as well, not just postgres. Have you considered using sqlalchemy as well?

I have, but I didn't think I can complete a PR using sqlchemy or similar library (someone suggested using pandasai in issue #28).

I already used this method for my experiment using Autogen with Odoo which uses Postgresql, so I didn't really do much for this PR.

I believe the more flexible way to do it is to have the these line as a callback:

                with pg.connect(self.dsn, row_factory=dict_row) as cnn:
                    with cnn.cursor() as cr:
                        cr.execute(code)
                        cres = cr.fetchall()
                        res_json = json.dumps(cres, default=str)

Link:

autogen/autogen/agentchat/contrib/postgresql_agent.py

Line 70 in 9304745

with pg.connect(self.dsn, row_factory=dict_row) as cnn:

This way the code becomes more generic and does not depend on any specific library, but one may argue this burdens the user with generic code.

Of course, using connector library as you suggest is also feasible.

Autogen's coding package is very flexible, this is just one example of it.

thinkall

Thank you very much for the PR, @yonitjio ! I left a few comments, but I'm OK to see the sql support being added as a code language support in another PR.

thinkall · 2024-07-07T11:56:48Z

autogen/agentchat/contrib/postgresql_agent.py

+from autogen.coding import CodeExecutor, CodeExtractor, MarkdownCodeExtractor, CodeBlock, CodeResult
+from autogen.runtime_logging import log_new_agent, logging_enabled
+
+class PostgreSqlQueryExecutor(CodeExecutor):


This class seems to be better put in the coding module.

thinkall · 2024-07-07T12:44:21Z

autogen/agentchat/contrib/postgresql_agent.py

+    def restart(self) -> None:
+        return
+
+class PostgreSqlAgent(ConversableAgent):


If we go even further, this class is not very much needed once we extend the existing code executor to support sql language. The cnn could be passed into the executor.

thinkall · 2024-07-07T12:49:31Z

setup.py

-    retrieve_chat_pgvector.extend(["psycopg>=3.1.18"])
+    psycopg.extend(["psycopg>=3.1.18"])
+
+retrieve_chat_pgvector.extend(psycopg)


Why would this PR change the setting of retrieve_chat_pgvector? Is there an issue with current retrieve_chat_pgvector?

thinkall

You can fix the code-formatting error with: pre-commit run --all-files.
To run pre-commit as part of git workflow, use pre-commit install.

yonitjio · 2024-07-08T01:50:03Z

Thank you very much for the PR, @yonitjio ! I left a few comments, but I'm OK to see the sql support being added as a code language support in another PR.

@thinkall I could not agree more with you on this. This PR should not be merged to main branch as there are better ways for Autogen to support SQL with the coding package. I'm going to close this PR. I apologize for the inconvenience.

I think we should continue the discussion in issue #28 for better implementation.

yonitjio added 2 commits June 22, 2024 11:58

PostgreSqlAgent

5ab5731

Add docstring and fix argument type hint.

aeab4d8

qingyun-wu requested a review from thinkall June 22, 2024 18:26

qingyun-wu requested a review from sonichi June 22, 2024 18:29

Merge branch 'main' into postgresql_agent

d02303a

Merge branch 'microsoft:main' into postgresql_agent

9304745

sonichi requested review from Knucklessg1, WaelKarkoub and Hk669 July 5, 2024 02:49

Hk669 reviewed Jul 5, 2024

View reviewed changes

WaelKarkoub reviewed Jul 5, 2024

View reviewed changes

thinkall reviewed Jul 7, 2024

View reviewed changes

yonitjio closed this Jul 8, 2024

yonitjio mentioned this pull request Jul 8, 2024

Please add a SQL agent for autogen . #28

Closed

yonitjio had a problem deploying to openai1 July 24, 2024 09:26 — with GitHub Actions Failure

yonitjio had a problem deploying to openai1 July 24, 2024 09:26 — with GitHub Actions Error

yonitjio had a problem deploying to openai1 July 24, 2024 09:26 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Postgresql Agent #3001

Postgresql Agent #3001

yonitjio commented Jun 22, 2024

yonitjio commented Jun 22, 2024

qingyun-wu commented Jun 22, 2024

qingyun-wu commented Jun 22, 2024

gitguardian bot commented Jun 23, 2024 •

edited

Loading

yonitjio commented Jun 24, 2024

qingyun-wu commented Jun 27, 2024

yonitjio commented Jun 27, 2024

Hk669 Jul 5, 2024

yonitjio Jul 5, 2024

Hk669 Jul 5, 2024

yonitjio Jul 5, 2024

Hk669 Jul 5, 2024

yonitjio Jul 5, 2024

WaelKarkoub left a comment

yonitjio commented Jul 5, 2024

thinkall left a comment

thinkall Jul 7, 2024

thinkall Jul 7, 2024

thinkall Jul 7, 2024

thinkall left a comment

yonitjio commented Jul 8, 2024


		This will execute SQL SELECT query statement on PostgreSQL server.

		The code blocks are executed or save in the order they are received.

	The code blocks are executed or save in the order they are received.
	The code blocks are executed or saved in the order they are received.

Postgresql Agent #3001

Postgresql Agent #3001

Conversation

yonitjio commented Jun 22, 2024

Why are these changes needed?

Related issue number

Checks

yonitjio commented Jun 22, 2024

qingyun-wu commented Jun 22, 2024

qingyun-wu commented Jun 22, 2024

gitguardian bot commented Jun 23, 2024 • edited Loading

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

yonitjio commented Jun 24, 2024

qingyun-wu commented Jun 27, 2024

yonitjio commented Jun 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WaelKarkoub left a comment

Choose a reason for hiding this comment

yonitjio commented Jul 5, 2024

thinkall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thinkall left a comment

Choose a reason for hiding this comment

yonitjio commented Jul 8, 2024

gitguardian bot commented Jun 23, 2024 •

edited

Loading