Pin Pandas version and move back to native to_sql #1053
Conversation
|
| ) | ||
| ) | ||
|
|
||
| flat_transaction_data.to_sql(name=TMP_DATA_TABLENAME, con=conn, schema=DATA_SCHEMA, index=False, if_exists="append") |
There was a problem hiding this comment.
Won't this create a very long sql statement?
There was a problem hiding this comment.
Won't this create a very long sql statement?
It was the way all our Python scripts worked before SQL alchemy got updated from v1 to v2. Pandas v1 was incompatible with SQL alchemy v2. The batching function I wrote to replace pandas to_sql ended up taking about 5x as long to run as the original. To the point where large updates would take ~30 hours.
So we need to update Python from 3.7 to 3.8, in order to pull in pandas v2 and get the original fast to_sql back. It does smart batching.
There was a problem hiding this comment.
If it does batching, we good.
No description provided.