I'm facing an issue with the loading time of data from PostgreSQL to python Polars. I have 7 tables. two tables have size of 1.5 gb and others around 500 mb. I load the table into various polars data frames and the joined based on single column. it takes around 20 minutes to load all table and joins to make a single data frame. Also the memory is so high when joining. Is there any solution which can reduce the all the time to 10 -30 seconds. Also lower time is appreciated. any tools or any postgresql tricks
n_live_tup is rows count.
-[ RECORD 1 ]------------------
schemaname | public
relname | table1
n_live_tup | 7467575
-[ RECORD 2 ]------------------
schemaname | public
relname | table2
n_live_tup | 1341294
-[ RECORD 3 ]------------------
schemaname | public
relname | table3
n_live_tup | 888511
-[ RECORD 4]------------------
schemaname | public
relname | table4
n_live_tup | 112289
-[ RECORD 5 ]------------------
schemaname | public
relname | table5
n_live_tup | 47007
-[ RECORD 6 ]-----------------
schemaname | public
relname | table6
n_live_tup | 12722 Re: clarification for the questions you ve asked
- Software versions for Postgres and Polars.
Version is 15.6
- Where is the Polars client located relative to the Postgres database, same machine, same network, remote network?
Polars python code is on remote server... ang postgres is on another remote server.
- The code you are using to fetch the data. –
I'm using the python and sqlalchemy build polars dataframe. @Adrian Klaver
Источник: https://stackoverflow.com/questions/780 ... and-polars
