Основная логика и код, которые я использую:
шаг 1:
проверьте, имеют ли существующие файлы локально тот же размер, что и на FTP:ftp.nlist() # get file names on ftp
for file in ftp_files:
response = ftp.sendcmd('SIZE', file)
size = int(response.split()[1].rstrip('.'))
# then compare size against local file
Шаг 2:
загрузка файлов
Я использую следующий способ:
command = f"wget -r -nd --user={ftp_user} --password={ftp_pwd} ftp://{path_to_file}"
subprocess.run(command, shell=True, cwd=destination_directory)
Этот скрипт работает нормально, если я запускаю его на удаленном компьютере. Но если я запущу его с воздушным потоком (у меня есть 20 заданий на загрузку разных данных), некоторые из них зависнут. Никакая ошибка не вылетает, а просто застревает там.
Я спросил у службы передачи данных, и у нас нет ограничений на загрузку файлов через этот ftp.
Обновлено 24 июля:
Я добавил несколько журналов ниже. На этом работа и застревает.
[2024-07-24, 17:19:51 UTC] {ssh.py:524} INFO - 2024-07-24 13:19:51 EDT|windmaster_utils.py === Start to download FinancialNews_N_202407240325.xml.gz ===
[2024-07-24, 17:19:51 UTC] {ssh.py:529} WARNING - --2024-07-24 13:19:51-- ftp://xxx/FinancialNews/xml/daily/Finan ... 325.xml.gz
[2024-07-24, 17:19:51 UTC] {ssh.py:529} WARNING - => ‘xxx/FinancialNews/xml/daily/.listing’
[2024-07-24, 17:19:51 UTC] {ssh.py:529} WARNING - Connecting to xxx...
[2024-07-24, 17:19:51 UTC] {ssh.py:529} WARNING - connected.
[2024-07-24, 17:19:51 UTC] {ssh.py:529} WARNING - Logging in as yyy ...
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - Logged in!
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - ==> SYST ...
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - done. ==> PWD ...
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - done.
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - ==> TYPE I ...
[2024-07-24, 17:19:52 UTC] {ssh.py:529} WARNING - done. ==> CWD (1) /FinancialNews/xml/daily ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - done.
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ==> PASV ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - done. ==> LIST ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - done.
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - 0K .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .....
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - . .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .. .
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ..
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ..
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .. ..
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - ......
[2024-07-24, 17:19:53 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - . ......
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .... 117K
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - 50K ...
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ....
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ..
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ....
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ... ...
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ......
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .......
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - 16.9M=0.4s
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ==> PASV ...
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - done. ==> LIST ...
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - done.
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - 0K .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - ..
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .....
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - . .
[2024-07-24, 17:19:54 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ..
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - .
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - .. .
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ......... .
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - .....
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ...
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - . ........
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - .. 118K
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - 50K ..
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ........ .......... .......... ....... 15.9M
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - =0.4s
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - 2024-07-24 13:19:55 (411 KB/s) - ‘xxx/FinancialNews/xml/daily/.listing’ saved [179980]
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - Removed ‘xxx/FinancialNews/xml/daily/.listing’.
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - --2024-07-24 13:19:55-- ftp://xxx/FinancialNews/xml/daily/Finan ... 325.xml.gz
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - => ‘xxx/FinancialNews/xml/daily/FinancialNews_N_202407240325.xml.gz’
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ==> CWD not required.
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - ==> PASV ...
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - done. ==> RETR FinancialNews_N_202407240325.xml.gz ...
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - done.
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - Length: 48619
[2024-07-24, 17:19:55 UTC] {ssh.py:529} WARNING - (47K)
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - 0K ........
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - .. ....
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - ...... ..........
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - .......... ..
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - ..... 100%
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - 78.4K=0.6s
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING -
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - 2024-07-24 13:19:56 (78.4 KB/s) - ‘xxx/FinancialNews/xml/daily/FinancialNews_N_202407240325.xml.gz’ saved [48619]
[2024-07-24, 17:19:56 UTC] {ssh.py:524} INFO - 2024-07-24 13:19:56 EDT|windmaster_utils.py === Finish downloading FinancialNews_N_202407240325.xml.gz ===
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - FINISHED --2024-07-24 13:19:56--
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - Total wall clock time: 5.1s
[2024-07-24, 17:19:56 UTC] {ssh.py:529} WARNING - Downloaded: 1 files, 47K in 1.5s (32.5 KB/s)
Подробнее здесь: https://stackoverflow.com/questions/787 ... -get-stuck
