Как спроектировать последовательные вычисления с привязкой к вводу-выводу и ЦП внутри итераций ⇐ Python

[Гость]

Я делюсь фрагментом кода, в котором хочу продемонстрировать свой вариант использования, циклом, в котором я перебираю группы задач

Код: Выделить всё

   for idx, task in enumerate(tasks):
try:
manager.open_connection(idx)
for jdx, subtask in enumerate(task):
subtask.io_bound_read_from_sql()
subtask.io_bound_read_from_fs()
subtask.cpu_bound_compute()
subtask.io_bound_write_to_sql()
finally:
manager.close_connection(idx)

To date, the instance "manager" wraps SQLAlchemy engine objects in order to manage SQL connections by pools of connections. Both the engine and pool objects are said to be "lazy" as the actual connection is establish when doing the transactions as far as I understood.
Both methods

Код: Выделить всё

io_bound_read_from_sql()

and

Код: Выделить всё

io_bound_write_to_sql()

actually run SQL transactions by the "engines" of SQL Alchemy on a dedicated database identified by

Код: Выделить всё

idx

The idea is to group related tasks referring to the same database, labeled as "subtasks" so that the outer loop iterates over these groups using the same database and the subtasks perform transaction on the same database (i.e. without switching to another database). The idea was to micro-manage these groups so that not to open all the connections/pools of connections together and then keep them alive even if not used and to avoid timeouts/reconnections.
The

Код: Выделить всё

open_connection(idx)

and

Код: Выделить всё

close_connection(idx)

methods actually wrap, among other things, the call to "create_engine" and "engine.dispose" of SQL Alchemy.
Also,

Код: Выделить всё

io_bound_read_from_fs()

wraps, among other things, asyncio-based concurrent IO functions.
The sequence of functions called on the

Код: Выделить всё

subtask

inside the inner loop are sequential, but indipendent. For instance, if

Код: Выделить всё

subtask

with

Код: Выделить всё

jdx

counter equal to

Код: Выделить всё

1

is doing

Код: Выделить всё

io_bound_write_to_sql()

, I would like

Код: Выделить всё

subtask

with

Код: Выделить всё

jdx

2 to start concurrently

Код: Выделить всё

io_bound_read_from_sql()

.
My question is how to design this proof of concepts. I wish not to use

Код: Выделить всё

multiprocessing

because it is too complicated to me, and I wish to leave multiprocessing for horizontal scaling in the futures. I wish to use only 1 process, and use

Код: Выделить всё

multithreading

or asyncio to concurrently manage the works of the tasks. I'm entusiastic about

Код: Выделить всё

asyncio

, and I saw SQL Alchemy also has asyncio-based adapters to work with

Код: Выделить всё

asyncio

instead of

Код: Выделить всё

multithreading

\

Код: Выделить всё

concurrent.futures

(?).
Do you have any suggestions or experience/insights to share?


Источник: https://stackoverflow.com/questions/781 ... iterations