Многопроцессорность Python блокирует внутри асинхронной функции ⇐ Python

[Гость]

I have this non-async code which is running in async function

Basically ast.NodeTransformer transformations

def is_lgf_in_tree(tree: ast.Expression) -> bool: """Walk tree while lgf call is not found""" for n in ast.walk(tree): if ( isinstance(n, ast.Call) and isinstance(n.func, ast.Name) and n.func.id == "lgf" ): return True return False def transform(lgf: LGF, nested_lgfs: dict[str, LGF]) -> Tuple[LGF, ast.Expression]: transformer = LGFTransformer(nested_lgfs=nested_lgfs) tree = ast.parse(lgf.expression, mode="eval") while is_lgf_in_tree(tree): # make flat layer by layer tree = transformer.visit(tree) return lgf, tree In case if NodeTransformer code could help

class LGFTransformer(ast.NodeTransformer): def __init__(self, nested_lgfs: Dict[str, LGF]): super().__init__() self.nested_lgfs = nested_lgfs def visit_Call(self, node: ast.Call): if not isinstance(node.func, ast.Name): return self.generic_visit(node) if not (node.func.id == "lgf" and len(node.args) == 2 and len(node.keywords) == 0): return self.generic_visit(node) if not isinstance(node.args[0], ast.Constant): return self.generic_visit(node) _args = [arg.value for arg in node.args if isinstance(arg, ast.Constant)] lgf = self.nested_lgfs.get(node.args[0].value, None) if not lgf: return node exp = ast.parse(lgf.expression, mode="eval") return exp.body Example of straightforward code results = [] for lgf in initial_lgfs: results.append(transform(lgf, lgfs_by_name))


Example of my attempt to multiprocess it
from multiprocess.pool import Pool arguments = [(lgf, lgfs_by_name) for lgf in initial_lgfs] with Pool(processes=4) as pool: results = pool.starmap(transform, arguments)



My idea is to run ast module in parallel to lower computation time

Results are unpleasant for me. What can I change/play with to transform multiple functions with ast.NodeTransformer. Why those new processes are trying to wait on each other. Is it because I share mutual data? Or async function is the problem? Could deepcopy solve it?

What am I missing and why is it trying to synchronize processes?


Источник: https://stackoverflow.com/questions/781 ... c-function