Есть ли хороший способ ускорить вывод глубокого обучения без пакетной обработки выборок? ⇐ Python

[Гость]

For algorithmic reasons, I would like to perform inference sequentially, sample by sample instead of performing a batched sample. It is Deep Neuronal network with the following structure:

Код: Выделить всё

Model: "sequential" _________________________________________________________________  Layer (type)                Output Shape              Param #    =================================================================  dense (Dense)               (None, 64)                2624                                                                          dense_1 (Dense)             (None, 128)               8320                                                                          dense_2 (Dense)             (None, 128)               16512                                                                         dense_3 (Dense)             (None, 1)                 129                                                                          ================================================================= Total params: 27585 (107.75 KB) Trainable params: 27585 (107.75 KB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________ None 

My testing shows that the batched inference is considerably faster (1.34 seconds):

Код: Выделить всё

start = time.time() y_pred = model.predict(x_test_feature_space) end = time.time() print("Exeuction time: ", end - start) print("Execution time per sample: ", (end-start)/len(x_test_feature_space)) # 672/672 [==============================] - 1s 1ms/step # Exeuction time:  1.345379114151001 # Execution time per sample:  6.26194607470794e-05 

Whereas, the sequential, sample by sample inference is very slow (24 minutes):

Код: Выделить всё

y_pred = [] timings = [] start_total = time.time() for x in x_test_feature_space:     start = time.time()     y_pred.append(model.predict(np.array([x,]), verbose=False))     end = time.time()     timings.append(end-start) end_total = time.time() print("Exeuction time: ", end_total - start_total) print("Average time per sample: ", sum(timings)/len(timings)) # Exeuction time:  1467.2437105178833 # Average time per sample:  0.06825089837496631 

Seeing this contrast, makes me believe the inference cost is lower than simply loading up the model.

Is there any approach I could investigate to improve the timing of sequential (sample by sample) inference? What are the best practices?


Источник: https://stackoverflow.com/questions/781 ... ng-samples