Tensorflow замораживает перетасованный набор данных ⇐ Python

[Гость]

For training model i am using shuffled dataset wich created so:

Код: Выделить всё

train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)

but after training i want to check the prediction results with some simple test function

Код: Выделить всё

def detect_wrong_ts(model,ds,label_encoder):
# Take first elements batch
ds = ds.take(1)
# Do data preprocessing
# it converts strings from initial dataset to indexes "word1 word2" -> [ 1, 2 ]
mapper = lambda in1,in2,out : (model.preprocessor(in1),out)
train = ds.map(mapper)

# Do predictions on postprocessed dataset
x = model.predict(train)

# Now i try to print results with the information based
# on the
# 1 ) initial dataset
# 2 ) postprocessed dataset
# 3 ) prediction
for batch,outp in zip(ds,train):
inp = batch[0]
outp_code = outp[0]
for i,inp in enumerate(inp):
inp_str = inp.numpy().decode("utf-8")
inp_codes = model.preprocessor([inp_str])[0].numpy()
postprocess_codes = outp_code[i].numpy()
print(
f"#{i} {inp_str=} {inp_codes=} {postprocess_codes=}",
)

But i see that the results of this two datasets are shuffled and don't match each other.
The reason is clear : after starting a new iterator operator shuffling starts again.
I can switch off full shuffling but it is rather complicated for the full pipeline.
I am thinking about some option which can freeze the dataset so that all iterations on it produce same results.
Is it possible to solve this problem this way ?


Источник: https://stackoverflow.com/questions/781 ... ed-dataset