Как получить очищенные данные из зашумленного tsv-файла с помощью регулярного выражения? ⇐ Python

[Anonymous]

Мне нужно очистить данные, хранящиеся в tsv, в фрейме данных с помощью регулярного выражения, для справки: данные выглядят так

Код: Выделить всё

ps8trw17rlo16s  dh7r1wjixjse72  Theoretical movements expensive. In rural areas, especially at The Fox  positive    2020-06-01 00:00:00
psw4o545h8gc2h  dhykkf6486p9ra  Ave SW components on the East in 1498, and soon
Educational campaigns encounter difficulty. To Socrates, a person will experience a continental    positive    2020-06-01 00:07:00
pscnx5eqtjocca  dhn4dhhp3wm5lt  Kinds are larger, possibly over two million. The country is the city's average. Hollywood    Fril Functional four national
Pp. Cayton, Cage is     Has opened (RTA) coordinates the operation positive    2020-06-01 00:14:00

данные имеют неверный формат, и мне приходится разделять данные на pid, uid, text, предсказание, datetile
Это мой код

Код: Выделить всё

# Sample file path
input_file_path = 'logs.tsv'
output_file_path = 'cleaned_logs.txt'

# Read the malformed TSV file and write it to a text file
with open(input_file_path, 'r', encoding='utf-8') as infile, open(output_file_path, 'w', encoding='utf-8') as outfile:
for line in infile:
outfile.write(line)
import pandas as pd
import re

# Define regex patterns for each column
patterns = {
'pid': r'[A-Za-z0-9]+',  # Matches a word (alphanumeric characters)
'uid': r'[A-Za-z0-9]+',  # Matches a word (alphanumeric characters)
'text': r'([^\t]+)\t([^\t]+)\t([^\t]+)',  # Matches the entire text (paragraph)
'prediction': r'(positive|negative)',  # Matches 'positive' or 'negative'
'datetime': r'[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]{1,3})?'  # Matches datetime format 'YYYY-MM-DD HH:MM:SS'
}

# Function to clean each row based on its pattern
def clean_row(row):
columns = row.split('\t')
if len(columns) != 5:
return [None, None, None, None, None]  # Return a list of Nones if the row is malformed

cleaned_data = []
cleaned_data.append(re.match(patterns['pid'], columns[0]).group(0) if re.match(patterns['pid'], columns[0]) else None)
cleaned_data.append(re.match(patterns['uid'], columns[1]).group(0) if re.match(patterns['uid'], columns[1]) else None)
cleaned_data.append(re.match(patterns['text'], columns[2]).group(0) if re.match(patterns['text'], columns[2]) else None)
cleaned_data.append(re.match(patterns['prediction'], columns[3]).group(0) if re.match(patterns['prediction'], columns[3]) else None)
cleaned_data.append(re.match(patterns['datetime'], columns[4]).group(0) if re.match(patterns['datetime'], columns[4]) else None)
return cleaned_data

# Read the cleaned text file
with open(output_file_path, 'r', encoding='utf-8') as file:
rows = file.readlines()

# Initialize a list to collect cleaned rows
cleaned_rows = []

# Process each row with its corresponding pattern
for row in rows:
row = row.strip()  # Remove leading/trailing whitespace
cleaned_row = clean_row(row)
cleaned_rows.append(cleaned_row)

# Convert the cleaned rows to a DataFrame
df = pd.DataFrame(cleaned_rows, columns=['pid', 'uid', 'text', 'prediction', 'datetime'])

# Display the DataFrame
print(df)

# Optionally, save the DataFrame to a CSV file
df.to_csv('cleaned_logs.csv', index=False)

Какую строку регулярного выражения мне следует использовать для решения такой проблемы?
Я пробовал работать построчно, но мне все равно не удалось захватить весь текст< /п>

Подробнее здесь: https://stackoverflow.com/questions/784 ... sing-regex