Как обрабатывать пустой массив внутри Transform() в pyspark ⇐ Python
-
Гость
Как обрабатывать пустой массив внутри Transform() в pyspark
I have a nested dataframe, where I am trying to update a certain element within a struct of a array. I am able to achieve that using F.transform(). However the problem arises, when the array within the struct is empty, then getting syntax error :
\[INVALID_EXTRACT_BASE_FIELD_TYPE\] Can't extract a value from "namedlambdavariable()". Need a complex type \[STRUCT, ARRAY, MAP\] but got "STRING". I have a dataframe in pyspark where the nested structure is as follows:
\`|-- details: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- lineempid: array (nullable = true) | | | |-- element: struct (containsNull = true) | | | | |-- empidname: string (nullable = true)\` My objective is to update the value of empidname. I am using the following syntax, which is working as expected
df.withColumn( "details", F.transform( "details", lambda x: x.withField( "lineempid", F.transform( x.lineempid, lambda y: y.withField("empidname", when(F.lower(y.empidname) == "tom","Tom").when(F.lower(y.empidname) == "roy", "Roy").when(F.lower(y.empidname) == "greg", "Greg").otherwise(F.lit(y.empidname))))))) As empidname is of NULL types sometimes in dataframe, value of empidname is NOT populating, so in that case dataframe is populating as follows:
\`|-- details: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- lineempid: array (nullable = true) | | | |-- element: string (containsNull = true)\` In this schema, the above code base is failing.
I am getting the below error :
\[INVALID_EXTRACT_BASE_FIELD_TYPE\] Can't extract a value from "namedlambdavariable()". Need a complex type \[STRUCT, ARRAY, MAP\] but got "STRING". Any leads will be really appreciated.
TIA
Источник: https://stackoverflow.com/questions/778 ... in-pyspark
I have a nested dataframe, where I am trying to update a certain element within a struct of a array. I am able to achieve that using F.transform(). However the problem arises, when the array within the struct is empty, then getting syntax error :
\[INVALID_EXTRACT_BASE_FIELD_TYPE\] Can't extract a value from "namedlambdavariable()". Need a complex type \[STRUCT, ARRAY, MAP\] but got "STRING". I have a dataframe in pyspark where the nested structure is as follows:
\`|-- details: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- lineempid: array (nullable = true) | | | |-- element: struct (containsNull = true) | | | | |-- empidname: string (nullable = true)\` My objective is to update the value of empidname. I am using the following syntax, which is working as expected
df.withColumn( "details", F.transform( "details", lambda x: x.withField( "lineempid", F.transform( x.lineempid, lambda y: y.withField("empidname", when(F.lower(y.empidname) == "tom","Tom").when(F.lower(y.empidname) == "roy", "Roy").when(F.lower(y.empidname) == "greg", "Greg").otherwise(F.lit(y.empidname))))))) As empidname is of NULL types sometimes in dataframe, value of empidname is NOT populating, so in that case dataframe is populating as follows:
\`|-- details: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- lineempid: array (nullable = true) | | | |-- element: string (containsNull = true)\` In this schema, the above code base is failing.
I am getting the below error :
\[INVALID_EXTRACT_BASE_FIELD_TYPE\] Can't extract a value from "namedlambdavariable()". Need a complex type \[STRUCT, ARRAY, MAP\] but got "STRING". Any leads will be really appreciated.
TIA
Источник: https://stackoverflow.com/questions/778 ... in-pyspark
