Приложение Spark и неясное исключение о prelaunch.err не существует в узлах диспетчера узлов ⇐ Linux

[Anonymous]

мы используем структурированную потоковую передачу Spark с Spark 3.x. и запускаем множество приложений Spark в кластере Hadoop, который включает 323 машины с менеджерами узлов.
примечания:

• каждая машина с менеджером узлов включает службу узла данных для использования файловой системы HDFS.
• у нас есть кластер Hadoop, основанный на HDP 2.6. Версия .5 и служба диспетчеров узлов управляются службами YARN (активные/резервные службы резервирования)

Одно из искровых приложений не удалось, и мы все еще расследуем проблему проблема, но одно интересное исключение касается сообщения: prelaunch.err не существует
, которое приходит из журнала - /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-worker1 .hadoop-cluster-poc.com , и то же исключение определено и на других компьютерах с менеджерами узлов
2024-04-09 10:54:55,268 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
java.io.FileNotFoundException: File /data/sdd/hadoop/yarn/local/application_17847938726_0108/container_e1771_1712601932026_0108_04_000019/prelaunch.err does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:624)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.handleContainerExitWithFailure(ContainerLaunch.java:442)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:379)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:87)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

2024-04-09 10:57:51,942 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(249)) - Exit code from container container_e1771_1712601932026_0108_07_000013 is : 2
2024-04-09 10:57:51,942 WARN nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:launchContainer(255)) - Exception from container-launch with container ID: container_e1771_1712601932026_0108_07_000013 and exit code: 2
ExitCodeException exitCode=2:

что означает - prelaunch.err не существует ? это происходит из исключения, как описано,
здесь также другой пример, показывающий сохранение исключения в журнале
grep -i error /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-worker1.hadoop-cluster-poc.com

2024-04-09 15:53:07,889 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:53:32,821 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:54:05,233 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:56:40,718 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:57:04,488 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:57:11,070 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:57:35,905 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:57:48,465 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:58:19,757 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:59:02,691 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 15:59:09,309 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:01:28,173 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:02:46,909 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:02:53,515 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:03:05,929 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:03:39,537 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:03:46,160 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
2024-04-09 16:03:52,755 ERROR launcher.ContainerLaunch (ContainerLaunch.java:handleContainerExitWithFailure(455)) - Failed to get tail of the container's prelaunch error log file
.
.
.
.


Подробнее здесь: https://stackoverflow.com/questions/782 ... exist-from