Как Log4J Clean Up на самом деле работает в Hive Metastore? ⇐ JAVA

Программисты JAVA общаются здесь
Anonymous
Как Log4J Clean Up на самом деле работает в Hive Metastore?

Сообщение Anonymous »

Я пытаюсь настроить журналы для Hive Metastore. Я использую эту конфигурацию: < /p>

Код: Выделить всё

status = INFO
name = HiveLog4j2
packages = org.apache.hadoop.hive.ql.log

property.hive.log.level = INFO
property.hive.root.logger = DRFA
property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name}
property.hive.log.file = hive.log
property.hive.perflogger.log.level = INFO

appenders = console, DRFA

appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n

appender.DRFA.type = RollingRandomAccessFile
appender.DRFA.name = DRFA
appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd-hh-mm}
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.DRFA.policies.type = Policies
appender.DRFA.policies.time.type = SizeBasedTriggeringPolicy
appender.DRFA.policies.time.size = 4Kb
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 3

appender.DRFA.strategy.fileIndex = nomax
appender.DRFA.strategy.action.type = Delete
appender.DRFA.strategy.action.basepath =  ${sys:hive.log.dir}
appender.DRFA.strategy.action.condition.type = IfFileName
appender.DRFA.strategy.action.condition.glob = ${sys:hive.log.file}*
appender.DRFA.strategy.action.condition.ifAny.type = IfAny
appender.DRFA.strategy.action.condition.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
appender.DRFA.strategy.action.condition.ifAny.ifAccumulatedFileSize.exceeds = 30Kb

# list of all loggers
loggers = NIOServerCnxn, ClientCnxnSocketNIO, DataNucleus, Datastore, JPOX, PerfLogger, AmazonAws, ApacheHttp
logger.NIOServerCnxn.name = org.apache.zookeeper.server.NIOServerCnxn
logger.NIOServerCnxn.level = WARN
logger.ClientCnxnSocketNIO.name = org.apache.zookeeper.ClientCnxnSocketNIO
logger.ClientCnxnSocketNIO.level = WARN
logger.DataNucleus.name = DataNucleus
logger.DataNucleus.level = ERROR
logger.Datastore.name = Datastore
logger.Datastore.level = ERROR
logger.JPOX.name = JPOX
logger.JPOX.level = ERROR
logger.AmazonAws.name=com.amazonaws
logger.AmazonAws.level = INFO
logger.ApacheHttp.name=org.apache.http
logger.ApacheHttp.level = INFO
logger.PerfLogger.name = org.apache.hadoop.hive.ql.log.PerfLogger
logger.PerfLogger.level = ${sys:hive.perflogger.log.level}
# root logger
rootLogger.level = ${sys:hive.log.level}
rootLogger.appenderRefs = root
rootLogger.appenderRef.root.ref = ${sys:hive.root.logger}
< /code>
Так что это должно: < /p>
- rolling logs every minute
- keep log files size `4Kb`
- keep only 3 logs files
- clean up old log files when entire folder grows more than `30Kb`
< /code>
Что на самом деле делает:
- Есть гораздо более 3 журналов
- Все файлы усекаются до 8Kb < /code>
- Размер папки намного больше 30 КБ < /code>
- некоторые файлы, такие как hive.log.2025-06-06-25 < /code hive.log.2025-06-26-06-26
были удалены с помощью 5 и 10 минут LAG
могут быть мои настройки слишком малы, и е с т ь з а д е р ж к а д л я о ч и с т к и ж у р н а л о в ? Е с т ь м ы с л и ? < c o d e > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 2 8 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 9 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 9 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 9 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 9 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 3 0 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 6 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 7 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 9 < b r / > 8 . 0 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 3 0 < b r / > h i v e @ h i v e - m e t a s t o r e : / o p t / h i v e $ d u - h s / t m p / h i v e / * < b r / > 4 . 0 K / t m p / h i v e / h i v e . l o g < b r / > 1 2 K / t m p / h i v e / h i v e . l o g . 2 0 2 5 - 0 6 - 2 6 - 0 6 - 2 5 < b r / > 8 . 0 K / t m p / h i v e / h i ve.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
hive@hive-metastore:/opt/hive$
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-26
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
8.0K /tmp/hive/hive.log.2025-06-26-06-37
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-27
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
8.0K /tmp/hive/hive.log.2025-06-26-06-37
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
8.0K /tmp/hive/hive.log.2025-06-26-06-37
8.0K /tmp/hive/hive.log.2025-06-26-06-39
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-29
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
8.0K /tmp/hive/hive.log.2025-06-26-06-37
8.0K /tmp/hive/hive.log.2025-06-26-06-39
hive@hive-metastore:/opt/hive$ du -hs /tmp/hive/*
4.0K /tmp/hive/hive.log
8.0K /tmp/hive/hive.log.2025-06-26-06-30
8.0K /tmp/hive/hive.log.2025-06-26-06-31
8.0K /tmp/hive/hive.log.2025-06-26-06-35
8.0K /tmp/hive/hive.log.2025-06-26-06-36
8.0K /tmp/hive/hive.log.2025-06-26-06-37
8.0K /tmp/hive/hive.log.2025-06-26-06-39
8.0K /tmp/hive/hive.log.2025-06-26-06-45

После последнего звонка я вижу 7 файлов, а не 3

Подробнее здесь: https://stackoverflow.com/questions/796 ... -metastore
Вернуться к началу

Вернуться в «JAVA»