Мобильная версияУ нас есть приложение Java, которое использует указанные ниже пакеты для пула баз данных HikariCP.
Код: Выделить всё
+--- com.custom.gwtext:gwtext-server
| +--- org.hibernate:hibernate-hikaricp:5.4.6.Final
| | +--- org.hibernate:hibernate-core:5.4.6.Final (*)
| | \--- com.zaxxer:HikariCP:3.2.0 -> 3.3.0
| | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.36
| +--- com.zaxxer:HikariCP:3.3.0 (*)
Это конфигурация для HikariCP
Код: Выделить всё
00:06:05.521 [main] DEBUG com.zaxxer.hikari.HikariConfig - HikariPool-2 - configuration:
00:06:05.522 [main] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."oracle.jdbc.OracleDriver"
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:oracle:thin:@1.1.1.1:1521:test
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........200000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................18000000
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................90
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................15
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - password........................
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-2"
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - username........................"user"
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
- После запуска приложения Java и входа в систему под пользователем, через несколько минут (вероятно, LeakDetectionThreshold) мы замечаем ниже запись журнала, в которой говорится, что сразу две утечки
Код: Выделить всё
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection@5ae99303 on thread pool-10-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Код: Выделить всё
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection@7b911a8f on thread pool-6-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at jdk.internal.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.gwtext.server.utils.DbUtilImpl.doWork(DbUtilImpl.java:516)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:117)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:67)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQueryInternal(DbUtilImpl.java:266)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQuery(DbUtilImpl.java:173)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:147)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:74)
at com.custom.pins.server.data.helper.QueryHelper.executeSelectFirstRow(QueryHelper.java:54)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstRow(QueryHelper.java:36)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstValueRequired(QueryHelper.java:80)
at com.custom.pins.server.business.bdt.BusinessDataTableDataUtil.existsBusinessTable(BusinessDataTableDataUtil.java:20)
at com.custom.pins.server.business.bdt.BusinessDataTableBuilderImpl.existsTable(BusinessDataTableBuilderImpl.java:225)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.createWsRequestConfig(WebServiceLoggerConfigImpl.java:116)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.getWsRequestConfig(WebServiceLoggerConfigImpl.java:75)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.getWsRequestSpecificConfiguration(WebServiceAuditLogService.java:88)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.logIfConfigEnabled(WebServiceAuditLogService.java:71)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.lambda$getInternalWsLogConsumer$0(WebServiceAuditLogService.java:62)
at com.custom.gwtext.server.security.audit.AuditLogServiceImpl.lambda$createEventHandler$2(AuditLogServiceImpl.java:90)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:150)
at java.base/java.lang.Thread.run(Thread.java:834)
Код: Выделить всё
[HikariPool-2 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-2 - After cleanup stats (total=17, active=2, idle=15, waiting=0)
- Затем через несколько часов это может быть 3–10 часов, так что конкретного диапазона нет, возможно, зависит от активности и т. д., понятия не имею, это появляется в журналах
Код: Выделить всё
[pool-10-thread-1] WARN c.zaxxer.hikari.pool.ProxyConnection - HikariPool-2 - Connection oracle.jdbc.driver.T4CConnection@5ae99303 marked as broken because of SQLSTATE(08006), ErrorCode(17002)
java.sql.SQLRecoverableException: IO Error: Connection reset by peer
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:929)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyCallableStatement.execute(HikariProxyCallableStatement.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:34)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
... 29 common frames omitted
[pool-10-thread-1] INFO com.zaxxer.hikari.pool.ProxyLeakTask - Previously reported leaked connection oracle.jdbc.driver.T4CConnection@5ae99303 on thread pool-10-thread-1 was returned to the pool (unleaked)
[HikariPool-2 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-2 - Closing connection oracle.jdbc.driver.T4CConnection@5ae99303: (connection is broken)
И сразу после этого наше Java-приложение начинает выдавать много (даже около 10 в секунду, иногда меньше) ошибок ниже и становится очень медленным
Код: Выделить всё
[pool-10-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
[pool-10-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Connection is closed
[pool-10-thread-1] ERROR l.p.p.s.i.DatabaseSequenceIdPool - error executing work
org.hibernate.exception.GenericJDBCException: error executing work
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:311)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy307.prepareCall(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.prepareCall(ProxyConnection.java:290)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareCall(HikariProxyConnection.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:31)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
... 16 common frames omitted
Может ли это быть связано с HikariCP? Похоже, мы используем довольно устаревшую версию Hikari и Hibernate, но каким-то образом эти версии встроены в какой-то специальный пакет com.custom, и мы не знаем, как мы можем «перезаписать» эти версии. Мы используем Gradle для создания файла WAR.
Да, и еще добавлю: при работе без Kubernetes таких проблем не возникает. Возможно, это связано с сетью/брандмауэрами, но я почему-то думаю, что HikariCP должен обойти эти разорванные соединения в случае их возникновения.
Я был бы очень признателен за помощь или даже за некоторые подсказки, где/что искать.
Подробнее здесь: https://stackoverflow.com/questions/730 ... eset-by-pe
