feat: watchdog v4 + check-alert v2 + collect-business v2 + DUMMY补数据脚本
This commit is contained in:
commit
7ab75364b1
22 files changed
+1795
No files matched your search
@@ -0,0 +1,49 @@
|
||||
DUMMY 补数据 · 执行与回滚方案
|
||||
==============================
|
||||
|
||||
目标:根治 updateBizState 死循环 → 推送线程饿死 → Resin 反复重启
|
||||
表:ecology_message_info_read → 只 INSERT,不删不改
|
||||
脚本:/opt/oa-monitor/bin/fix-info-read-dummy.py
|
||||
执行时间:2026-11-04 下午
|
||||
|
||||
问题一句话
|
||||
----------
|
||||
泛微归档流程的 info_read 记录被清空了(归档 389 万条 vs info_read 归档=0 条)。
|
||||
updateBizState 线程池扫归档流程 → 找不到 info_read → 10 线程全部死循环占满 →
|
||||
推送线程饿死 → message_log 停写 → Resin 被 watchdog 重启。5 天 7 次重启。
|
||||
|
||||
根因证据
|
||||
--------
|
||||
info_read 归档记录: 0 条
|
||||
归档流程总量: 389 万条
|
||||
10-26 jstack 抓到 updateBizState 10 线程全 WAITING(实锤)
|
||||
10-26 补 60/138 两人 DUMMY → 立即恢复正常 2 天(治疗验证)
|
||||
11-04 上午 updateBizState 扫到未补的用户 → 3 小时死 3 次(复现)
|
||||
watchdog 日志 20 次 count=0 → 重启
|
||||
|
||||
执行步骤
|
||||
--------
|
||||
Phase 0: 前置检查(Resin HTTP 200 / MySQL Threads_running<15 / 现有 DUMMY 存在)
|
||||
Phase 1: 备份(CREATE TABLE ... AS SELECT)
|
||||
Phase 2: 分批 INSERT
|
||||
Phase 2a: CREATER TOP 100 业务用户(userid != 1)
|
||||
Phase 2b: LASTOPERATOR TOP 100 业务用户(userid != 1)
|
||||
每批 5000 条 + sleep 1s + 每批前检查 Resin/MySQL 健康
|
||||
Phase 3: 后置验证 + 钉钉通知
|
||||
|
||||
锁库风险控制
|
||||
----------
|
||||
每批 INSERT 5000 条,单次持有表锁 < 1s
|
||||
批次间隔 sleep(1)
|
||||
总批数 ≈ 48 批,总时长 60-90 min
|
||||
|
||||
回滚方案(3 种粒度)
|
||||
-------------------
|
||||
粒度1 全部回滚: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%";
|
||||
粒度2 按用户: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%_1723_%";
|
||||
粒度3 按时间: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%" AND CREATEDATE >= "2026-11-04";
|
||||
|
||||
紧急刹车
|
||||
--------
|
||||
kill $(ps -eo pid,cmd | grep fix-info-read | grep -v grep | awk "{print \\$1}")
|
||||
断点续插: 重跑脚本即可
|
||||
Reference in new issue
Block a user