Files
oa-monitor/docs/DUMMY-fix-plan.md

50 lines
2.0 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
DUMMY 补数据 · 执行与回滚方案
==============================
目标:根治 updateBizState 死循环 → 推送线程饿死 → Resin 反复重启
表:ecology_message_info_read → 只 INSERT,不删不改
脚本:/opt/oa-monitor/bin/fix-info-read-dummy.py
执行时间:2026-11-04 下午
问题一句话
----------
泛微归档流程的 info_read 记录被清空了(归档 389 万条 vs info_read 归档=0 条)。
updateBizState 线程池扫归档流程 → 找不到 info_read → 10 线程全部死循环占满 →
推送线程饿死 → message_log 停写 → Resin 被 watchdog 重启。5 天 7 次重启。
根因证据
--------
info_read 归档记录: 0 条
归档流程总量: 389 万条
10-26 jstack 抓到 updateBizState 10 线程全 WAITING(实锤)
10-26 补 60/138 两人 DUMMY → 立即恢复正常 2 天(治疗验证)
11-04 上午 updateBizState 扫到未补的用户 → 3 小时死 3 次(复现)
watchdog 日志 20 次 count=0 → 重启
执行步骤
--------
Phase 0: 前置检查(Resin HTTP 200 / MySQL Threads_running<15 / 现有 DUMMY 存在)
Phase 1: 备份(CREATE TABLE ... AS SELECT)
Phase 2: 分批 INSERT
Phase 2a: CREATER TOP 100 业务用户(userid != 1)
Phase 2b: LASTOPERATOR TOP 100 业务用户(userid != 1)
每批 5000 条 + sleep 1s + 每批前检查 Resin/MySQL 健康
Phase 3: 后置验证 + 钉钉通知
锁库风险控制
----------
每批 INSERT 5000 条,单次持有表锁 < 1s
批次间隔 sleep(1)
总批数 ≈ 48 批,总时长 60-90 min
回滚方案(3 种粒度)
-------------------
粒度1 全部回滚: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%";
粒度2 按用户: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%_1723_%";
粒度3 按时间: DELETE FROM ecology_message_info_read WHERE ID LIKE "DUMMY_B_%" AND CREATEDATE >= "2026-11-04";
紧急刹车
--------
kill $(ps -eo pid,cmd | grep fix-info-read | grep -v grep | awk "{print \\$1}")
断点续插: 重跑脚本即可