test: stabilize distributed load verification

docs/test/IM-group2k-r6-群聊复测分析-20260928.md

@@ -0,0 +1,93 @@
+# IM 2000 人群聊 r6 复测分析
+
+日期：2026-09-28
+
+Run ID：`group2k-r6-20260928-01`
+
+## 1. 结论
+
+r6 比 r5 有改善，但仍未通过。A 去掉 Fake-IP 后 ACK 数从 426 增加到 758，统计窗口断线从 657 降到 213；然而 `api.118im.com` 的真实公网 IP 仍经 `utun4`，A 在正式开始前就发生 28 次连接抖动，正式开始后继续成批断线。后续排查确认该 `utun4` 由 OpenVPN Connect 的全局路由持有，而不是 LightXtreme。B 也出现 2 次断线和 1 次发布结果未知，不再是完全零错误。
+
+历史落库仍然可靠：ACK 台账和未 ACK 观测台账合计核验 1870/1870，持久化丢失为 0。FD 修复已验证生效，A 的存活连接从 1000 降至 787 时，open FDs 从约 1013 同步降至约 799，不再滞留断开的 socket。
+
+历史限速和退出行为仍会污染服务端队列指标：历史查询阶段约产生 436 次 Topic 队列满；A 结束时一次性关闭大量群订阅连接，又集中产生约 598 次队列满。
+
+总体结论：**落库正确性通过；FD 修复通过；实时可靠性失败；A 的 OpenVPN/TUN 链路仍未绕过；历史回放与集中断连仍需改进。**
+
+## 2. 汇总结果
+
+| 指标 | A | B | 合计 |
+|---|---:|---:|---:|
+| READY | 1000 | 1000 | 2000 |
+| 发送尝试 | 1000 | 1000 | 2000 |
+| ACK | 758 | 999 | 1757 |
+| ACK 超时 | 122 | 0 | 122 |
+| socket/连接结果未知 | 120 | 1 | 121 |
+| 实际明确拒绝 | 0 | 0 | 0 |
+| 断线 | 213 | 2 | 215 |
+| ACK 台账内应收 | 1757000 | 1757000 | 3514000 |
+| ACK 台账内实收 | 1468873 | 1756514 | 3225387 |
+| ACK 台账内实时缺失 | 288127 | 486 | 288613 |
+| ACK 台账外额外投递 | 94965 | 112984 | 207949 |
+| 历史核验 | 1870/1870 | 0/0 | 1870/1870 |
+| 未 ACK 观测消息核验 | 113/113 | 0/0 | 113/113 |
+
+服务端实际处理消息至少为 `1757 + 113 = 1870` 条。其余 130 条结果未知没有 ACK 或实时观测证据，不能证明服务端接受。
+
+## 3. 报告口径修正
+
+合并报告当前显示 91 次明确拒绝和 152 次结果未知，这是错误分类：
+
+- A 的 91 次 `use of closed network connection` 被归类为 `other`。
+- 该错误是连接关闭后的写失败，应该归类为 `socket` 结果未知。
+- 修正后本轮明确拒绝为 0，结果未知为 243。
+
+此外，群历史只由 A 执行。A 已证明全部 1757 条 ACK 消息落库，但 B 的 `realtime_lost_but_persisted` 无法读取 A 的历史结果，因此 B 的 486 次实时缺失没有计入“实时缺失但已落库”。合并报告需要共享全局历史核验结果后再计算该字段。
+
+## 4. A 网络与 FD
+
+本轮 DNS 返回真实地址：
+
+```text
+64.118.158.131
+216.236.31.189
+216.236.31.191
+```
+
+但到真实地址的系统路由仍为：
+
+```text
+gateway: 172.27.240.1
+interface: utun4
+```
+
+这说明 Fake-IP 已关闭，但流量仍进入 VPN/TUN。A 在 barrier 提交前持续掉线重连，正式开始后约 50 秒开始掉线，最终只剩约 787 个连接。后续确认 OpenVPN Connect 下发了 `0/1` 和 `128.0/1` 两条全局路由；LightXtreme 的 DIRECT 规则无法绕过另一套 OpenVPN TUN。r7 前已为 `api.118im.com` 当前三个 IP 添加经 `192.168.1.1/en1` 的 `/32` 主机路由，保留 OpenVPN 内网访问的同时让压测 HTTP/WS 直连。
+
+FD 时间序列证明 read loop 修复有效：
+
+- READY 1000 时 open FDs 约 1013。
+- READY 901 时 open FDs 约 913。
+- READY 787 时 open FDs 约 799。
+
+连接每减少一个，FD 基本同步减少一个。
+
+## 5. Topic 队列满
+
+实时发送和 drain 阶段仍没有 Topic/Proxy 满队列。`im-server-2` 的 1034 次 `topic's broadcast queue is full` 分为：
+
+- `00:35:59–00:36:11 UTC`：约 436 次，发生在历史回查阶段。
+- `00:37:19 UTC`：约 598 次，发生在 A agent 完成后批量关闭连接阶段。
+
+20 条/500ms 虽有改善，但 Topic 历史数据仍以突发方式回送。建议：
+
+1. 历史每页进一步降至 10 条，查询间隔保持 500ms。
+2. 每页请求后等待本页目标消息全部收到或超时，再发送下一页，而不是只按固定时间节流。
+3. agent 退出时按 10–20ms 间隔分批关闭连接，避免 1000 个群订阅同时离线。
+
+## 6. 后续修复
+
+1. 已将 `use of closed network connection` 分类为 socket 结果未知。
+2. 已在 coordinator 共享两机历史命中 ID，使 B 也能正确计算实时缺失但已落库。
+3. 历史回查已改为每页 10 条：同一 Topic 每页完成确认后再请求下一页，不同 Topic 最多 12 路并行。
+4. agent 已改为在 10 秒窗口内渐进关闭 WebSocket。
+5. r7 开始前重新解析 `api.118im.com`，确认每个 IP 都通过 `/32` 主机路由走 `en1`；OpenVPN 重连、DNS 变更或机器重启后必须重新核验。

loadtest/im-pressure/MACHINE-B.md

@@ -148,8 +148,8 @@ mkdir -p "loadtest/im-pressure/results/${RUN_ID}"
 3. 如果机器 A 尚未 READY，当前命令会等待，这是正常现象。
 4. 等待期间连接断开会自动重连并重新订阅；在 A/B 都实时保持 1000 条连接并完成最终屏障前，agent 不会退出或开始发压。
 5. coordinator 下发 `run_id`、场景和统一 UTC 开始时间。
-6. agent 完成发送和 WS 到达统计后，先向协调器提交未 ACK 但已观测的消息台账；等待 `history_settle`（默认 5 分钟）后，以限速分页方式执行历史持久化回查。等待期间不要停止进程。
-7. agent 写出 summary、上报 coordinator，然后正常退出。
+6. agent 完成发送和 WS 到达统计后，先向协调器提交未 ACK 但已观测的消息台账；等待 `history_settle`（默认 5 分钟）后，以同 Topic 串行、不同 Topic 并行的限速分页方式执行历史持久化回查。等待期间不要停止进程。
+7. 两机历史回查完成后会通过 coordinator 汇总命中消息，B 也会据此计算 `realtime_lost_but_persisted`；随后 agent 在默认 10 秒窗口内渐进关闭连接、写出 summary 并正常退出。
 
 机器 B 不需要传场景参数。`p2p`、`group` 或 `mixed` 由机器 A coordinator 决定。
 

loadtest/im-pressure/README.md

@@ -44,6 +44,7 @@
 - agent 默认每 20 秒发送 WebSocket Ping，覆盖等待另一台机器、发压、drain 和历史核验阶段；`ws_ping_interval` 必须小于公网入口的空闲连接超时。
 - 正式开始前 agent 会持续检查实际存活连接；连接断开后自动重新登录并订阅 P2P、群 Topic。初次连接不齐或等待另一台机器期间掉线不会退出。
 - coordinator 使用两阶段屏障：两边实时连接数都达到配置值后进入验证倒计时；倒计时内任一方掉线就撤销屏障。两边再次确认满连接后才提交最终开始时间。
+- 压测机如保持 OpenVPN 等全局 TUN，仅在另一套代理中设置 `DIRECT` 无法绕过它。起压前必须确认 `http_base` 和 WS 域名的每个当前 IP 都走物理网卡；需保留 VPN 访问内网时，可为这些 IP 配置经局域网关的 `/32` 主机路由。VPN 重连、DNS 变更或机器重启后需重新核验。
 
 ## 统计口径
 
@@ -52,7 +53,8 @@
 - 群消息设置 `noecho=false`，每条服务端接受的群消息期望 2000 个连接各收到一次。
 - 实时 drain 结束后，两机先汇总 ACK 台账和“已实时观测但未收到 ACK”的消息台账；等待 `history_settle`（默认 5 分钟）让 Kafka 自然排空，再通过 `{get what=data}` 按 Topic/Seq 回查两类消息。
 - 每轮建立 WS 后、正式计时前，以 32 个 worker 对本机 1000 个账号做好友关系、P2P Topic 和发布权限预检；任一异常会终止本轮。
-- 历史回查默认每机使用 1 个 worker、每页 20 条、查询间隔 500ms、总超时 3 分钟，并在超时内持续重试缺失区间。该限速用于避免正确性核验本身制造 Topic/Proxy 洪峰；历史读取吞吐应使用独立场景测试。
+- 历史回查默认每机最多使用 12 个 worker、每页 10 条、同一 Topic 两页之间至少间隔 500ms，并在每页请求后等待目标消息全部返回（最长 3 秒）再继续。同一 Topic 严格串行，不同 Topic 才并行；因此群聊只会有 1 个活跃回查 worker，单聊可并行核验不同 Topic。总超时默认 3 分钟，并在超时内持续重试缺失区间。该限速用于避免正确性核验本身制造 Topic/Proxy 洪峰；历史读取吞吐应使用独立场景测试。
+- agent 完成后默认在 10 秒内均匀关闭本机 WebSocket，避免 1000 个群订阅同时离线冲击 Topic 队列。两机退出时间可能相差数秒，协调器完成以两份 summary 都已提交为准。
 - publish 错误完整分类为 `403`、`502`、`503`、`ctrl_timeout`、`socket_error` 和 `other`。`publish_rejected` 只统计明确拒绝，超时和 socket 错误单独计入 `publish_outcome_unknown`。
 - `delivery_local_p95_ms` 使用发送方本机收到 echo 的同机时钟样本，是投递延迟门禁口径；兼容字段 `delivery_p95_ms` 与它保持一致。
 - `delivery_cross_machine_p95_ms` 是跨机器单向观测值，只有两机时钟同步达标时才有意义；`cross_machine_negative_latency_samples > 0` 明确表示存在时钟偏差。

loadtest/im-pressure/agent.go

@@ -234,13 +234,20 @@ func runAgent(ctx context.Context, cfg *Config, manifest *Manifest, machine, coo
 		return err
 	}
 	history := run.reconcileHistory(ctx, verificationLedger, ledger)
+	if err = coord.submitHistory(ctx, machine, history.foundIDs); err != nil {
+		return err
+	}
+	storedMessageIDs, err := coord.history(ctx)
+	if err != nil {
+		return err
+	}
 	summary.HistoryChecked = int64(history.checked)
 	summary.HistoryFound = int64(history.found)
 	summary.HistoryUnackedChecked = int64(history.unackedChecked)
 	summary.HistoryUnackedFound = int64(history.unackedFound)
 	persistenceMissing := history.missing
 	summary.PersistenceLost = int64(len(persistenceMissing))
-	summary.RealtimeLostStored = run.recorder.countRealtimeLostStored(ledger)
+	summary.RealtimeLostStored = run.recorder.countRealtimeLostStored(ledger, storedMessageIDs)
 	if len(persistenceMissing) > 10000 {
 		summary.PersistenceMissing = append([]string(nil), persistenceMissing[:10000]...)
 		summary.Errors = append(summary.Errors, "persistence missing details truncated at 10000")
@@ -261,11 +268,12 @@ func runAgent(ctx context.Context, cfg *Config, manifest *Manifest, machine, coo
 }
 
 type historyQuery struct {
-	client *WSClient
-	topic  string
-	since  int64
-	before int64
-	limit  int
+	client   *WSClient
+	topic    string
+	since    int64
+	before   int64
+	limit    int
+	expected []string
 }
 
 type historyResult struct {
@@ -274,6 +282,7 @@ type historyResult struct {
 	unackedChecked int
 	unackedFound   int
 	missing        []string
+	foundIDs       []string
 }
 
 func (r *agentRun) reconcileHistory(ctx context.Context, ledger, ackLedger []AcceptedMessage) historyResult {
@@ -316,8 +325,7 @@ func (r *agentRun) reconcileHistory(ctx context.Context, ledger, ackLedger []Acc
 			if end > len(messages) {
 				end = len(messages)
 			}
-			queries = append(queries, historyQuery{client: client, topic: topic,
-				since: messages[start].Seq, before: messages[end-1].Seq + 1, limit: pageSize})
+			queries = append(queries, newHistoryQuery(client, topic, messages[start:end], pageSize))
 		}
 	}
 	historyTimeout, _ := configDuration(r.cfg.HistoryTimeout)
@@ -340,8 +348,12 @@ func (r *agentRun) reconcileHistory(ctx context.Context, ledger, ackLedger []Acc
 		missingIDs = append(missingIDs, message.ClientMid)
 	}
 	sort.Strings(missingIDs)
-	result := historyResult{checked: len(expected), found: len(expected) - len(missing), missing: missingIDs}
+	result := historyResult{checked: len(expected), found: len(expected) - len(missing), missing: missingIDs,
+		foundIDs: make([]string, 0, len(expected)-len(missing))}
 	for mid := range expected {
+		if r.recorder.historyFound(mid) {
+			result.foundIDs = append(result.foundIDs, mid)
+		}
 		if acked[mid] {
 			continue
 		}
@@ -350,6 +362,7 @@ func (r *agentRun) reconcileHistory(ctx context.Context, ledger, ackLedger []Acc
 			result.unackedFound++
 		}
 	}
+	sort.Strings(result.foundIDs)
 	return result
 }
 
@@ -374,48 +387,62 @@ func (r *agentRun) historyQueriesForMessages(messages []AcceptedMessage) []histo
 			if end > len(topicMessages) {
 				end = len(topicMessages)
 			}
-			queries = append(queries, historyQuery{client: client, topic: topic,
-				since: topicMessages[start].Seq, before: topicMessages[end-1].Seq + 1, limit: pageSize})
+			queries = append(queries, newHistoryQuery(client, topic, topicMessages[start:end], pageSize))
 		}
 	}
 	return queries
 }
 
 func (r *agentRun) issueHistoryQueries(ctx context.Context, queries []historyQuery) {
+	byTopic := make(map[string][]historyQuery)
+	for _, query := range queries {
+		byTopic[query.topic] = append(byTopic[query.topic], query)
+	}
 	workers := r.cfg.HistoryWorkers
-	if workers > len(queries) {
-		workers = len(queries)
+	if workers > len(byTopic) {
+		workers = len(byTopic)
 	}
 	if workers == 0 {
 		return
 	}
-	jobs := make(chan historyQuery)
+	jobs := make(chan []historyQuery)
 	interval := 500 * time.Millisecond
 	if r.cfg != nil && r.cfg.HistoryInterval != "" {
 		interval = mustDuration(r.cfg.HistoryInterval)
 	}
-	limiter := time.NewTicker(interval)
-	defer limiter.Stop()
+	pageWait := 3 * time.Second
+	if r.cfg != nil && r.cfg.HistoryPageWait != "" {
+		pageWait = mustDuration(r.cfg.HistoryPageWait)
+	}
 	var wg sync.WaitGroup
 	for i := 0; i < workers; i++ {
 		wg.Add(1)
 		go func() {
 			defer wg.Done()
-			for query := range jobs {
-				select {
-				case <-limiter.C:
-				case <-ctx.Done():
-					return
-				}
-				if err := query.client.GetData(ctx, query.topic, query.since, query.before, query.limit); err != nil && ctx.Err() == nil {
-					r.addError(fmt.Sprintf("history get topic=%s range=%d-%d: %v", topicHash(query.topic), query.since, query.before, err))
+			for topicQueries := range jobs {
+				for index, query := range topicQueries {
+					if index > 0 {
+						if err := waitContext(ctx, interval); err != nil {
+							return
+						}
+					}
+					if err := query.client.GetData(ctx, query.topic, query.since, query.before, query.limit); err != nil && ctx.Err() == nil {
+						r.addError(fmt.Sprintf("history get topic=%s range=%d-%d: %v", topicHash(query.topic), query.since, query.before, err))
+						continue
+					}
+					r.waitHistoryPage(ctx, query.expected, pageWait)
 				}
 			}
 		}()
 	}
-	for _, query := range queries {
+	topics := make([]string, 0, len(byTopic))
+	for topic := range byTopic {
+		topics = append(topics, topic)
+	}
+	sort.Strings(topics)
+	for _, topic := range topics {
 		select {
-		case jobs <- query:
+		case jobs <- byTopic[topic]:
 		case <-ctx.Done():
 			close(jobs)
 			wg.Wait()
@@ -426,11 +453,39 @@ func (r *agentRun) issueHistoryQueries(ctx context.Context, queries []historyQue
 	wg.Wait()
 }
 
+func newHistoryQuery(client *WSClient, topic string, messages []AcceptedMessage, limit int) historyQuery {
+	expected := make([]string, 0, len(messages))
+	for _, message := range messages {
+		expected = append(expected, message.ClientMid)
+	}
+	return historyQuery{client: client, topic: topic, since: messages[0].Seq,
+		before: messages[len(messages)-1].Seq + 1, limit: limit, expected: expected}
+}
+
+func (r *agentRun) waitHistoryPage(ctx context.Context, expected []string, timeout time.Duration) {
+	deadline := time.NewTimer(timeout)
+	defer deadline.Stop()
+	ticker := time.NewTicker(25 * time.Millisecond)
+	defer ticker.Stop()
+	for {
+		if r.recorder.historyFoundAll(expected) {
+			return
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-deadline.C:
+			return
+		case <-ticker.C:
+		}
+	}
+}
+
 func (r *agentRun) historyPageSize() int {
 	if r.cfg != nil && r.cfg.HistoryPageSize > 0 {
 		return r.cfg.HistoryPageSize
 	}
-	return 20
+	return 10
 }
 
 func missingHistory(recorder *deliveryRecorder, expected map[string]AcceptedMessage) []AcceptedMessage {
@@ -720,7 +775,7 @@ func (r *agentRun) classifyPublishError(err error) {
 	case strings.Contains(err.Error(), "ctrl timeout") || errors.Is(err, context.DeadlineExceeded):
 		r.publishTimeout.Add(1)
 	case strings.Contains(err.Error(), "websocket") || strings.Contains(err.Error(), "broken pipe") ||
-		strings.Contains(err.Error(), "connection reset"):
+		strings.Contains(err.Error(), "connection reset") || strings.Contains(err.Error(), "closed network connection"):
 		r.publishSocket.Add(1)
 	default:
 		r.publishOther.Add(1)
@@ -802,8 +857,16 @@ func (r *agentRun) drainDuration() (time.Duration, error) {
 }
 
 func (r *agentRun) close() {
-	for _, client := range r.clientsSnapshot() {
+	clients := r.clientsSnapshot()
+	var interval time.Duration
+	if r.cfg != nil && r.cfg.CloseRamp != "" && len(clients) > 1 {
+		interval = mustDuration(r.cfg.CloseRamp) / time.Duration(len(clients)-1)
+	}
+	for index, client := range clients {
 		client.Close()
+		if interval > 0 && index < len(clients)-1 {
+			time.Sleep(interval)
+		}
 	}
 }
 

loadtest/im-pressure/agent_history_test.go

@@ -1,6 +1,9 @@
 package main
 
-import "testing"
+import (
+	"errors"
+	"testing"
+)
 
 func TestHistoryQueriesForMessagesGroupsRangesByTopic(t *testing.T) {
 	p2pClient := &WSClient{}
@@ -25,7 +28,7 @@ func TestHistoryQueriesForMessagesGroupsRangesByTopic(t *testing.T) {
 		switch query.topic {
 		case "p2p-a":
 			p2pQueries++
-			if query.client != p2pClient || query.limit != 200 {
+			if query.client != p2pClient || query.limit != 200 || len(query.expected) == 0 {
 				t.Fatalf("unexpected p2p query: %#v", query)
 			}
 		case "grp-a":
@@ -40,6 +43,14 @@ func TestHistoryQueriesForMessagesGroupsRangesByTopic(t *testing.T) {
 	}
 }
 
+func TestClassifyPublishErrorTreatsClosedConnectionAsUnknownSocketError(t *testing.T) {
+	run := &agentRun{}
+	run.classifyPublishError(errors.New("write tcp: use of closed network connection"))
+	if run.publishSocket.Load() != 1 || run.publishOther.Load() != 0 {
+		t.Fatalf("socket=%d other=%d", run.publishSocket.Load(), run.publishOther.Load())
+	}
+}
+
 func TestMergeMessageLedgersPrefersPeerObservationAndAck(t *testing.T) {
 	senderEcho := AcceptedMessage{ClientMid: "mid-1", SenderMachine: "a", PeerMachine: "a", Seq: 10}
 	peerDelivery := AcceptedMessage{ClientMid: "mid-1", SenderMachine: "a", PeerMachine: "b", PeerIndex: 7, Seq: 10}

loadtest/im-pressure/config.example.json

@@ -17,11 +17,13 @@
   "ws_ping_interval": "20s",
   "preflight_workers": 32,
   "skip_p2p_preflight": false,
-  "history_workers": 1,
-  "history_page_size": 20,
+  "history_workers": 12,
+  "history_page_size": 10,
   "history_query_interval": "500ms",
+  "history_page_wait": "3s",
   "history_settle": "5m",
   "history_timeout": "3m",
+  "close_ramp": "10s",
   "p2p_messages_per_user": 10,
   "p2p_interval": "1s",
   "drain_p2p": "30s",

loadtest/im-pressure/coordinator.go

@@ -39,6 +39,7 @@ type coordinatorState struct {
 	armed          map[string]bool
 	ledgers        map[string][]AcceptedMessage
 	observed       map[string][]AcceptedMessage
+	history        map[string][]string
 	summaries      map[string]AgentSummary
 	complete       chan struct{}
 	completeOnce   sync.Once
@@ -58,14 +59,15 @@ func runCoordinator(ctx context.Context, listen, runID, scenario, output string,
 		start: StartInfo{RunID: runID, Scenario: scenario}, startDelay: startDelay, commitDelay: 5 * time.Second,
 		expectedAgents: expectedAgents,
 		ready:          make(map[string]AgentReady), ledgers: make(map[string][]AcceptedMessage),
-		observed: make(map[string][]AcceptedMessage),
-		armed:    make(map[string]bool), summaries: make(map[string]AgentSummary), complete: make(chan struct{}),
+		observed: make(map[string][]AcceptedMessage), history: make(map[string][]string),
+		armed: make(map[string]bool), summaries: make(map[string]AgentSummary), complete: make(chan struct{}),
 	}
 	mux := http.NewServeMux()
 	mux.HandleFunc("/v1/ready", state.handleReady)
 	mux.HandleFunc("/v1/start", state.handleStart)
 	mux.HandleFunc("/v1/ledger", state.handleLedger)
 	mux.HandleFunc("/v1/observed", state.handleObserved)
+	mux.HandleFunc("/v1/history", state.handleHistory)
 	mux.HandleFunc("/v1/summary", state.handleSummary)
 	mux.HandleFunc("/v1/status", state.handleStatus)
 	server := &http.Server{Addr: listen, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
@@ -94,6 +96,52 @@ func runCoordinator(ctx context.Context, listen, runID, scenario, output string,
 	return nil
 }
 
+func (s *coordinatorState) handleHistory(w http.ResponseWriter, r *http.Request) {
+	machine := strings.ToLower(strings.TrimSpace(r.URL.Query().Get("machine")))
+	switch r.Method {
+	case http.MethodPost:
+		if machine != "a" && machine != "b" {
+			http.Error(w, "invalid machine", http.StatusBadRequest)
+			return
+		}
+		var messageIDs []string
+		if !decodeLimitedJSON(w, r, &messageIDs) {
+			return
+		}
+		s.mu.Lock()
+		if s.history == nil {
+			s.history = make(map[string][]string)
+		}
+		s.history[machine] = messageIDs
+		count := len(s.history)
+		s.mu.Unlock()
+		writeJSON(w, http.StatusOK, map[string]int{"submitted_agents": count})
+	case http.MethodGet:
+		s.mu.Lock()
+		defer s.mu.Unlock()
+		if len(s.history) < s.expectedAgents {
+			writeJSON(w, http.StatusAccepted, map[string]int{"submitted_agents": len(s.history), "expected_agents": s.expectedAgents})
+			return
+		}
+		unique := make(map[string]bool)
+		for _, messageIDs := range s.history {
+			for _, mid := range messageIDs {
+				if mid != "" {
+					unique[mid] = true
+				}
+			}
+		}
+		merged := make([]string, 0, len(unique))
+		for mid := range unique {
+			merged = append(merged, mid)
+		}
+		sort.Strings(merged)
+		writeJSON(w, http.StatusOK, merged)
+	default:
+		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+	}
+}
+
 func (s *coordinatorState) handleObserved(w http.ResponseWriter, r *http.Request) {
 	machine := strings.ToLower(strings.TrimSpace(r.URL.Query().Get("machine")))
 	switch r.Method {
@@ -264,7 +312,8 @@ func (s *coordinatorState) handleStatus(w http.ResponseWriter, _ *http.Request)
 	defer s.mu.Unlock()
 	writeJSON(w, http.StatusOK, map[string]any{
 		"start": s.start, "ready": s.ready, "armed": s.armed,
-		"ledger_agents": len(s.ledgers), "observed_agents": len(s.observed), "summary_agents": len(s.summaries),
+		"ledger_agents": len(s.ledgers), "observed_agents": len(s.observed),
+		"history_agents": len(s.history), "summary_agents": len(s.summaries),
 	})
 }
 
@@ -275,8 +324,12 @@ func (s *coordinatorState) writeOutput(path string) error {
 		Ready     map[string]AgentReady        `json:"ready"`
 		Ledgers   map[string][]AcceptedMessage `json:"ledgers"`
 		Observed  map[string][]AcceptedMessage `json:"observed"`
+		History   map[string][]string          `json:"history"`
 		Summaries map[string]AgentSummary      `json:"summaries"`
-	}{s.start, s.ready, s.ledgers, s.observed, s.summaries}
+	}{
+		Start: s.start, Ready: s.ready, Ledgers: s.ledgers, Observed: s.observed,
+		History: s.history, Summaries: s.summaries,
+	}
 	s.mu.Unlock()
 	raw, err := json.MarshalIndent(data, "", "  ")
 	if err != nil {
@@ -387,6 +440,27 @@ func (c *coordinatorClient) observed(ctx context.Context) ([]AcceptedMessage, er
 	}
 }
 
+func (c *coordinatorClient) submitHistory(ctx context.Context, machine string, messageIDs []string) error {
+	_, err := c.request(ctx, http.MethodPost, "/v1/history?machine="+machine, messageIDs, nil)
+	return err
+}
+
+func (c *coordinatorClient) history(ctx context.Context) ([]string, error) {
+	for {
+		var messageIDs []string
+		status, err := c.request(ctx, http.MethodGet, "/v1/history", nil, &messageIDs)
+		if err != nil {
+			return nil, err
+		}
+		if status == http.StatusOK {
+			return messageIDs, nil
+		}
+		if err = waitContext(ctx, time.Second); err != nil {
+			return nil, err
+		}
+	}
+}
+
 func (c *coordinatorClient) submitSummary(ctx context.Context, summary AgentSummary) error {
 	_, err := c.request(ctx, http.MethodPost, "/v1/summary", summary, nil)
 	return err

loadtest/im-pressure/coordinator_test.go

@@ -100,6 +100,16 @@ func TestCoordinatorClientWaitsForBothAgentsAndMergesLedger(t *testing.T) {
 	if err != nil || len(observed) != 1 || observed[0].ClientMid != "unknown" {
 		t.Fatalf("observed=%+v err=%v", observed, err)
 	}
+	if err = client.submitHistory(context.Background(), "a", []string{"mid-2", "mid-1"}); err != nil {
+		t.Fatal(err)
+	}
+	if err = client.submitHistory(context.Background(), "b", []string{"mid-2", "mid-3"}); err != nil {
+		t.Fatal(err)
+	}
+	history, err := client.history(context.Background())
+	if err != nil || len(history) != 3 || history[0] != "mid-1" || history[1] != "mid-2" || history[2] != "mid-3" {
+		t.Fatalf("history=%+v err=%v", history, err)
+	}
 }
 
 func testCoordinatorMux(state *coordinatorState) *http.ServeMux {
@@ -108,6 +118,7 @@ func testCoordinatorMux(state *coordinatorState) *http.ServeMux {
 	mux.HandleFunc("/v1/start", state.handleStart)
 	mux.HandleFunc("/v1/ledger", state.handleLedger)
 	mux.HandleFunc("/v1/observed", state.handleObserved)
+	mux.HandleFunc("/v1/history", state.handleHistory)
 	mux.HandleFunc("/v1/summary", state.handleSummary)
 	mux.HandleFunc("/v1/status", state.handleStatus)
 	return mux

loadtest/im-pressure/model.go

@@ -35,8 +35,10 @@ type Config struct {
 	HistoryWorkers   int                 `json:"history_workers"`
 	HistoryPageSize  int                 `json:"history_page_size"`
 	HistoryInterval  string              `json:"history_query_interval"`
+	HistoryPageWait  string              `json:"history_page_wait"`
 	HistorySettle    string              `json:"history_settle"`
 	HistoryTimeout   string              `json:"history_timeout"`
+	CloseRamp        string              `json:"close_ramp"`
 	PreflightWorkers int                 `json:"preflight_workers"`
 	SkipP2PPreflight bool                `json:"skip_p2p_preflight"`
 }
@@ -84,13 +86,13 @@ func loadConfig(path string) (*Config, error) {
 		cfg.DrainGroup = "60s"
 	}
 	if cfg.HistoryWorkers <= 0 {
-		cfg.HistoryWorkers = 1
+		cfg.HistoryWorkers = 12
 	}
 	if cfg.HistoryWorkers > 128 {
 		return nil, errors.New("history_workers must not exceed 128")
 	}
 	if cfg.HistoryPageSize <= 0 {
-		cfg.HistoryPageSize = 20
+		cfg.HistoryPageSize = 10
 	}
 	if cfg.HistoryPageSize > 200 {
 		return nil, errors.New("history_page_size must not exceed 200")
@@ -101,6 +103,12 @@ func loadConfig(path string) (*Config, error) {
 	if _, err = configDuration(cfg.HistoryInterval); err != nil {
 		return nil, fmt.Errorf("history_query_interval: %w", err)
 	}
+	if cfg.HistoryPageWait == "" {
+		cfg.HistoryPageWait = "3s"
+	}
+	if _, err = configDuration(cfg.HistoryPageWait); err != nil {
+		return nil, fmt.Errorf("history_page_wait: %w", err)
+	}
 	if cfg.HistoryTimeout == "" {
 		cfg.HistoryTimeout = "3m"
 	}
@@ -113,6 +121,12 @@ func loadConfig(path string) (*Config, error) {
 	if _, err = configDuration(cfg.HistoryTimeout); err != nil {
 		return nil, fmt.Errorf("history_timeout: %w", err)
 	}
+	if cfg.CloseRamp == "" {
+		cfg.CloseRamp = "10s"
+	}
+	if _, err = configDuration(cfg.CloseRamp); err != nil {
+		return nil, fmt.Errorf("close_ramp: %w", err)
+	}
 	if cfg.PreflightWorkers <= 0 {
 		cfg.PreflightWorkers = 32
 	}

loadtest/im-pressure/model_test.go

@@ -16,7 +16,8 @@ func TestLoadConfigAppliesBoundedHistoryDefaults(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if cfg.HistoryWorkers != 1 || cfg.HistoryPageSize != 20 || cfg.HistoryInterval != "500ms" ||
+	if cfg.HistoryWorkers != 12 || cfg.HistoryPageSize != 10 || cfg.HistoryInterval != "500ms" ||
+		cfg.HistoryPageWait != "3s" || cfg.CloseRamp != "10s" ||
 		cfg.HistorySettle != "5m" || cfg.HistoryTimeout != "3m" ||
 		cfg.PreflightWorkers != 32 || cfg.WSPingInterval != "20s" {
 		t.Fatalf("unexpected defaults: %+v", cfg)

loadtest/im-pressure/recorder.go

@@ -165,6 +165,17 @@ func (r *deliveryRecorder) historyFound(mid string) bool {
 	return r.history[mid]
 }
 
+func (r *deliveryRecorder) historyFoundAll(messageIDs []string) bool {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	for _, mid := range messageIDs {
+		if !r.history[mid] {
+			return false
+		}
+	}
+	return true
+}
+
 func (r *deliveryRecorder) unackedObservedMessages(ledger []AcceptedMessage, scenario string) []AcceptedMessage {
 	accepted := make(map[string]bool, len(ledger))
 	for _, message := range ledger {
@@ -274,13 +285,17 @@ func (r *deliveryRecorder) summarize(base AgentSummary, ledger []AcceptedMessage
 	return base
 }
 
-func (r *deliveryRecorder) countRealtimeLostStored(ledger []AcceptedMessage) int64 {
+func (r *deliveryRecorder) countRealtimeLostStored(ledger []AcceptedMessage, storedMessageIDs []string) int64 {
 	runID, _ := r.runID.Load().(string)
+	stored := make(map[string]bool, len(storedMessageIDs))
+	for _, mid := range storedMessageIDs {
+		stored[mid] = true
+	}
 	r.mu.Lock()
 	defer r.mu.Unlock()
 	var count int64
 	for _, message := range ledger {
-		if message.RunID != runID || !r.history[message.ClientMid] {
+		if message.RunID != runID || !stored[message.ClientMid] {
 			continue
 		}
 		obs := r.observed[message.ClientMid]

loadtest/im-pressure/recorder_test.go

@@ -43,11 +43,23 @@ func TestDeliveryRecorderSeparatesHistoryFromRealtime(t *testing.T) {
 	if !recorder.historyFound(message.ClientMid) {
 		t.Fatal("history message was not recorded")
 	}
-	if got := recorder.countRealtimeLostStored([]AcceptedMessage{message}); got != 1 {
+	if got := recorder.countRealtimeLostStored([]AcceptedMessage{message}, []string{message.ClientMid}); got != 1 {
 		t.Fatalf("realtime lost but persisted=%d", got)
 	}
 }
 
+func TestDeliveryRecorderUsesSharedHistoryForRealtimeLostStored(t *testing.T) {
+	recorder := newDeliveryRecorder("run1", "b", 1)
+	message := AcceptedMessage{RunID: "run1", Kind: "group", ClientMid: "run1-g-a-0001-01", Topic: "grp-one", Seq: 10}
+
+	if got := recorder.countRealtimeLostStored([]AcceptedMessage{message}, []string{message.ClientMid}); got != 1 {
+		t.Fatalf("shared history count=%d want=1", got)
+	}
+	if got := recorder.countRealtimeLostStored([]AcceptedMessage{message}, nil); got != 0 {
+		t.Fatalf("empty shared history count=%d want=0", got)
+	}
+}
+
 func TestDeliveryRecorderExportsObservedMessagesWithoutAck(t *testing.T) {
 	recorder := newDeliveryRecorder("run1", "b", 2)
 	acked := AcceptedMessage{RunID: "run1", Scenario: "group", Kind: "group", ClientMid: "run1-g-a-0001-01",