Some messages in the (azure) kafka cluster are too large for rdkafka clients to be able to decompress them
The content copy to S3 uses a journal client to do its work.
This work has been stuck for a few weeks now, because the client is breaking on some messages at the head of the swh.journal.objects.content topic.
After investigating, the error messages emitted by the client look like a decompression issue. We've had such issues in the graph export process, where we had to increase the journal client buffer size substantially to allow it to decompress all messages.
However, it looks like the current message size setting for the content topic overflows the max size that rdkafka is able to allocate for its decompression buffer.
We need to investigate this further, before we perform the backfill of objects in the new kafka cluster.
Dump of the current offsets of the s3 copy consumer group for further investigation:
0 43,808,814 38,508,457 5,300,357 content-replayer-001-2415c0f8-3149-49d1-90ee-aacc96e48fee:192.168.100.101/192.168.100.101
1 43,764,359 38,461,588 5,302,771 content-replayer-002-cad3546a-fe3c-4298-ba45-9a225e49f317:192.168.100.101/192.168.100.101
2 43,766,028 38,462,888 5,303,140 content-replayer-003-9fd121b8-d4bb-46ec-9b78-f9a7f7a1d99a:192.168.100.101/192.168.100.101
3 43,749,343 38,451,637 5,297,706 content-replayer-004-37c245d0-c264-4e3a-91a6-b25fc125ac14:192.168.100.101/192.168.100.101
4 43,778,588 38,475,127 5,303,461 content-replayer-005-d0045e23-21a5-4e1d-b74c-d53912664930:192.168.100.101/192.168.100.101
5 43,774,127 38,478,528 5,295,599 content-replayer-006-9243d5b6-0981-45e3-a2f4-64c7ad4c7668:192.168.100.101/192.168.100.101
6 43,784,620 38,489,348 5,295,272 content-replayer-007-c373c9ce-5ae7-4fba-8b92-a183255f1e03:192.168.100.101/192.168.100.101
7 43,785,259 38,480,366 5,304,893 content-replayer-008-e4326b1b-75d6-42a9-9318-e9d9a1322e33:192.168.100.101/192.168.100.101
8 43,782,736 38,480,816 5,301,920 content-replayer-009-421d0635-6d40-4114-865a-79dc9bced90a:192.168.100.101/192.168.100.101
9 43,770,757 38,468,204 5,302,553 content-replayer-010-74b4cae2-635f-4070-ab08-2ca605497411:192.168.100.101/192.168.100.101
10 43,778,211 38,478,808 5,299,403 content-replayer-011-61913913-2daa-4851-bded-973c98ac1ff2:192.168.100.101/192.168.100.101
11 43,769,834 38,470,166 5,299,668 content-replayer-012-44a4e668-fe2f-4b1c-9f1b-4948416185a8:192.168.100.101/192.168.100.101
12 43,766,071 38,468,480 5,297,591 content-replayer-013-18cfb1b3-7c92-4638-ad18-49f9e253bd55:192.168.100.101/192.168.100.101
13 43,771,359 38,474,437 5,296,922 content-replayer-014-cdc2a04d-378c-43b1-a878-d6dee3d62850:192.168.100.101/192.168.100.101
14 43,783,204 38,477,149 5,306,055 content-replayer-015-7233b907-9827-4c4b-be01-325ce999a96b:192.168.100.101/192.168.100.101
15 43,755,937 38,458,452 5,297,485 content-replayer-016-5818339f-ef42-4d19-8d24-6b72a5a7dd02:192.168.100.101/192.168.100.101
16 43,773,488 38,468,915 5,304,573 content-replayer-017-821bf608-28b7-4dca-b0c2-be34a234c203:192.168.100.101/192.168.100.101
17 43,798,030 38,498,789 5,299,241 content-replayer-018-9897b14b-dbf6-465d-adeb-8c9895970983:192.168.100.101/192.168.100.101
18 43,778,690 33,332,557 10,446,133 content-replayer-019-c9c45e6d-b914-4452-b83f-cdd31ddc7379:192.168.100.101/192.168.100.101
19 43,768,898 38,466,640 5,302,258 content-replayer-020-c56c2116-d9c1-47be-97e3-d8cc42963aee:192.168.100.101/192.168.100.101
20 43,772,490 38,474,979 5,297,511 content-replayer-021-c432e391-79e9-4a85-a6a8-46c9c1ac4a31:192.168.100.101/192.168.100.101
21 43,753,994 38,452,094 5,301,900 content-replayer-022-3da52e78-1f75-4433-b045-8b2e3569b69e:192.168.100.101/192.168.100.101
22 43,780,022 38,474,986 5,305,036 content-replayer-023-3c794fa0-0f56-4f87-9ec8-e1385138428a:192.168.100.101/192.168.100.101
23 43,773,399 38,470,376 5,303,023 content-replayer-024-08d21842-ff32-4fd4-960e-a39958bb810a:192.168.100.101/192.168.100.101
24 43,795,442 38,494,594 5,300,848 content-replayer-025-86e1cfe9-f180-4d4d-98d5-1ea5b151d573:192.168.100.101/192.168.100.101
25 43,786,240 38,481,916 5,304,324 content-replayer-026-d72ec6e5-1c5f-4db6-a650-2d25c06c9f6a:192.168.100.101/192.168.100.101
26 43,789,380 38,489,797 5,299,583 content-replayer-027-88d20a54-6823-47db-9da1-266c84c8c0dc:192.168.100.101/192.168.100.101
27 43,788,829 38,484,875 5,303,954 content-replayer-028-047ab787-4fe2-4f86-89d5-9c54c12397a5:192.168.100.101/192.168.100.101
28 43,778,021 38,478,584 5,299,437 content-replayer-029-7cb416f1-4ee3-400c-a904-25fd73d3142d:192.168.100.101/192.168.100.101
29 43,771,217 38,475,656 5,295,561 content-replayer-030-195e41ee-9c68-4a1a-92a7-a30cbd27b67f:192.168.100.101/192.168.100.101
30 43,775,710 38,475,824 5,299,886 content-replayer-031-29c5ce7e-4c3b-40af-a41e-b2efa50cf34f:192.168.100.101/192.168.100.101
31 43,758,531 38,460,661 5,297,870 content-replayer-032-052a6e10-38cf-43bf-a24f-38d1f9f169db:192.168.100.101/192.168.100.101
32 43,762,530 38,455,674 5,306,856 content-replayer-033-152cdcd0-a3fe-4f08-a52e-34f562f79523:192.168.100.101/192.168.100.101
33 43,760,864 38,457,085 5,303,779 content-replayer-034-3b7d62e9-dd03-4d76-82d4-d69b01ff4d4f:192.168.100.101/192.168.100.101
34 43,778,277 38,474,271 5,304,006 content-replayer-035-69c8a357-3051-4378-a619-0e25e88f98da:192.168.100.101/192.168.100.101
35 43,772,836 38,469,713 5,303,123 content-replayer-036-835b6db2-7fd5-4255-ae0f-5eb7c56b4988:192.168.100.101/192.168.100.101
36 43,783,402 38,483,318 5,300,084 content-replayer-037-d8d4c4b4-a868-4f1f-8dec-c32d6d4d9504:192.168.100.101/192.168.100.101
37 43,781,593 38,477,029 5,304,564 content-replayer-038-425d6a72-3614-4ec3-9b03-5c72f62a89e8:192.168.100.101/192.168.100.101
38 43,775,490 38,478,452 5,297,038 content-replayer-039-fe8220cf-1cfd-464d-9c8a-c33693339375:192.168.100.101/192.168.100.101
39 43,762,516 38,460,320 5,302,196 content-replayer-040-c91b786d-00f4-4cc3-bd7a-4f50afd8cac3:192.168.100.101/192.168.100.101
40 43,778,957 38,477,170 5,301,787 content-replayer-041-6be7b6fe-67e3-42cc-9e9f-b20e2bc2ffae:192.168.100.101/192.168.100.101
41 43,791,067 38,486,096 5,304,971 content-replayer-042-1e003314-9b78-4a5d-a546-1a6eb38ec9cb:192.168.100.101/192.168.100.101
42 43,766,031 38,465,732 5,300,299 content-replayer-043-11b86415-3531-48d6-8975-da9cbd75ae49:192.168.100.101/192.168.100.101
43 43,776,144 38,476,523 5,299,621 content-replayer-044-2b1a9ca7-15dd-4151-a90e-21a684052413:192.168.100.101/192.168.100.101
44 43,770,876 38,473,632 5,297,244 content-replayer-045-49fef889-57ce-48a0-99a0-16c3a47bb160:192.168.100.101/192.168.100.101
45 43,757,011 38,457,309 5,299,702 content-replayer-046-8bb5c048-e2f9-4a51-93e6-c29f703dd901:192.168.100.101/192.168.100.101
46 43,766,930 38,463,372 5,303,558 content-replayer-047-cf4af560-016f-4f59-93a0-8a4514b25b56:192.168.100.101/192.168.100.101
47 43,786,162 38,488,466 5,297,696 content-replayer-048-0a283464-fe51-4153-b93a-13d431558069:192.168.100.101/192.168.100.101
48 43,796,390 38,492,876 5,303,514 content-replayer-049-7079028a-9704-417e-b016-5ab0493b73cb:192.168.100.101/192.168.100.101
49 43,778,919 38,481,660 5,297,259 content-replayer-050-84b515be-d58d-4533-b7f5-285143b91ddf:192.168.100.101/192.168.100.101
50 43,762,132 38,462,065 5,300,067 content-replayer-051-e362d521-3c89-4479-be6c-9acdc15e9e36:192.168.100.101/192.168.100.101
51 43,758,103 38,459,197 5,298,906 content-replayer-052-c8429783-77c6-482d-aacf-d99b067f3ac4:192.168.100.101/192.168.100.101
52 43,779,788 38,477,287 5,302,501 content-replayer-053-5cc848cc-0253-4d77-8f4a-c8552df0bd99:192.168.100.101/192.168.100.101
53 43,764,912 38,468,235 5,296,677 content-replayer-054-56ae1086-d0bb-4463-9458-0bce9286b7cd:192.168.100.101/192.168.100.101
54 43,770,778 38,471,619 5,299,159 content-replayer-055-d20cb6dc-be3d-4824-b1c6-acef266159ae:192.168.100.101/192.168.100.101
55 43,772,019 38,471,246 5,300,773 content-replayer-056-2a936d9a-efac-4968-a78a-d37eeccc4d21:192.168.100.101/192.168.100.101
56 43,754,511 38,456,316 5,298,195 content-replayer-057-5508a50c-5485-4922-9d73-e7f6d70f7ac1:192.168.100.101/192.168.100.101
57 43,763,310 38,461,341 5,301,969 content-replayer-058-4f32ef85-8d4c-4288-99b0-dcfeb911d73c:192.168.100.101/192.168.100.101
58 43,796,248 38,494,029 5,302,219 content-replayer-059-710a878a-f574-4a27-9698-82d836b733a1:192.168.100.101/192.168.100.101
59 43,769,705 38,468,983 5,300,722 content-replayer-060-6d2ed7e5-a5ff-4355-872c-a6ff03d040bc:192.168.100.101/192.168.100.101
60 43,774,695 38,472,929 5,301,766 content-replayer-061-7e73ee12-975f-492b-b190-8dc227f75111:192.168.100.101/192.168.100.101
61 43,753,497 38,455,881 5,297,616 content-replayer-062-2724ce1b-4b25-417b-8ed0-7a9253706641:192.168.100.101/192.168.100.101
62 43,762,932 38,456,275 5,306,657 content-replayer-063-5368fe2d-d0c0-4577-ae16-5495d0e3e471:192.168.100.101/192.168.100.101
63 43,740,624 38,146,048 5,594,576 content-replayer-064-884a910d-b283-45fb-b5ff-f6ffad86099a:192.168.100.101/192.168.100.101
64 43,779,366 38,474,102 5,305,264 content-replayer-065-65fb5524-7f73-4300-a5c6-622a1e379008:192.168.100.101/192.168.100.101
65 43,761,014 38,463,531 5,297,483 content-replayer-066-ac8f6a14-e0f4-458f-9d68-8a63e91fcac5:192.168.100.101/192.168.100.101
66 43,800,626 38,498,165 5,302,461 content-replayer-067-b8545453-9257-4676-8f55-1b64e374b8ff:192.168.100.101/192.168.100.101
67 43,787,515 38,483,819 5,303,696 content-replayer-068-c7e9bf24-0022-4e32-b69e-6ba9581e3452:192.168.100.101/192.168.100.101
68 43,771,044 38,472,573 5,298,471 content-replayer-069-b9374949-7f1c-4df8-b585-f801a6273e48:192.168.100.101/192.168.100.101
69 43,757,448 38,455,690 5,301,758 content-replayer-070-07861521-1384-4ebb-947b-dc15c94659b7:192.168.100.101/192.168.100.101
70 43,789,111 38,490,441 5,298,670 content-replayer-071-d841e6cb-535b-4287-aa36-0c1a321ea2e8:192.168.100.101/192.168.100.101
71 43,783,000 38,479,549 5,303,451 content-replayer-072-5ff89acc-096f-45b7-8e30-68148281ef71:192.168.100.101/192.168.100.101
72 43,739,919 38,439,393 5,300,526 content-replayer-073-649becda-6971-4a54-92fc-b6157bb8c503:192.168.100.101/192.168.100.101
73 43,779,607 38,483,036 5,296,571 content-replayer-074-cd6fc315-5f60-4d2a-bdcc-3613ebfd501f:192.168.100.101/192.168.100.101
74 43,806,272 38,501,300 5,304,972 content-replayer-075-22419a81-fe75-4fd6-93d8-3f9ce45c8523:192.168.100.101/192.168.100.101
75 43,741,622 38,441,328 5,300,294 content-replayer-076-2526305a-d990-4855-96ec-6b1853301fbc:192.168.100.101/192.168.100.101
76 43,781,295 38,475,032 5,306,263 content-replayer-077-736931e3-2f0e-4cb5-8f65-3a2eaa021b38:192.168.100.101/192.168.100.101
77 43,796,830 38,494,109 5,302,721 content-replayer-078-544053cb-2cc3-47d2-99a1-6b29d9b4ba5d:192.168.100.101/192.168.100.101
78 43,806,179 38,505,035 5,301,144 content-replayer-079-3d7737f7-2a54-4ab4-8edd-ae3fe7d6e665:192.168.100.101/192.168.100.101
79 43,779,140 38,481,774 5,297,366 content-replayer-080-5b467d8e-28ab-45de-9c33-09c20e7e7ed1:192.168.100.101/192.168.100.101
80 43,773,388 38,461,946 5,311,442 content-replayer-081-38fe52c0-10c8-4849-b554-7c9cf5b8a85c:192.168.100.101/192.168.100.101
81 43,745,945 38,445,507 5,300,438 content-replayer-082-5b81e979-adf1-41f3-85ff-df01a8349b56:192.168.100.101/192.168.100.101
82 43,770,418 38,474,220 5,296,198 content-replayer-083-b04531d9-919b-4163-82a8-59da3e9b6864:192.168.100.101/192.168.100.101
83 43,776,040 38,472,673 5,303,367 content-replayer-084-be97701c-ee81-4de6-a037-fabe2c457788:192.168.100.101/192.168.100.101
84 43,798,037 38,496,814 5,301,223 content-replayer-085-b158a38b-43a2-4452-8f4d-0d49d808fedd:192.168.100.101/192.168.100.101
85 43,751,490 38,455,408 5,296,082 content-replayer-086-c2907974-1269-4ed5-bc33-299a4011d69b:192.168.100.101/192.168.100.101
86 43,781,265 38,480,532 5,300,733 content-replayer-087-6576f8f3-2b5b-45d8-9b0c-c28ca403cad6:192.168.100.101/192.168.100.101
87 43,786,507 38,484,885 5,301,622 content-replayer-088-b6ddd64a-2bc2-4f86-b87f-9828b6f1b25b:192.168.100.101/192.168.100.101
88 43,779,666 38,475,032 5,304,634 content-replayer-089-37175b8c-10b5-46d8-be60-404b804e1142:192.168.100.101/192.168.100.101
89 43,744,607 38,449,547 5,295,060 content-replayer-090-414405f8-2288-48b3-a567-1e16d27dddcb:192.168.100.101/192.168.100.101
90 43,796,029 38,496,076 5,299,953 content-replayer-091-406b76dc-094a-46ec-822e-7df0d2fc1522:192.168.100.101/192.168.100.101
91 43,770,468 38,467,502 5,302,966 content-replayer-092-cadd2515-4d26-44e5-bb04-4eb0ae92dbce:192.168.100.101/192.168.100.101
92 43,748,039 38,449,785 5,298,254 content-replayer-093-b7ce8cbf-9a3b-4d2c-847f-1d131f9c91b6:192.168.100.101/192.168.100.101
93 43,743,286 38,444,471 5,298,815 content-replayer-094-1c2d6bba-4869-45d6-a56c-b57a4491bc17:192.168.100.101/192.168.100.101
94 43,786,614 38,482,271 5,304,343 content-replayer-095-1a10c3df-3b7c-4a3a-b1b4-c95ced1ef8b7:192.168.100.101/192.168.100.101
95 43,772,930 38,469,412 5,303,518 content-replayer-096-8e77d068-3af4-47eb-b8c5-1612a359cabd:192.168.100.101/192.168.100.101
96 43,784,284 38,234,780 5,549,504 content-replayer-097-540e8924-8066-4688-8b02-9e98a51cba57:192.168.100.101/192.168.100.101
97 43,766,666 38,468,052 5,298,614 content-replayer-098-4dcaba45-ee70-4fcc-8930-b396e3ef753b:192.168.100.101/192.168.100.101
98 43,780,009 38,478,366 5,301,643 content-replayer-099-6b766c2d-5cf8-4452-81cf-65b6731c2a66:192.168.100.101/192.168.100.101
99 43,766,414 38,458,674 5,307,740 content-replayer-100-37c3d7e4-a24e-43f6-9c2a-fc6761ba647a:192.168.100.101/192.168.100.101
100 43,773,866 38,474,856 5,299,010 content-replayer-101-915877ab-c788-4d87-ae2f-770a5d22b7e1:192.168.100.101/192.168.100.101
101 43,782,788 38,480,446 5,302,342 content-replayer-102-a0c147cb-ea42-4306-b88b-a91fa5017e73:192.168.100.101/192.168.100.101
102 43,731,781 38,429,251 5,302,530 content-replayer-103-0721e260-66af-4de9-9e5d-cfd741fcd8c6:192.168.100.101/192.168.100.101
103 43,784,080 38,483,313 5,300,767 content-replayer-104-14d6b221-a72b-4a09-a409-7aba743d6d63:192.168.100.101/192.168.100.101
104 43,775,839 38,479,591 5,296,248 content-replayer-105-2e11945a-12d3-4bf8-80db-a36ccbe1cbee:192.168.100.101/192.168.100.101
105 43,751,959 38,456,153 5,295,806 content-replayer-106-d9ddc7c5-83a6-4e92-b557-dd292da934d8:192.168.100.101/192.168.100.101
106 43,778,361 38,476,841 5,301,520 content-replayer-107-f840f792-73ef-45d6-9d1a-9f6c8f86556e:192.168.100.101/192.168.100.101
107 43,793,787 38,492,815 5,300,972 content-replayer-108-600a5081-6dc1-458b-affa-0e01f8563216:192.168.100.101/192.168.100.101
108 43,807,179 38,508,130 5,299,049 content-replayer-109-c5b4f274-9b2a-4937-aac1-a333b07813b7:192.168.100.101/192.168.100.101
109 43,776,083 38,475,949 5,300,134 content-replayer-110-bba3dbf6-7d2a-42af-9519-5fc2ebb705b6:192.168.100.101/192.168.100.101
110 43,789,661 38,483,000 5,306,661 content-replayer-111-b33536c1-ee5a-47a3-93f9-c614a37987a4:192.168.100.101/192.168.100.101
111 43,768,298 38,467,092 5,301,206 content-replayer-112-c9d51275-a191-4005-85c6-789dfe79d8f0:192.168.100.101/192.168.100.101
112 43,788,915 38,487,413 5,301,502 content-replayer-113-92d47242-7687-4f75-8348-be472da3b125:192.168.100.101/192.168.100.101
113 43,781,639 38,254,660 5,526,979 content-replayer-114-96a14196-03d3-4362-ac82-f6b8e29332a6:192.168.100.101/192.168.100.101
114 43,811,108 38,511,905 5,299,203 content-replayer-115-5b29f0df-bffd-4e0e-809a-0519a874d538:192.168.100.101/192.168.100.101
115 43,767,742 38,469,259 5,298,483 content-replayer-116-266a144d-3486-4a51-a7ac-dc7dfb3e7054:192.168.100.101/192.168.100.101
116 43,767,690 38,459,723 5,307,967 content-replayer-117-59f26700-17c7-46c4-aafe-cc7775d80403:192.168.100.101/192.168.100.101
117 43,786,291 38,485,332 5,300,959 content-replayer-118-b1905c0d-1339-405a-ae76-510982a992de:192.168.100.101/192.168.100.101
118 43,779,420 38,479,931 5,299,489 content-replayer-119-287d303d-66c7-4c86-940f-1213f51570c1:192.168.100.101/192.168.100.101
119 43,792,351 38,489,055 5,303,296 content-replayer-120-37303e43-a73f-444b-a530-f0da5d1ead2b:192.168.100.101/192.168.100.101
120 43,788,904 38,493,324 5,295,580 content-replayer-121-415a1d60-59ee-46cf-b592-4228db6c6922:192.168.100.101/192.168.100.101
121 43,764,372 38,465,597 5,298,775 content-replayer-122-f98c2473-f012-4fd3-b399-b9900f6784c1:192.168.100.101/192.168.100.101
122 43,774,027 38,475,590 5,298,437 content-replayer-123-1c73d224-5afc-41f2-bf5d-f7a2b92fed81:192.168.100.101/192.168.100.101
123 43,740,485 38,440,625 5,299,860 content-replayer-124-34a28249-f359-4e72-a58a-999bd1418897:192.168.100.101/192.168.100.101
124 43,756,674 38,452,328 5,304,346 content-replayer-125-c4fa56a4-3806-4885-abe5-99d8e63ee13b:192.168.100.101/192.168.100.101
125 43,782,351 38,476,689 5,305,662 content-replayer-126-91cbaa79-99e2-45f5-a776-ef8b44bf1988:192.168.100.101/192.168.100.101
126 43,804,386 38,497,870 5,306,516 content-replayer-127-8307ba0a-f2a1-4996-aae8-4e144a5f7c70:192.168.100.101/192.168.100.101
127 43,798,314 38,493,701 5,304,613 content-replayer-128-68755487-33db-4b99-be43-95c1cc633c5e:192.168.100.101/192.168.100.101
128 43,781,708 38,479,021 5,302,687 content-replayer-129-bb5b90c4-a117-4d47-89b0-eff062648956:192.168.100.101/192.168.100.101
129 43,758,012 38,457,420 5,300,592 content-replayer-130-7c313107-27f7-4ad4-aa5c-2d90d62a6916:192.168.100.101/192.168.100.101
130 43,773,682 38,470,181 5,303,501 content-replayer-131-1e1f4716-9996-404f-9ce2-d9f94aeac3fe:192.168.100.101/192.168.100.101
131 43,783,030 38,482,031 5,300,999 content-replayer-132-3be181f8-e024-46fd-805c-4b7beb114140:192.168.100.101/192.168.100.101
132 43,781,353 38,481,072 5,300,281 content-replayer-133-5392e25e-01f0-4ca3-810b-7d5b4979827e:192.168.100.101/192.168.100.101
133 43,762,186 38,458,065 5,304,121 content-replayer-134-306eb0b0-2d6f-4857-abad-1050962a2695:192.168.100.101/192.168.100.101
134 43,787,509 38,481,551 5,305,958 content-replayer-135-82cc56fe-64c3-4bc0-ac74-b69352a8fb39:192.168.100.101/192.168.100.101
135 43,768,141 38,463,591 5,304,550 content-replayer-136-e1261697-bba5-4c96-87cc-482100a71460:192.168.100.101/192.168.100.101
136 43,783,840 38,479,842 5,303,998 content-replayer-137-24c8ac55-36fa-43d3-8626-5e3012c95516:192.168.100.101/192.168.100.101
137 43,782,771 38,482,079 5,300,692 content-replayer-138-16c5b42b-383e-478e-aee0-e3858a214dd2:192.168.100.101/192.168.100.101
138 43,794,455 38,498,633 5,295,822 content-replayer-139-3f5c42da-2076-4331-a5e5-9d5a5e01f605:192.168.100.101/192.168.100.101
139 43,758,501 38,460,286 5,298,215 content-replayer-140-67f5aa3b-6e41-46a2-bc25-14d001de3c79:192.168.100.101/192.168.100.101
140 43,782,354 38,481,589 5,300,765 content-replayer-141-2a33c360-7089-4c9f-a11b-fd39da80eaf7:192.168.100.101/192.168.100.101
141 43,766,958 38,465,081 5,301,877 content-replayer-142-554aeb95-9868-427c-a1a0-71e21cfc984d:192.168.100.101/192.168.100.101
142 43,785,607 38,486,377 5,299,230 content-replayer-143-5d34de77-3f9c-426d-9cb8-8ff89b4ae87f:192.168.100.101/192.168.100.101
143 43,770,060 38,468,902 5,301,158 content-replayer-144-466a2415-7bd3-47b3-9449-87dc637463b9:192.168.100.101/192.168.100.101
144 43,784,543 38,483,489 5,301,054 content-replayer-145-5638ce60-5634-483d-8d2d-55b97de05d3a:192.168.100.101/192.168.100.101
145 43,785,913 38,480,562 5,305,351 content-replayer-146-36b7e0a3-e56d-46a6-bbc2-91a45c54993a:192.168.100.101/192.168.100.101
146 43,765,864 38,466,031 5,299,833 content-replayer-147-f8ee80f3-ad34-4c4a-80ee-e8ab479c5f82:192.168.100.101/192.168.100.101
147 43,771,280 38,460,828 5,310,452 content-replayer-148-8785c4a3-277c-4dd0-9188-7a2f4bd62f4d:192.168.100.101/192.168.100.101
148 43,783,406 38,475,307 5,308,099 content-replayer-149-2ce1b2b5-d9dc-4418-b227-5fa5185dc4b0:192.168.100.101/192.168.100.101
149 43,760,654 38,453,722 5,306,932 content-replayer-150-d9f8d088-efab-4321-ac40-fa71a9304f8f:192.168.100.101/192.168.100.101
150 43,791,068 38,487,239 5,303,829 content-replayer-151-cffc47c7-99ef-4f63-a981-6aa2eb9a44c7:192.168.100.101/192.168.100.101
151 43,781,763 38,481,834 5,299,929 content-replayer-152-cfd24649-0c2b-420d-a9ab-bb3b7526649d:192.168.100.101/192.168.100.101
152 43,763,520 38,465,432 5,298,088 content-replayer-153-55e5ef9b-dfef-4ca5-b043-ba1c13f1cc9e:192.168.100.101/192.168.100.101
153 43,772,096 38,475,014 5,297,082 content-replayer-154-c9f80e53-dc9b-4370-87a7-5a6c1ede14b8:192.168.100.101/192.168.100.101
154 43,810,895 38,510,104 5,300,791 content-replayer-155-39c15af5-9287-4d79-85e8-3791516726c9:192.168.100.101/192.168.100.101
155 43,800,162 38,494,108 5,306,054 content-replayer-156-241d3f8c-f3bf-452a-a53f-4ebe9b2a711c:192.168.100.101/192.168.100.101
156 43,794,969 38,493,505 5,301,464 content-replayer-157-836c61c2-cc16-4a33-a1b1-e4ea5a1a2703:192.168.100.101/192.168.100.101
157 43,782,044 38,476,315 5,305,729 content-replayer-158-e4ae7bea-4b2c-4e63-8150-adae5fa5bd6b:192.168.100.101/192.168.100.101
158 43,762,846 38,462,533 5,300,313 content-replayer-159-0dc726e5-f7f7-43f7-b72b-0f26d0c5478b:192.168.100.101/192.168.100.101
159 43,742,298 38,439,503 5,302,795 content-replayer-160-41a59db1-f474-4eb5-90bd-b567ae84fba1:192.168.100.101/192.168.100.101
160 43,760,706 38,456,266 5,304,440 content-replayer-161-8795ec89-af1a-45e7-8fd4-8f3248a0d62f:192.168.100.101/192.168.100.101
161 43,759,677 38,456,420 5,303,257 content-replayer-162-948d2e57-d100-4244-a144-e8ecc8ce1322:192.168.100.101/192.168.100.101
162 43,760,194 38,455,669 5,304,525 content-replayer-163-2e8cc10e-773b-4277-9e6a-98e32fd530a5:192.168.100.101/192.168.100.101
163 43,774,266 38,477,420 5,296,846 content-replayer-164-c2928e2b-44db-46fd-b1c5-25a063d26670:192.168.100.101/192.168.100.101
164 43,765,863 38,470,774 5,295,089 content-replayer-165-2ca05b45-580b-4050-be04-3a7fda7fa935:192.168.100.101/192.168.100.101
165 43,765,780 38,459,168 5,306,612 content-replayer-166-56d28200-5f6c-4bda-aedd-e79575335b36:192.168.100.101/192.168.100.101
166 43,781,804 38,484,686 5,297,118 content-replayer-167-1584032c-2566-4f2c-b589-738ab0065212:192.168.100.101/192.168.100.101
167 43,790,021 38,479,235 5,310,786 content-replayer-168-3a4dd029-630d-4e13-a138-2e448ae5a65b:192.168.100.101/192.168.100.101
168 43,789,415 38,488,545 5,300,870 content-replayer-169-dd8b786d-d8c8-4ca9-8f9e-d074b6b0baf7:192.168.100.101/192.168.100.101
169 43,781,918 38,485,573 5,296,345 content-replayer-170-bc78aeb4-9eb7-485b-8564-ffb9d52db56b:192.168.100.101/192.168.100.101
170 43,763,522 38,457,943 5,305,579 content-replayer-171-c1d85ee4-5186-4231-9c41-ff6060d97c4d:192.168.100.101/192.168.100.101
171 43,755,858 38,453,242 5,302,616 content-replayer-172-4fea9050-9144-4e22-ab46-2977564b213a:192.168.100.101/192.168.100.101
172 43,803,295 38,496,135 5,307,160 content-replayer-173-9e135df3-c45b-47e8-85f2-6c76af48bbb6:192.168.100.101/192.168.100.101
173 43,790,704 38,488,255 5,302,449 content-replayer-174-919d627a-f2c9-4768-bb3f-ffc6ccdf2c86:192.168.100.101/192.168.100.101
174 43,794,028 38,492,262 5,301,766 content-replayer-175-14b3f1c8-56c8-460d-b3db-efeb7e24e292:192.168.100.101/192.168.100.101
175 43,800,849 38,499,366 5,301,483 content-replayer-176-2631f913-2abd-49b1-9d9f-a0196787465f:192.168.100.101/192.168.100.101
176 43,766,173 38,463,245 5,302,928 content-replayer-177-b082e319-d03d-4a31-9d1c-e1d7ff47663e:192.168.100.101/192.168.100.101
177 43,769,576 38,463,744 5,305,832 content-replayer-178-ce104385-3522-4b6d-99d5-e5aa2564c9d4:192.168.100.101/192.168.100.101
178 43,772,571 38,470,594 5,301,977 content-replayer-179-799bb83d-3e77-4dd8-bf29-f3d2d2e4d8ba:192.168.100.101/192.168.100.101
179 43,781,680 38,476,212 5,305,468 content-replayer-180-d44886c2-d73e-4585-b4ff-afa6bf6a9c0e:192.168.100.101/192.168.100.101
180 43,797,807 38,495,875 5,301,932 content-replayer-181-2306e083-aabe-44d1-a00e-7c97d938e6c2:192.168.100.101/192.168.100.101
181 43,767,369 38,466,932 5,300,437 content-replayer-182-dd8b14e0-b545-4686-9a7a-325d2026d14a:192.168.100.101/192.168.100.101
182 43,777,677 38,473,582 5,304,095 content-replayer-183-3b83b6b1-0604-4ceb-b6ed-3fab3323ed18:192.168.100.101/192.168.100.101
183 43,768,211 38,469,031 5,299,180 content-replayer-184-5ecf84df-0358-4906-aea4-fb357155e27a:192.168.100.101/192.168.100.101
184 43,772,895 38,470,097 5,302,798 content-replayer-185-71bd1373-2b41-458f-9be1-fd60d85a23ce:192.168.100.101/192.168.100.101
185 43,797,720 38,491,226 5,306,494 content-replayer-186-fe7881e2-95b4-4220-9ac4-9e05c7329528:192.168.100.101/192.168.100.101
186 43,794,874 38,495,258 5,299,616 content-replayer-187-ab883284-fe83-4cdf-a2e1-a345c90e5775:192.168.100.101/192.168.100.101
187 43,786,711 38,481,069 5,305,642 content-replayer-188-14284ae8-347c-4596-a5bc-b93c5349f9e0:192.168.100.101/192.168.100.101
188 43,759,119 38,460,579 5,298,540 content-replayer-189-724d4e37-f749-47ef-9957-31a68fe6e646:192.168.100.101/192.168.100.101
189 43,756,197 38,449,002 5,307,195 content-replayer-190-c37a29df-443e-4fb5-8088-60a021cff072:192.168.100.101/192.168.100.101
190 43,753,417 38,445,819 5,307,598 content-replayer-191-adf92845-d37f-4644-97b8-d913ff0f0d0c:192.168.100.101/192.168.100.101
191 43,760,327 38,459,770 5,300,557 content-replayer-192-a142e409-402e-4d41-a01f-5caa5cc483f7:192.168.100.101/192.168.100.101
192 43,774,430 38,471,070 5,303,360 content-replayer-193-d674f413-a9e8-4ead-9b5a-36d94fe67172:192.168.100.101/192.168.100.101
193 43,757,239 38,459,581 5,297,658 content-replayer-194-5c67e292-45dd-4cd8-883f-62190a76e15c:192.168.100.101/192.168.100.101
194 43,763,181 38,461,893 5,301,288 content-replayer-195-3088302b-1a2d-49b7-b38b-0be45d929f77:192.168.100.101/192.168.100.101
195 43,754,529 38,451,515 5,303,014 content-replayer-196-a5576466-9f69-4791-bd30-378c2d8495ae:192.168.100.101/192.168.100.101
196 43,796,587 38,497,136 5,299,451 content-replayer-197-c53ce3fc-4c4d-4ada-bf73-7eac59be878f:192.168.100.101/192.168.100.101
197 43,775,972 38,478,282 5,297,690 content-replayer-198-da7a120c-eee7-4f7c-99fa-f37531647be3:192.168.100.101/192.168.100.101
198 43,814,410 38,514,237 5,300,173 content-replayer-199-0598e961-02f0-41e0-a33a-d158d634592b:192.168.100.101/192.168.100.101
199 43,775,484 38,470,406 5,305,078 content-replayer-200-e9f1efaa-9ae0-4be3-a055-7c7fc7c6a197:192.168.100.101/192.168.100.101
200 43,762,904 38,464,437 5,298,467 content-replayer-201-c469a355-3a26-4c67-ad1c-bbef2c5ec4ff:192.168.100.101/192.168.100.101
201 43,774,292 38,470,269 5,304,023 content-replayer-202-6943368e-3964-4205-9f77-f3535103db22:192.168.100.101/192.168.100.101
202 43,786,147 38,485,163 5,300,984 content-replayer-203-db066a94-4b2c-44df-bf58-4c269368ffbb:192.168.100.101/192.168.100.101
203 43,773,372 38,473,404 5,299,968 content-replayer-204-50ad512a-aed3-4778-a506-2c7d17b3a487:192.168.100.101/192.168.100.101
204 43,794,669 38,493,167 5,301,502 content-replayer-205-007367a7-cec6-44e4-8268-83d96dee9016:192.168.100.101/192.168.100.101
205 43,791,233 38,492,111 5,299,122 content-replayer-206-0d2c074f-aff6-471a-a06d-96928b67389f:192.168.100.101/192.168.100.101
206 43,794,174 38,490,602 5,303,572 content-replayer-207-7af6169a-950d-40a1-8925-481ab350d05e:192.168.100.101/192.168.100.101
207 43,757,664 38,458,276 5,299,388 content-replayer-208-8a150277-c667-4b2a-964b-3fbe3be0df12:192.168.100.101/192.168.100.101
208 43,785,303 38,482,822 5,302,481 content-replayer-209-3257663a-c0ec-4dd9-b0ac-fb9e1e3d9a8a:192.168.100.101/192.168.100.101
209 43,785,100 38,482,722 5,302,378 content-replayer-210-91d3745a-320a-44b7-b35f-773ce27dcb65:192.168.100.101/192.168.100.101
210 43,803,382 38,501,459 5,301,923 content-replayer-211-8ad8868e-9494-4003-9635-1659fb63487e:192.168.100.101/192.168.100.101
211 43,773,646 38,475,907 5,297,739 content-replayer-212-46af40e5-0537-421b-81d9-a3a74b305167:192.168.100.101/192.168.100.101
212 43,770,546 38,470,736 5,299,810 content-replayer-213-3c06f844-0bb1-4583-a706-8edbad305799:192.168.100.101/192.168.100.101
213 43,758,849 38,456,959 5,301,890 content-replayer-214-a925c053-e2d9-4147-944c-78f6852d6ec2:192.168.100.101/192.168.100.101
214 43,772,180 38,472,697 5,299,483 content-replayer-215-c0a43040-19b9-483e-879d-290991a81317:192.168.100.101/192.168.100.101
215 43,817,458 38,515,768 5,301,690 content-replayer-216-ac43b5cb-b9c8-4af7-8b12-c103e2f9727e:192.168.100.101/192.168.100.101
216 43,800,084 38,497,846 5,302,238 content-replayer-217-851259ae-16d0-4b46-8045-c355d7d5d734:192.168.100.101/192.168.100.101
217 43,754,241 38,453,578 5,300,663 content-replayer-218-10f87bbe-d77c-41e5-ba6c-826fc6e998a2:192.168.100.101/192.168.100.101
218 43,773,650 38,476,359 5,297,291 content-replayer-219-0db15257-8f98-41d5-89c4-24fcbfad56a5:192.168.100.101/192.168.100.101
219 43,751,352 38,454,594 5,296,758 content-replayer-220-b66cbddc-5e49-4241-885a-f5b117a27d80:192.168.100.101/192.168.100.101
220 43,770,930 38,470,926 5,300,004 content-replayer-221-02801534-388b-4a0d-b8b2-ec508dec85ad:192.168.100.101/192.168.100.101
221 43,781,613 38,482,391 5,299,222 content-replayer-222-4f49c0dd-e500-4cae-af85-259c9428b517:192.168.100.101/192.168.100.101
222 43,744,978 38,446,740 5,298,238 content-replayer-223-6a5b6445-ade3-46ee-92bd-c1a61feb28fa:192.168.100.101/192.168.100.101
223 43,772,645 38,463,506 5,309,139 content-replayer-224-6a52822b-9f88-4ad8-b982-090dfad7c9e3:192.168.100.101/192.168.100.101
224 43,768,362 38,467,544 5,300,818 content-replayer-225-32a7c833-ecc9-437c-8f29-390efeff11ef:192.168.100.101/192.168.100.101
225 43,764,895 38,459,600 5,305,295 content-replayer-226-a3a637a5-b228-4c1d-a7f1-ae8f6f94e7b7:192.168.100.101/192.168.100.101
226 43,775,753 38,477,037 5,298,716 content-replayer-227-9280b09f-bdeb-4c09-806d-21fa9f41d978:192.168.100.101/192.168.100.101
227 43,773,196 38,468,082 5,305,114 content-replayer-228-acb58472-4b39-468f-8aa0-a358aec90c68:192.168.100.101/192.168.100.101
228 43,805,270 38,510,358 5,294,912 content-replayer-229-d82aa003-78d6-48e4-9654-2906f18035ee:192.168.100.101/192.168.100.101
229 43,775,205 38,479,047 5,296,158 content-replayer-230-acd3e033-b85e-419e-b451-b90cccfc2479:192.168.100.101/192.168.100.101
230 43,752,232 38,453,742 5,298,490 content-replayer-231-9904aa8d-48f8-47f7-860d-5eff89c6aac6:192.168.100.101/192.168.100.101
231 43,771,997 38,470,397 5,301,600 content-replayer-232-7680271b-a07c-46ae-ad13-3ef10c88026a:192.168.100.101/192.168.100.101
232 43,788,234 38,491,614 5,296,620 content-replayer-233-893c9c6a-fe53-4c26-bf66-e5836ccb9392:192.168.100.101/192.168.100.101
233 43,775,132 38,477,167 5,297,965 content-replayer-234-196e3d55-4376-4dde-8f9c-1eb471759880:192.168.100.101/192.168.100.101
234 43,795,433 38,491,023 5,304,410 content-replayer-235-4424a7ea-5168-4c84-899f-0911b8960f88:192.168.100.101/192.168.100.101
235 43,808,099 38,508,860 5,299,239 content-replayer-236-9ca9d2d3-4588-44d8-8059-892854632a99:192.168.100.101/192.168.100.101
236 43,801,129 38,495,168 5,305,961 content-replayer-237-ae32ee6a-9d14-4cce-adf3-83c1fa4d11a9:192.168.100.101/192.168.100.101
237 43,766,514 38,464,964 5,301,550 content-replayer-238-e378fcaf-85e5-47c8-9a89-30adbe2844a0:192.168.100.101/192.168.100.101
238 43,786,003 38,485,739 5,300,264 content-replayer-239-3a9aa47f-66c7-4f9c-b7bf-40fb8be0b552:192.168.100.101/192.168.100.101
239 43,775,743 38,475,455 5,300,288 content-replayer-240-9e88caf2-a791-4644-9fbd-2f4d7323638b:192.168.100.101/192.168.100.101
240 43,797,667 38,496,325 5,301,342 content-replayer-241-fb79acdf-3aeb-4942-9a5c-2bf000bcec0c:192.168.100.101/192.168.100.101
241 43,772,618 38,471,182 5,301,436 content-replayer-242-9a191b33-4912-4299-90fa-a36155117629:192.168.100.101/192.168.100.101
242 43,769,786 38,468,056 5,301,730 content-replayer-243-dd1a86d0-e576-486c-b47a-696a663faddc:192.168.100.101/192.168.100.101
243 43,768,081 38,466,972 5,301,109 content-replayer-244-2ec96423-d712-410f-90aa-b50fc42d6916:192.168.100.101/192.168.100.101
244 43,764,804 38,462,702 5,302,102 content-replayer-245-8d1a8c6e-f610-4be2-ba2f-6c7a5bad6470:192.168.100.101/192.168.100.101
245 43,817,059 38,513,231 5,303,828 content-replayer-246-c1b26524-10dd-4223-b554-d395632af58c:192.168.100.101/192.168.100.101
246 43,822,637 38,517,625 5,305,012 content-replayer-247-373a224c-a74e-45e2-bd86-aacb1b89d1fc:192.168.100.101/192.168.100.101
247 43,779,343 38,486,693 5,292,650 content-replayer-248-b763225b-1014-44e2-ad31-84aa25fff33c:192.168.100.101/192.168.100.101
248 43,768,316 38,466,864 5,301,452 content-replayer-249-33d254fa-785d-4de3-886c-b7e94c12e103:192.168.100.101/192.168.100.101
249 43,755,187 38,449,577 5,305,610 content-replayer-250-9145a218-c866-42b6-86a8-9e3d74781704:192.168.100.101/192.168.100.101
250 43,776,400 38,247,096 5,529,304 content-replayer-251-a33e2c74-5b92-47d0-9cff-3d28c0b093d7:192.168.100.101/192.168.100.101
251 43,766,793 38,468,986 5,297,807 content-replayer-252-95f63adc-83d5-47b9-b700-e44443569fcd:192.168.100.101/192.168.100.101
252 43,777,183 38,469,717 5,307,466 content-replayer-253-044a9f74-5ea7-4b8c-9512-f4775fc91850:192.168.100.101/192.168.100.101
253 43,748,887 38,453,691 5,295,196 content-replayer-254-dd1f0d29-4b4c-46b2-a901-7be741e8e1d5:192.168.100.101/192.168.100.101
254 43,760,841 38,466,146 5,294,695 content-replayer-255-7077b144-4cc2-4c07-bfec-111972312c99:192.168.100.101/192.168.100.101
255 43,783,688 38,479,227 5,304,461 content-replayer-256-1e1f0f78-bcdd-44e8-b363-17f838282645:192.168.100.101/192.168.100.101
Migrated from T2543 (view on Phabricator)