Skip to content
Snippets Groups Projects

Missing objects detected by the references scrubbers

    • Open Issue created by Vincent Sellier @vsellier

    The scrubber-storagechecker-directory-references scrubber seems to detect missing contents in postgresql. These contents are present in cassandra:

    swh-scrubber=# select config_id, cast(first_occurrence as date), count(id) from missing_object where first_occurrence >= '2024-11-01' group by 1,2;
 config_id | first_occurrence | count
-----------+------------------+-------
         5 | 2024-11-01       |   978
         5 | 2024-11-02       |  1555
         5 | 2024-11-03       |  1140
         5 | 2024-11-04       |  1025
         5 | 2024-11-05       |  1116
         5 | 2024-11-06       |   793
         5 | 2024-11-07       |   748
         5 | 2024-11-08       |   898
         5 | 2024-11-09       |   695
         5 | 2024-11-10       |   518
         5 | 2024-11-11       |   565
         5 | 2024-11-12       |   513
         5 | 2024-11-13       |   840
         5 | 2024-11-14       |   677
         5 | 2024-11-15       |   847
         5 | 2024-11-16       |   841
         5 | 2024-11-17       |   813
         5 | 2024-11-18       |   344
(18 rows)

    For the last month:

    swh-scrubber=# copy (select config_id, first_occurrence, id from missing_object where first_occurrence >= '2024-11-01') to '/tmp/missing_objects.csv' with CSV quote ';';
COPY 14993
     cut -f5 -d ":" /tmp/missing_objects.csv | sort | uniq -c
  13350 cnt
   1643 dir

    Pocking at some contents

    (1 rows)
admin@cqlsh:swh> select * from content_by_sha1_git where sha1_git=0xe6a77355ec32f794ea12b65cfadd514a5a6a3be3 limit 1;

 sha1_git                                   | target_token
--------------------------------------------+---------------------
 0xe6a77355ec32f794ea12b65cfadd514a5a6a3be3 | -575363835627303098

(1 rows)
admin@cqlsh:swh> select toDate(writetime(status)/1000) from content where token(sha256) = -575363835627303098;

 system.todate(writetime(status) / 1000)
-----------------------------------------
                              2024-10-07

(1 rows)
admin@cqlsh:swh> select * from content_by_sha1_git where sha1_git=0x9f126249c01d281761b729e1d59e557284b34834 limit 1;

 sha1_git                                   | target_token
--------------------------------------------+---------------------
 0x9f126249c01d281761b729e1d59e557284b34834 | 4505255356418193379

(1 rows)
admin@cqlsh:swh> select toDate(writetime(status)/1000) from content where token(sha256) = 4505255356418193379;

 system.todate(writetime(status) / 1000)
-----------------------------------------
                              2024-10-16

admin@cqlsh:swh> select * from content_by_sha1_git where sha1_git=0xf3b57ab0a641dd8b0a405080aed77aa45f0efb33 limit 1;

 sha1_git                                   | target_token
--------------------------------------------+---------------------
 0xf3b57ab0a641dd8b0a405080aed77aa45f0efb33 | 2030982574245679696

(1 rows)
admin@cqlsh:swh> select toDate(writetime(status)/1000) from content where token(sha256) = 2030982574245679696;

 system.todate(writetime(status) / 1000)
-----------------------------------------
                              2024-07-24

(1 rows)
    swh-scrubber=# select count(*) from missing_object where config_id='5';
  count
---------
 1577641
(1 row)

    There is also missing contents not present in cassandra and in postgresql, which should not be possible.

    There is no errors logged in the redis associated to the replayer.

    We should check if this is only an issue in staging and it's not the same in production

    Edited

    Designs

      Child items ...

      2. Activity

      • Vincent Sellier changed title from Incorrect missing object detected by the references scrubbers to Missing object detected by the references scrubbers

        changed title from Incorrect missing object detected by the references scrubbers to Missing object detected by the references scrubbers

      • Vincent Sellier added environment: staging label

        added environment: staging label

      • Vincent Sellier changed the description

        changed the description

      • Vincent Sellier
        Vincent Sellier @vsellier ·
        Author Owner

        A small snippet to check if a content is present in postgresql and in cassandra:

        $1642

      • Vincent Sellier changed title from Missing object detected by the references scrubbers to Missing objects detected by the references scrubbers

        changed title from Missing object detected by the references scrubbers to Missing objects detected by the references scrubbers

      • Guillaume Samson assigned to @guillaume

        assigned to @guillaume

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Check in production on the last 10k occurences of storage-primary-references-directory:

        ᐅ ./check_missing_object.sh production > last_10k_missing_objects_storage-primary-references-directory_production 2>&1
        ᐅ awk '/^Only in/' last_10k_missing_objects_storage-primary-references-directory_production
Only in cassandra  0xeb05e0fd956c4c631f7d7e98939680c2071e789a  cnt 2024-09-25
Only in cassandra  0xdfc8655296e1dbeeb7c512ec502331b92b1ab668  cnt 2024-09-25
Only in cassandra  0xf68939343bd9aa3acce85aece2bc7d8debb9e0aa  cnt 2024-09-25
Only in cassandra  0xea9730b97f09893aae772c618abd3907aa4a92a1  cnt 2024-09-25
Only in cassandra  0xeedca4d102b017fdfdddc9ac07277ae7ac67896c  cnt 2024-09-25
Only in cassandra  0xee66a2aa48d9790c71ef56aa059a56832f808873  cnt 2024-09-25
Only in cassandra  0x5fbac6b9558ef6255e12528367c84948f8fc4308  cnt 2024-09-25
Only in cassandra  0xcc14b234d6076db8c6e7800dab2f05a0afd4b03b  cnt 2024-09-25
Only in cassandra  0x3bb7e33352528c519c471cef9eaac9df88f806d9  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x623c3fa0069d3a40d2901d83183fdc9a8b89bfb5  cnt 2024-11-14
Only in cassandra  0xcc7cf5895024b1ddd7f69533db7f778836e9abec  cnt 2024-09-25
Only in cassandra  0x7e553b1035ddc126a0a80761859f1417f7f7bec7  cnt 2024-09-25
Only in cassandra  0x603efeb7db1824a2a8feef5dd5c960c5f66c5989  cnt 2024-09-25
Only in cassandra  0x41ca986502ccd7fa87534cec87e00bcc3b646395  cnt 2024-09-25
Only in cassandra  0x8937742da07b265a76c781d3c4c2e187fc77efbb  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x6b1b40331f1d55f01429171966c8717213f5bae1  cnt 2024-09-25
Only in cassandra  0x0cef7f78f0d67a2e2af9b5ec5e97a29557e1992b  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0xa77f0d0092c4b3102d8bf3a1ae1b12c7b478255e  cnt 2024-09-25
Only in cassandra  0xf665da6864c4531d96ebacfdf4ca5beceabd9741  cnt 2024-09-25
Only in cassandra  0x17223d31a4d329880430d75f1978cf92f0861936  cnt 2024-09-25
Only in cassandra  0x80b09944856d2d3a75deb43f77d52715689c7df4  cnt 2024-10-26
Only in cassandra  0xdc074d4760983e562739ba16472a3607b011fbfa  cnt 2024-09-25
Only in cassandra  0x033f2aebbae17774b8b7a62539d408bc35d8b86c  cnt 2024-11-19
Only in cassandra  0xd3dd8ed6fcf10b1021e4139b5ceee38961e42aa5  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x1928a679fe330d0d7c7f53b78d1238d9ce977039  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x7b1cfb09b837ddd915c81f924cb8a590b98a6cdb  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x55f74b32a6ae6977ab92699a957084696503cf88  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x43279c64392a450cdd7a7df5552d092566b1ce2a  cnt 2024-09-25
Only in cassandra  0xb816e7c8496723420595d3e057fbb7aa7f7c5a77  cnt 2024-10-31
Only in cassandra  0x12ef3374dec9a1083bde78e22bed8cb4fb3ae804  cnt 2024-09-25
Only in cassandra  0xe884b280a0b1e1e0613f7cfbd49654f39520fe5d  dir <stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined column name sha256 in table swh.directory"
Only in cassandra  0x547ccb19340801d9451f3b1acbcb8c9c2b4d1fee  cnt 2024-09-25
        ᐅ awk '/^ko: both absent/' last_10k_missing_objects_storage-primary-references-directory_production | wc -l
9953
        ᐅ awk '/^ko: both absent/{}END{print}' last_10k_missing_objects_storage-primary-references-directory_production
ko: both absent cnt 12a37ea04074cabe2d41eed7d190927dc4ec8d62
        ᐅ awk '/^ok: both exist/' last_10k_missing_objects_storage-primary-references-directory_production | wc -l
15

        The vast majority of missing content is missing from cassandra and postgresql.
        I will fix the script error on retrieving directory first occurrence date and run the script again with the last 100k occurrences.

        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Owner

          Production

          Scanning

          Checking the last 100k occurrences (order by first_occurrence desc limit 100000) of storage-primary-references-directory in scrubber table missing_object:

          ᐅ ./check_missing_object.sh production > last_100k_missing_objects_storage-primary-references-directory_production 2>&1

          Parsing the results file with awk:

          ᐅ ./missing_objects_counters.awk last_100k_missing_objects_storage-primary-references-directory_production
# cs pg status   # hits   # rates
--------------   ------   -------
only_in_pg       44       0.04
only_in_cs       167      0.17
both_exist       43112    43.09
both_absent      56643    56.62
total            100047   100.00
---
# cs only dates  # hits   # rates
---------------  ------   -------
2023-04-10       1        0.60
2023-05-01       1        0.60
2023-05-02       1        0.60
2024-10-26       1        0.60
2024-11-14       1        0.60
2024-11-18       1        0.60
2023-04-16       2        1.20
2024-10-29       2        1.20
2024-11-19       2        1.20
2024-12-03       2        1.20
2024-12-08       2        1.20
2023-04-15       3        1.80
2024-10-28       3        1.80
2023-04-11       4        2.40
2023-04-14       6        3.59
2023-04-12       7        4.19
2023-04-13       7        4.19
2024-10-31       11       6.59
2024-09-25       81       48.50
---
errors:          110      0.11
---
Reported on Wed Jan 15 03:16:31 PM CET 2025

          There are some errors due to connection timeout:

          ᐅ awk 'BEGIN{count=0;last_line=""}{if($0 ~ /^Connection error/){count++;last_line=$0}}END{printf "Error: `%s`\nTotal: %s\n",last_line,count}' last_100k_missing_objects_storage-primary-references-directory_production 
Error: `Connection error: ('Unable to connect to any servers', {'192.168.100.181:9042': TimeoutError(110, "Tried connecting to [('192.168.100.181', 9042)]. Last error: Connection timed out")})`
Total: 81

          Connection error: ('Unable to connect to any servers', {'192.168.100.181:9042': TimeoutError(110, "Tried connecting to [('192.168.100.181', 9042)]. Last error: Connection timed out")})

          A very small number of id were not processed (directory has no ingestion date and I forgot a line break):

          ᐅ awk '$0~/Only in cassandra/&&NF>12' last_100k_missing_objects_storage-primary-references-directory_production
Only in cassandra  0x8bfbe7ccac92b1277d0a7ab90cc8d633169228e9  dir Only in cassandra  0x7820373dd5634ce05df6ffd47daa41483f0e9e64  dir Only in cassandra  0x6f734246208126aa489cd1efb3c68b157b758075  dir ko: both absent cnt 1687154f36ed8ff1a74b71668924bb772293619b
Only in cassandra  0xd3dd8ed6fcf10b1021e4139b5ceee38961e42aa5  dir Only in cassandra  0x1928a679fe330d0d7c7f53b78d1238d9ce977039  dir Only in cassandra  0x7b1cfb09b837ddd915c81f924cb8a590b98a6cdb  dir Only in cassandra  0x55f74b32a6ae6977ab92699a957084696503cf88  dir ko: both absent cnt 4aad046e3824cfebd96ffa3d0714007747ee6bd9

          Conclusion

          In summary 1, it seems that the vast majority (99.71%) of swhid in the scrubber missing_object table for the storage-primary-references-directory are in these two cases:

          • object exists in postregsql and cassandra (43.09%);
          • object is absent in postgresql and cassandra(56.62%).

          Notes

          `check_missing_object.sh` 
          #!/usr/bin/env bash

set -u

[[ $# == 1 ]] || exit 1
[[ "$1" =~ ^production|staging$ ]] || exit 1

source "$HOME"/cqlshEnv/bin/activate

typeset -r DB_CMD="select id,first_occurrence from missing_object where config_id = 5 order by first_occurrence desc limit 100000;"
typeset -A TYPE
TYPE["cnt"]="content"
TYPE["dir"]="directory"
if [[ "$1" == "production" ]];then
  typeset -r SCRUBBER_PG_ARGS="dbname=swh-scrubber host=db.internal.softwareheritage.org xxx"
  typeset -r SWH_PG_ARGS="dbname=softwareheritage host=db.internal.softwareheritage.org xxx"
  typeset -r SWH_CASS_ARGS="--cqlshrc=$HOME/.config/swh/prod_cqlshrc"
elif [[ "$1" == "staging" ]];then
  typeset -r SCRUBBER_PG_ARGS="dbname=swh-scrubber host=db1.internal.staging.swh.network xxx"
  typeset -r SWH_PG_ARGS="dbname=swh host=db1.internal.staging.swh.network xxx"
  typeset -r SWH_CASS_ARGS="--cqlshrc=$HOME/.config/swh/stg_cqlshrc"
fi

HASHES=$(psql -d "$SCRUBBER_PG_ARGS" -c "$DB_CMD" -At | awk -F "|" '{print $1}')

for swhid in $HASHES; do
  type=$(awk -F ":" '{print $3}' <<< "$swhid")
  hash=$(awk -F ":" '{print $4}' <<< "$swhid")
  if [[ "$type" == "cnt" ]];then
  pg=$(psql -d "$SWH_PG_ARGS" -c "select sha1_git from ${TYPE[$type]} where sha1_git = '\x${hash}';" -At)
  cs=$(cqlsh "$SWH_CASS_ARGS" -e "select * from content_by_sha1_git where sha1_git = 0x${hash};" | awk '$1~/^0x/')
  fi
  if [[ "$type" == "dir" ]];then
  pg=$(psql -d "$SWH_PG_ARGS" -c "select id from ${TYPE[$type]} where id = '\x${hash}';" -At)
  cs=$(cqlsh "$SWH_CASS_ARGS" -e "select * from ${TYPE[$type]} where id = 0x${hash};" | awk '$1~/^0x/')
  fi
  if [[ -n "$pg" ]] && [[ -n "$cs" ]];then
    echo ok: both exist "$type $hash"
    continue
  fi
  if [[ -z "$pg" ]] && [[ -z "$cs" ]];then
    echo ko: both absent "$type $hash"
    continue
  fi
  [[ -n "$pg" ]] && echo "Only in postgresql $pg"
  if [[ -n "$cs" ]];then
   echo -n "Only in cassandra ${cs%%|*} $type "
   token=$(awk -F '|' '{print $2}' <<< "$cs")
   if [[ "$type" == "cnt" ]]
   then
    cqlsh "$SWH_CASS_ARGS" -e "select toDate(writetime(status)/1000) from ${TYPE[$type]} where token(sha256) = $token;" | \
    awk 'NF==1&&$1~/^20/{gsub (" ", "", $0); print}'
   else
    echo
   fi
  fi
done
          `missing_objects_counters.awk` 
          #!/usr/bin/awk -f
# vim: ai ts=4 sts=4 sw=4

BEGIN{
format="%-16s %-8s %s\n"
counters["total"]=0
}
{
#print
counters["total"]++
if ($0 ~ /^Only in cassandra/){
    counters["only_in_cs"]++
    if ($6 ~ /^[^20]/){
        errors_counter["errors"]++
    } else {
        cs_date_counter[$6]++
    }
}
if ($0 ~ /^Only in postgresql/){
        counters["only_in_pg"]++
    if ($6 ~ /^[^20]/){
        errors_counter["errors"]++
    }
}
if ($0 ~ /^Connection error/){errors_counter["errors"]++}
if ($0 ~ /^ok: both exist/){counters["both_exist"]++}
if ($0 ~ /^ko: both absent/){counters["both_absent"]++}
}
END{
"date" | getline date
close("date")
printf format, "# cs pg status", "# hits","# rates"
printf format, "--------------", "------","-------"
format="%-16s %-8s %.2f\n"
# sort by description name
#PROCINFO["sorted_in"] = "@ind_str_asc"
# sort by counters
PROCINFO["sorted_in"] = "@val_num_asc"
for (i in counters)
        printf  format, i, counters[i], counters[i]/counters["total"]*100
format="%s\n"
printf format, "---"
format="%-16s %-8s %s\n"
printf format, "# cs only dates", "# hits","# rates"
printf format, "---------------", "------","-------"
format="%-16s %-8s %.2f\n"
for (i in cs_date_counter)
        printf  format, i, cs_date_counter[i], cs_date_counter[i]/counters["only_in_cs"]*100
format="%s\n"
printf format, "---"
format="%-16s %-8s %.2f\n"
printf format, "errors:", errors_counter["errors"], errors_counter["errors"]/counters["total"]*100
printf "---\nReported on %s\n", date
}

          1. we can conclude that awk is really awesome (as usual). ↩

          Edited by Guillaume Samson
        • Vincent Sellier
          Vincent Sellier @vsellier ·
          Author Owner

          we can conclude that awk is really awesome (as usual).

          :laughing:

          More seriously it's s good news that the errors are not so frequent

          Maybe it would be worth generating the complete list of missing contents in Cass or PG to check if they can be repaired, but it could be done in another dedicated issue. I've no idea if it's realistic or not in term of execution time.

          Edited by Vincent Sellier
        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Owner

          Scanning 100k occurrences took approximately 24 hours on my laptop.
          I will see if these scripts run in a debug pod created from the toolbox.

          Edited by Guillaume Samson
        • Please register or sign in to reply
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Here is the results file: last_100k_missing_objects_storage-primary-references-directory_production

      • Guillaume Samson added environment: production label

        added environment: production label

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Staging

        Scanning

        Checking the last 10k occurrences (order by first_occurrence desc limit 10000) of storage-references-directory in scrubber table missing_object:

        ᐅ ./check_missing_object.sh staging > last_10k_missing_objects_storage-primary-references-directory_staging 2>&1
        ᐅ ./missing_objects_counters.awk last_10k_missing_objects_storage-primary-references-directory_staging
☰ status         ⛁ hits     ⚙ rates
---------        -------    --------
both_exist       21         0.21
both_absent      2249       22.45
only_in_cs       7730       77.16
total            10018      100.00

☰ only_in_cs     ⛁ hits     ⚙ rates
------------     -------    --------
2024-10-24       109        1.41
2024-10-11       129        1.67
2024-07-26       147        1.90
2024-10-03       154        1.99
2024-07-29       198        2.56
2024-10-08       203        2.63
2024-10-04       224        2.90
2024-08-05       312        4.04
2024-08-13       313        4.05
2024-11-06       403        5.21
2024-10-07       412        5.33
2024-11-20       437        5.65
2024-10-26       484        6.26
2024-10-14       508        6.57
2024-11-27       527        6.82
2024-07-23       571        7.39
2024-07-24       671        8.68
2024-10-16       737        9.53
2024-11-14       825        10.67

☰ errors         ⛁ hits     ⚙ rates
---------        -------    --------
unknown          2          0.02
conn_timeout     4          0.04
conn_shutdown    25         0.25

Reported on Thu Jan 16 10:42:57 AM CET 2025
        Edited by Guillaume Samson
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Staging

        After:

        • creating a debug pod from toolbox; 
          ᐅ kbs debug -n swh-cassandra -it swh-toolbox-5bbc6c7575-f9nn9 \
-it --copy-to=checking-missing-contents --container=swh-toolbox -- bash
        • copying scripts and configs files in the debug pod;
        • creating a venv with cqlsh in the debug pod.

        Everything seems to works fine:

        ᐅ kbs exec -ti -n swh-cassandra -c swh-toolbox checking-missing-contents -- bash
        swh@checking-missing-contents:~$ check_missing_object staging
ko: both absent dir 09cd2b52b31402b2e34cca97956e932a087a6af5
ko: both absent dir 0d3a983d21acdb0614c47cd12542d74a112a8c74
Only in cassandra cnt  0xa5550dd359146561f0372191c818df603d3c3ff2  2024-10-16
Only in cassandra cnt  0xd126bc8cde3503e55a81452f07f306e8058b5fee  2024-07-24
ko: both absent dir ebe84ddacbca167fdc659ae8cedfc99597fd10bc
ko: both absent cnt 53776a65301dcf3637e8cf75879f2002389cf3fd
ko: both absent dir 8c0ee48e872933185db2fd3e19bb4a69dfd12fc1
Only in cassandra cnt  0xcbc5b755d0fb5c5ca49aed22f57789b5b1e44fb4  2024-11-27
Only in cassandra cnt  0x69dc7e5eac4a18e063860bc9db5bb483e516d7cc  2024-11-14
Only in cassandra cnt  0x2a88e1e4d8068dc2adc0469aea477ed387cd0018  2024-11-14
        swh@checking-missing-contents:~$ check_missing_object staging asc
ok: both exist cnt 917f473632e5ba47e613604622b2ab83b6c7ffcb \x917f473632e5ba47e613604622b2ab83b6c7ffcb  0x917f473632e5ba47e613604622b2ab83b6c7ffcb | -2263884508483967155
ok: both exist cnt 003b77527d968c6fa60e12333deb8b75371bf4d5 \x003b77527d968c6fa60e12333deb8b75371bf4d5  0x003b77527d968c6fa60e12333deb8b75371bf4d5 | 7073853444204460170
ok: both exist cnt 3220a2029b15056659440a4d8d1bade024ce8893 \x3220a2029b15056659440a4d8d1bade024ce8893  0x3220a2029b15056659440a4d8d1bade024ce8893 | -7874141961238189960
ok: both exist cnt 3099fb5a937644a1d92f86f07c85b7436a6cde95 \x3099fb5a937644a1d92f86f07c85b7436a6cde95  0x3099fb5a937644a1d92f86f07c85b7436a6cde95 | 3189955565862811696
ok: both exist cnt 3c46c7f7c9a89596b58636d23fad67bd8fe8dd3b \x3c46c7f7c9a89596b58636d23fad67bd8fe8dd3b  0x3c46c7f7c9a89596b58636d23fad67bd8fe8dd3b | 8162213414131792167
ok: both exist cnt 5a32bf57bc347eb9dc8adc32e66befc074a4b069 \x5a32bf57bc347eb9dc8adc32e66befc074a4b069  0x5a32bf57bc347eb9dc8adc32e66befc074a4b069 | 8048195691597345888
ok: both exist cnt 697bbecfd56890d69063019cecc10feb68dd7dad \x697bbecfd56890d69063019cecc10feb68dd7dad  0x697bbecfd56890d69063019cecc10feb68dd7dad | 6516629249060593557
ok: both exist cnt 60622ad1398423cfe0d2aceb7d7c330949896e94 \x60622ad1398423cfe0d2aceb7d7c330949896e94  0x60622ad1398423cfe0d2aceb7d7c330949896e94 | -6895134055073361785
ok: both exist cnt 35592b133028e44781cb9b9bf9b48f0454b21543 \x35592b133028e44781cb9b9bf9b48f0454b21543  0x35592b133028e44781cb9b9bf9b48f0454b21543 | 4807782215291803383
ok: both exist cnt 315d0884829a38016b3a7243df9408ac464efa2b \x315d0884829a38016b3a7243df9408ac464efa2b  0x315d0884829a38016b3a7243df9408ac464efa2b | -1589591585623535194

        swh@checking-missing-contents:~$ check_missing_object staging | missing_objects_counters.awk
# status         # hits     # rates
--------         ------     -------
only_in_cs       5          50.00
both_absent      5          50.00
total            10         100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-10-16       1          20.00
2024-07-24       4          80.00

# errors         # hits     # rates
--------         ------     -------

Reported on Thu Jan 16 11:51:12 UTC 2025

        I will create the same debug pod in production.

        Edited by Guillaume Samson
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        There are a deployment with a ceph-rbd volume, check-missing-contents, in staging and production.

        • staging:

          ~ ᐅ kbs get po,deploy,pvc -l app=check-missing-contents -n swh
NAME                                         READY   STATUS    RESTARTS   AGE
pod/check-missing-contents-ddd9d746f-bp8wq   1/1     Running   0          5d16h

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/check-missing-contents   1/1     1            1           5d16h

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-ab5ad099-fe51-4b35-9162-a810b7967d20   5Gi        RWO            ceph-rbd       5d19h

        • production:

          ~ ᐅ kbp get po,deploy,pvc -l app=check-missing-contents -n swh
NAME                                         READY   STATUS    RESTARTS   AGE
pod/check-missing-contents-ddd9d746f-c7mmj   1/1     Running   0          5d16h

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/check-missing-contents   1/1     1            1           5d16h

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-33d8b07a-37b5-4ed6-9ec8-f8f6376b5725   5Gi        RWO            ceph-rbd       5d17h

        The ceph-rbd volume is mounted on /data.

        ᐅ kbs get po -l app=check-missing-contents -n swh -o yaml | \
yq '.items[].spec.containers[].volumeMounts[]|
select(.name == "cmc-persistent-storage")'
mountPath: /data
name: cmc-persistent-storage

        Scripts and results are on the ceph-rbd volume.
        Here are the current results since Tue Jan 21 ~18:00.

        • staging:

          ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \
/data/swh/missing_objects_counters.awk /data/swh/storage-references-directory_staging_asc_all
# status         # hits     # rates
--------         ------     -------
both_exist       298837     96.13
both_absent      12023      3.87
total            310860     100.00

Reported on Mon Jan 27 11:45:59 UTC 2025

        • production:

          ᐅ kbp exec -ti -n swh $(kbp get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \
/data/swh/missing_objects_counters.awk /data/swh/storage-primary-references-directory_production_asc_all
# status         # hits     # rates
--------         ------     -------
both_exist       2347       0.67
only_in_cs       26         0.01
both_absent      348518     99.32
total            350891     100.00

Reported on Mon Jan 27 11:46:03 UTC 2025
          ᐅ kbp exec -ti -n swh $(kbp get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \
awk '/Only in cassandra/{$1=$2=$3="";print}' /data/swh/storage-primary-references-directory_production_asc_all
   dir 0x4fe52078b920f21590b94711745c8eca5685cc23
   dir 0x17986680582470dd2892628328bb8bdecfab97ff
   dir 0x2b7b931415030aa39aec46a561ef98e3046c0f63
   dir 0xf4083baebe5e34d3dfef60b83181c5edc0c37e6b
   dir 0xe48aaf0b054d76547ce57d4e33ff07f5daefb8ea
   dir 0x394a721548af2f8c112083ba0de6d1df3b8fc275
   dir 0x47610f1ee3db2ec6fe35d0ac8ca35de74afd43a9
   dir 0x8f183e6ff56aa49c5b4a9f6cd3ff55d2898e9e64
   dir 0xeb38c99d70702d7c32c97eaa70baa1daa4ae7916
   dir 0x134c2e069bb76173116a785fa4f05ee1ea2a1f70
   dir 0x47ab8b9120aa0603b2d7df07f2db4e984032e666
   dir 0x943636de3622202057c77067e49ac6e7e85c271c
   dir 0x29c2e3747e982b76a4f64ed3ff76d048f778df0d
   dir 0xff40c4aa40451d1f0567d4727907cc45527452f3
   dir 0xb77db38fef828b643a1e87d7705cb03e87a65e8a
   dir 0x7543ca30ad74939e3a8c85d4f7337bd7242e99e0
   dir 0xc44061a20faa0f673a4d0c31a757d9a58feb5030
   dir 0xdf9792fe933ce345314feabf9fdfac6ab06e9f10
   dir 0x537f33ff7b4f483f2d588d6f86bd7ea25602149b
   dir 0x14f4473ea8b3e91601f68e05f4a4f638c7a91b6a
   dir 0xaf917e57a6aad60f0416fcd4b626f4e3fb771e39
   dir 0x803c1fdb0ea16d85b731508738c464a2d6e67bf2
   dir 0x6800e99940f13b8106313c6a423735a5d1d49130
   dir 0x50a8084d995d90035d6f085e87e0ab8258ba2fa6
   dir 0x9b0a7f63eeed2834fdb328c35181d21d9376aa71
   dir 0xb14e07fc965363fbd1dd60b6af0ed1cd1d7a8415
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Here are the results in production:

        ᐅ kbp exec -ti -n swh $(kbp get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \     
/data/swh/missing_objects_counters.awk /data/swh/storage-primary-references-directory_production_asc_all
# status         # hits     # rates
--------         ------     -------
only_in_pg       555        0.03
both_exist       553315     31.95
only_in_cs       1074       0.06
both_absent      1176655    67.95
total            1731599    100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-09-25       124        11.55
2024-10-29       3          0.28
2023-04-10       2          0.19
2023-05-02       3          0.28
2023-05-01       2          0.19
2024-12-03       2          0.19
2023-04-29       2          0.19
2023-04-13       9          0.84
2023-04-19       1          0.09
2024-07-01       1          0.09
2024-10-31       41         3.82
2023-04-16       3          0.28
2023-04-15       8          0.74
2023-04-12       6          0.56
2023-05-04       3          0.28
2024-10-28       3          0.28
2022-09-24       5          0.47
2023-04-11       6          0.56
2023-04-30       2          0.19
2023-05-03       2          0.19
2024-11-19       1          0.09
2023-04-14       7          0.65

Reported on Tue Feb 18 09:00:37 UTC 2025

        storage-primary-references-directory_production_asc_all.gz

        I removed the deployment but for now kept the pvc:

        ᐅ kbp get po,deploy,pvc -n swh -l app=check-missing-contents
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-33d8b07a-37b5-4ed6-9ec8-f8f6376b5725   5Gi        RWO            ceph-rbd       27d

        I stopped the scan in staging 10 days ago (the pod used a lot of memory; > 20G), I will relaunch it.

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        New deployment in staging:

        ᐅ kbs get po,deploy,pvc -n swh -l app=check-missing-contents 
NAME                                         READY   STATUS    RESTARTS   AGE
pod/check-missing-contents-ddd9d746f-rmzg6   1/1     Running   0          23m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/check-missing-contents   1/1     1            1           23m

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-ab5ad099-fe51-4b35-9162-a810b7967d20   5Gi        RWO            ceph-rbd       27d

        Current results:

        ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \   
/data/swh/missing_objects_counters.awk /data/swh/storage-references-directory_staging_asc_all
# status         # hits     # rates
--------         ------     -------
only_in_pg       1          0.00
both_exist       896003     96.08
only_in_cs       20         0.00
both_absent      36517      3.92
total            932541     100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-01-08       20         100.00

Reported on Tue Feb 18 09:51:42 UTC 2025

        Memory usage is better:

        ᐅ kbs top po -n swh -l app=check-missing-contents
NAME                                     CPU(cores)   MEMORY(bytes)   
check-missing-contents-ddd9d746f-rmzg6   1009m        304Mi
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        Current results in staging:

        ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \
/data/swh/missing_objects_counters.awk /data/swh/storage-references-directory_staging_asc_all
# status         # hits     # rates
--------         ------     -------
only_in_pg       54394      4.94
both_exist       1001229    90.97
only_in_cs       20         0.00
both_absent      44946      4.08
total            1100589    100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-01-08       20         100.00

Reported on Fri Feb 21 09:40:56 UTC 2025
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        In staging environment the deployment was recreated:

        ᐅ kbs get po -n swh -l app=check-missing-contents
NAME                                     READY   STATUS    RESTARTS   AGE
check-missing-contents-ddd9d746f-bfnrd   1/1     Running   0          37h

        So the check is no longer running:

        ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- ps auxfwww
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
swh           31  0.0  0.0   8648  1848 pts/0    Rs+  09:58   0:00 ps auxfwww
swh            1  0.0  0.0   5788  1796 ?        Ss   Feb24   0:00 /bin/bash /opt/swh/entrypoint.sh
swh            9  0.0  0.0   4284   416 ?        S    Feb24   0:00 sleep infinity

        I relaunched it:

        swh@check-missing-contents-ddd9d746f-bfnrd:~$ /data/swh/check_missing_object staging asc all 2>&1 >> /data/swh/storage-references-directory_staging_asc_all &
[1] 46

        Current results:

        ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \
/data/swh/missing_objects_counters.awk /data/swh/storage-references-directory_staging_asc_all
# status         # hits     # rates
--------         ------     -------
only_in_pg       285070     21.23
both_exist       1001229    74.55
only_in_cs       20         0.00
both_absent      56758      4.23
total            1343078    100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-01-08       20         100.00

Reported on Wed Feb 26 10:02:36 UTC 2025
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Owner

        The check is complete in staging:

        ᐅ kbs exec -ti -n swh $(kbs get po -n swh -l app=check-missing-contents -o jsonpath='{.items[0].metadata.name}') -- \   
/data/swh/missing_objects_counters.awk /data/swh/storage-references-directory_staging_asc_all
# status         # hits     # rates
--------         ------     -------
only_in_pg       285071     17.01
both_exist       1152114    68.76
only_in_cs       139338     8.32
both_absent      99093      5.91
total            1675617    100.00

# only_in_cs     # hits     # rates
------------     ------     -------
2024-11-14       7136       5.12
2024-10-10       19         0.01
2024-10-04       3051       2.19
2024-07-26       2766       1.99
2025-02-02       53         0.04
2024-10-29       291        0.21
2024-07-14       13         0.01
2024-07-23       9299       6.67
2024-08-08       5          0.00
2024-10-14       6944       4.98
2024-01-08       33         0.02
2025-01-15       16         0.01
2024-11-07       331        0.24
2024-10-25       88         0.06
2024-11-15       62         0.04
2024-10-17       1213       0.87
2025-02-03       66         0.05
Only             2          0.00
2025-02-20       1313       0.94
2024-10-26       5612       4.03
2025-02-19       1          0.00
2024-07-29       4650       3.34
2024-10-12       172        0.12
2024-10-21       5          0.00
2024-10-02       32         0.02
2025-01-23       9          0.01
2025-02-04       9          0.01
2024-09-17       3          0.00
2024-10-08       2951       2.12
2024-11-06       2246       1.61
2024-11-18       15         0.01
2024-10-22       7          0.01
2024-08-06       65         0.05
2024-11-20       4469       3.21
2024-10-16       9739       6.99
2024-07-30       87         0.06
2024-07-24       25068      17.99
2025-02-18       1931       1.39
2024-08-05       6908       4.96
2024-07-18       3417       2.45
2024-11-27       4135       2.97
2024-10-11       4215       3.03
2024-07-12       50         0.04
2024-10-03       2170       1.56
2025-02-05       16         0.01
2024-10-28       603        0.43
2024-11-12       2          0.00
2024-08-13       4713       3.38
2024-10-23       6          0.00
2025-01-21       2649       1.90
2024-10-15       221        0.16
2024-10-07       18935      13.59
2025-02-01       17         0.01
2025-01-26       21         0.02
2024-10-24       1421       1.02
2025-02-22       1          0.00
2024-09-18       23         0.02

Reported on Fri Mar  7 12:43:55 UTC 2025

        storage-references-directory_staging_asc_all.gz

        The deployment check-missing-contents has been deleted.
        The pvc in staging and production has been kept:

        ᐅ kbs get po,deploy,pvc -l app=check-missing-contents -n swh        
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-ab5ad099-fe51-4b35-9162-a810b7967d20   5Gi        RWO            ceph-rbd       44d
        ᐅ kbp get po,deploy,pvc -l app=check-missing-contents -n swh
NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/cmc-pvc   Bound    pvc-33d8b07a-37b5-4ed6-9ec8-f8f6376b5725   5Gi        RWO            ceph-rbd       44d
        Edited by Guillaume Samson
      Please register or sign in to reply