Skip to content
Snippets Groups Projects
New look: Off

cassandra08 fails to restart

    • Closed Issue created by Guillaume Samson @guillaume

    The cassandra08 node fails to restart:

    root@cassandra01:~# /opt/cassandra/bin/nodetool -u cassandra -pw siesta-subway-composer-reproduce status -r | awk '/^DN/'
DN  cassandra08.internal.softwareheritage.org  9.03 TiB  16      29.9%             247cd9e3-a70c-465c-bca1-ea9d3af9609a  rack1

    Here are the error logs:

    root@cassandra08:~# tail -100 /var/log/cassandra/instance1/system.log | grep ERROR
ERROR [main] 2024-03-28 08:35:52,422 LogReplicaSet.java:194 - Found too many lines for , giving up
ERROR [main] 2024-03-28 08:35:52,423 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [main] 2024-03-28 08:35:52,424 LogTransaction.java:551 - Unexpected disk state: failed to read transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [main] 2024-03-28 08:35:52,428 CassandraDaemon.java:911 - Cannot remove temporary or obsoleted files for swh.extid due to a problem with transaction log files. Please check records with problems in the log messages above and fix them. Refer to the 3.0 upgrading instructions in NEWS.txt for a description of transaction log files.

    Here is the log of the failed transaction:

    root@cassandra08:~# cat /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log
ABORT:[,0,0][737437348]
ADD:[/srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb-14885-big-,0,8][3021221038]
    root@cassandra08:~# ls -trl /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb-14885-big*
-rw-r--r-- 1 cassandra cassandra 0 Mar 27 16:14 /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb-14885-big-Index.db
-rw-r--r-- 1 cassandra cassandra 0 Mar 27 16:14 /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb-14885-big-Data.db

    Designs

      Child items ...

      2. Activity

      • Guillaume Samson added cassandra label

        added cassandra label

      • Guillaume Samson assigned to @guillaume

        assigned to @guillaume

      • Vincent Sellier
        Vincent Sellier @vsellier ·
        Owner

        The first occurence of the errors was the 2024-03-27 16:14:41 UTC

        vsellier@cassandra08 /var/log/cassandra/instance1
 % zgrep "Failed to read records"  debug.log.2024-03-27.2.zip
vsellier@cassandra08 /var/log/cassandra/instance1
 % zgrep "Failed to read records"  debug.log.2024-03-27.3.zip |head -n 10
ERROR [prometheus-http-1-1] 2024-03-27 16:14:41,802 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:14:42,808 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:15:11,751 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:15:12,878 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:15:41,327 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:15:42,346 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:16:11,540 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:16:12,913 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:16:41,327 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
ERROR [prometheus-http-1-1] 2024-03-27 16:16:42,398 LogFile.java:162 - Failed to read records for transaction log [nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log in /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22]
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        Resolution:

        • stop the service:
        systemctl stop cassandra@instance1.service
        • remove the failed transaction log:
        mkdir restart_failure_20240328
mv /srv/cassandra/instance1/data/swh/extid-ba0c7ec02d2e11edb23fb91705299a22/nb_txn_stream_14660670-ec55-11ee-b784-f378cae1562c.log restart_failure_20240328/
        • restart the service:
        systemctl start cassandra@instance1.service
      • Guillaume Samson added 1h 30m of time spent

        added 1h 30m of time spent

      • Guillaume Samson closed

        closed

      • Vincent Sellier
        Vincent Sellier @vsellier ·
        Owner

        A full repair was launched on the extid table through reaper. It finished without any error.

        Edited by Vincent Sellier
      • Vincent Sellier reopened

        reopened

      • Vincent Sellier closed

        closed

      • Vincent Sellier mentioned in issue #5389 (closed)

        mentioned in issue #5389 (closed)

      Please register or sign in to reply