Skip to content
Snippets Groups Projects

Deploy loader.svn v1.1 and analyze remaining errors

    • Closed Issue created by Antoine R. Dumont @ardumont

    When fixes (T3870) land, this will need a new release cycle.

    Migrated from T3875 (view on Phabricator)

    Designs

      Child items ...

      2. Activity

      • Antoine R. Dumont added SVN Loader System administration priority:Normal labels

        added SVN Loader System administration priority:Normal labels

      • Antoine R. Dumont added state:wip label

        added state:wip label

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        staging:

        • Package python3-swh.loader.svn upgraded
        • Service swh-worker@loader_{svn,high_priority} restarted
      • Antoine R. Dumont changed title from Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1

        changed title from Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1

      • Antoine Lambert
        Antoine Lambert @anlambert ·
        Maintainer

        Subversion origins that failed to load or with a partial visit status when using swh-loader-svn v1.0.0 (T3870#77580) have been resubmitted on staging to check how many can now be fully loaded using swh-loader-svn v1.1.0.

        http://kibana0.internal.softwareheritage.org:5601/app/dashboards#/view/3c126be0-7944-11ec-b7b0-c7cd6a2a1525?_g=h@66049b4

        anlambert@carnavalet:~/swh$ cat svn_with_externals_to_reload_with_v1.1.txt | sed s/^/svn,/ | swh web --config-file ~/.config/swh/global.yml save submit-request
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/disy-commons/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gate/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ivs-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jvcl/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lignedetemps/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/matplotlib/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediboard/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/restore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/turbogears1/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ubraa/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ugmdatabase/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ui4w/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/umlmodeler/svncode)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/unixpiger/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/upsurge/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urbanmediator/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/urt41banguid/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usboblivion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/usvn/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/w3af/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/whack-a-mole/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wideimage/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikipad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wikiup/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/windsofscrum/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wp-blog/svn)
INFO:root:Submitted origin (svn, https://svn.blender.org/svnroot/bf-translations/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aiadsocialnet/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/aoia/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/apertium/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/arp2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asicms/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/asunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ats-lang/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/auber/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/axiomengine/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/azsmrc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bibdesk/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bitpim/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/borgworx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/boson/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/brlcad/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bsml/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/bzflag/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cesaachen/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/charon-suite/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/chocolate-doom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/coati-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/codeblocks/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/collada-dom/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crackpot/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/crossfire/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-battletrack/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/csipc2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/cs-project/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/czretouch/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dafizilla/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/daimonin/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/delphiasiovst/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dlvhex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/docbook/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dooble/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/doubleagent/code/)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/drdivx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/dsao-md/code-code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/emgucv/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/epidoc/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ergatis/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/exponent/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/eyeqhtml5/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ffdshow-tryout/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fireclipse/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/firemox/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flightpred/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flog/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/flylinkdc-client/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/foray/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/fredslibraries/esp-clibs)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/freeorion/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/funambol/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gallery/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/garabagh/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gar/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/geshi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/getdata/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/gexperts/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/giews/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/graphsharp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/helix/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/homefinance/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/htmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ieatbrainz/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/itelex/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jamwiki/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jedit/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jgerman/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jsk-ros-pkg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/jython/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/kdenlive/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/libpong/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/librestudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lima-vva/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lithron/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/loconetovertcp/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/logilogi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lucifersstand/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/lw-framework/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/m2bee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/madwifi/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/magware/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/makingthings/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/manatee/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marathon/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/marsyas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mclarobotics/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mediainfo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/meshlab/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/microhal/code-0)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mingw-w64/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/misesquipulas/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/modelbuilder/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/moose/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mplayerx-osx/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/mwclient/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ndlb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/nukeshtmlunit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/obi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/olympos/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/oof/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openautomation/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openimscore/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/openrocket/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pascalscada/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pcpe/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pencil-planner/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/perp/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpaxes/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phpbbproject/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/phphtmllib/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/plonepm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pmml2svg/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/potassco/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/prism-api/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/projectorria/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/psycle/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pure-data/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/pywebsvcs/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/rand-mh/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/razorbacktm/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/roadnav/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/ruby-gnome2/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sashimi/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/seabassshowdown/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/shareaza/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/silex/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/simplectest/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/skanner3d/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/smartweb/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/stemkit/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/strawhat/tango-ds)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sugarcadesnake/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/sumo/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook/source)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/svnbook-zh/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/texstudio/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tigefa4u/trac)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tikiwiki/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tora/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/tortoisesvn/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/trinlibr/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/virtualslight/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/wesnoth-umc-dev/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xreal/svn)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvidcap/code)
INFO:root:Submitted origin (svn, https://svn.code.sf.net/p/xvrstudio/code)
INFO:root:Submitted origin (svn, http://svn.code.sf.net/p/tortoisesvn/code/trunk)
INFO:root:Submitted origin (svn, http://svn.osdn.net/svnroot/tortoisesvn/)
      • Antoine Lambert changed title from staging: Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1 and analyze remaining errors

        changed title from staging: Deploy loader.svn v1.1 to staging: Deploy loader.svn v1.1 and analyze remaining errors

      • Sentry Integration
        Sentry Integration @swh-sentry-integration ·
        Maintainer

        Sentry issue: SWH-LOADER-SVN-54

      • Phabricator Migration user mentioned in commit swh/devel/swh-loader-svn@34f61cb5

        mentioned in commit swh/devel/swh-loader-svn@34f61cb5

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        python3-swh.loader.svn v1.1.1 deployed on staging, production ongoing.

      • Antoine R. Dumont changed title from staging: Deploy loader.svn v1.1 and analyze remaining errors to Deploy loader.svn v1.1 and analyze remaining errors

        changed title from staging: Deploy loader.svn v1.1 and analyze remaining errors to Deploy loader.svn v1.1 and analyze remaining errors

      • Antoine Lambert
        Antoine Lambert @anlambert ·
        Maintainer

        Below are the results of the loading of the subversion origins listed above, I used a filtered version of the webapp database on staging.

        13:36 $ sqlite3 swh/web/settings/db.sqlite3 
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> select visit_status, count(*) from save_origin_request group by visit_status;
failed|18
full|76
partial|83

        So 76 origins that could not be fully loaded with swh-loader-svn v1.0.0 are now correctly processed with v1.1.0.

        They are still failures and partial loading but most of them comes from network errors while dumping subversion repositories, see example below:

        [2022-01-25 17:00:23,243: ERROR/ForkPoolWorker-14] Loading failure, updating to `failed` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 335, in load
    self.prepare()
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 811, in prepare
    dump_path = self.dump_svn_revisions(self.svn_url, last_loaded_svn_rev)
  File "/usr/lib/python3/dist-packages/swh/loader/svn/loader.py", line 771, in dump_svn_revisions
    % (last_dumped_rev, last_loaded_svn_rev)
Exception: Last dumped subversion revision (48) is lesser than the last one loaded into the archive (10498)

        This last batch of tests allowed me to detect some small remaining issues that got fixed in swh-loader-svn v1.1.1.

      • Antoine R. Dumont removed state:wip label

        removed state:wip label

      • Antoine R. Dumont closed

        closed

      • Antoine R. Dumont mentioned in issue #3864 (closed)

        mentioned in issue #3864 (closed)

      • Phabricator Migration user mentioned in merge request swh/devel/swh-loader-svn!109 (closed)

        mentioned in merge request swh/devel/swh-loader-svn!109 (closed)

      • Phabricator Migration user mentioned in issue swh/devel/swh-loader-svn#3870 (closed)

        mentioned in issue swh/devel/swh-loader-svn#3870 (closed)

      Please register or sign in to reply