sentry 5823: fill-in the large-repository hole

changed title from fill in the hole from sentry to sentry 5823: fill-in the large-repository hole

Dataset [1]

I'm gonna extract the non github.com origins and shuffle those and push them directly to the oneshot queue.

[1]

$ wc -l 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt
10686 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt
$ grep -v github.com 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt | wc -l
769

[1] 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt

I'm gonna extract the non github.com origins and shuffle those and push them directly to the oneshot queue.

done ^

Splitting dataset with github/non-github origins [1]
Scheduling immediately the shuffled list of origins [2]

[1]

$ grep -v github.com 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt > 20230411-1010-loader-git.pack-file-too-big-issue-5823.non-github-urls.txt
$ grep github.com 20230411-1010-loader-git.pack-file-too-big-issue-5823.urls.txt > 20230411-1010-loader-git.pack-file-too-big-issue-5823.github-urls.txt
$ wc -l 20230411-1010-loader-git.pack-file-too-big-issue-5823.non-github-urls.txt
769 20230411-1010-loader-git.pack-file-too-big-issue-5823.non-github-urls.txt
$ wc -l 20230411-1010-loader-git.pack-file-too-big-issue-5823.github-urls.txt
9917 20230411-1010-loader-git.pack-file-too-big-issue-5823.github-urls.txt

[2]

$ export SWH_CONFIG_FILENAME=~/.config/swh/scheduler.yml; cat ../sentry/20230411-1010-loader-git.pack-file-too-big-issue-5823.non-github-urls.txt | shuf | ./schedule_with_queue_length_check.py --queue-name oneshot-large-git

changed the description

changed milestone to %Extend archive coverage [Roadmap - Collect]

added activity::Processing label

assigned to @ardumont

With [1], extraction of the priority.list [2], the process ran most of the day.

I'll schedule those tomorrow (the queue is already up to 25k so no matter the wait).

[2] priority.list.github

$ wc -l priority.list.github
2242 priority.list.github

[1]

#!/usr/bin/env bash

set -ex

[ -z "${GH_TOKEN}" ] \
    && echo "Missing env variable GH_TOKEN set with a gh bearer token." \
    && exit 1

[ -z "${SWH_TOKEN}" ] \
    && echo "Missing env variable SWH_TOKEN set with a swh bearer token." \
    && exit 1

tmpdir="swh-check-repositories-$$"
tmppath="/tmp/${tmpdir}"
mkdir -p $tmppath

# once debug is done, uncomment the following to remove temporary working directory
# trap 'rm -rf "$tmppath"' EXIT

INPUT=${1-fulldata.txt}
OUTPUT=${2-priority.list.github}

FULLDATA_LOG=$tmppath/fulldata.log
FORKED_LOG=$tmppath/forked.log
FORKED_DATA=$tmppath/forked.data
LIST_WORKING_DATA=$tmppath/fulldata.data
LIST_NON_FORKS=$tmppath/nonfork.list
LIST_FORKS=$tmppath/forked.list
LIST_PRIORITY=$tmppath/priority.list

python3 get-repos-info.py -t "$GH_TOKEN" -a "$SWH_TOKEN" \
  $INPUT > $LIST_WORKING_DATA 2> $FULLDATA_LOG

# Extract the nonfork repositories that are still in GitHub, and sort them by number of stars
grep -v ISFORK $LIST_WORKING_DATA | sed 's/;.*;/;/' | sort -t \; -k 2 -n -r \
  | grep -v NOTINGITHUB > $LIST_NON_FORKS

# Extract original repository from fork projects
grep ISFORK $LIST_WORKING_DATA | sed 's/.*ISFORK;//' | sed 's/[^;]*;//' \
  | sed 's/;.*//' | sort -u > $LIST_FORKS

# Process the forked list, extract repos still to be archived, sort them by number of
# stars
python3 get-repos-info.py -t "$GH_TOKEN" -a "$SWH_TOKEN" \
  $LIST_FORKS > $FORKED_DATA 2> $FORKED_LOG

egrep -v "NOTING|UPTODATE|NOWPRIVATE|TOUPDATE" $FORKED_DATA | sed 's/;.*;/;/' | \
  sort -t \; -k 2 -n -r

# Merge with the nonfork.list
cat $LIST_FORKS $LIST_NON_FORKS | sort -u > $LIST_PRIORITY

# Remove number of stars
sed 's/;.*//' $LIST_PRIORITY > $OUTPUT

@rdicosmo jsyk ^ is it worth a PR to have the script within your upstream repository ?

@ardumont, sure!

great ^

done in [1]

[1] https://github.com/rdicosmo/swh-check-repositories/pull/1

I'll schedule those tomorrow (the queue is already up to 25k so no matter the wait).

meh, i was already on it so might as well be done with it...

$ export SWH_CONFIG_FILENAME=~/.config/swh/scheduler.yml; cat ~/downloads/20230411-2013-priority.list.github | shuf | ./schedule_with_queue_length_check.py --queue-name oneshot-large-git
...
{'args': [], 'kwargs': {'url': 'https://github.com/YetAnotherMorty/Audio', 'lister_name': 'github', 'lister_instance_name': 'github', 'pack_size_bytes': 34359738368}}
{'args': [], 'kwargs': {'url': 'https://github.com/zusterben/plan_b', 'lister_name': 'github', 'lister_instance_name': 'github', 'pack_size_bytes': 34359738368}}
{'args': [], 'kwargs': {'url': 'https://github.com/DSASC/yergaliyev2020', 'lister_name': 'github', 'lister_instance_name': 'github', 'pack_size_bytes': 34359738368}}

changed the description

added 2h of time spent

mentioned in commit swh/infra/ci-cd/swh-charts@546a1960

mentioned in issue #4873 (closed)

closed

added 2h of time spent

mentioned in issue #4871

sentry 5823: fill-in the large-repository hole

Designs

Child items ...

Activity