guix: Request to archive Bioconductor sources

changed milestone to %Extend archive coverage [Roadmap - Collect]

added activity::MRO label

changed title from Request to archive Bioconductor sources to guix: Request to archive Bioconductor sources

changed the description

I did not find the time to process this issue earlier this week.

Could you schedule archival and let us know how it goes?

Yes, we can.

I kinda expected a remote json manifest like the lister consumes though.

Is that possible on your end?

Thanks in advance.

Cheers,

@zimoun, @samplet: for some reason completion doesn't find you but I'm glad you're here. :-)

@samplet, could you share an updated JSON file as @ardumont asks?

@samplet I have some scripts for generating some sources.json. However, it appears to me easier if you could generate using the PoG database the sources.json containing all Bioconductor origins. Let me know.

@ardumont, I'm not sure I understand "remote JSON manifest". Do you mean one that has the original URLs as well as the content-addressed ones? Or do you mean a JSON file hosted on the Web? I'm happy to help, just confused!

If possible, please load this sources file. It contains all of the non-VCS Guix sources that I know of that are missing from SWH. It also has the original URLs. There are some false positives (sources I think are missing that are actually there), but I know the loader will ignore those. There are around 6K sources, including those from Bioconductor. I realize this is slightly beyond the scope of this issue, but we discussed loading these elsewhere, and I figured this would be the easiest way to solve both problems.

If not, let me know and I will prepare a Bioconductor version.

@ardumont, I'm not sure I understand "remote JSON manifest". ...Or do you mean a JSON file hosted on the Web?

^ json file hosted on the web (which i don't have to compute myself too ;)

If possible, please load this sources file.

yes, on my todo list for the week (i'll try to do it soon). On monday morning, it's meeting time so asap. As mentioned earlier, i'll do it on staging first so you can have a look too (prior to actually trigger in production).

Cheers,

As mentioned earlier, i'll do it on staging first so you can have a look too (prior to actually trigger in production).

Listing triggered.

And ingestion ongoing. I've bumped a bit the directory ones so it goes faster.

An issue has been identified and a fix is ongoing (both staging & prod) [1] I'll trigger the fill-in-the-hole in production once the fix is landed and deployed (probably next week) [and fully ingested on staging first].

[1] #5230 (closed)

The fix has been deployed. Listed origins which failed the ingestion (most occurrences being because of the issue I mentioned) have been rescheduled on staging btw.

mentioned in commit swh/infra/ci-cd/swh-charts@44698510

Current ingestion of the fill-in-the-hole dataset [1] and the standard ingestion [2] is mostly done on staging.

@samplet Can you please check on your side whether the coverage is going up on your side as well?

[1]

Forge ngyro.com (nixguix) has 5801 scheduled ingests in the scheduler.
failed       : 248
None         : 2
not_found    : 462
successful   : 5089
total        : 5801
success rate : 87.7262540941217%

[2]

Forge guix.gnu.org (nixguix) has 33369 scheduled ingests in the scheduler.
failed       : 862
None         : 5394
not_found    : 56
successful   : 27057
total        : 33369
success rate : 81.08423986334621%

I pointed my script at the staging instance, and it checked 5923 missing SWHIDs and found 2255 of them. This is an amazing improvement (), but it's not quite what your numbers suggest.

Some of the still "missing" sources are in the staging archive, just not in the way they usually are. Usually a tarball is extracted into a directory (as if running tar -C DIR ...) and then that directory is stored as a node in the SWH graph. The few "missing" tarballs I investigated have the single subdirectory of the tarball in the graph, but not the parent directory. Take jq-1.7 for example:

wget 'https://github.com/jqlang/jq/releases/download/jq-1.7/jq-1.7.tar.gz'
mkdir out
tar -C out -xf jq-1.7.tar.gz
guix hash -S git -H sha1 -f hex out
# => 0244a41d27f273afe243f07eae359f88891a5051
guix hash -S git -H sha1 -f hex out/jq-1.7
# => 1fcdad1847e7c69d395f6a74ddbf010388fdead7

The first ID is missing, but the second is there.

It's weird because some (about half guessing by the numbers) work the usual way. An example is Lilypond 2.19.80, which has a parent directory node. Here are the origin pages: jq-1.7; lilypond-2.19.80. The snapshots for both point to the subdirectories. Personally, I like the consistency of having the top-level directory even if everyone is too polite to pack multiple directories into a tarball. Whatever the design goal is, the fact that some are one way and some are the other is probably worth some investigation!

As always, thank you @ardumont for considering and sorting through these issues. Getting 5K new Guix-referenced sources into SWH will be a huge improvement.

Hello,

I'm not sure how to reply ;) And what to do next... So i'll just reply along the way.

The fact that the staging instance is a append-only test archive (with a messy dataset inside, since it never got emptied since its start) does not help the analysis... (or the will to go and dig...). Fortunately, we have some tasks(/epic) to work on to try and have scratchable instances for, among other things, improved analysis of ingestion stack... Hopefully, we'll start that work soon.

I pointed my script at the staging instance, and it checked 5923 missing SWHIDs and found 2255 of them.

What's the script you are using?

What kind of swhids we are discussing here?

For that last part, I gather it's directory (given the discussion below), but better ask ;)

This is an amazing improvement (),

yes!

but it's not quite what your numbers suggest.

Those numbers are heuristics about the scheduler/ingestion, unbeknownst of what the swhid (or/and ExtIDs) are about. This discrepancy though sounds like something is off somewhere.

Personally, I like the consistency of having the top-level directory even if everyone is too polite to pack multiple directories into a tarball. Whatever the design goal is, the fact that some are one way and some are the other is probably worth some investigation!

We do like consistency as well ;)

It seems the code [1] is currently not providing the top-level directory holding the extracted archive but directly the contents of the tarball. And when it's one top-level directory, it starts the ingestion from there.

Now, how come some have a different structure? That must be other package loaders which can archive the same tarballs but differently, which does not sound too good.

It seems that's the issue ^ [2]. So we may need to adapt to be consistent with what the other package loaders are ingesting tarballs [2] They seem to just pass along directly the uncompressed path (without doing some extra check like [1]).

[1] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/core/loader.py?ref_type=heads#L1141-1151

[2] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

As always, thank you @ardumont for considering and sorting through these issues. Getting 5K new Guix-referenced sources into SWH will be a huge improvement.

sure

@anlambert What do think about the issue and adaptation on the tarball directory loader to ingest as package loader do?

It seems that's the issue ^ [2]. So we may need to adapt to be consistent with what the other package loaders are ingesting tarballs [2] They seem to just pass along directly the uncompressed path (without doing some extra check like [1]).

Tentatively pushed [1], there may need to bump the extid version too.

[1] swh/devel/swh-loader-core!516 (merged)

Although, we do have the nixpkgs dataset to contend with too...

I recall the development started with theirs, hence that strange if (see [1] from previous comment). Because nix does not use the top-level directory to compute the tarball nar hash...

Because nix does not use the top-level directory to compute the tarball hashes...

Indeed, see example for that tarball where nix computes the NAR hash of its content:

{
      "outputHash": "18pn3dqyk9hp82gva92c6y3vk52s366rrx74rdnvahswdr5dmq4d",
      "outputHashAlgo": "sha256",
      "outputHashMode": "recursive",
      "type": "url",
      "urls": [
        "https://github.com/DFgraphics/phoebus/archive/47.05.tar.gz"
      ],
      "integrity": "sha256-jeDaSm5cQ7Vty+T0nI0ZWpS5hzdMJLWfQBem6XEb9qI=",
      "inferredFetcher": "fetchzip"
},

We can see that the top level directory is ignored when computing the NAR hash.

$ curl -LO https://github.com/DFgraphics/phoebus/archive/47.05.tar.gz
$ mkdir phoebus
$ tar -xvzf 47.05.tar.gz
$ mv Phoebus-47.05 phoebus/
$ cd phoebus
$ nix-hash --base32 --type sha256 .
0kdw8yvcvrvqq2a2n9215zwm5cp4pc3xldaryfndaw6jc6vwg368
$ nix-hash --base32 --type sha256 Phoebus-47.05/
18pn3dqyk9hp82gva92c6y3vk52s366rrx74rdnvahswdr5dmq4d

Where does this sources.json snippet come from?

On Guix side, we do not extract but compute outputHashMode: flat

from https://nix-community.github.io/nixpkgs-swh/sources-unstable-full.json

Hum, I guess it is generated from this repository: https://github.com/nix-community/nixpkgs-swh.git

Well, I do not speak Nix language. :-) Hum, I am missing something:

  urls = map (drv: {
    url = head (drv.urls or [ drv.url ]);
    outputHash = drv.outputHash;
    outputHashAlgo = drv.outputHashAlgo;
    name = drv.name;
    outputHashMode = drv.outputHashMode;
    postFetch = drv.postFetch or "";
  }) fetchurlDependencies;

https://github.com/nix-community/nixpkgs-swh/blob/master/scripts/find-tarballs.nix#L17-L24

It should not come from that snippet but from something else.

This one comes from fetchFromGitHub. Nix has a way of downloading an archive and extracting it as a fixed-output derivation (which means the hash is the nar hash of the output directory). It looks like they use fetchFromGitHub as an optimization: it will download the generated archive file and unpack it rather than cloning the repo. In Guix, we had trouble with those archives being unstable, but maybe if you hash the contents it doesn't matter. They also have the underlying fetchzip, which sometimes gets used for stable tarball/zip releases. I don't know if they have a policy for when to use fetchzip vs. fetchurl.

Well, I do not speak Nix language. :-) Hum, I am missing something:

(HTH) I think the nixpkgs-swh code is only about reading the nixpkgs repositories and extracting derivation information into a json manifest. They don't recompute anything.

This one comes from fetchFromGitHub. Nix has a way of downloading an archive and extracting it as a fixed-output derivation (which means the hash is the nar hash of the output directory). It looks like they use fetchFromGitHub as an optimization: it will download the generated archive file and unpack it rather than cloning the repo. In Guix, we had trouble with those archives being unstable, but maybe if you hash the contents it doesn't matter. They also have the underlying fetchzip, which sometimes gets used for stable tarball/zip releases. I don't know if they have a policy for when to use fetchzip vs. fetchurl.

[3]

Implementation wise, for tarballs, the code was already consuming a path generator of one filepath, the unique top-level directory if any or the top-level uncompressed archive path otherwise. Hence, probably, the source of the discrepancy for some tarballs @samplet mentioned. In the current mr referenced [1] (not yet deployed), that generator now provides both the top-level uncompressed tarball path first and the unique top-level directory within it if any.

The remaining part of the generation consumption stays the same, it's checking the nar checksums on the path up until the nar checksums match. If nothing match, that fails. Otherwise, it goes on and ingest the tarball. So both Guix (alway top-level uncompressed archive path) and Nixpkgs (top-level directory of the tarballs if any) datasets should be ingested appropriately.

The identified impact is that we have to bump the extid_version (already done in mr [1]). Meaning, we'll have to reingest tarballs. It should be mostly noop (we already have the source code), but for the visit snapshot (swhid) & extid mapping. That way, it should become complete, relevant and consistent at least for the guix tarball dataset (& another consistency pattern for nixpkgs tarball datasets... [3]). That way you should be able to find the 2.5k missing swhid mentioned.

On your side, you'll have to adapt the extid query api call to pass along an extra '?extid_version=1' [1] [2] in your calls to find the right mapping.

[1] swh/devel/swh-loader-core!516 (merged)

[2] It's currently not set, hence defaulting to the version 0

[3] We cannot do anything about this unless asking nix to change their ways... which does not sound like a sensible inquiry... We might as well ask them (and you ;) to use swhid directly

The mr got merged [1]

After running a bit on staging (deployment implementation detail), we can see the extid new version coming [2].

So i've merged the new version in staging [3]. This will run during the week end, we should see the hole subside.

[1] swh/devel/swh-loader-core!516 (merged)

[2] https://webapp.staging.swh.network/api/1/extid/target/swh:1:dir:06ba116ce16643f6cb0bf5399c4a3a4ccc51a000/

[3] swh/infra/ci-cd/swh-charts!347 (merged)

mentioned in commit ardumont/swh-loader-core@dc29da66

mentioned in merge request swh/devel/swh-loader-core!516 (merged)

mentioned in commit ardumont/swh-loader-core@00cc2dd8

mentioned in commit ardumont/swh-loader-core@f09df8b6

mentioned in commit ardumont/swh-loader-core@de890c8d

mentioned in commit ardumont/swh-loader-core@590db4b1

mentioned in commit ardumont/swh-loader-core@96109eb2

mentioned in commit ardumont/swh-loader-core@01eb6cb4

mentioned in commit ardumont/swh-loader-core@46b837cc

mentioned in commit ardumont/swh-loader-core@cc2b441a

mentioned in commit ardumont/swh-loader-core@c9b51f8b

mentioned in merge request swh/infra/ci-cd/swh-charts!347 (merged)

@olasd my answers below

17:18:19 Let me try to rephrase, because I still don't get why an extid version bump is needed... Let's suppose tarball tarball.tar.gz is referenced by both the guix and nix sources.json. Extracting tarball.tar.gz yields a single directory a/, containing a bunch of files. The swhid of the a subdirectory is S_a; The swhid of the toplevel directory is S_top (with a single entry a pointing to the swhid S_a).

ok

The snapshot that we would generate for the tarball.tar.gz should always have had a single branch that points to S_top (and a HEAD alias). Is that correct?

no, before, with <= v5.16.1, that depended on the manifest & tarball structure [1]. Snapshot would have targeted the directory that matched the checksums [2] (for nar or flat checksums). I realized I probably answered you wrong this morning (orally)...

[2] https://gitlab.softwareheritage.org/swh/devel/swh-loader-core/-/blob/master/swh/loader/package/loader.py?ref_type=heads#L829-837

[1] swh/infra/ci-cd/swh-charts!347 (merged)

If I understand correctly, guix's nar computation for tarball.tar.gz (let's call the value nar_guix) is a hash of the toplevel directory (which has a single 'a' subdirectory); whereas nix's nar computation (let's call it nar_nix) skips the toplevel and uses directly the contents of the 'a' subdirectory. Is that correct?

yes.

Which nar extids did we store? I assume we would have tried to store (nar_nix, S_top) and (nar_guix, S_top), is that right?

No, the old version loader (still in prod) ingested the directory matching the checksums (same goes for the extid mapping computed). So a combination of one of [(nar_nix, S_top), (nar_nix, S_a)] and one of [(nar_guix, S_top), (nar_guix, S_a)] depending on the structure of the tarball too (because of the conditional [2] in there pulled by nixpkgs' dataset iirc the train of thoughts at the time)

Which nar extids are we storing now? My guess would be (nar_nix, S_a) and (nar_guix, S_top)?

yes, that's it.

---- to which @anlambert responded more succintely & exhaustively. It's definitely worth quoting ;p

17:28:37 <anlambert> olasd: the tarball hashes referenced in guix sources.json
manifest are flat ones (checksums of tarballs bytes) not recursive ones (NAR
hashes of contained source trees) so the checksum verification took place
after we downloaded a tarball and succeeded but unfortunately the tarball
loader that was deployed skip the top level directory so we ended up having
extid of type checksum-sha256 targetting a wrong directory SWHID
17:31:14 <anlambert> if we want to avoid bumping the extid version, we must
remove the wrong checksum-sha256 extid mapping from the storage and reload the
concerned origins but I guess this is not something we want to do
17:33:31 <olasd> ah ! OK, that makes a ton of sense

changed the description

Scheduling the fill-in-the-hole dataset in production [1] [2]

[1]

swh scheduler -C $SWH_CONFIG_FILENAME task add list-nixguix \
  url='https://ngyro.com/pog-reports/latest/sources.json' instance='guix-backlog-20240220' origin_upstream='https://git.savannah.gnu.org/cgit/guix.git/'
Created 1 tasks

Task 416351618
  Next run: today (2024-02-20T16:42:21.972212+00:00)
  Interval: 1 day, 0:00:00
  Type: list-nixguix
  Policy: recurring
  Args:
  Keyword args:
    instance: 'guix-backlog-20240220'
    origin_upstream: 'https://git.savannah.gnu.org/cgit/guix.git/'
    url: 'https://ngyro.com/pog-reports/latest/sources.json'

[2]

2024-02-20 17:29:49 softwareheritage-scheduler@albertina:5432 λ select * from listers where name='nixguix';
+-[ RECORD 1 ]--+--------------------------------------+
| id            | eadece7c-bf18-42d7-891b-6c39582a3376 |
| name          | nixguix                              |
| instance_name | guix-backlog-20240220                |
| created       | 2024-02-20 16:42:26.814699+00        |
| current_state | {}                                   |
| updated       | 2024-02-20 16:42:26.814699+00        |
+-[ RECORD 2 ]--+--------------------------------------+
| id            | e4683926-804f-43ca-8774-a21e4656de56 |
| name          | nixguix                              |
| instance_name | guix.gnu.org                         |
| created       | 2024-01-24 15:21:36.788728+00        |
| current_state | {}                                   |
| updated       | 2024-01-24 15:21:36.788728+00        |
+-[ RECORD 3 ]--+--------------------------------------+
| id            | 6637842f-6da1-4af6-ba7f-5861f297f5a6 |
| name          | nixguix                              |
| instance_name | nix-community.github.io              |
| created       | 2024-01-24 16:17:39.87412+00         |
| current_state | {}                                   |
| updated       | 2024-01-24 16:17:39.87412+00         |
+---------------+--------------------------------------+

marked the checklist item production: Schedule fill-in-the-hole dataset as completed

production: ^ Listing

Done [1], now the ingestion will take place [2]

[2]

swh@swh-toolbox-665d47b677-lmvhn:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins nixguix guix-backlog-20240220

Forge guix-backlog-20240220 (nixguix) has 5769 scheduled ingests in the scheduler.
failed      : 54
None        : 5544
not_found   : 1
successful  : 170
total       : 5769
success rate: 2.95%

[1]

listers {"asctime": "2024-02-20 16:45:32,599", "threadName": "MainThread", "pathname": "/opt/swh/.local/lib/python3.10/
site-packages/celery/app/trace.py", "lineno": 131, "funcName": "info", "task_name": null, "task_id": null, "name": "cel
ery.app.trace", "levelname": "INFO", "message": "Task swh.lister.nixguix.tasks.NixGuixListerTask[bfc5f857-8573-4a60-81e
3-a5a8d1f8179a] succeeded in 186.1982779763639s: {'pages': 6016, 'origins': 5801}", "data": {"id": "bfc5f857-8573-4a60-
81e3-a5a8d1f8179a", "name": "swh.lister.nixguix.tasks.NixGuixListerTask", "return_value": "{'pages': 6016, 'origins': 5
801}", "runtime": 186.1982779763639, "args": "()", "kwargs": "{'url': 'https://ngyro.com/pog-reports/latest/sources.jso
n', 'instance': 'guix-backlog-20240220', 'origin_upstream': 'https://git.savannah.gnu.org/cgit/guix.git/'}"}}

changed the description

fwiw, i've been monitoring the process and it's steadily growing up [1]

From early today to now, it's keeping up and should keep up with the old extid_version (0). As mentioned earlier in the discussion, extid_version 0 represents the old mapping (with the issue on tarball ingested at the wrong level in regards to how guix expects the directory representation).

Data with extid_version 1 is the new version (with the correct directory representation).

We'll see how to deal with [2] (very few hg* origins, in regards to those mappings).

[2] #5256 (closed)

[1]

select extid_type, extid_version, count(*) from extid where extid_type in ('nar-sha256', 'checksum-sha256') group by extid_type, extid_version order by extid_type, extid_version;
+-----------------+---------------+-------+
|   extid_type    | extid_version | count |
+-----------------+---------------+-------+
| checksum-sha256 |             0 | 32034 |
| checksum-sha256 |             1 |  1243 |
| nar-sha256      |             0 | 17112 |
| nar-sha256      |             1 |   203 |
+-----------------+---------------+-------+
(4 rows)

Time: 110.344 ms

Tue 20 Feb 2024 06:05:58 PM CET (every 300s)

+-----------------+---------------+-------+
|   extid_type    | extid_version | count |
+-----------------+---------------+-------+
| checksum-sha256 |             0 | 32034 |
| checksum-sha256 |             1 | 27943 |
| nar-sha256      |             0 | 17122 |
| nar-sha256      |             1 |  7648 |
+-----------------+---------------+-------+
(4 rows)

Time: 1086.312 ms (00:01.086)

Note: I expect those numbers to converge and then exceed the version 0 since I deployed the 'guix-backlog-20240220' dataset too.

changed the description

Status update:

Wed 21 Feb 2024 09:52:07 AM CET (every 300s)

+-----------------+---------------+-------+
|   extid_type    | extid_version | count |
+-----------------+---------------+-------+
| checksum-sha256 |             0 | 32034 |
| checksum-sha256 |             1 | 36108 |
| nar-sha256      |             0 | 17128 |
| nar-sha256      |             1 |  9645 |
+-----------------+---------------+-------+
(4 rows)

Time: 735.832 ms

mentioned in commit anlambert/swh-web@41eb0679

mentioned in merge request swh/devel/swh-web!1245 (merged)

mentioned in commit anlambert/swh-web@42a73204

Status update:

2024-02-23 10:43:50 softwareheritage@albertina:5432 λ select extid_type, extid_version, target_type, count(*) from extid where extid_type in ('nar-sha256', 'checksum-sha256') group by extid_type, extid_version, target_type order by extid_type, extid_version, target_type;
+-----------------+---------------+-------------+-------+
|   extid_type    | extid_version | target_type | count |
+-----------------+---------------+-------------+-------+
| checksum-sha256 |             0 | content     |   502 |
| checksum-sha256 |             0 | directory   | 31532 |
| checksum-sha256 |             1 | content     |   527 |
| checksum-sha256 |             1 | directory   | 36144 |
| nar-sha256      |             0 | directory   | 17151 |
| nar-sha256      |             1 | directory   | 16484 |
+-----------------+---------------+-------------+-------+
(6 rows)

Time: 2662.260 ms (00:02.662)

changed the description

The scheduling is done. The ingestion is done too. There is a slight discrepancy in numbers.

We have 16905 nar-sha256 (version 1) where we had 17151 (version 0), so 246. That could be explained by the 250 failed visits mentioned.

Other types ([checksums|nar]-sha256) are exceeding their previous versions which was expected [2].

[1]

swh@swh-toolbox-55f7fcf794-m7rwf:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins nixguix guix-backlog-20240220

Forge guix-backlog-20240220 (nixguix) has 5801 scheduled ingests in the scheduler.
failed      : 250
None        : 0
not_found   : 464
successful  : 5087
total       : 5801
success rate: 87.69%

2024-02-27 10:06:53 softwareheritage@albertina:5432 λ select extid_type, extid_version, target_type, count(*) from extid where extid_type in ('nar-sha256', 'checksum-sha256') group by extid_type, extid_version, target_type order by extid_type, extid_version, target_type;
+-----------------+---------------+-------------+-------+
|   extid_type    | extid_version | target_type | count |
+-----------------+---------------+-------------+-------+
| checksum-sha256 |             0 | content     |   502 |
| checksum-sha256 |             0 | directory   | 31532 |
| checksum-sha256 |             1 | content     |   528 |
| checksum-sha256 |             1 | directory   | 36525 |
| nar-sha256      |             0 | directory   | 17151 |
| nar-sha256      |             1 | directory   | 16905 |
+-----------------+---------------+-------------+-------+
(6 rows)

Time: 2041.637 ms (00:02.042)

[2] #5222 (comment 165773)

I checked my collection of Guix SWHIDs against the staging instance, and it found 5406 (out of 5923 that were missing as of a month ago). That's a much better result! (I assume the extra 319 found their way into to the archive by other means over the last month.) As far as I can tell the new loader is working well. The new coverage numbers for Guix are very exciting! I'm looking forward to these landing in the main archive.

guix: Request to archive Bioconductor sources

Designs

Child items ...

Activity