Ideas/notes about building a reduced provenance index:

• filters related to revision date:

  • no commit date <= (EPOCH + epsilon)
  • no commit date <= (VCS birth date): eg. no git commit anterior to git birth date, etc. but then it makes provenance ignore a posteriori rebuilt history ("burn baby burn")
  • no commit date in the future (but these are not an issue in terms of provenance, just probably useless content)

• select ingested revisions:

  • releases
  • VCS branch heads; pruning noise ones (MRs, review, CI related ones)

• use some a priori knowledge to white/black list revisions(/origins?):

  • prune content for which we know there are a lot of references (typically a license file), as identified by the "most famous content" datasets

Note that for the dates filtering:

• filter commit date <= EPOCH is debatable (same argument than for VCS birth date based filtering)
• the main issue is identify and to prune buggy/rogue revisions; buggy revisions are mostly in the [EPOCH, EPOCH+1day] range (typically a badly configured vcs conversion tool); rogue ones are harder to identify
• a possible solution could also involve whitelisting revisions which commit date is <EPOCH from a list of known origins (eg. rebuilt git history of famous vintage code base, like Apollo)

For the record, in the current complete provenance index (2.3B ingested revisions so far), around EPOCH we have:

So filtering from 1970-01-03 seems good enough; 1970-01-31 is suspicious; more digging might be useful.

Query: with toto as ( select sha1, date_trunc('day', date) as date from revision where date<'1970-02-01') select date, count(sha1) from toto group by date;

Here is more complete version of the query above: revs-by-date

See swh/devel/swh-provenance#4706