swh-model merge requestshttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests2023-11-07T14:39:32Zhttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/329Add discovery model2023-11-07T14:39:32ZFranck BretAdd discovery modelIt adds primitives for finding the unknown parts of disk contents
efficiently.
It provides ArchiveDiscoveryInterface, BaseDiscoveryGraph and
RandomDirSamplingDiscoveryGraph classes for discovery algorithms.
This is a follow up of comme...It adds primitives for finding the unknown parts of disk contents
efficiently.
It provides ArchiveDiscoveryInterface, BaseDiscoveryGraph and
RandomDirSamplingDiscoveryGraph classes for discovery algorithms.
This is a follow up of comments made in swh-scanner!64 about splitting
some code to common module
Related swh-scanner!64
Related swh-scanner#4591
Also related to https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/269#note_128820https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/267model: Add payload to ExtID class2023-10-31T15:41:03ZTimothy Samplemodel: Add payload to ExtID classThis revision adds payload and payload type fields to
the ExtID class. The intent is to use these fields to store
Disarchive specifications to support recovering source tarballs.
See https://sympa.inria.fr/sympa/arc/swh-devel/2022-02/m...This revision adds payload and payload type fields to
the ExtID class. The intent is to use these fields to store
Disarchive specifications to support recovering source tarballs.
See https://sympa.inria.fr/sympa/arc/swh-devel/2022-02/msg00022.html
where Stefano suggests using a generic payload mechanism for ExtIDs
and https://sympa.inria.fr/sympa/arc/swh-devel/2022-05/msg00027.html
where we decided on using object storage for the payload.
----
*Migrated from D8759 ([view on Phabricator](https://forge.softwareheritage.org/D8759))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/327Add 'discovery' model2023-02-27T13:20:05ZFranck BretAdd 'discovery' modelIt adds primitives for finding the unknown parts of disk contents efficiently.
It provides ArchiveDiscoveryInterface and BaseDiscoveryGraph classes
for discovery algorithms.
This is a follow up of comments made in D8539 about splitting ...It adds primitives for finding the unknown parts of disk contents efficiently.
It provides ArchiveDiscoveryInterface and BaseDiscoveryGraph classes
for discovery algorithms.
This is a follow up of comments made in D8539 about splitting some code
to common module
Related swh/devel/swh-scanner!64
Related swh/devel/swh-scanner#4591
Replaces https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/269https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/269Draft: Add 'discovery' model2023-02-27T12:43:17ZFranck BretDraft: Add 'discovery' modelIt adds primitives for finding the unknown parts of disk contents efficiently.
It provides ArchiveDiscoveryInterface and BaseDiscoveryGraph classes
for discovery algorithms.
This is a follow up of comments made in swh/devel/swh-scanner!...It adds primitives for finding the unknown parts of disk contents efficiently.
It provides ArchiveDiscoveryInterface and BaseDiscoveryGraph classes
for discovery algorithms.
This is a follow up of comments made in swh/devel/swh-scanner!64 about splitting some code
to common module
Related swh/devel/swh-scanner!64
Related swh/devel/swh-scanner#4591
----
*Migrated from D8937 ([view on Phabricator](https://forge.softwareheritage.org/D8937))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/300Add an extid_version field to ExtIDs2023-01-08T18:24:52ZNicolas Dandrimontolasd@softwareheritage.orgAdd an extid_version field to ExtIDsThis allows distinguishing multiple potential versions of the mapping
between external objects and their counterparts archived in Software
Heritage, for instance when a loader has a backwards-incompatible change
that should result in obj...This allows distinguishing multiple potential versions of the mapping
between external objects and their counterparts archived in Software
Heritage, for instance when a loader has a backwards-incompatible change
that should result in objects being loaded again.
The field defaults to zero, in which case it's backwards-compatible with
the previous implementation in terms of identifier computation.
Related to [T3418](https://forge.softwareheritage.org/T3418 'view original for T3418 on Phabricator')
## Test Plan
new objects added to existing tests
----
*Migrated from D6019 ([view on Phabricator](https://forge.softwareheritage.org/D6019))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/296swh-identify: Hide tracebacks if Click or Dulwich is not installed2023-01-08T18:20:40Zvlorentzswh-identify: Hide tracebacks if Click or Dulwich is not installedAnd show nice human-readable errors instead
Resolves [T3226](https://forge.softwareheritage.org/T3226 'view original for T3226 on Phabricator')
----
*Migrated from D5466 ([view on Phabricator](https://forge.softwareheritage.org/D5466))*And show nice human-readable errors instead
Resolves [T3226](https://forge.softwareheritage.org/T3226 'view original for T3226 on Phabricator')
----
*Migrated from D5466 ([view on Phabricator](https://forge.softwareheritage.org/D5466))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/295Remove accidental dependency of 'swh-identify' on swh-core2023-01-08T18:20:31ZvlorentzRemove accidental dependency of 'swh-identify' on swh-coreWas added in !137
Resolves [T3220](https://forge.softwareheritage.org/T3220 'view original for T3220 on Phabricator')
----
*Migrated from D5460 ([view on Phabricator](https://forge.softwareheritage.org/D5460))*Was added in !137
Resolves [T3220](https://forge.softwareheritage.org/T3220 'view original for T3220 on Phabricator')
----
*Migrated from D5460 ([view on Phabricator](https://forge.softwareheritage.org/D5460))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/285SWHID spec: add support for IRI2023-01-08T18:04:06ZStefano ZacchiroliSWHID spec: add support for IRICloses [T2379](https://forge.softwareheritage.org/T2379 'view original for T2379 on Phabricator')
----
*Migrated from D3068 ([view on Phabricator](https://forge.softwareheritage.org/D3068))*Closes [T2379](https://forge.softwareheritage.org/T2379 'view original for T2379 on Phabricator')
----
*Migrated from D3068 ([view on Phabricator](https://forge.softwareheritage.org/D3068))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/276model.OriginVisit: Make obsolete fields optional2023-01-08T18:02:26ZAntoine R. Dumontmodel.OriginVisit: Make obsolete fields optionalIn order to smoothly transition to origin visit without those fields, let's
make them optional first.
Then we'll be able to drop them as !275 suggests without breaking the world.
Related to [T2310](https://forge.softwareheritage.org/T2...In order to smoothly transition to origin visit without those fields, let's
make them optional first.
Then we'll be able to drop them as !275 suggests without breaking the world.
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## Test Plan
tox
----
*Migrated from D3340 ([view on Phabricator](https://forge.softwareheritage.org/D3340))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/275model.OriginVisit: Drop obsolete fields2023-01-08T18:02:21ZAntoine R. Dumontmodel.OriginVisit: Drop obsolete fieldsRelated to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## Test Plan
tox
----
*Migrated from D3337 ([view on Phabricator](https://forge.softwareheritage.org/D3337))*Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## Test Plan
tox
----
*Migrated from D3337 ([view on Phabricator](https://forge.softwareheritage.org/D3337))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/282model: Rename OriginVisitUpdate to OriginVisitStatus2023-01-08T18:01:48ZAntoine R. Dumontmodel: Rename OriginVisitUpdate to OriginVisitStatusThis also adapts the hypothesis strategies, using the plural form
origin_visit_statuses. That plural form is acceptable because in our context,
the statuses are countable.
Related to [T2310](https://forge.softwareheritage.org/T2310 'vie...This also adapts the hypothesis strategies, using the plural form
origin_visit_statuses. That plural form is acceptable because in our context,
the statuses are countable.
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
Related to [1]
To be able to update names in the current swh/devel/swh-storage!919 swh/devel/swh-storage!920 swh/devel/swh-storage!921 (they use origin
visit update).
- [1] https://hebdo.framapad.org/p/9g09-0jjinnno5y?lang=en
## Test Plan
tox
----
*Migrated from D3001 ([view on Phabricator](https://forge.softwareheritage.org/D3001))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/273docs/data-model: Update visits chapter definition2023-01-08T18:01:31ZAntoine R. Dumontdocs/data-model: Update visits chapter definition(Proposal according to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator'))
Hinting at the origin_visit_update model
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for ...(Proposal according to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator'))
Hinting at the origin_visit_update model
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## Test Plan
tox
----
*Migrated from D2941 ([view on Phabricator](https://forge.softwareheritage.org/D2941))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/272model: Add OriginVisitUpdate model object2023-01-08T18:01:31ZAntoine R. Dumontmodel: Add OriginVisitUpdate model object(pairing with @vlorentz)
- Need to be updated according to @douardda's latest improvments on model (if that got merged before ;)
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## ...(pairing with @vlorentz)
- Need to be updated according to @douardda's latest improvments on model (if that got merged before ;)
Related to [T2310](https://forge.softwareheritage.org/T2310 'view original for T2310 on Phabricator')
## Test Plan
tox
----
*Migrated from D2880 ([view on Phabricator](https://forge.softwareheritage.org/D2880))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/34Add origin persistent identifiers.2023-01-08T17:49:51ZvlorentzAdd origin persistent identifiers.
----
*Migrated from D1523 ([view on Phabricator](https://forge.softwareheritage.org/D1523))*
----
*Migrated from D1523 ([view on Phabricator](https://forge.softwareheritage.org/D1523))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/115Allow negative_utc to be None in normalize_timestamp()2023-01-08T16:59:55ZDavid DouardAllow negative_utc to be None in normalize_timestamp()thus in TimestampWithTimezone.from_dict(). This is needed to help consuming
existing (invalid) messages from kafka.
Not sure about the couple of tests added in there... this is far from covering properly this function and may give the ...thus in TimestampWithTimezone.from_dict(). This is needed to help consuming
existing (invalid) messages from kafka.
Not sure about the couple of tests added in there... this is far from covering properly this function and may give the false impression it's properly tested while it's not.
----
*Migrated from D3263 ([view on Phabricator](https://forge.softwareheritage.org/D3263))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/140cli: make SWHIDParamType return SWHID type instead of string2023-01-07T23:31:32ZThibault Allançoncli: make SWHIDParamType return SWHID type instead of string
----
*Migrated from D4139 ([view on Phabricator](https://forge.softwareheritage.org/D4139))*
----
*Migrated from D4139 ([view on Phabricator](https://forge.softwareheritage.org/D4139))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/12model.hashutil: Open new endpoint to allow to hash stream2023-01-07T23:26:54ZAntoine R. Dumontmodel.hashutil: Open new endpoint to allow to hash streamAnd open swh.model.hashutil.hash_stream endpoint
Related swh/devel/swh-loader-core#421
## Test Plan
make test
----
*Migrated from D410 ([view on Phabricator](https://forge.softwareheritage.org/D410))*And open swh.model.hashutil.hash_stream endpoint
Related swh/devel/swh-loader-core#421
## Test Plan
make test
----
*Migrated from D410 ([view on Phabricator](https://forge.softwareheritage.org/D410))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/113Use Tuple instead of List in model declarations2023-01-07T23:19:05ZDavid DouardUse Tuple instead of List in model declarationsUse Tuple instead of List in model declaration.
----
*Migrated from D3177 ([view on Phabricator](https://forge.softwareheritage.org/D3177))*Use Tuple instead of List in model declaration.
----
*Migrated from D3177 ([view on Phabricator](https://forge.softwareheritage.org/D3177))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/112Tag model entities with their "type"2023-01-07T23:12:38ZDavid DouardTag model entities with their "type"this aims at preventing constant usage of isinstance() based dispatch
code when writing generic code handling model entities.
For example, the "object_type" argument of JournalWriter.write_addition() has
become superflous now we only pa...this aims at preventing constant usage of isinstance() based dispatch
code when writing generic code handling model entities.
For example, the "object_type" argument of JournalWriter.write_addition() has
become superflous now we only pass model entities, etc.
This idea comes olasd's reading of mypy doc:
https://mypy.readthedocs.io/en/latest/literal_types.html#tagged-unions
----
*Migrated from D3152 ([view on Phabricator](https://forge.softwareheritage.org/D3152))*https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/279Ignore special file when computing content hashes2023-01-07T23:05:03ZAntoine R. DumontIgnore special file when computing content hashesRef. [T255](https://forge.softwareheritage.org/T255 'view original for T255 on Phabricator')
----
*Migrated from D179 ([view on Phabricator](https://forge.softwareheritage.org/D179))*Ref. [T255](https://forge.softwareheritage.org/T255 'view original for T255 on Phabricator')
----
*Migrated from D179 ([view on Phabricator](https://forge.softwareheritage.org/D179))*