swh-model merge requestshttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests2024-03-26T12:43:14Zhttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/237Add 'evolve' method to BaseModel objects2024-03-26T12:43:14ZvlorentzAdd 'evolve' method to BaseModel objectsThis allows implicitly updating the id when altering data,
which is very useful in tests of other packages
Depends on !236.
## Test Plan
~~This is missing a bunch of tests; I will write them if you are fine with this idea.~~
----
*Mi...This allows implicitly updating the id when altering data,
which is very useful in tests of other packages
Depends on !236.
## Test Plan
~~This is missing a bunch of tests; I will write them if you are fine with this idea.~~
----
*Migrated from D8151 ([view on Phabricator](https://forge.softwareheritage.org/D8151))*vlorentzvlorentzhttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/344from_disk: Add optional information callback2024-03-04T10:47:17ZFranck Bretfrom_disk: Add optional information callbackAdd an optional callback to `from_disk` method. It basically returns
`local_scanning` as context with the number of computed entries for
top entries traversed.
This is useful for CLI, in particular to display progress information
for SWH...Add an optional callback to `from_disk` method. It basically returns
`local_scanning` as context with the number of computed entries for
top entries traversed.
This is useful for CLI, in particular to display progress information
for SWH Scanner.https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/336model: Add payload to ExtID class2024-02-22T13:33:18ZTimothy Samplemodel: Add payload to ExtID classThis is a follow up to !267, which I cannot update due to the Phabricator migration.
The changes are almost exactly the same. I clarified in the docs that an ExtID does not store the payload but rather refers to it by its `sha1-git` ha...This is a follow up to !267, which I cannot update due to the Phabricator migration.
The changes are almost exactly the same. I clarified in the docs that an ExtID does not store the payload but rather refers to it by its `sha1-git` hash.
part of https://gitlab.softwareheritage.org/swh/devel/swh-model/-/issues/2430https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/338drop async and allow for "real-time" callback when doing discovery2024-01-16T15:14:48ZPierre-Yves Daviddrop async and allow for "real-time" callback when doing discoveryThis update the API to remove now unnecessary async and add an optionnal callback to unlock some swh scanner feature.This update the API to remove now unnecessary async and add an optionnal callback to unlock some swh scanner feature.https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/343Blackify and fix a few type annotations2023-12-04T17:52:51ZDavid DouardBlackify and fix a few type annotationshttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/342Apply swh-py-template 0.1.62023-12-03T19:16:58ZDavid DouardApply swh-py-template 0.1.6https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/340Migrate to copier-based template2023-11-28T20:37:03ZDavid DouardMigrate to copier-based templatehttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/339docs: include the README file in the main index page2023-11-20T13:20:26ZDavid Douarddocs: include the README file in the main index pageConvert README from markdown to ReST to make it embeddable in
docs/index.rst
Requires https://gitlab.softwareheritage.org/swh/devel/swh-docs/-/merge_requests/391Convert README from markdown to ReST to make it embeddable in
docs/index.rst
Requires https://gitlab.softwareheritage.org/swh/devel/swh-docs/-/merge_requests/391https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/236Rename HashableObject to BaseHashableModel and make it a subclass of BaseModel2023-11-15T09:14:21ZvlorentzRename HashableObject to BaseHashableModel and make it a subclass of BaseModelinstead of a mix-in class.
A future commit will add a method implemented by both with different
signatures that mypy cannot unify yet.
----
*Migrated from D8150 ([view on Phabricator](https://forge.softwareheritage.org/D8150))*instead of a mix-in class.
A future commit will add a method implemented by both with different
signatures that mypy cannot unify yet.
----
*Migrated from D8150 ([view on Phabricator](https://forge.softwareheritage.org/D8150))*vlorentzvlorentzhttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/337Add a way of also filtering files in `Directory`2023-11-14T17:41:37ZRaphaël GomèsAdd a way of also filtering files in `Directory`Is a dependency of https://gitlab.softwareheritage.org/swh/devel/swh-scanner/-/merge_requests/88Is a dependency of https://gitlab.softwareheritage.org/swh/devel/swh-scanner/-/merge_requests/88https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/333Move most of the discovery module from `swh-loader-core` here2023-11-07T14:35:31ZRaphaël GomèsMove most of the discovery module from `swh-loader-core` hereThe initial implementation of the discovery algorithm was incorrectly
done in `swh-loader-core`, all the generic code should live here, since
it uses `swh-model` types.The initial implementation of the discovery algorithm was incorrectly
done in `swh-loader-core`, all the generic code should live here, since
it uses `swh-model` types.https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/335hypothesis_strategies: Fix mypy error2023-09-26T09:35:34ZAntoine Lamberthypothesis_strategies: Fix mypy errorUse a list instead of a tuple to keep mypy happy with latest
hypothesis version.Use a list instead of a tuple to keep mypy happy with latest
hypothesis version.https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/334model: Implement repr for parents field of Revision model2023-08-29T08:38:06ZAntoine Lambertmodel: Implement repr for parents field of Revision modelAs with other fields containing sha1_git values, display hexadecimal
representation of parent revision ids.As with other fields containing sha1_git values, display hexadecimal
representation of parent revision ids.https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/332Bundle of tox.ini changes2023-06-14T15:06:54ZNicolas Dandrimontolasd@softwareheritage.orgBundle of tox.ini changesTo prepare for running tests on multiple python versions (swh/meta#5040), tweak the tox.ini syntax:
- fix a line continuation weirdness in tox.ini parsing when no factors are set
- make the default "py3" testenv properly parametric
- ...To prepare for running tests on multiple python versions (swh/meta#5040), tweak the tox.ini syntax:
- fix a line continuation weirdness in tox.ini parsing when no factors are set
- make the default "py3" testenv properly parametric
- use minversion=4 to save ourselves from reinstalling tox all the timeDesign and test next generation CI Automation [Roadmap - Tooling and infrastructure]Nicolas Dandrimontolasd@softwareheritage.orgNicolas Dandrimontolasd@softwareheritage.orghttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/330Add several helper methods returning SWHIDs2023-04-05T12:22:24ZJérémy Bobbio (Lunar)Add several helper methods returning SWHIDsTo remove objects from the archive, we need to traverse the archive from one object to the ones they reference. Having helper methods that returns SWHID directly makes it easier to record which objects have been seen or match data provid...To remove objects from the archive, we need to traverse the archive from one object to the ones they reference. Having helper methods that returns SWHID directly makes it easier to record which objects have been seen or match data provided by `swh.graph`.
These methods makes a first step for #3034.Tooling for takedown notices [Roadmap - Collect]Jérémy Bobbio (Lunar)Jérémy Bobbio (Lunar)https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/328mypy: Bump to 1.0 and fix new typing errors2023-02-17T14:33:28ZAntoine Lambertmypy: Bump to 1.0 and fix new typing errorsLet's start mypy bump with `swh-model` as a typing issue related to `ImmutableDict`
propagates in other swh modules.
Related to swh/meta#4960Let's start mypy bump with `swh-model` as a typing issue related to `ImmutableDict`
propagates in other swh modules.
Related to swh/meta#4960https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/326Use tox 4 command line syntax2023-02-16T16:36:00ZJérémy Bobbio (Lunar)Use tox 4 command line syntaxSee: https://tox.wiki/en/latest/upgrading.html#updating-usage-with-eSee: https://tox.wiki/en/latest/upgrading.html#updating-usage-with-ehttps://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/325Update and clean tox configuration for version 42023-02-16T15:12:07ZJérémy Bobbio (Lunar)Update and clean tox configuration for version 4Related to swh/meta#4959Related to swh/meta#4959https://gitlab.softwareheritage.org/swh/devel/swh-model/-/merge_requests/324collections: Add index to ImmutableDict to speedup look up by key2023-02-16T12:52:12ZAntoine Lambertcollections: Add index to ImmutableDict to speedup look up by keyPreviously when looking up data by key in an ImmutableDict, the inner
tuple storing keys and values was iterated until finding the requested
key.
This is not really efficient when the ImmutableDict contains a lot of
entries, typically f...Previously when looking up data by key in an ImmutableDict, the inner
tuple storing keys and values was iterated until finding the requested
key.
This is not really efficient when the ImmutableDict contains a lot of
entries, typically for an origin snapshot containing a lot of branches.
So add an index to speedup look up by key operations and improve loader
performances.
Before these changes, this is the timing we obtain when performing a new
visit of the v8 repository (for the record I am using a [homemade storage proxy](https://gitlab.softwareheritage.org/anlambert/uploads/-/snippets/1551)
that reads from production storage and writes in a memory storage):
```
14:04 $ time swh loader -C ~/.config/swh/loader.yml run git https://github.com/v8/v8
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/v8/v8' with type 'git'
Enumerating objects: 106, done.
Counting objects: 100% (106/106), done.
Compressing objects: 100% (96/96), done.
Total 106 (delta 48), reused 56 (delta 8), pack-reused 0
INFO:swh.loader.git.loader:Listed 30888 refs for repo https://github.com/v8/v8
INFO:swh.loader.git.loader.GitLoader:Fetched 107 objects; 107 are new
{'status': 'eventful'} for origin 'https://github.com/v8/v8'
real 0m59,440s
user 0m33,682s
sys 0m0,847s
```
And this is the timing we obtain after applying these changes:
```
14:07 $ time swh loader -C ~/.config/swh/loader.yml run git https://github.com/v8/v8
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://github.com/v8/v8' with type 'git'
Enumerating objects: 106, done.
Counting objects: 100% (106/106), done.
Compressing objects: 100% (96/96), done.
Total 106 (delta 48), reused 56 (delta 8), pack-reused 0
INFO:swh.loader.git.loader:Listed 30888 refs for repo https://github.com/v8/v8
INFO:swh.loader.git.loader.GitLoader:Fetched 107 objects; 107 are new
{'status': 'eventful'} for origin 'https://github.com/v8/v8'
real 0m28,338s
user 0m3,613s
sys 0m0,945s
```
So a x2 speedup !