Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

TargetType is far too generic and introduces confusion with
ReleaseTargetType. So we rename it to a clearer name.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

ObjectType is far too generic and introduces confusion with
swhids.ObjectType. So we rename it to a clearer name.

A Content object with no attached data or get_data function could
no longer be converted to a dict as MissingData exception was raised.

vlorentz authored 10 months ago and Antoine Lambert committed 10 months ago

Having the escaped URL in `swhid.origin` is inconsistent with self.path
(which is always escaped) and never what we want, because it is only
useful while serializing, which is already handled by `__str__`.

This led to swh-indexer#4738
where swh-deposit parsed a qualified SWHID, then used `.origin` to get
an origin URL.

Additionally, as serialization always escapes the `origin` qualifier,
this means that deserializing then re-serializing a qualified SWHID
would double-escape it.

Finally, fixing this made the test uncover that `%` was not escaped
while serializing, while `;` was, leading to incorrect (and ambiguous)
escaped URLs.

There are two other package using DiskBackedContent "swh-loader-svn" and
"swh-loader-cvs". Both use it to check "DiskBackedContent.object_type"
at the same time as "model.Content.object_type".

so we do this small hack to avoid breaking these other module until
they migrate.

This sets the pieces in place to finally cleanup the confusion from the
various object_type attributes. They now have different type, so we
should be able to start detecting error at some point.

As for FromDiskType, we keep compatibility with string value for now.
This avoid breaking existing code.

Instead of having multiple class and `object_type` value, we just adds
a few lines in the main `model.Content` class to retrieved data on
demand. The `with_data` logic already existed there anyway.

This will avoid having from_disk extending the model from the outside.

This is part of a wider effort to differentiate the various type of
"object_type" attribute around the model code base.

This requires cleaning various item along the ways. Which is probably an
added benefit. Especially, mypy now consider BaseModel to hold a
object_type attribute.

Check inline comment for details.

Currently the only limit is "enforced" by PostgreSQL.

This makes sure that origins created after we switch to Cassandra as the
primary storage remain compatible with a PostgreSQL-based storage.

Add an optional progress callback to `from_disk` method. It can
returns the number of computed entries for each top entries traversed.
This is useful for CLI, in particular to display progress information
for SWH Scanner.

Related to swh/meta#5075.

Right now, the discovery process offered by `filter_known_objects`
returns all results after the discovery is complete. The new callback
provides a way to get information "in real time" which is useful for at
least a couple of planned use case in the SWH scanner:
- displaying progress information while processing
- update a graphical UI in real time.

This simple callback fits this need without too much troubles.

For some reason, mypy complained about the existing type hint in this
file for unclear reason. So I fixed them.

The web Client is no longer using async so we no longer needs it.

When building package documentation outside tox by calling make in the
docs directory, the include of Makefile.sphinx inside the docs Makefile
was failing as its relative path was invalid.

So adapt this relative path according if the SWH_PACKAGE_DOC_TOX_BUILD
environment variable is set or not.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

instead of a mix-in class.

A future commit will add a method implemented by both with different signatures that mypy cannot unify yet.

`dir_filter` only filters directories. `swh-scanner` needs to
accurately filter out ignored files before making expensive requests
to the web API. We introduce a more general `path_filter` that allows
us to differentiate between files and folders.

`dir_filter` is now deprecated and will be removed once the remaining
users in other packages are migrated over to the new API.

`accept_all_directories` is also deprecated, because it only implies
accepting *directories* even though its behavior also accepts
non-directory entries when used with `path_filter`.

Use a list instead of a tuple to keep mypy happy with latest
hypothesis version.

Antoine Lambert authored 1 year ago and vlorentz committed 1 year ago

As with other fields containing sha1_git values, display hexadecimal
representation of parent revision ids.

This separate package was introduced recently and is needed for our CLIs
to pass type checking.

This allows using the "system" tox, if it's recent enough, instead of
always provisioning an internal .tox venv with tox 4.

Instead of going back to py3, pass through the environment name, so that
it can be called with an arbitrary interpreter version.

When parsing the configuration, tox would complain about the unfollowed
line continuation (which is what happens when the testenv was qualified
with neither full nor minimal).

Moving {posargs} to be unqualified allows the line continuation
character to always have something behind it.

This adds several helper methods returning SWHIDs to model objects,
namely:

- SkippedContent.swhid()
- DirectoryEntry.swhid()
- SnapshotBranch.swhid()
- Release.target_swhid()
- Revision.directory_swhid() and Release.parent_swhids()
- OriginVisitStatus.origin_swhid() and
  OriginVisitStatus.snapshot_swhid()

Better using latest mypy release.

Related to swh/meta#4960