Pierre-Yves David authored 10 months ago and Pierre-Yves David committed 10 months ago

git_objects.snapshot_git_object wants Snapshot argument, lets move
things to Snapshot object.

Pierre-Yves David authored 10 months ago and Pierre-Yves David committed 10 months ago

object_dicts still need to be migrated.

Pierre-Yves David authored 10 months ago and Pierre-Yves David committed 10 months ago

And automatically generate it as we are at it.

Pierre-Yves David authored 10 months ago and Pierre-Yves David committed 10 months ago

We also take this as an opportunity to use the blacklist_types feature
in that test.

Pierre-Yves David authored 10 months ago and Pierre-Yves David committed 10 months ago

We also typed the function, which turned out more painful than
anticipated.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

This test work with cpython because as we do not assign the open file to
a variable, the reference counting garbage collect it right after the
`write` call. Deleting the file object mean the write is flushed.

On an interpreter without reference counting, as pypy, the file object might be
garbaged collected too late, breaking the test.

So we give the file object a clear life cycle.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

As for the previous commit, using deprecated actually change the source
Class leading all usage to raise deprecation warning. So we have to
remove the deprecated and keep the compatibility silently for a small
while.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

It turns out that calling @deprecated on Content alter the class, and
any instantion of "Content" will be wrongly marked as deprecated… So use
a different approach too preserve compatibility (and we won't keep it
long).

It do works, but it is now discouraged.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

TargetType is far too generic and introduces confusion with
ReleaseTargetType. So we rename it to a clearer name.

Pierre-Yves David authored 10 months ago and vlorentz committed 10 months ago

ObjectType is far too generic and introduces confusion with
swhids.ObjectType. So we rename it to a clearer name.

A Content object with no attached data or get_data function could
no longer be converted to a dict as MissingData exception was raised.

vlorentz authored 10 months ago and Antoine Lambert committed 10 months ago

Having the escaped URL in `swhid.origin` is inconsistent with self.path
(which is always escaped) and never what we want, because it is only
useful while serializing, which is already handled by `__str__`.

This led to swh/devel/swh-indexer#4738
where swh-deposit parsed a qualified SWHID, then used `.origin` to get
an origin URL.

Additionally, as serialization always escapes the `origin` qualifier,
this means that deserializing then re-serializing a qualified SWHID
would double-escape it.

Finally, fixing this made the test uncover that `%` was not escaped
while serializing, while `;` was, leading to incorrect (and ambiguous)
escaped URLs.

There are two other package using DiskBackedContent "swh-loader-svn" and
"swh-loader-cvs". Both use it to check "DiskBackedContent.object_type"
at the same time as "model.Content.object_type".

so we do this small hack to avoid breaking these other module until
they migrate.

This sets the pieces in place to finally cleanup the confusion from the
various object_type attributes. They now have different type, so we
should be able to start detecting error at some point.

As for FromDiskType, we keep compatibility with string value for now.
This avoid breaking existing code.

Instead of having multiple class and `object_type` value, we just adds
a few lines in the main `model.Content` class to retrieved data on
demand. The `with_data` logic already existed there anyway.

This will avoid having from_disk extending the model from the outside.

This is part of a wider effort to differentiate the various type of
"object_type" attribute around the model code base.

This requires cleaning various item along the ways. Which is probably an
added benefit. Especially, mypy now consider BaseModel to hold a
object_type attribute.

Check inline comment for details.

Currently the only limit is "enforced" by PostgreSQL.

This makes sure that origins created after we switch to Cassandra as the
primary storage remain compatible with a PostgreSQL-based storage.

Add an optional progress callback to `from_disk` method. It can
returns the number of computed entries for each top entries traversed.
This is useful for CLI, in particular to display progress information
for SWH Scanner.

Related to swh/meta#5075.

Right now, the discovery process offered by `filter_known_objects`
returns all results after the discovery is complete. The new callback
provides a way to get information "in real time" which is useful for at
least a couple of planned use case in the SWH scanner:
- displaying progress information while processing
- update a graphical UI in real time.

This simple callback fits this need without too much troubles.

For some reason, mypy complained about the existing type hint in this
file for unclear reason. So I fixed them.

The web Client is no longer using async so we no longer needs it.

When building package documentation outside tox by calling make in the
docs directory, the include of Makefile.sphinx inside the docs Makefile
was failing as its relative path was invalid.

So adapt this relative path according if the SWH_PACKAGE_DOC_TOX_BUILD
environment variable is set or not.