According to the SWHID specification, it is not forbidden for a qualifier
value to contain a '=' character (for instance in origin URL).

So update parsing code to handle that special case.

Some ValidationError exceptions could not be serialized to string due
to these format errors.

Related to T3234

this object aims at being able to keep in the SWH Archive an
  SWHID <-> External object ID
map, e.g. to be able to keep track of Mercurial ids so the Mercurial
loader can be made more efficient.

Related to T2849.

The rounding algorithm wasn't specified

Serializing as ISO8601 makes the hash brittle, because the database may
change the timezone silently and/or lose precision in the microseconds.

As we do not need precise timestamp, using an integer is good enough,
and is consistant with the git format.

The manifest also does not need to contain a timezone, as it only
represents the timezone of the system that fetched this metadata,
which is useless data.

This will be used to compute an intrisic identifier for RawExtrinsicMetadata;
which can be used for deduplication and refering to it like any other sha1_git
instead of needed to use a tuple of its fields.

Other packages don't use it anymore.

This is a useful shorthand when generating SWHIDs in tests.

* Quote/unquote path
* Fix line parsing and serializing to properly handle None
* Fix error raised by check_visit/check_anchor

by making them all derive from an abstract class.

Following the discussion on T3034, we decided to replace SWHID with
two or three classes:

* QualifiedSWHID to replace the existing SWHID (standard types + qualifiers)
* CoreSWHID, for "core SWHID" only (standard types + no qualifiers)
* ExtendedSWHID for internal use in Software Heritage (extra types + no qualifiers)

This commit adds the last one.

It also removes "ori" as a valid object type for CoreSWHID and
QualifiedSWHID, as it now only belongs in ExtendedSWHID.

It is cleaner, avoids warnings, and will be needed when introducing
ExtendedSWHID in a future commit.

And store their parsed values (CoreSWHID, tuple of ints, etc.) instead of string.

Following the discussion on T3034, we decided to replace SWHID with
two or three classes:

* QualifiedSWHID to replace the existing SWHID (standard types + qualifiers)
* CoreSWHID, for "core SWHID" only (standard types + no qualifiers)
* ExtendedSWHID for internal use in Software Heritage (extra types + no qualifiers)

This commit adds the second one

Following the discussion on T3034, we decided to replace SWHID with
two or three classes:

* QualifiedSWHID to replace the existing SWHID (standard types + qualifiers)
* CoreSWHID, for "core SWHID" only (standard types + no qualifiers)
* ExtendedSWHID for internal use in Software Heritage (extra types + no qualifiers)

Since migrating from SWHID will break existing code, this commit uses
the opportunity to modernize it a little, ie.:

* `keyword`-only constructor, to get rid of the hacky default values for
  `object_type` and `object_id`
* enum instead of strings for the object type
* `bytes` instead of an hex string for the object id
* rename `metadata` to `qualifiers`

Before this change there was a lot of overlap between parse_swhid() and the
attrs-based validators in the SWHID class. Also, the validation implementation
in parse_swhid() was done by hand.

With this change the coarse-grained validation done by parse_swhid() is now
delegated to a regex. The semantic validation of SWHIDs is left to attrs
validators. The regex is also exposed as a module attribute, to be used by
client code that want to syntactically validate SWHIDs without necessarily
instantiate SWHID classes (we have several other modules doing that already,
and they are using slightly different hand-made regexs, which isn't great).

As part of this change we also clean up the use of ValidationError exceptions,
systematically passing the problematic parts of SWHID as arguments, and uniform
error messages.

This change also brings some speed up in SWHID parsing. On a benchmark parsing
~30 M valid SWHIDs, the previous implementation took ~3:06 minutes, the new one
~2:50 minutes, or a ~9% speedup.

Closes T2788

Related to T2729

So parse_swhid raises a ValidationError when that is detected.

Related to T2769

Related to T2769

Related to T2769

This collapses the shared logic between these two identifier computations into a
few more explicit steps:
 - generate data for the manifest (in either identifier computation);
 - format the manifest (in the new format_manifest function);
 - hash the manifest (in the new hash_manifest function).

This will enable reusing this logic for more object types, as well as stronger
typing for the manifest computation.

Use the new SWHID naming convention instead of SWH PID.

Related to T2610

As NamedTuple inherits from tuple, msgpack serializes it like a tuple,
which makes it indistinguishable from a tuple when deserializing,
which is an issue for the RPC API.

Add a new extra_headers attribute on Revision and use it for computing
the revision's id instead of extract it from the metadata field.

Only accept (bytes, bytes) as extra_header.

Add a post init hook to Revision to initialize this new attribute from
given metadata, if any, for bw compat.

Also amend the revision_d hyptothesis strategy to generate extra_headers.

When Software Heritage persistent identifiers were introduced, they were not
yet abbreviated as SWHIDs.

Now that abbreviation is growing adoption, rename some functions and types in
swh.model.identifiers for consistency:

  - PersistentId -> SWHID

  - persistent_identifier -> swhid

  - parse_persistent_identifier -> parse_swhid

Backward compatibility with previous naming is maintained but deprecation
warnings are introduced to encourage the use of the new names.

Numerous variables in swh.model codebase have also been renamed accordingly.

Also rework and improve documentation.

thus in TimestampWithTimezone.from_dict(). This is needed to help consuming
existing (invalid) messages from kafka.

Warning: tests added in this revision do not cover the whole
normalize_timestamp() function.

- blackify all the python files,
- enable black in pre-commit,
- add a black tox environment.

It should be cheap enough to do it, and it makes tests easier.

Daniele Serafini authored 5 years ago and Stefano Zacchiroli committed 5 years ago

... from test_persistent_identifier.

Closes T1986