Running metadata tests with multiple hyothesis examples uncovered those
issues in api-1-raw-extrinsic-metadata-swhid Web API view:

  - ExtendedSWHID.from_string must be used to parse extended SWHID.

  - link-next URL for pagination was invalid.

  - next_page_token must be encoded before providing it to urlsafe_b64encode.

When running swh-web tests using 'make test-full', multiple hypothesis examples
are provided as test inputs instead of a single one when running 'make test'.
In that case some tests were failing mostly due to the fact they were not stateless
between test runs.

That commit fixes those tests execution and ensures stateless test runs by:

  - turning some hypothesis strategies into stateless ones

  - turning the archive_data fixture into a function scope one

  - using subtest fixture from pytest-subtesthack when it is required to reset
    the archive_data fixture for a test between hypothesis example runs

As a consequence, tests will be longer to execute as global state will be reseted
between each test.

Nevertheless, metadata related tests are still failing when running with multiple
hypothesis examples, fix will be handled in next commit.

Related to T1695

Contents with mimetype starting with application/ but with
no detected programming language were not processed for
display but they can be regular text files.

Closes T3486

Useful for the dev environments (a connection to the vpn is needed)

This diff introduces a new config parameter "enable_ql"

If it's set to yes/True, the swh-search query language can be used for the archive search

While using the /origin/search API endpoint, "use_ql=true" can be
passed to use the url_pattern as the search query language input

api_raw_extrinsic_metadata_swhid: Make the docstring consistent with api_raw_extrinsic_metadata_swhid_authorities

Reword some sentences and add step showing SWHID with lines qualifer.

Fix redirection at end of tour when started from homepage.

Related to T3437

D5982 did not fix the flaky cypress test.

Removing cy.intercept hack required some times ago seems
to do the trick.

It will enable to exploit those data in the archive coverage widget
implementation to get deposit counts by origins type.

Related to T3127

Footer is put in cache by Varnish so an old version will be displayed
while the layout template is not updated.

So only display the version in development mode.

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

Add a Web UI guided tour implemented using the intro.js library.

The tour showcases the main browsing features offered by Software
Heritage web application by highlighting the relevant DOM elements
and displaying help texts in tooltips.

The tour can span multiple web pages and steps are configured in
a YAML file for help texts writing commodity.

Currently, the homepage, the main origin browsing view and content
browsing view are covered by the tour.

Related to T3202

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

This new major version of highlight.js brings notable improvements in
terms of language parsing.

As the github theme we were using has been updated to match the current
github.com one, we now use a custom theme adapted from the old github
one to keep the same look and feel for code highlighting as before.

Closes T3368

The visit is already recursive, we don't need to add this explicit
recursion.

Children nodes can be dispatched to visit_field_name and
visit_field_body instead.

Motivation:

This commit started as a simple change: I wanted to replace:

```
`<type> <IRI>`
```

with:

```
``<type> <IRI>``
```

Unfortunately, this syntax looks too much like XML for its own good,
so it was stripped by the `process_paragraph` method, because it reads
the docutils pseudo-XML representation and strips every tag it doesn't
know about.
(I'm saying pseudo-XML, because my poor `<type> <IRI>` string was not
escaped with XML entities, so it was in fact undistinguishable from
actual XML tags).

Changes:

Therefore, stops using the XML-like string representation of docutils
trees, and visits tree nodes directly instead.
Conveniently, this is already in a node visit, so we can reuse that;
simply by iterating recursively instead of stopping the recursion
as soon as we see a known node (ie. the visitors actually visited
only nodes very close to the root).

This means that we needed to add methods to handle each node type,
and produce its ReST output. And since we don't have a global view
anymore, we need to return the produced ReST instead of appending
directly to `self.data["description"]`, because handlers of parent
nodes may need to re-indent their children's output.o

This results in cleaner code (and also closer to what we expect from
a visitor transformer), so it's a win too.

This has some other nice side-effects:

* our custom role code is now neatly restricted in `visit_problematic`,
  so it can't overflow, because docutils runs `visit_problematic` with
  *only* the role's string as child
* it detects unexpected nodes, such as the `title_reference` roles,
  which is usually produced when accidentally using single-backquotes
  instead of double-backquotes to wrap inline code blocks (it happens
  a lot when one is used to markdown)

When clicking on the "Go to origin" icon, open a new browser tab to
keep Software Heritage one.

Cypress commands should not be invoked in a loop as it is an anti-pattern
that can produce flaky tests.

The issue was spotted by a test that start failing since
rDSEA2e1fb863871c82c5561b6a1cbd94c7064a1a392d got landed.

Hypothesis origin strategy in swh-model has been recently updated
to generate IRIs instead of just URIs.

It allows easier discovery for humans, as clients need to know the authority
type/url to list metadata on a given SWHID.

Create a missing new line at end of file icon by combining two Material Design
ones instead of reusing a SVG copy of the old GitHub one.

This removes a copyright infringement while having a better looking icon.

Ensure to declare a variable with const when it is not reassigned.

This change enables the possibility to search with release names inside
the browse origin releases page.A search textbox is added, which is
connected to a JS function and that reloads the page with the
release_name_include query param.
Filtering with the name is happening at the storage level.

Related to T3404

This code enables the possibility to search for branch names inside
the browse origin branches page.A newly added search input is connected
to a JS function and that reloads the page on press of return.
Filtering with name is happening at the storage level.

Related to T3157

So metadata files are always downloaded as attachment (when accessed from a browser),
and has a filename that contains the SWHID of the metadata target.

This maximizes the use of the origin visit status to determine the save code now request
final status (succeeded or failed). When such data is not yet available, the task is
probably not yet scheduled, scheduled or currently running. So, it fallbacks to use the
scheduler task/task_run to deduce such non-final statuses (e.g scheduled/running/not yet
scheduled/...).

Computing the save code now status that way will decrease the potential discrepancy [1]
between the status displayed and the origin not being browsable yet.

[1] Most likely due to some replication lag

Related to T3378

When a save code now request is submitted for a github repository,
ensure to use the canonical repository URL by getting it from the
github Web API.

The purpose is to avoid adding duplicated github origins in the
archive.

Related to T1110
Related to T2187

To allow users to request only once their save code now origins, once the first
ingestion is successfully ingested, we also mark it as recurrent origin to crawl.

Implementation wise, the scheduling routine in charge of updating the save code now
statuses reported in the save code now ui is in charge of this.

Related to T1524

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

It enables to obtain a more readable code with less indentation levels.

When the user searches with a valid, archived URL, it makes more sense
to redirect directly to the browse page rather than showing search
results.Two generic functions added are 'isValidURL' and
'isArchivedOrigin'. Both returns boolean.

Some HTTP hosted tarballs have been archived by the Internet Archive.
In that case to check URL validity and get tarball metadata, HEAD requests
must follow redirection and info regarding last modified date or content
length must be retrieved from different HTTP response headers.

Related to T3365