Skip to content
Snippets Groups Projects

gitea, gogs: Ensure query parameters are not duplicated in API URLs

Merged Antoine Lambert requested to merge anlambert/swh-lister:gogs-lister-url-fix into master
1 unresolved thread

Gitea API return next pagination link with all query parameters provided to an API request.

As we were also passing a dict of fixed query parameters to the page_request method, some query parameters ended up having multiple instances in the URL for fetching a new page of repositories data. So each time a new page was requested, new instances of these parameters were appended to the URL which could result in a really long URL if the number of pages to retrieve is high and make the request fail.

Also remove a debug log already present in http_request method.

Fixes SWH-LISTER-AY.

Edited by Antoine Lambert

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • added 1 commit

    • 323e2774 - gitea, gogs: Ensure query parameters are not duplicated in API URLs

    Compare with previous version

  • Antoine Lambert changed the description

    changed the description

  • Jenkins job DLS/gitlab-builds #361 succeeded .
    See Console Output and Coverage Report for more details.

  • Jenkins job DLS/gitlab-builds #362 succeeded .
    See Console Output and Coverage Report for more details.

    • Are there really query params that aren't in the next_link and that we need to carry over to the next request? I would expect the remote to already repeat the relevant query parameters from the incoming request.

    • Gitea does it right but not Gogs unfortunately, see below.

      Gitea instance:

      $ curl -i "https://codeberg.org/api/v1/repos/search?limit=1"
      HTTP/2 200 
      access-control-expose-headers: Link, X-Total-Count
      cache-control: max-age=0, private, must-revalidate, no-transform
      content-type: application/json;charset=utf-8
      link: <https://codeberg.org/api/v1/repos/search?limit=1&page=2>; rel="next",<https://codeberg.org/api/v1/repos/search?limit=1&page=93772>; rel="last"
      vary: Origin
      x-total-count: 93772
      date: Wed, 05 Jun 2024 14:04:51 GMT
      strict-transport-security: max-age=63072000; includeSubDomains; preload
      permissions-policy: interest-cohort=()
      x-frame-options: sameorigin
      x-content-type-options: nosniff
      
      {"ok":true,"data":[{"id":42396,"owner":{"id":46225,"login":"Zombie","login_name":"","full_name":"","email":"zombie@noreply.codeberg.org","avatar_url":"https://codeberg.org/avatars/fb9a1252c7aef70b39c19659791347be","language":"","is_admin":false,"last_login":"0001-01-01T00:00:00Z","created":"2022-04-20T00:36:45Z","restricted":false,"active":false,"prohibit_login":false,"location":"","pronouns":"","website":"","description":"","visibility":"public","followers_count":0,"following_count":0,"starred_repos_count":0,"username":"Zombie"},"name":"-","full_name":"Zombie/-","description":"","empty":false,"private":false,"fork":false,"template":false,"parent":null,"mirror":false,"size":1074,"language":"HTML","languages_url":"https://codeberg.org/api/v1/repos/Zombie/-/languages","html_url":"https://codeberg.org/Zombie/-","url":"https://codeberg.org/api/v1/repos/Zombie/-","link":"","ssh_url":"git@codeberg.org:Zombie/-.git","clone_url":"https://codeberg.org/Zombie/-.git","original_url":"","website":"","stars_count":0,"forks_count":0,"watchers_count":1,"open_issues_count":0,"open_pr_counter":0,"release_counter":0,"default_branch":"pages","archived":false,"created_at":"2022-04-21T00:54:34Z","updated_at":"2023-02-17T18:32:32Z","archived_at":"1970-01-01T00:00:00Z","permissions":{"admin":false,"push":false,"pull":true},"has_issues":true,"internal_tracker":{"enable_time_tracker":true,"allow_only_contributors_to_track_time":true,"enable_issue_dependencies":true},"has_wiki":true,"wiki_branch":"master","has_pull_requests":true,"has_projects":true,"has_releases":true,"has_packages":false,"has_actions":false,"ignore_whitespace_conflicts":false,"allow_merge_commits":true,"allow_rebase":true,"allow_rebase_explicit":true,"allow_squash_merge":true,"allow_fast_forward_only_merge":false,"allow_rebase_update":true,"default_delete_branch_after_merge":false,"default_merge_style":"merge","default_allow_maintainer_edit":false,"avatar_url":"","internal":false,"mirror_interval":"","object_format_name":"sha1","mirror_updated":"0001-01-01T00:00:00Z","repo_transfer":null}]}

      Gogs instance:

      $ curl -i "https://git.loquico.me/api/v1/repos/search?q=_&limit=1"
      HTTP/2 200 
      strict-transport-security: max-age=63072000; includeSubdomains; preload
      referrer-policy: strict-origin-when-cross-origin
      content-type: application/json; charset=UTF-8
      link: <https://git.loquico.me/api/v1/repos/search?page=2>; rel="next",<https://git.loquico.me/api/v1/repos/search?page=16>; rel="last"
      x-content-type-options: nosniff
      x-frame-options: sameorigin
      date: Wed, 05 Jun 2024 14:05:51 GMT
      content-length: 789
      set-cookie: lang=en-US; Path=/; Max-Age=2147483647
      set-cookie: i_like_gogs=523e4bdbe1edde23; Path=/; HttpOnly
      set-cookie: _csrf=YyC1z2IyOJEi7CFQVxn7D517aqY6MTcxNzU5NjM1MTI5NTM1MDc3MA; Path=/; Domain=git.loquico.me; Expires=Thu, 06 Jun 2024 14:05:51 GMT; HttpOnly; Secure
      server: Loquicom Systems
      
      {"data":[{"id":1,"owner":{"id":1,"username":"Loquicom","login":"Loquicom","full_name":"","email":"abrandao@loquico.me","avatar_url":"https://secure.gravatar.com/avatar/ba13d1ff5cabfde6e085dc53e9beab1e?d=identicon"},"name":"jdr-roll","full_name":"Loquicom/jdr-roll","description":"Lanceur de dés virtuels avec salon pour des JDR","private":false,"fork":false,"parent":null,"empty":false,"mirror":false,"size":397312,"html_url":"https://git.loquico.me/Loquicom/jdr-roll","ssh_url":"ssh://git@git.loquico.me:8228/Loquicom/jdr-roll.git","clone_url":"https://git.loquico.me/Loquicom/jdr-roll.git","website":"","stars_count":0,"forks_count":0,"watchers_count":1,"open_issues_count":0,"default_branch":"master","created_at":"2022-02-19T15:28:05Z","updated_at":"2024-01-06T20:17:27Z"}],"ok":true}
    • Please register or sign in to reply
  • Nicolas Dandrimont approved this merge request

    approved this merge request

  • Antoine Lambert merged manually

    merged manually

Please register or sign in to reply
Loading