Commits · 300d534c78b8dfc3fda05c44e20b39519b049f20 · Platform / Development / snippets

Apr 26, 2018
- seirl: dedup: send sql inserts in batch · 300d534c
  Antoine Pietri authored 7 years ago
  
  300d534c
Apr 13, 2018
- morane/crossminer_launch: Make the script use starting range · 88eca5dd
  Antoine R. Dumont authored 7 years ago
  
  88eca5dd
- morane/crossminer_launch: Fix origin_scan function to use range · a546af39
  Antoine R. Dumont authored 7 years ago
  
  + add docstring
  a546af39
- morane/crossminer_launch: Fix pep8 violations · 7fbe9a1d
  Antoine R. Dumont authored 7 years ago
  
  7fbe9a1d
- Add crossminer_launch snippet · 777ab0c0
  Morane Otilia Gruenpeter authored 7 years ago
  
  777ab0c0
Apr 06, 2018
- seirl: swh-dedup: working prototype with preseeded methods · 9e7eb6aa
  Antoine Pietri authored 7 years ago
  
  9e7eb6aa
- seirl: swh-dedup: working prototype with preseeded methods · 57c3c638
  Antoine Pietri authored 7 years ago
  
  57c3c638
- seirl: ase-vault-test: update get_all_trees and add fetch_vault · ff7d7338
  Antoine Pietri authored 7 years ago
  
  ff7d7338
Mar 26, 2018
- seirl: ase-vault-test: add send_vault · ee08a2a4
  Antoine Pietri authored 7 years ago
  
  ee08a2a4
- seirl: ase-vault-test: add github queries · 028e25c3
  Antoine Pietri authored 7 years ago
  
  028e25c3
Mar 21, 2018
- Use last 50 characters to group by exception message · d550280d
  Antoine R. Dumont authored 7 years ago
  
  d550280d
- scheduling: Adapt mercurial/svndump/svn arguments setup · f30a6764
  Antoine R. Dumont authored 7 years ago
  
  f30a6764
Mar 14, 2018
- seirl: add swh-dedup · b496be4e
  Antoine Pietri authored 7 years ago
  
  b496be4e
Feb 22, 2018
- add SQL snippets to dump (meaningful parts of) the origin table · d43f2eaf
  Stefano Zacchiroli authored 7 years ago
  
  d43f2eaf
- swh-dedup: fix db name in driver shell script · 3d51335f
  Stefano Zacchiroli authored 7 years ago
  
  3d51335f
Feb 12, 2018
- group_by_exception: Add mercurial loader type · f2dbd149
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  f2dbd149
- kibana_fetch_logs: Add configuration abilities to fetching tool · 1cb7d409
  Antoine R. Dumont authored 7 years ago
  
  Related T682 Related P221
  1cb7d409
Feb 09, 2018
- scheduling: Fix mercurial arguments setup · 26ea29b2
  Antoine R. Dumont authored 7 years ago
  
  Related T329
  26ea29b2
- scheduling: Fix default case scheduling · 4d182dbe
  Antoine R. Dumont authored 7 years ago
  
  4d182dbe
Jan 12, 2018

scheduling: Check queues' state to trigger new task scheduling · 68c7d748

Use case: indexers.  To send new tasks for scheduling to the
orchestrator queue (input queue), we need to check the mimetype and
license indexers queues (not only the mimetype one).

68c7d748

Jan 11, 2018
- create_snapshots: be more efficient · 2c49fd80
  Nicolas Dandrimont authored 7 years ago
  
  2c49fd80
Jan 08, 2018
- swh-dedup-blocks: add sample driver script (run.sh) · 998d88cb
  Stefano Zacchiroli authored 7 years ago
  
  to streamline running experiments with different parameters
  998d88cb
- swh-dedup-blocks: add effective average block size to summary · 9b50858a
  Stefano Zacchiroli authored 7 years ago
  
  9b50858a
- swh-dedup-blocks: fix typo in avg block size configuration · 75e0a2d6
  Stefano Zacchiroli authored 7 years ago
  
  also, updated Rabin fingerprint parameters for better compression rates
  75e0a2d6
Jan 07, 2018
- swh-dedup-blocks: multiprocess implementation · 359fea82
  Stefano Zacchiroli authored 7 years ago
  
  down to 25m for linux.git, from 1h for the single process version, from 14h for the sqlite3 version
  359fea82
- swh-dedup-blocks: use postgres to store deduplication results · b9d742ee
  Stefano Zacchiroli authored 7 years ago
  
  instead of sqlite3 even if we're not using COPY, it is *much* faster
  b9d742ee
- move swh-dedup-blocks.py to dedicated dir · 567ad94a
  Stefano Zacchiroli authored 7 years ago
  
  567ad94a
- swh-dedup-blocks: new script to play with (rabin) deduplication · e26a6cee
  Stefano Zacchiroli authored 7 years ago
  
  e26a6cee
Jan 04, 2018
- sql/anon-dump: fix typo in SQL statement · d6ccd489
  Stefano Zacchiroli authored 7 years ago
  
  d6ccd489
Jan 03, 2018
- sql/anon-dump: fix bytea output format · d180b165
  Stefano Zacchiroli authored 7 years ago
  
  also, add driver shell script
  d180b165
- check-in id-to-path, to have it have a stable place · 10f626fd
  Stefano Zacchiroli authored 7 years ago
  
  10f626fd
- SQL queries (+ script) to produce an email-anonymized dump of the DB · 70bd9cbf
  Stefano Zacchiroli authored 7 years ago
  
  70bd9cbf
Dec 22, 2017
- Add snapshot migration snippet · 0118a624
  Nicolas Dandrimont authored 7 years ago
  
  0118a624
Dec 21, 2017
- Prepare googlecode mercurial origins scheduling · 4b224005
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  4b224005
- Update filtering tool to integrate the nature in the built url · 8bea10f0
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  8bea10f0
Dec 14, 2017

filter_path_clash: Update filtering path clash tool · 5c8391fa

Antoine R. Dumont authored 7 years ago

This will help to filter the origins that should not be googlecode
ones but eclipselabs or apache-extras one

5c8391fa

Dec 13, 2017
- volatile scheduling: Permit batch size override from cli · b85d77ec
  Antoine R. Dumont authored 7 years ago
  
  Related T864
  b85d77ec
- volatile scheduling: Improve state information output · 3ee2a9cb
  Antoine R. Dumont authored 7 years ago
  
  Related T864
  3ee2a9cb
Dec 11, 2017
- sort_by_size: Add size to a given path · 5b8d6e1d
  Antoine R. Dumont authored 7 years ago
  
  5b8d6e1d
- scheduling: Permit to schedule other tasks (indexer) the same way · 2d0923a6
  Antoine R. Dumont authored 7 years ago
  
  Related T864
  2d0923a6