reth_stages::stages

Function fetch

Source 
pub async fn fetch(
    filename: &str,
    target_dir: &Path,
    url: &str,
    concurrent: u64,
    file_hash: Option<B256>,
) -> Result<(), DownloaderError>
Expand description

Downloads file from url to data file path.

If a file_hash is passed, it will verify it at the end.

§Details

  1. A Metadata file is created or opened in {target_dir}/download/{filename}.metadata. It tracks the download progress including total file size, downloaded bytes, chunk sizes, and ranges that still need downloading. Allows for resumability.
  2. The target file is preallocated with the total size of the file in {target_dir}/download/{filename}.
  3. Multiple workers are spawned for downloading of specific chunks of the file.
  4. Orchestrator manages workers, distributes chunk ranges, and ensures the download progresses efficiently by dynamically assigning tasks to workers as they become available.
  5. Once the file is downloaded:
    • If file_hash is Some, verifies its blake3 hash.
    • Deletes the metadata file
    • Moves downloaded file to target directory.