1 min read

[linux] Scrape entire website with a single line command

[linux] Scrape entire website with a single line command

I've tried many online tools in order to scrape an entire website but none of them allow me to freely download all pages.

Next i found the webhttrack tool for Windows/Ubuntu/Mac, it works well but needs installation.

The final tested solution is a single line command...


➜  ~ wget \                       
     --recursive \
     --no-clobber \
     --page-requisites \
     --html-extension \
     --convert-links \
     --restrict-file-names=windows \
     --domains target.com \
     --no-parent \
         https://target.com/shop

Tweets by YBacciarini