How to crawl external links (on-site) in bulk?

If you're trying to capture all the external URLs like YouTube, links to other websites, or other calls to external sites that are happening from your front end website, then follow these steps to see all external URLs on-site in a spreadsheet.

  1. Open Screaming Frog.
  2. Type in the full URL to the website you want to crawl.
  3. Once the website is crawled 100% then in the Overview tab in the right-hand side pane, under "External" click "HTML" filter.
  4. Then in the results sort the "Address" field to organize the common websites together (to easily read).
  5. Highlight all items or just the websites you want to export to a spreadsheet (click the first one and then Shift-Click to highlight.
  6. Click Export
How to capture youTube video links on site.

You will now have a spreadsheet that shows the "From" page which is the page on your website the External URL is located. Then you will see "To" which is the External URL address that your page is linking/calling out to.

Thank you for visiting Pelican Commerce. We're a full-service E-Commerce agency servicing small business owners around the world. Located in Windermere, Florida USA, we specialize in website migrations to Shopify and bulk product uploading, data crawling, and helping clients get fully set up online.

Want us to take care of it?

Blog Pages
arrow-right linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram