How does Pelican Commerce data crawling work?

Migrating websites can be quite challenging since each site has its fingerprint of data structure, which is why Pelican Commerce takes a two-step approach to capture the most complete dataset possible to use in our website migration service. Here’s what we do.

Our Data Capturing Methods

  • We first export a necessary file from your website admin into a spreadsheet format like .csv, .xls, or .xlsx.
  • We then crawl and capture your front end website product and metadata into a structured format.
  • We then combine these two datasets to create the most robust, rich dataset we can capture to later use for your migration.

Behind each web page is a page full of code and meta tags. Each tag for different types of data is mapped for the crawler to seek when scraping the data from your site.

pelican commerce crawling process example meta tags

Common Types Of Crawled Data

  • Product Title
  • Pricing
  • SKU
  • UPC Code or GTIN Code.
  • Features (Bullet Points).
  • Rich Description (in plain text and HTML format).
  • Image Asset URLs (The original file gets transferred to Shopify’s CDN when migrating).
  • Additional product images (Multi-Product Shots).
  • Embedded image URLs (Images found in the description area).
  • Embedded link URLs (links added to the description area).
  • Title and Description meta data (For SEO retention).
  • Canonical URL (For 301-Redirects).
  • Product Variation label and option names, prices, and SKU.
  • Other custom data types specific to the project that need to be captured.

How Much Time Does It Take?

Each project is completely unique and is mapped out at the time your migration begins with Pelican Commerce. We first start by mapping each of the primary page types for all the useful product data and metadata available. Then we start the crawler and begin capturing that data. Each site is a bit different because some servers take longer than others to resolve, and some websites have more code behind each page to crawl through. So time can vary, so be patient while this automated process captures your data.

In most cases, it takes 5-7 business days.

Once the data is captured, we then combine that data with the exported data and start to format and massage into the Shopify format and begin the upload and validation process.

If you have any questions or need your website data crawled, then reach out to Pelican Commerce anytime. We offer a full-service website migration service to Shopify and our primary focus is making sure your data is moved properly and you retain your existing Search Engine Rank (SEO Rank).

Thank you for visiting Pelican Commerce. We're a full-service E-Commerce agency servicing small business owners around the world. Located in Windermere, Florida USA, we specialize in website migrations to Shopify and bulk product uploading, data crawling, and helping clients get fully set up online.

Want us to take care of it?

Blog Pages
arrow-right linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram