Blogs

How does Pelican Commerce data crawling work?

Migrating websites can be quite challenging since each site has its fingerprint of data structure, which is why Pelican Commerce takes a two-step approach to capture the most complete dataset possible to use in our website migration service. Here’s what we do.

Our Data Capturing Methods

  • We first export a necessary file from your website admin into a spreadsheet format like .csv, .xls, or .xlsx.
  • We then crawl and capture your front end website product and metadata into a structured format.
  • We then combine these two datasets to create the most robust, rich dataset we can capture to later use for your migration.

Behind each web page is a page full of code and meta tags. Each tag for different types of data is mapped for the crawler to seek when scraping the data from your site.

pelican commerce crawling process example meta tags

Common Types Of Crawled Data

  • Product Title
  • Pricing
  • SKU
  • UPC Code or GTIN Code.
  • Features (Bullet Points).
  • Rich Description (in plain text and HTML format).
  • Image Asset URLs (The original file gets transferred to Shopify’s CDN when migrating).
  • Additional product images (Multi-Product Shots).
  • Embedded image URLs (Images found in the description area).
  • Embedded link URLs (links added to the description area).
  • Title and Description meta data (For SEO retention).
  • Canonical URL (For 301-Redirects).
  • Product Variation label and option names, prices, and SKU.
  • Other custom data types specific to the project that need to be captured.

How Much Time Does It Take?

Each project is completely unique and is mapped out at the time your migration begins with Pelican Commerce. We first start by mapping each of the primary page types for all the useful product data and metadata available. Then we start the crawler and begin capturing that data. Each site is a bit different because some servers take longer than others to resolve, and some websites have more code behind each page to crawl through. So time can vary, so be patient while this automated process captures your data.

In most cases, it takes 5-7 business days.

Once the data is captured, we then combine that data with the exported data and start to format and massage into the Shopify format and begin the upload and validation process.

If you have any questions or need your website data crawled, then reach out to Pelican Commerce anytime. We offer a full-service website migration service to Shopify and our primary focus is making sure your data is moved properly and you retain your existing Search Engine Rank (SEO Rank).

Handcrafted Migrations

Our service is a bit different than other automated systems. Our handcrafted migration solutions are designed to be handled with care, for well established companies, and to retain existing search engine rank (SEO).

Take Your Business To The Next Level

All Clients Receive Support 7-Days A Week
Ongoing Support
Project Management Portal
Scheduled Phone Calls
Quick Turnaround for Requests
Free Link QA Crawling
Strategic Site Launches
Contact Us

Book A Free 30-Min Discovery Call

Speak to an experienced project manager and take your business to the next level!
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
arrow-leftarrow-rightcheckmark-circle linkedin facebook pinterest youtube rss twitter instagram facebook-blank rss-blank linkedin-blank pinterest youtube twitter instagram