In this blog, we will discuss web scraping with ParseHub combined with cold emailing with Instantly.ai for effective lead generation!

Two lead generation strategies, web scraping and cold emailing, are widely used and effective. However, when combined, they become a powerful duo. Our free web scraping tool, ParseHub, and cold emailing tool, Instantly.ai, produce great results when used together.

ParseHub is a free web scraping tool that you can use even if you don't know how to code. It lets you collect data from websites effortlessly, giving you valuable information, prospect emails and other details to boost your lead generation efforts.

How ParseHub Works

ParseHub is our free web scraping tool that systematically collects data from websites. When using it, you have the option to select certain pieces of information that you want to gather, like contact information, business details, prospective emails, cities, user reviews, or product descriptions.

ParseHub can handle websites of any complexity, including those with interactive elements and sophisticated designs, as well as simple, straightforward sites.

Web Scraping For Lead Generation

How does ParseHub help with lead generation?

To generate more leads, it's essential to understand your target audience including their interests and how your product or service can meet their needs. With ParseHub's web scraping tool, you can collect this data directly from your potential customers and use it to your advantage. Data is crucial for successful lead generation.

If your business serves other businesses (B2B), you can use ParseHub to extract information from business directories or LinkedIn profiles. This can help you identify potential leads that would benefit from your product or service. For those in the business-to-customer (B2C) sector, ParseHub can scrape data from review sites, forums, or social media platforms, giving you insights into consumer profiles, behaviour and preferences. By analyzing this data, you can adjust your product or service to meet the needs of your potential customers and attract more leads.

Introducing Cold Emailing with Instantly.ai

After using ParseHub to collect the required data, you can use Instantly.ai to contact potential leads easily. Instantly.ai is an effective tool for cold emailing that streamlines the process of connecting with potential leads.

How Instantly.ai Works

With Instantly.ai, you can easily create customized cold emails optimized for conversions. Its advanced features include automated follow-ups, real-time analytics, and integration with CRM. Additionally, you can add unlimited email accounts, ensuring your campaigns are sent from multiple emails, reducing your chances of going into spam.

Check out Instantly.ai and its resources for more details!

Cold Email For Lead Generation

To generate leads, cold emailing can be highly effective as it lets you directly approach potential customers. However, personalization is crucial to cold emailing success. It's essential to ensure that the email is customized to the individual's needs. This is where the information acquired through ParseHub can be useful.

By utilizing the gathered data, you have the ability to customize every email according to the receiver's preferences, requirements, and actions. To illustrate, if you are promoting business software, you can emphasize how it can address the particular issues of the company you are communicating with.

With Instantly.ai, you can ensure that your emails are both timely and relevant. It has automated follow-up features that prevent leads from slipping through the cracks. Also, it offers real-time analytics that provides insights into the effectiveness of your emails. You can use these insights to refine your email strategy and increase your success rate.

Web Scraping & Cold Emailing for Success

With ParseHub and Instantly.ai, you can generate leads effortlessly. First, ParseHub extracts the necessary data about potential leads, which is then used by Instantly.ai to create customized cold emails. This collaboration enables you to reach out to your prospects in a precise, tailored, and successful manner.

In the end, using ParseHub for web scraping and Instantly.ai for cold emailing are effective ways to improve your lead generation. By using the right tools and strategies, you can create a scalable and efficient lead generation process. In the business world, leads are essential for success, and with these tools at your disposal, you can increase your chances of achieving that success.

Here are some guides on websites you may scrape for lead generation:

• How to Scrape Freelancers on Freelancer.com
• How to Scrape Capterra Reviews and Data
• Job Board Web Scraping – How To Scrape Workopolis
• How to Scrape Yelp Data: Business Information, Ratings, Reviews and more…
• How to Scrape Redfin Listings

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

In this blog, we will show you our scraped laptop prices from Amazon and Best Buy, so you can make an informed decision before purchasing a new laptop! We will analyze pricing, availability, and customer options to provide insights for those searching for a laptop online. Although there are many e-commerce sites available, Amazon and Best Buy are popular options to consider when shopping for laptops, in addition to buying directly from manufacturers.

We will also show you how you can use ParseHub for price monitoring and comparisons.

Web Scraped Data

Here is our scraped data, powered by ParseHub, which we used to scrape laptop details from Amazon and Best Buy:

Click here to view the full file on Google Sheets.

Our Findings

When using the web scraping tool, ParseHub, to investigate laptops available on Amazon and Best Buy. Here are the main findings:

Mac Computers Take the Crown: When it comes to luxury laptops, Mac computers reign supreme. Across both Amazon and Best Buy, Macs consistently claimed the title of the most expensive laptops. With their sleek designs and cutting-edge features, Macs offer a premium experience for those who prioritize style and performance.

Wallet-Friendly Wonders Await Amazon customers: For budget-conscious shoppers, Amazon emerges as a treasure trove of affordable laptops. Prices on Amazon generally lean towards the lower end, thanks to competitive sellers and frequent discounts. If you’re looking to strike a balance between cost and quality, Amazon might just be the ideal destination for your laptop hunt.

Best Buy's Open Box Option: A Hidden Gem: Best Buy has a secret weapon for deal-hunters—the open box option. This feature allows customers to purchase new products not in their original packaging for many reasons, often sold at discounted prices. It is an intelligent choice for those seeking quality laptops while keeping a close eye on their budget.

Dell dominates the Availability Game: Among the plethora of laptop brands, Dell stands out as the most readily available on both Amazon and Best Buy. With a wide range of Dell laptops, online shoppers can enjoy a diverse selection that caters to their specific needs. Dell's popularity is a testament to its reliability and appeal in the laptop market.

Concluding Thoughts

As online platforms continue to revolutionize how we shop, understanding the nuances of different platforms becomes crucial. Our analysis of Amazon and Best Buy’s laptop offerings has shed light on key aspects such as pricing, availability, and customer options.

Whether you aspire for high-end luxury, seek budget-friendly deals, or desire a reliable brand, there’s a laptop waiting for you in the vast online market. With this knowledge, embark on your laptop shopping journey confidently and find the perfect device to suit your needs and preferences.

Here are some other studies done with ParseHub:

• We Scraped Tim Hortons Locations – Canada vs. UAE
• Web Scraping Car Prices to Track Inflation
• We Scraped Black Friday Deals - Are They Worth Waiting a Year?
• Microsoft Teams or Cisco Webex? – 2020 vs. 2022 Stats
• Rising Cost of Groceries in Canada: Web Scraping Reveals Inflation

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

In this blog, you will learn how to scrape Belk.com products, prices and reviews with ParseHub, our free web scraper!

Belk, Inc. was founded in 1888 in North Carolina, first named New York Racket, then Belk Brothers, and finally Belk. The company has over 300 locations spanning 16 different states, over 17,000 employees and has generated over 3.3 billion in revenue in 2022. The Belk department stores and website primarily sell women’s, men’s and kid’s clothing, shoes, handbags, jewelry, beauty, bed and bath products, and much more!

Let’s begin scraping Belk.com!

Step 1: Scraping Products

1. Firstly, run the ParseHub application and log in.
2. Click the New Project button.
3. Enter the Belk.com URL you would like to scrape, we will be scraping men’s shoes: https://www.belk.com/men/shoes/
4. When the page loads, click the first product’s name, the rest should turn yellow.
5. Click the next product’s name to train the algorithm; you may need to repeat this on missing products.
6. Rename this selection on the left to “product”.

Step 2: Scraping Relative Data

We will use ParseHub’s Relative Select tool to scrape additional details from each product:

1. Begin by clicking the PLUS(+) button next to your product selection.
2. Choose “Relative Select”.
3. Click the first product’s name, then its price.
4. Do this for the next product as well to train the algorithm; you may need to do this for other products, where the price is not extracted.
5. Rename this selection on the left to “price”.
6. You can do these steps again for other data such as the reviews.

Step 3: Pagination

1. Scroll all the way down until you see the page navigation button.
2. Click the PLUS(+) button next to your page selection, not to be confused with product.
3. Choose “Select” and click the next button chevron.
4. Rename the selection to “pagination” and expand it.
5. Then, delete the two extractions in there.
6. Click the PLUS(+) button next to your pagination extraction and choose “Click”.
7. On the popup, choose “Yes” and then enter the additional amount of pages to scrape!
   

Step 4: Bypassing Blocks

During our test scrape, our results came back empty, which means that we ran into a scrape block. To bypass this, we need to use ParseHub’s IP Rotation; note this is a paid feature.

1. Begin by clicking the Gear icon, at the top left of ParseHub.
2. Click “Settings” and then tick the Rotate IP Addresses checkbox.
3. Click “Ok” and then you are ready to scrape!

Step 5: Begin Scraping

Great job! To begin scraping all the pages you specified in our pagination step, click the green Get Data button on the left pane of ParseHub. You can Test, Run or Schedule your scrape. Testing is great for finding faults in your project, Run will run your scrape on ParseHub’s servers, and Scheduling will help with getting up-to-date data!

Here is what our sample export looked like:

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

In this guide, you will learn how to scrape Michaels, the arts and crafts retailer’s website, for free with ParseHub!

Michaels was founded 50 years ago in 1973 by Michael Dupey in Texas. There are over 1,200 physical locations in the US and Canada, with over 40,000 employees. In 2010, Michaels hosted their e-commerce website, selling everything from art supplies, framing, decor, wedding and party products, and a variety of other items. The company generated more than $5 billion in sales in 2020, and continues to grow as more people decide to create their own art, crafts and decor!

Ready to scrape Michaels.com? Let’s begin!

Step 1: Scraping Products

1. Begin by running ParseHub on your PC, Mac or Linux system.
2. Click the blue “New Project” button to start a new project.
3. Enter the Michaels URL you wish to scrape from, we will be scraping acrylic paints with this URL: https://canada.michaels.com/en/paint/acrylic/845161011
4. Wait for the page to load, then click the first product’s name to extract it.
5. Click the next product’s name, which should be yellow, to train the ParseHub algorithm.
6. All products should now be extracted, rename this selection on the left to “product”.

Step 2: Scraping Additional Details

To scrape relative data, such as prices and reviews, we need to use ParseHub’s Relative Select tool!

1. Firstly, click the PLUS(+) button next to ‘Select product’.
2. Choose the “Relative Select” tool.
3. Click the first product’s name, then click its price.
4. Redo this for the next product’s price which should be yellow.
5. Rename this selection on the left to “price”.
6. You can redo these 4 steps again for other details, such as the reviews!

Step 3: Pagination

1. Let’s scrape multiple pages, firstly, scroll down the page until you see the “Next” button.
2. Click the PLUS(+) button next to ‘Select page’ and choose Select.
3. Click the “Next” button to extract it.
4. Rename this selection to “pagination” and expand it to delete the two extractions.
5. Click the PLUS(+) button again, this time next to ‘Select pagination’ and choose “Click”.
6. Choose “Yes” and finally enter the additional amount of pages you wish to scrape.

Step 4: Begin Scraping

Congratulations, you have successfully set up your Michaels scraping project on ParseHub!

To begin scraping, click the green “Get Data” button on the left pane. You can choose to Test, Run or Schedule your scrape on ParseHub’s servers. Scheduling is a good option for getting up-to-date data on a consistent basis!

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

In this guide, we will show you how to scrape Chewy.com, the American online pet food, supply and toy retailer!

Chewy was founded in 2011 and has generated $2 billion dollars, which is 51% of all online pet food sales by 2017. Chewy was acquired in the same year by PetSmart for over $3 billion dollars, the largest e-commerce purchase at the time. The company has generated over 9 billion dollars in 2021 and has over 21,000 employees. As an e-commerce company, there are no physical stores, but Chewy has 13 fulfilment centers where it ships products from. Chewy hosts over 3,000 brands of pet foods and items and has 24/7 customer service available to assist customers. Are you ready to scrape dog, cat, fish, bird, reptile, farm, horse and other pet products?

Let’s begin!

Step 1: Scraping Products

1. Firstly, open ParseHub and log in.
2. Click “New Project” to start a new project.
3. Enter the Chewy.com URL you would like to scrape, we will be scraping dog plush toys with this URL: https://www.chewy.com/b/plush-toys-320
4. Once the page loads, press CTRL/CMD+1 a few times with your cursor on the first product’s name, until an A tag appears, then click to extract the product.
5. Redo this for the next product’s name to train the algorithm.
6. You may need to scroll down to the other product section and click there too to extract every product on the first page.
7. Rename this selection on the left to “product”.

Step 2: Scraping Prices & Reviews

Now that we have our products, we can use ParseHub’s Relative Select tool to scrape relative data for each product!

1. Begin by clicking the PLUS(+) button next to your product selection.
2. Choose Relative Select and click the first product’s name then its price.
3. Scroll down to the next product section and redo the product click, then the price.
4. Rename this selection on the left to “price”.
5. You can redo these steps for other details, such as product reviews or ratings!

Note: If extra text shows up in the price, you can add: ([0-9,.]+) to the regex, which will only show the price.

Step 3: Pagination

1. To scrape multiple pages, begin by scrolling down the page until you see the nav bar.
2. Click the PLUS(+) button next to your page selection and choose Select.
3. Click the Next page button to extract it.
4. Rename this selection on the left to “pagination”.
5. Expand the selection and delete the two extractions.
6. Click the PLUS(+) button next to your pagination selection and choose “Click”.
7. Click “Yes” and enter the additional amount of pages you would like to scrape, entering zero will scrape every single page!

Step 4: Bypassing Blocks

At the time of this guide, it seems like IP Rotation is required to scrape Chewy.com. Without it you may run into blocks, resulting in an empty data file.

To enable IP Rotation, click the settings cog on the top left of ParseHub, and then click the settings tab. Finally, tick the “Rotate IP Addresses” box.

Note, this is a paid ParseHub feature.

Step 5: Start Scraping

Now that you have set up your ParseHub project to extract products, their prices, and other details such as reviews, and have set up pagination and IP Rotation, it is time to begin scraping!

Click the green “Get Data” button on the left, and choose whether you want to test, run or schedule your scrape. We chose to run the scrape a single time.

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

Playing the lotto is a gamble.

But despite their seemingly random results, you might wonder if there are any lucky numbers that win more often than others.

With this in mind, we went off and scraped 10 years worth of Powerball lottery results (US) and Lotto Max results (Canada) to see how these two lotto giants compare to each other.

How we found the luckiest lotto numbers to play

For this project, we used ParseHub, a free and powerful web scraper that can extract data from any website. We used it to pull the winning numbers from 2011 to 2020 from both the Powerball and Lotto Max.

We then analyzed the data we scraped to come up with the luckiest numbers for each lotto / country.

Luckiest Powerball Lotto Numbers (USA)

When playing Powerball, any number from 1 to 69 can come up as one of the winning 5 numbers for each draw (white balls). While a 6th number is pulled from a pool of red balls from 1 to 26.

While we scraped 10 years of Powerball results, the Powerball actually changed the format of their game in October of 2015, with more numbers being added to the pool. As a result, we based our analysis on the past 5 years of results.

After scraping Powerball results for the last 5 years, the luckiest white ball Powerball numbers were:

• 23 being drawn on 2.08% of all draws.
• 32 on 2.05% of all draws
• 61 on 1.86% of all draws
• 53 and 69 on 1.83% of all draws
• 64 on 1.79% of all draws
• 3 on 1.72% of all draws
• 21, 27 and 62 on 1.68% of all draws

But how lucky are really these numbers? On average, these 10 numbers were drawn an average of 50 times.

In comparison, the 10 least common (or unluckiest) numbers were drawn an average of 30 times. That means these numbers are almost twice as likely to be drawn as their unlucky counterparts (1.66 times as likely to be more precise).

Unluckiest Powerball Numbers

So, which are the unluckiest Powerball white ball numbers that get drawn less frequently?

Let’s breakdown the unluckiest numbers:

• 24 and 58 drawn on 1.21% of all draws
• 4, 26, 38 and 65 on 1.17% of all draws
• 51 and 35 on 1.06% of all draws
• And lastly, the unluckiest numbers are 46 and 34 on 1.02% of all draws.

You can now see that the difference in draws between the luckiest and the unluckiest numbers.

Luckiest Lotto Max Numbers

Now we turn our attention to the Lotto Max from Canada.

For this draw, 8 numbers from 1 to 49 are drawn (up to 50 since 2019, we will only look at 1-49 for this analysis).

We analyzed 10 years worth of Lotto Max draws and the luckiest numbers were:

• 39 being drawn on 2.49% of all draws
• 19 and 38 on 2.29% of all draws
• 18 and 44 on 2.27% of all draws
• 28, 1, 15, 22 and 24 on 2.24% of all draws

On average, these numbers have been drawn 110.9 times. In contrast, the bottom 10 unluckiest numbers average 85.1 draws.

Unluckiest Lotto Max Numbers

So, which are the unluckiest numbers for Lotto Max draws?

Let’s break them down:

• 43 being drawn on 1.85% of all draws
• 32 on 1.83% of all draws
• 21, 49 on 1.79% of all draws
• 29 and 34 on 1.77% of all draws
• 13 on 1.71% of all draws
• 4 on 1.69% of all draws
• 33 on 1.67% of all draws
• And the unluckiest number is 35 only being drawn on 1.65% of all draws in the last 10 years

So, What are the Luckiest Lottery Numbers?

Well, as anyone with statistical knowledge will tell you, this historical data of numbers drawn has zero impact on future drawings.

Meaning that using this historical data to pick your next series of numbers is as effective as just choosing them at random.

However, it is fun and interesting to look at all the numbers in hindsight and see what the “luckiest” numbers have been over the years.

In this case, that’s 23 in the US and 39 in Canada.

Want to start your own web scraping project? Check our beginner guide on web scraping.

View the other research pieces we have done:

• Is the Pink Tax Real?
• Best and worst IPOs of 2020
• How much are Pokemon cards worth on eBay
• Web Scraping Car Prices to Track Inflation
• Rising Cost of Groceries in Canada: Web Scraping Reveals Inflation

Last updated: 5/5/2023
Status: Scraping behind a login works great with the following 2023 updated guide.

Web Scraping provides anyone with access to massive amounts of data from any website.

As a result, some websites might hide their content and data behind login screens. This practice actually stops most web scrapers as they cannot log in to access the data the user has requested.

However, there is a way to simply get past a login screen and scrape data while using a free web scraper.

Web Scraping Past Login Screens

ParseHub is a free and powerful web scraper that can log in to any site before it starts scraping data.

You can then set it up to extract the specific data you want and download it all to an Excel or JSON file.

To get started, make sure you download and install ParseHub for free.

Before We Start

Before we get scraping, we recommend consulting the terms and conditions of the website you will be scraping. After all, they might be hiding their data behind a login for a reason.

For reference, we recommend you read our guide on the legality of web scraping.

You can also check out our blog post about the ethics of web scraping.

Note, if you want our dedicated team to legally and ethically scrape large amounts of data for you, check out ParseHub Plus.

Next, if you’re scraping a website where account creation is free, we recommend that you create a dummy account for your scraping purposes.

To do this, feel free to use a new email account from a free email provider. For most cases, we recommend creating a dummy Gmail account.

Scraping a Website with a Login Screen

Every login page is different, but for this example, we will setup ParseHub to login past the Reddit login screen. You might be interested in scraping data from a private Subreddit.

1. Open ParseHub and enter the URL of the site you’d like to scrape. ParseHub will now render the page inside the app.

2. Start by clicking in the “Log In” button to select it. In the left sidebar, rename your selection to login.



3. Click on the PLUS(+) sign next to your login selection and choose the Click command.



4. A pop-up will appear asking you if this is “Next Page” button. Click on “No”, name your template to login_page and click “Create New Template”.



5. A new browser tab and new scraping template will open in ParseHub.
6. Start, by clicking on the username field. ParseHub will automatically ask you for the text to enter in this field, enter your account username and rename the selection to username.



7. Click on the PLUS(+) sign next to your page selection and use the Select command.



8. Now, select the password field and just like in step 6, enter your password details and rename your selection to password.



9. Click on the PLUS(+) sign next to your page selection and choose the Select command.



10. With the select command, click on the blue “Sign In” button and rename your selection to sign_in.
11. Click on the PLUS(+) sign next to your sign_in selection and choose the Click command.



12. A pop-up will appear asking you if this is a “Next Page” button. Click on “No” and create a new template. In this case, we will name it homepage.



Closing Thoughts

You now know how to easily get past any login screen while web scraping.

You can now go ahead and create the rest of your scraping project (more on this below).

Although we know that not every website is built the same, if you run into any issues while setting up this project, reach out to us via email or chat and we’ll be happy to assist you with your project.

While you’re at it, want to learn how to scrape data from Reddit? Read our guide on how to scrape Reddit posts and data.

Looking to scrape another website? Here’s how to scrape websites into Excel spreadsheets.

We have many other guides such as web scraping any ecommerce website as well!

Or better yet, why not become a certified web scraping expert? Check out our FREE Web Scraping Certification courses and get certified today!

Happy Scraping!

Download ParseHub for free

In this tutorial, we will show you how to scrape the Slickdeals website for free, so you can extract the latest products, deals, coupons and more!

Slickdeals was founded in 1999 in Las Vegas, and has since been helping American users find the best deals, offers, coupons and products in the US. The app and website both allow members to post deals they find purchase-worthy, and once live, the deals can be rated, shared or commented on. The comment feature is great, as you can discuss specific deals, if they’re legit or still in stock, and discuss any other questions! Slickdeals is one of the world’s biggest online deal-sharing communities, and hosts many deals from numerous categories, such as video games, TVs, computers, credit cards, and even travel deals; which means a lot of data to scrape!

Ready to scrape the best deals? Let’s begin!

Step 1: Extracting Products & Deals

1. Begin by opening ParseHub on your computer.
2. Click the blue “New Project” button to create a new project.
3. Enter the Slickdeals URL you would like to scrape from, we will be scraping TV deals: https://slickdeals.net/tv-deals/
4. Wait until the page loads; once it’s loaded, click the first product’s name to extract it.
5. Click the next product’s name to train the algorithm. You may need to do this for missing entries.
6. All 40 products and deals should now be extracted.
7. Finally, rename this selection on the left to “product” or “deal”.

Step 2: Scraping Prices & Details

To scrape additional data, we need to use ParseHub’s Relative Select tool.

1. First, click the PLUS(+) button next to Select product, and choose Relative Select.
2. Click the first product’s name again, then its price.
3. Rename this selection on the left to “price”.
4. Redo these steps for other details, such as the vendor, the likes and comments!

Step 3: Pagination

Although these deals are all on a single page, some categories have deals across multiple pages, when there are more than 40 to show.

To parse through multiple pages, view our e-commerce tutorial or pagination help doc!

Step 4: Running Your Scrape

You have successfully extracted products and deals, their listed prices and perhaps other data such as the vendor, the user who posted the deal, the amount of likes and comments and more!

To begin scraping on ParseHub’s servers, first, click the green “Get Data” button on the left pane. You can choose to Test, Run or Schedule your scrape. Scheduling your scrape on a consistent basis can be useful for finding up-to-date deals!

Here is what our data export looks like:

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

In this tutorial, you will learn how to scrape Poshmark products, prices, sizes, sellers and more using ParseHub, our free web scraper!

Poshmark is an e-commerce website and social platform that allows users to sell and buy new or used fashion, electronics and other products. With over 80 million users, and 200 million listings, Poshmark is one of the largest online marketplaces in the world. Poshmark generates over $262 million dollars and has over 800 employees, with its headquarters in Redwood City, California. Listing your items on Poshmark is very simple, all you need is their app and photos, and that’s it. Shipping is also very simple for sellers, as the labels are generated for you. Similar to StockX, which we also did a tutorial on, Poshmark also ensures brand authenticity.

Are you ready to scrape tons of new and used Poshmark listings? Let’s begin!

Step 1: Extracting Products

1. Firstly, open ParseHub and log in.
2. Click “New Project” to start a new project.
3. Enter the Poshmark URL you wish to scrape from, we will be scraping Ed Hardy items for sale, with this URL: https://poshmark.ca/brand/Ed_Hardy-Women
4. Click the first product’s name to extract it.
5. Click the next product’s name to train the algorithm.
6. Rename the selection on the left to “product”.

Step 2: Scraping Additional Details

1. Begin by clicking the PLUS(+) button next to your product selection.
2. Choose “Relative Select” and click the first product.
3. Move the arrow to its respective price, and click to extract it.
4. Rename this selection on the left to “price”.
5. Redo these steps for the product size, the seller, and other details you may require!

Step 3: Pagination

1. To scrape multiple pages, first, scroll down to the page navigation bar.
2. Click the PLUS(+) button next to Select page, and choose “Select”.
3. Click the Next button to extract it, it should have a Button tag.
4. Rename this selection to “pagination”.
5. Expand the selection and delete the extractions.
6. Click the PLUS(+) button next to your pagination selection and choose “Click”.
7. Choose “Yes” as this is a next-page button.
8. Finally, choose the additional number of pages you want to scrape, entering zero will scrape every single page available!

Step 4: Start Scraping

Now that you have extracted products, prices, sellers, sizes and other details, and have set up your pagination on ParseHub, you are ready to begin scraping!

To scrape on ParseHub’s servers, click the green “Get Data” button on the left. You can Test, Run, or Schedule your scrape.

Running into scraping issues? Contact our live chat support!

Happy Scraping! 💻

Shopping for groceries in Canada is becoming more and more costly, with worrisome inflation patterns. The outcry from shoppers can be heard across the country as everyday items become increasingly expensive; Google Trends reveals a steady growth in search terms such as "Canadian grocery inflation." It's not hard to see that this issue has taken hold of Canadians' attention and isn't likely to let go any time soon.

Recently, Loblaws went under fire for pricing chicken breast much higher than major competitors, nearly $40 Canadian for a pack:

I beg your pardon pic.twitter.com/3IIlc7xtY6

— Siobhan Morris (@siomoCTV) January 4, 2023

One might think it was a price mistake, however, the price was indeed correct:

Loblaws confirmed the price of the chicken breast was correct, justifying it as being a 'premium' product. https://t.co/LuZ82E6NWN

— CTV Toronto (@CTVToronto) January 6, 2023

In this blog post, we'll explore grocery prices via our web scraping, which can shed light on the extent of grocery and food inflation in Canada. We'll also discuss potential factors, solutions and long-term implications for Canadian households.

How ParseHub Can Reveal Inflation Trends

Web scraping allows for the extraction of data from websites for deeper insights into consumer behaviours and market volatility. For example, by exploring grocery store sites for product prices across Canada, we are able to uncover patterns in the cost of groceries, so that inflation rates become much clearer. Historical prices can be seen with digital flyers, or statistic and analytic websites.

Using ParseHub, our free web scraping tool, we can aggregate data from multiple sources like supermarkets and export them on a scheduled basis, to a CSV or JSON file, or via our API to be used for analysis. In fact, many companies use ParseHub for price monitoring, which can bring discoveries to light, such as the average cost inflations on various types of food products or regional discrepancies in price hikes. We also wrote a piece about Pink Tax, and how female products often cost more than their male counterparts. With big data related to food pricing, shoppers will be better equipped to make informed decisions; while policymakers devise aimed measures that address economic inflationary issues at its core.

Average food and groceries prices from 2015 to 2023:

Milk (4L): $5.52 to over $6.63
Bananas: Relatively stable.
Steak (1kg): $24.74 to over $27.49
Eggs (Dozen): $3.33 to over $3.87

According to Trading Economics, the cost of food and non-alcoholic beverages increased by 9.7% in February 2023, a little less than the 10.4% from the prior month.

Factors Driving Canadian Grocery Inflation

The cost of grocery items in Canada is rising due to a combination of factors, including global supply chain disruptions, labour shortages, and the effects of climate change on agriculture. Furthermore, the COVID-19 pandemic has further exacerbated these issues by disrupting food production and distribution systems. Animal feed, raw materials and transportation costs have increased which consequently leads to higher prices for consumers when they shop at their local grocers.

Consumers are accusing supermarkets of 'price-gouging', despite laws existing to prevent it. Fortunately, web scraping and Statistics Canada offer the ability to track price fluctuations and compare them to past trends in order to gain a better understanding of what is contributing toward grocery inflation. With data analysis, we can make precautionary decisions to reduce large price increases.

Potential Solutions and the Role of Technology

With Canadian grocery inflation becoming an increasing issue, it’s important to evaluate potential solutions and consider how modern technology can help alleviate its effects. Artificial intelligence (AI) and machine learning algorithms, when combined with web scraping and data harvesting, are viable options for optimizing supply chain management and streamlining logistics. By enhancing efficiency while reducing waste, these advanced technologies aid in keeping costs low; ultimately leading to decreased prices for consumers.

Web scraping can monitor the effectiveness of these algorithms, by tracking their adoption rates and impact on grocery prices. By collecting and analyzing data over time, governments or corporations can choose strategies that are most successful in curbing inflation, and make better policies and better pricing decisions to benefit society.

Concluding Thoughts

Canadian grocery inflation is an increasing issue affecting families nationwide. However, with the power of web scraping, we can gain greater insight into the increased prices, and their root causes and search for viable solutions. As technology progresses, there's hope that groundbreaking methods in food production and supply chain management will help lessen the impact of inflation on shoppers.

For now, the best thing Canadian consumers can do is stay informed about price trends, adjust their shopping habits, and look for good bargains with digital flyer apps, such as Flipp, to navigate through this challenging financial landscape. Ultimately, with the help of web scraping, data analytics and statistics, we can continue monitoring the rising costs of groceries in Canada, and in the world. We can work together to address the rising food costs before it becomes unaffordable.

Check out some of our other articles related to grocery stores, inflation and prices:

• How to Scrape Costco Products
• Web Scraping Used Cars to Track Inflation
• How to Scrape Walgreens

Need help using ParseHubf for price tracking or monitoring? Contact our live chat support!

Happy Scraping! 💻

Web scraping is one of the most efficient and useful ways to extract data from a website, especially in 2023!

Some websites can contain a very large amount of invaluable data.

Stock prices, product details, sports stats, company contacts, you name it.

If you wanted to access this information, you’d either have to use whatever format the website uses or copy-paste the information manually into a new document. Here’s where web scraping can help.

What is Web Scraping?

Web scraping refers to the extraction of data from a website. This information is collected and then exported into a format that is more useful for the user. Be it a spreadsheet or an API.

Although web scraping can be done manually, in most cases, automated tools are preferred when scraping web data as they can be less costly and work at a faster rate.

But in most cases, web scraping is not a simple task. Websites come in many shapes and forms, as a result, web scrapers vary in functionality and features.

Please note that you may encounter captchas when attempting to scrape some websites, so we suggest reading several guides on how to avoid & bypass captchas before scraping a website:

• How to avoid and bypass captchas
• Solving Captcha (for all Paid plans)

If you want to find the best web scraper for your project, make sure to read on.

Is web scraping legal?

In short, the action of web scraping isn't illegal. However, some rules need to be followed. Web scraping becomes illegal when non publicly available data becomes extracted.

This comes as no surprise given the growth of web scraping and many recent legal cases related to web scraping.

If you want to learn more about the legality of web scraping, you can continue reading here: Is web scraping legal?

How do Web Scrapers Work?

So, how do web scrapers work? Automated web scrapers work in a rather simple but also complex way. After all, websites are built for humans to understand, not machines.

First, the web scraper will be given one or more URLs to load before scraping. The scraper then loads the entire HTML code for the page in question. More advanced scrapers will render the entire website, including CSS and Javascript elements.

Then the scraper will either extract all the data on the page or specific data selected by the user before the project is run.

Ideally, the user will go through the process of selecting the specific data they want from the page. For example, you might want to scrape an Amazon product page for prices and models but are not necessarily interested in product reviews.

Lastly, the web scraper will output all the data that has been collected into a format that is more useful to the user.

Most web scrapers will output data to a CSV or Excel spreadsheet, while more advanced scrapers will support other formats such as JSON which can be used for an API.

What Kind of Web Scrapers are There?

Web scrapers can drastically differ from each other on a case-by-case basis.

For simplicity’s sake, we will break down some of these aspects into 4 categories. Of course, there are more intricacies at play when comparing web scrapers.

• self-built or pre-built
• browser extension vs software
• User interface
• Cloud vs Local

Self-built or Pre-built

Just like how anyone can build a website, anyone can build their own web scraper.

However, the tools available to build your own web scraper still require some advanced programming knowledge. The scope of this knowledge also increases with the number of features you’d like your scraper to have.

On the other hand, there are numerous pre-built web scrapers that you can download and run right away. Some of these will also have advanced options added such as scrape scheduling, JSON and Google Sheets exports and more.

Browser extension vs Software

In general terms, web scrapers come in two forms: browser extensions or computer software.

Browser extensions are app-like programs that can be added to your browsers such as Google Chrome or Firefox. Some popular browser extensions include themes, ad blockers, messaging extensions and more.

Web scraping extensions have the benefit of being simpler to run and being integrated right into your browser.

However, these extensions are usually limited by living in your browser. Meaning that any advanced features that would have to occur outside of the browser would be impossible to implement. For example, IP Rotations would not be possible in this kind of extension.

On the other hand, you will have actual web scraping software that can be downloaded and installed on your computer. While these are a bit less convenient than browser extensions, they make up for it in advanced features that are not limited by what your browser can and cannot do.

User Interface

The user interface between web scrapers can vary quite extremely.

For example, some web scraping tools will run with a minimal UI and a command line. Some users might find this unintuitive or confusing.

On the other hand, some web scrapers will have a full-fledged UI where the website is fully rendered for the user to just click on the data they want to scrape. These web scrapers are usually easier to work with for most people with limited technical knowledge.

Some scrapers will go as far as integrating help tips and suggestions through their UI to make sure the user understands each feature that the software offers.

Cloud vs Local

From where does your web scraper actually do its job?

Local web scrapers will run on your computer using its resources and internet connection. This means that if your web scraper has a high usage of CPU or RAM, your computer might become quite slow while your scrape runs. With long scraping tasks, this could put your computer out of commission for hours.

Additionally, if your scraper is set to run on a large number of URLs (such as product pages), it can have an impact on your ISP’s data caps.

Cloud-based web scrapers run on an off-site server which is usually provided by the company that developed the scraper itself. This means that your computer’s resources are freed up while your scraper runs and gathers data. You can then work on other tasks and be notified later once your scrape is ready to be exported.

This also allows for very easy integration of advanced features such as IP rotation, which can prevent your scraper from getting blocked from major websites due to their scraping activity.

What are Web Scrapers Used For?

By this point, you can probably think of several different ways in which web scrapers can be used. We’ve put some of the most common ones below (plus a few unique ones).

Real Estate Listing Scraping

Many real estate agents use web scraping to populate their database of available properties for sale or for rent.

For example, a real estate agency will scrape MLS listings to build an API that directly populates this information onto their website. This way, they get to act as the agent for the property when someone finds this listing on their site.

Most listings that you will find on a Real Estate website are automatically generated by an API.

Industry Statistics and Insights

Many companies use web scraping to build massive databases and draw industry-specific insights from these. These companies can then sell access to these insights to companies in said industries.

For example, a company might scrape and analyze tons of data about oil prices, exports and imports in order to sell their insights to oil companies across the world.

Comparison Shopping Sites

Some several websites and applications can help you to easily compare pricing between several retailers for the same product.

One way that these websites work is by using web scrapers to scrape product data and pricing from each retailer daily. This way, they can provide their users with the comparison data they need.

Lead Generation

One incredibly popular use of web scraping is lead generation. This use is so popular in fact, that we have written an entire guide on using web scraping for lead generation.

In short, web scraping is used by many companies to collect contact information about potential customers or clients. This is incredibly common in the business-to-business space, where potential customers will post their business information publicly online.

Check out our guides of how you can use web scraping for your business:

• Scraping stock prices into an app API
• Scraping data from YellowPages to generate leads
• Scraping data from a store locator to create a list of business locations
• Scraping product data from sites like Amazon or eBay for competitor analysis
• Scraping sports stats for betting or fantasy leagues
• Scraping site data before a website migration
• Scraping product details for comparison shopping
• Scraping financial data for market research and insights

The list of things you can do with web scraping is almost endless. After all, it is all about what you can do with the data you’ve collected and how valuable you can make it.

Read our Beginner's guide to web scraping to start learning how to scrape any website!

The Best Web Scraper

So, now that you know the basics of web scraping, you’re probably wondering what is the best web scraper for you?

The obvious answer is that it depends.

The more you know about your scraping needs, the better of an idea you will have about what’s the best web scraper for you. However, that did not stop us from writing our guide on what makes the Best Web Scraper.

Of course, we would always recommend ParseHub. Not only can it be downloaded for FREE but it comes with an incredibly powerful suite of features which we reviewed in this article. Including a friendly UI, cloud-based scrapping, awesome customer support and more.

Learn more about ParseHub and download it for free.

Want to become an expert on Web Scraping for Free? Take our free web scraping courses and become Certified in Web Scraping today!

If you are interested in getting the data you want right away without having to learn, we offer web scraping services. Our team of web scraping experts will extract any data from the most complex websites. Book a free call today!

PDF files are incredibly common on the internet.

There might be scenarios where you might have to download a long list of PDF files from a website.

If the number of files is large enough, you might be interested in automating the process.

In this updated guide, we will use a free web scraper to scrape a list of PDF files from a website and download them all to your drive.

Scraping a list of PDF Files

First, we’ll need to set up our web scraping project. For this, we will use ParseHub, a free and powerful web scraper that can scrape any website.

For this example, we will use the Province of British Columbia’s webpage for “Supreme Court Civil Rules Forms”. This page has a large list of links to PDF files.

We will use our scraper to extract the links to all these files and download them on to our Dropbox account.

Setting up Dropbox integration

For this example, we will download all files to our Dropbox account. So let’s set up our Dropbox integration in ParseHub.

NOTE: You can also download files via Amazon S3 or just download the list of links and use any download manager you’d want to use.

1. Make sure to log in to your ParseHub account through ParseHub.com
2. After downloading and installing ParseHub, boot it up and click on the Integrations tab on the left sidebar.

3. Click on the Dropbox option.
4. Enable the Integration.

5. You will be asked to login in to Dropbox. Login and allow ParseHub access. Your integration will now be enabled in ParseHub.

Setting up the web scraper

1. Now, go back to ParseHub’s new main menu and click on “New Project”
2. Enter the URL for the page you want to scrape. ParseHub will now load this page inside the app and let you make your first selection.

3. Scroll to the first link in the page and click on it to select it. The link will be highlighted in Green to indicate that it has been selected. The rest of the links will be highlighted in Yellow.

4. Click on the second link in the list. All the links will now be highlighted Green to indicate they have been selected.

5. In the left sidebar, we will get rid of the “name” extraction. As we are not interested in extracting the names of the links.

6. Now, click on the Extract URL command and open up the Extract dropdown under it. From the dropdown, choose the Download to Dropbox option.

Running your scrape

Now it’s time to run your scrape and have all your files downloaded directly into your Dropbox.

1. On the left sidebar, click on the green Get Data button.
2. You can now test, schedule or run your scrape. In this case, we will run our scrape right away.

3. The scraper will now run and automatically add all the PDF files to your Dropbox. If you’ve synced your Dropbox to your PC or Mac, all files will be downloaded to your file system as well.

Downloading all PDF Files in a website

You have now successfully scraped and downloaded all PDF files in a website with the help of a free web scraper!

If you run into any issues through this process, don’t hesitate to contact us via the live chat on our website or through support@parsehub.com.

Happy Scraping!

In this guide, you will learn how to scrape StockX products, prices and other details, with ParseHub, our free web scraper!

StockX is an e-commerce marketplace and apparel reselling service which was founded in 2015 in Detroit, Michigan. StockX has over 1,200 employees and has been since auctioning rare, or collectible clothing and other goods between sellers and buyers. The sellers usually send their products to StockX for inspection and authenticity and are then sold to buyers, which reduces the chances of getting scammed online. They also show historic pricing for specific items, which is cool to view, and can be useful for price monitoring and trajectory! If you’d like to conduct price monitoring, we also have a guide on scraping any e-commerce website, which you might find useful.

Let’s begin scraping StockX!

Step 1: Extracting Products

1. Begin by running the ParseHub application.
2. Click the blue “New Project” button to begin a new project.
3. Enter the StockX URL you wish to scrape from, we will be scraping shoes with this URL:
   https://stockx.com/shoes
4. Once the page loads, click the first product’s name to extract it.
5. The rest should now turn yellow, click the next one to train the algorithm.
6. Rename this selection on the left to “product”.

Step 2: Scraping Prices

1. Firstly, click the PLUS(+) button next to the product selection you just made.
2. Choose the “Relative Select” tool and click the first product’s name.
3. Now move your cursor to the respective product’s price, and click it.
4. All product prices on the first page should be extracted!
5. Rename it to “price” on the left.

Note: to harvest even more product data, you can use the “Click” function, next to your product selection, and create a new template.

Step 3: Pagination

1. Begin by scrolling down the page until you see the navigation bar.
2. Click the PLUS(+) button next to “Select page” and choose “Select”.
3. Click the next page icon, it should be an “svg” or “path” tag.
4. Rename this selection to “pagination” on the left, and expand it to delete the extractions.
5. Click the PLUS(+) button next to the pagination selection and choose “Click”.
6. On the popup, choose yes, as this is a next page button.
7. Finally, choose the additional amount of pages to scrape, inputting zero will scrape all available products!

Step 4: Starting Your Scrape

To begin scraping, click the green “Get Data” button on the left pane. You can test, run or schedule your scrape. Scheduling is a great option for up-to-date pricing information!

Note: If you are running into scrape blocks or empty files, view our IP Rotation tutorial.

Here is what our data export looked like:

Need help using ParseHub? Contact our live chat support!

Happy Scraping! 💻

In this tutorial, we will show you how to scrape new or used books and other products from the ThriftBooks website!

ThriftBooks was founded in 2003 in the United States and has since sold over 165 million books on its e-commerce platform. Although ThriftBooks hosts its own e-commerce website, they also sell used books on Amazon and eBay. ThriftBooks also offers used DVDs, CDs, video games, and old tapes which are often purchased online as collectibles. Aside from collectibles, many consumers purchase from ThriftBooks to save money when buying desired book titles and media.

Let’s begin scraping ThriftBooks!

Step 1: Extracting Products

1. Firstly, open ParseHub on your PC, Mac or Linux system.
2. Click “New Project” on the ParseHub application to start a new project.
3. Enter the ThriftBooks URL you wish to scrape from, we will be scraping all new and used bestsellers with this URL:
   https://www.thriftbooks.com/browse/#b.s=bestsellers-desc&b.p=1&b.pp=30&b.nr
4. Click the first book’s title to extract it, the rest should turn yellow.
5. Click the next book’s title to train the algorithm, all 30 books on this page should be extracted.
6. Rename this selection on the left to “product”.

Step 2: Scraping Additional Details

1. Begin by clicking the PLUS(+) button next to your product selection.
2. Choose “Relative Select” and click the first book’s title.
3. Now click the book’s author, to close the arrow.
4. Rename this selection on the left to “author”.
5. Redo this step for the price, and rename the selection to “price” on the left.
   

Note: to get even more book information, you will need to click into each book, create a new template, and scrape relative data with the Relative Select tool.

Step 3: Pagination

1. To scrape from multiple pages, scroll all the way down until you see the page nav bar.
2. Click the PLUS(+) button next to your page selection and choose “Select”.
3. Click the next page button to extract it, and rename the selection to “pagination”.
4. Expand the selection and delete the extraction.
5. Click the PLUS(+) button next to your pagination selection and choose “Click”.
6. Choose yes, as this is a next-page button.
7. Finally, choose the additional amount of pages you wish to scrape. Choosing zero will scrape every single page!

Step 4: Begin Scraping

You are now ready to begin scraping on ParseHub’s servers!

First, click the green “Get Data” button on the left pane. You can Test, Run or Schedule your scrape. We chose to run the scrape a single time, but testing can be useful to parse through your project, and scheduling is great for monitoring prices and data.

If you followed our steps, your data should look similar to this:

Need help scraping e-commerce websites?

Contact our live chat support!

Happy Scraping! 💻

In this tutorial, we will show you how to scrape products from the Calvin Klein website, using ParseHub our free web scraper!

Calvin Klein was founded in the 60s in New York by designers Calvin Klein and Barry K. Schwartz, and is one of the world’s top fashion and e-commerce brands.  It was acquired by PVH Corp in 2003 for over $700 million dollars. The Calvin Klein fashion line includes their world-famous Jeans, Performance, and Underwear brands. In 2021, Calvin Klein sold over 8.5 billion dollars worth of products. On the Calvin Klein website, you can shop for women’s, men’s, and kid’s clothing. The Calvin Klein e-commerce site hosts a large number of categorized products from tops, bottoms, activewear, swimwear, suits, shoes, accessories and much more.

Let’s begin scraping Calvin Klein products!

Step 1: Preloading Products With Scrolling

1. Begin by running the ParseHub application.
2. Click the blue “New Project” button to create a new project.
3. Enter the Calvin Klein URL you wish to scrape from, we will be scraping men’s accessories with this URL:
   https://www.calvinklein.ca/en/men/accessories?sz=120&start=0
4. Wait for the page to load, if there is a popup, you can close it with the Browse toggle at the top of ParseHub.
5. Move the cursor to the main product container, press and hold CTR/CMD+1 to zoom out until the whole container is highlighted, and click to extract it.
6. Rename this extraction to “container” and expand to delete the unneeded data.
7. Click the PLUS(+) icon next to your ‘select container’ selection and under advanced, choose Scroll.
8. Repeat the scroll a minimum of 10 times to make sure we load every single product, you may need to choose more depending on how many products are available!

If you need more help with the setup, we have a tutorial on how to scrape products that have an Infinite Scroll.

Step 2: Selecting Products

1. Begin by scrolling to the top of the page, and click the PLUS(+) button next to your page selection.
2. Choose Select and click the first product’s name, the rest should turn yellow.
3. Click the next product’s name to train the algorithm. You may need to scroll down to the 31st product and click that one too, to fully extract every product.
4. Finally, rename this selection on the left to “product”.

Step 3: Additional Product Details

1. Start by clicking the PLUS(+) button next to the product selection you created in the last step.
2. Choose “Relative Select” and click the product’s name, and then it’s price.
3. Rename this selection to “price” on the left.
4. Redo these steps for other relative data, such as the discount, and reviews.

Step 4: Bypassing Blocks

At the time of this guide, it seems like IP Rotation is required to scrape from Calvin Klein.

1. Begin by clicking the settings cog at the top of ParseHub.
2. Click Settings and tick the Rotate IP Addresses checkbox.
3. Click OK and now you are ready to begin scraping!

Step 5: Start Scraping

You have now successfully set up your project for scraping on ParseHub’s servers!

To begin your scraping project, click the green “Get Data” button on the left pane, or under by clicking the settings cog.

You can Test, Run or Schedule your scrape with ParseHub. Here’s our data export:

Need assistance with e-commerce web scraping? Message our live chat support!

Happy Scraping! 💻