Building a Python Web Scraper to Monitor and Analyze E-commerce Price Trends
Learn how to create a beginner-friendly Python web scraper to track and analyze price changes on e-commerce websites, helping you make smarter buying decisions.
Web scraping is a useful technique to extract data from websites, and an excellent project for beginners is building a price monitoring tool for e-commerce platforms. In this tutorial, we'll create a Python web scraper to track product prices, store the data, and do simple trend analysis. This helps you monitor when prices drop and make informed purchase decisions.
We'll use common Python libraries like requests to fetch web pages, BeautifulSoup to parse HTML content, and pandas to analyze data. This project keeps things beginner-friendly while demonstrating real-world applications of web scraping.
Let's get started by installing the libraries. Run this command in your terminal:
pip install requests beautifulsoup4 pandasNext, we'll write a simple function to scrape the price and name of a product from a sample e-commerce page. Note that different websites have different HTML structures, so you can inspect page elements in your browser to find the correct tags and classes.
import requests
from bs4 import BeautifulSoup
url = 'https://example.com/product-page'
headers = {'User-Agent': 'Mozilla/5.0'} # To mimic a browser
response = requests.get(url, headers=headers)
# Check request success
if response.status_code == 200:
soup = BeautifulSoup(response.text, 'html.parser')
# Example selectors - change based on the actual website
product_name = soup.find('h1', class_='product-title').text.strip()
price = soup.find('span', class_='price').text.strip()
print(f'Product: {product_name}')
print(f'Price: {price}')
else:
print('Failed to retrieve the page')Once you have successfully extracted name and price, you want to store this information with the date to monitor changes over time. Pandas is great for this purpose.
import pandas as pd
from datetime import datetime
# Example data (replace with scraped data)
data = {
'date': [datetime.now()],
'product_name': [product_name],
'price': [price]
}
df = pd.DataFrame(data)
# Save or append to CSV file
csv_file = 'price_data.csv'
try:
existing_df = pd.read_csv(csv_file)
df = pd.concat([existing_df, df])
df.drop_duplicates(inplace=True)
except FileNotFoundError:
pass
df.to_csv(csv_file, index=False)
print(f'Data saved to {csv_file}')Note that price values are often strings with currency symbols, so you'll want to clean and convert them to floats for analysis.
def clean_price(price_str):
# Remove currency symbols and commas, convert to float
return float(price_str.replace('$', '').replace(',', '').strip())
# Usage example
price_float = clean_price(price)
print(f'Cleaned price: {price_float}')After collecting price data over days or weeks, you can plot the trend using pandas and matplotlib.
import matplotlib.pyplot as plt
# Load saved data
prices_df = pd.read_csv(csv_file)
# Convert date column to datetime
prices_df['date'] = pd.to_datetime(prices_df['date'])
# Clean the price column
prices_df['price'] = prices_df['price'].apply(clean_price)
# Plot price over time
plt.figure(figsize=(10,5))
plt.plot(prices_df['date'], prices_df['price'], marker='o')
plt.title(f"Price Trend for {product_name}")
plt.xlabel('Date')
plt.ylabel('Price (USD)')
plt.grid(True)
plt.show()To automate this scraping daily, you can run the script on your computer using Task Scheduler, cron jobs, or cloud services. Always check the website's terms of service and robots.txt to comply with their scraping policies.
In summary, this beginner-friendly project demonstrates how Python helps you scrape, store, and analyze price trends on e-commerce websites. With some practice, you can extend this to multiple products or websites and build more complex alert systems.