Web Scraping

Let's say I need to pull product titles and prices from a category page on an e-commerce site. I start with one URL. I write a script in Python using requests and BeautifulSoup — it's lightweight and quick to iterate. I fetch the HTML, inspect the element in browser DevTools, and write a selector. For example, if titles are in h2.product-title and prices in span.price, I extract those. I test on one page until the output matches what I see on screen. Once it works, I scale by finding the pagination links — usually a "next" button or page numbers. I loop through each page, adding a delay of 2-3 seconds between requests to appear human. I also rotate user-agent strings to avoid basic fingerprinting. I store everything in a CSV with columns for URL, title, price, and timestamp. If a field is missing, I write null rather than skip the row — that way I can see gaps. I handle relative image URLs by joining with the base URL. For encoding, I set the response to UTF-8 explicitly. I wrap each extraction in try-except blocks so one bad row doesn't crash the whole run. Once I have the CSV, I open it in a spreadsheet to spot-check data. I also log URLs that returned unexpected status codes or missing elements. This discipline saves hours later when data looks off. If the site has thousands of pages, I parallelise with concurrent.futures but cap at 5 workers to keep load low. Some sites expose data via an API on the frontend. I check the Network tab before writing selectors. If I see XHR calls returning JSON, I hit that endpoint directly — it's faster and more reliable than parsing HTML. I also check if the site provides a [sitemap](/sitemap/) — that gives me all product URLs without needing to [pagination](/pagination/) manually. This step-by-step works for 90% of scraping tasks I've faced. The key is to start small and verify at each step before automating further.

A few things have tripped me up. JavaScript-rendered content is the most common. If the price appears after a script executes, requests returns an empty container. I've used Puppeteer in Node or Selenium in Python for these cases, but it's slow — each page takes seconds. I now check the Network tab first for hidden API endpoints; often the data is there in JSON format and I can call it directly. This is especially important for sites built with frameworks like React, where much of the content is injected after load. If you are dealing with heavy [JavaScript SEO](/javascript-seo/) challenges, expect this pattern. Another watch-out is anti-bot defences: sites return 403, 429, or block after a few requests. I rotate IPs through a proxy pool when needed, but I avoid scraping sites with aggressive protections — it's rarely worth the effort. Instead, I look for an official API or alternative source. I also pay attention to robots.txt. I've seen sites disallow the /p/ path for product pages, and scraping those could lead to a legal complaint. I always [check robots.txt](/robots-txt/) before scaling. Legal boundaries matter more than SEO goals. Another trap is relying on fragile selectors: class names like price_3f7a2 change after a site deploy. I try to select by more stable attributes like data-product-id or the element structure. If the site uses dynamic IDs, I use XPath with relative positions. I also handle pagination carefully — some sites have infinite scroll, which requires different handling. I use the browser DevTools to find the AJAX endpoint that loads the next batch. Lastly, I set proper user-agent headers and respect Cache-Control to avoid hammering servers. Over time, I've built a checklist that catches these issues before they blow up.

I used to think scraping was always a quick win. My biggest mistake was ignoring the cost of maintenance. I built a scraper for a client monitoring competitor prices on a major retail site. It worked perfectly for two weeks. Then the target site redesigned its HTML — all my selectors broke. I spent six hours fixing them over a weekend. The client expected continuous data, so I had to patch it quickly without full testing. I learned to write scrapers with fallbacks: if a selector fails, try an alternative based on sibling elements or data attributes. But the real lesson is that scraping is not a set-and-forget task. You must plan for site changes and budget time for maintenance. Another admission: I once scraped a site that had clear robots.txt disallows on the paths I needed. I went ahead anyway, thinking "it's just a suggestion". The site blocked my IP and sent a cease-and-desist letter. I was wrong — robots.txt is a legal tool in many jurisdictions and violating it can have consequences. Now I treat it as binding. I also underestimated how much time I'd spend debugging encoding issues or handling pages that load different layouts for mobile vs desktop. Scraping looks simple but the edge cases eat time — about 30% of my effort goes to error handling. Finally, I overestimated my ability to handle JavaScript-rendered sites without a headless browser. I spent a day reverse-engineering obfuscated API calls when I could have used Puppeteer from the start. Now I weigh the trade-off: headless is slower but more robust for complex pages. That change has saved me frustration and reduced rework.