Andromeda
Note

Web Scraping (Python)

Definition

The automated extraction of data from websites. It involves downloading web pages and parsing their content to retrieve specific information.

Why It Matters

The web is the world’s largest library, but it doesn’t have a “download all” button. Web scraping is the skill that turns the chaotic internet into a structured database for your personal use. It represents the shift from being a passive consumer of content to an active architect of intelligence.

Core Concepts

  • The Pipeline:
    1. webbrowser: Launching a browser to a specific URL (simplest form).
    2. requests: Programmatically downloading raw HTML/files.
    3. BeautifulSoup: Parsing the HTML and extracting data via CSS Selectors.
    4. Selenium: Controlling a real browser to interact with dynamic (JavaScript) sites.
  • Ethics and Limits: Always check a site’s robots.txt and terms of service. Avoid aggressive scraping that could be mistaken for a DoS attack.
import requests
from bs4 import BeautifulSoup

url = "https://example.com"
res = requests.get(url)
soup = BeautifulSoup(res.text, 'html.parser')

# Extract the title
title = soup.select('h1')[0].getText()
print(f"Page title: {title}")

Connected Concepts

Connected notes