Finding all links on a website is a fundamental task for SEO professionals, web developers, data engineers, and digital marketers. Whether you're conducting a comprehensive site audit, preparing for a content migration, building a web scraping pipeline, or analyzing a competitor's site structure, extracting URLs efficiently can save hours of manual work.

URL extraction can range from simple browser-based methods for grabbing links from one page, to sophisticated crawling operations that map thousands of URLs across complex site architectures.

In this guide, we'll cover everything from beginner-friendly browser tricks to advanced crawling techniques using command-line tools and enterprise scraping platforms. You'll learn how to handle JavaScript-rendered content, navigate pagination, detect broken links, and export clean URL lists for whatever workflow you're building.

Why you may want to extract all URLs from a website?

Understanding your use case helps you choose the right extraction method. Here are the most common scenarios:

SEO audits and site architecture analysis: SEO teams need comprehensive URL inventories to identify orphan pages (pages with no internal links), analyze internal linking structures, detect redirect chains, and map XML sitemaps against actual crawlable content. Extracting all URLs reveals the true scope of indexable pages versus what search engines can actually discover.

Content auditing and migration: Before migrating to a new CMS or redesigning a website, content teams need complete page inventories. URL extraction creates a master list for content gap analysis, helps identify outdated pages for consolidation, and provides the foundation for building 301 redirect maps.

Web scraping workflows: Data collection projects begin with target identification. If you're scraping product data, real estate listings, or job postings, you first need to extract all relevant URLs (product pages, listing pages, category pages) before running your extraction scripts against each target.

Broken link detection and maintenance: Large websites accumulate broken links over time. Extracting all URLs allows you to test each one systematically, identify 404 errors, dead external links, and redirect loops before they impact user experience or SEO performance.

Competitive research: Understanding a competitor's site structure reveals their content strategy. Extracting their URLs shows you how many product pages they maintain, their blog posting frequency, category organization, and potential keyword targeting across different sections.

Video and media URL extraction: Media teams managing large video libraries often need to extract direct video file URLs (MP4, WebM) or streaming URLs from platforms like YouTube, Vimeo, or Wistia for archival, migration, or analysis purposes.

Quick methods to find all links on a single webpage (beginner-friendly)

If you only need links from a single page rather than an entire domain, these methods work quickly without installing specialized software.

Use your browser's built-in tools (fastest option)

Every modern browser includes developer tools with powerful inspection capabilities:

Chrome/Edge method:

1. Navigate to the target webpage
2. Press F12 or right-click and select "Inspect"
3. Press Ctrl+F (Windows) or Cmd+F (Mac) to open the search box within DevTools
4. Search for href= - this will highlight all link attributes in the HTML
5. Click through each match to see the URLs, or switch to the Console tab and run:

Array.from(document.querySelectorAll('a')).map(a => a.href)

This JavaScript one-liner extracts all anchor tag URLs and displays them as an array. You can copy the output directly from the console.

Firefox method: Similar process, but Firefox's Inspector has a slightly different layout. Use Ctrl+Shift+C to open the inspector, then search for href= in the HTML pane.

Limitations: This method only captures links rendered in the initial HTML. If the page loads additional content via JavaScript (infinite scroll, lazy loading, AJAX requests), you won't see those links unless you manually trigger them first.

Use an online URL extractor (free tools)

Online URL extractors work by fetching a page's HTML and parsing out all link elements. They're convenient for quick jobs without setup:

How they work:

1. Paste the target URL into the tool
2. The tool fetches the page's HTML
3. It parses all <a href=""> tags and displays them in a list
4. Most tools offer export options (CSV, TXT, JSON)

Popular free options:

• Link Gopher (web version): Simple interface, extracts both internal and external links
• SEOToolsCentre Link Extractor: Separates internal/external links, shows anchor text
• SmallSEOTools URL Extractor: Basic extraction with copy-to-clipboard functionality

Pros: No installation required, works on any device with a browser, good for one-off extractions.

Cons: Most free tools have limitations - they can't handle JavaScript-heavy sites, enforce rate limits (usually 1-5 pages per session), and won't crawl entire domains. They also expose your target URLs to third-party services, which may be a concern for confidential projects.

Use a Chrome extension

Browser extensions offer more flexibility than online tools while remaining user-friendly:

Recommended extensions:

Link Klipper: Right-click any page and select "Extract Links" to get a sortable list of all URLs. Offers filtering by internal/external and domain. Good for basic extraction from rendered pages.

Instant Data Scraper: More powerful than simple link extractors - it auto-detects pagination and list structures. When activated, it identifies repeated patterns (like product listings) and can crawl through multiple pages automatically. Excellent for e-commerce and directory sites.

SimpleScraper: A visual scraper that lets you click elements to define what to extract. While overkill for just URLs, it's useful if you need URLs plus associated data (titles, descriptions, prices).

How to use (Link Klipper example):

1. Install the extension from Chrome Web Store
2. Navigate to your target page
3. Click the extension icon or right-click → "Link Klipper"
4. Choose "Extract all links"
5. Export as CSV or copy to clipboard

When extensions work well: Single-page extraction, sites with moderate JavaScript, quick ad-hoc tasks.

When they fail: Heavy JavaScript frameworks (React/Angular apps that render entirely client-side), sites with anti-bot protections, large-scale crawls (browser memory limits), or when you need to crawl thousands of pages systematically.

How to extract all links from an entire website (full domain crawl)

So you need more than just the links from one page, you want everything. Every product page, every blog post, every category, every forgotten "About Us" page buried three levels deep. This is where single-page methods fall apart and you need proper crawling.

Crawling a website means systematically visiting every discoverable page, following internal links recursively, and building a complete map of the site's structure. Think of it like a spider exploring a web, hopping from strand to strand until it's touched everything connected.

Let's look at your options, from GUI-based tools that SEO folks love, to command-line approaches that give you maximum control.

Method 1: Use a website crawler (Screaming Frog, Sitebulb, etc.)

Desktop crawlers are the workhorses of the SEO world. They're built specifically for this job, and they're really good at it.

Screaming Frog SEO Spider is probably the most popular. Here's the workflow:

1. Download and install Screaming Frog (free version handles up to 500 URLs)
2. Open the application and enter your target domain in the URL field at the top
3. Hit "Start" and watch it go—you'll see real-time stats as it discovers and crawls pages
4. The crawler follows every internal link it finds, building out your site structure automatically
5. Once complete, go to "Internal" tab to see all URLs discovered
6. Export everything via "Bulk Export" → "All Internal URLs" → save as CSV

The CSV gives you every URL along with metadata: status codes, page titles, meta descriptions, response times, word counts—the works. For just URLs, you can filter the export or grab the first column.

Why SEO teams love it: It's visual, it's fast (for most sites), and it catches things like redirect chains, canonicalization issues, and broken links automatically. The free version works fine for smaller sites, but you'll need the paid version (around £149/year) for anything over 500 pages.

Sitebulb is the fancier alternative—better reporting, prettier visualizations, more automated auditing. It's pricier (starting at $35/month) but gives you gorgeous charts showing how your site structure flows. Same basic process: enter domain, configure crawl settings, export URL list.

Other solid options:

• Netpeak Spider: Free, unlimited URLs, lightweight
• OnCrawl: Cloud-based, handles massive enterprise sites
• DeepCrawl (now Lumar): Enterprise-grade for technical SEO teams

When crawlers struggle: These desktop tools work great until they don't. JavaScript-heavy sites often break them—they'll miss entire sections that only load via React or Vue. They also hit rate limits on well-protected sites, and crawling a 100,000-page site on your laptop can take hours while maxing out your CPU.

Method 2: Use an online website extractor tool

Online extractors promise the convenience of cloud-based crawling without installing anything. In practice, they're hit-or-miss.

How they work: You enter a domain, the service crawls it on their servers, then gives you a downloadable URL list. Sounds great, right?

The reality: Most free online extractors are severely limited:

• Rate limits: Maybe 100-500 pages per crawl on free tiers
• Timeout restrictions: Crawls that take more than 5-10 minutes get killed
• Shallow crawling: Many only go 2-3 levels deep from the homepage
• JavaScript handling: Usually poor to nonexistent
• Queue position: Free users wait in line behind paid customers

A few that actually work decently:

XML-Sitemaps.com has a free crawler that generates sitemaps but also lets you export all discovered URLs. It's limited to 500 pages on the free tier, but it's reliable for small sites.

BeamUsUp is a desktop application that works online—sort of a hybrid approach. It's free, handles unlimited URLs, and it's surprisingly capable for basic crawling.

My take: Online extractors are fine for quick tests or very small sites, but don't rely on them for serious work. You'll be fighting arbitrary limitations, and you can't customize crawl behavior when things go wrong.

Method 3: Use a command-line scraper (wget, cURL, Python)

Now we're getting into the good stuff. Command-line tools give you complete control, no GUIs, no arbitrary limits—just you and the terminal.

Using wget (the classic approach):

wget is a Unix utility that's been around forever. It downloads web content, but with the right flags, it becomes a capable crawler:

wget --spider --recursive --no-parent --output-file=crawl.log https://example.com

Let me break down what's happening:

• --spider: Don't download files, just check they exist
• --recursive: Follow links recursively
• --no-parent: Don't crawl up to parent directories
• --output-file=crawl.log: Save all discovered URLs to a log

After running, grep through crawl.log to extract just the URLs:

grep -oP '(?<=--  )https?://[^\s]+' crawl.log > urls.txt

Pros: Fast, scriptable, works on any Unix system (Linux, Mac, WSL on Windows).

Cons: Limited JavaScript support, basic link extraction only, can get stuck in redirect loops if you're not careful.

Using Python with Scrapy:

If you're comfortable with Python, Scrapy is phenomenal. It's a full-featured scraping framework that handles crawling elegantly:

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.linkextractors import LinkExtractor

class URLSpider(scrapy.Spider):
    name = 'url_extractor'
    start_urls = ['https://example.com']
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.link_extractor = LinkExtractor()
        self.discovered_urls = set()
    
    def parse(self, response):
        # Extract all links from current page
        links = self.link_extractor.extract_links(response)
        
        for link in links:
            if link.url not in self.discovered_urls:
                self.discovered_urls.add(link.url)
                print(link.url)
                yield response.follow(link, callback=self.parse)

# Run the spider
process = CrawlerProcess(settings={
    'USER_AGENT': 'Mozilla/5.0 (compatible; URLExtractor/1.0)',
    'ROBOTSTXT_OBEY': True,
    'CONCURRENT_REQUESTS': 16,
})
process.crawl(URLSpider)
process.start()

This script crawls recursively, respects robots.txt, and handles JavaScript-free sites beautifully. You can extend it to handle authentication, custom headers, cookies, whatever you need.

For JavaScript-heavy sites, add Selenium or Playwright to render pages first:

from scrapy_playwright.page import PageRoute

# Configure Scrapy to use Playwright for rendering
# This executes JavaScript before extracting links

Why go this route: Complete flexibility. You control crawl speed, depth, user agents, headers, authentication, everything. You can handle edge cases that break GUI tools. And you can integrate URL extraction into larger data pipelines.

Why you might not: It requires actual programming knowledge. Debugging scrapers takes time. And for one-off jobs, it's overkill.

Method 4: Use a scraping platform like DataHen

DataHen is a web scraping service, which means you're not building or running scrapers yourself, you're outsourcing the entire job. You tell them what URLs you need extracted, and they handle everything from crawling to data cleaning to delivery of the final product.

How it works:

Instead of configuring crawlers or writing code, you provide DataHen with your target domain and requirements. Their team sets up the scraping job, and their infrastructure handles all the heavy lifting:

• JavaScript rendering: Their system uses real browsers to fully execute JavaScript-heavy sites (React, Angular, Vue), ensuring every dynamically loaded link gets captured
• Automatic pagination handling: Whether it's "next page" buttons, infinite scroll, or numbered pagination, they detect and follow it automatically
• Distributed crawling: Large sites get crawled in parallel across cloud infrastructure, 50,000 pages don't take 50,000 times longer than one page
• Broken link detection: Every URL discovered gets tested and flagged if it returns 404s, timeouts, or redirect loops
• Clean deliverables: You receive a polished URL list (CSV, JSON, or direct database integration) rather than raw crawl data you need to parse yourself

Real-world example: Let's say you need all product URLs from a competitor's e-commerce site with 50,000 products spread across complex category hierarchies. You'd give DataHen the domain and specify you want product page URLs. They configure the crawler to start at the homepage, discover category structures, handle all the pagination on listing pages, extract every product URL, remove duplicates, and deliver a clean CSV of all 50,000 URLs, while managing rate limits and proxy rotation to avoid getting blocked.

When a managed service makes sense:

• You need this done once or on a regular schedule, but don't want to maintain scraping infrastructure
• The target sites are technically challenging (heavy JavaScript, CAPTCHAs, authentication walls)
• You need guaranteed reliability, crawls can't fail halfway through because your script hit an edge case
• URL extraction is just the first step in a larger data project, and you want professionals handling the extraction so you can focus on analysis
• You don't have developers available to build and maintain scrapers

The tradeoff is cost and control, you're paying for expertise and infrastructure rather than doing it yourself. But if you need comprehensive URL extraction from complex sites without the technical overhead, services like DataHen deliver clean results without you touching a line of code.

How to extract URLs from HTML, text, or code snippets

Sometimes you don't need to crawl a live website, you already have the HTML source, a block of text with embedded URLs, or code snippets that contain links. Maybe you copied HTML from an email, exported content from a CMS, or grabbed source code from a git repository. You just need to pull out the URLs.

This is where regex (regular expressions) becomes your best friend, though there are simpler options if regex makes your eyes glaze over.

Quick and easy: Online text extractors

If you've got a chunk of text or HTML and just want the URLs out of it, paste it into an online extractor:

CyberChef (gchq.github.io/CyberChef) is phenomenal for this. It's a data manipulation tool from GCHQ, surprisingly powerful and free:

1. Go to CyberChef
2. Paste your HTML or text into the "Input" box
3. Search for "Extract URLs" in the operations panel and drag it to the recipe
4. All URLs automatically appear in the output

It handles messy input well and catches URLs even when they're embedded in JavaScript strings or JSON blobs.

LinkParser.com is simpler—paste text, click extract, get URLs. Less flexible than CyberChef but faster for basic jobs.

Using regex to extract URLs

If you're comfortable with regex, you can extract URLs from literally anything—log files, database dumps, code repositories, whatever.

Basic URL regex pattern:

https?://[^\s<>"]+

This matches any string starting with http:// or https:// followed by non-whitespace characters. It's not perfect (URLs can technically contain spaces when encoded), but it catches 95% of real-world cases.

More robust pattern that handles edge cases:

https?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&/=]*)

This looks intimidating, but it properly handles subdomains, query parameters, URL-encoded characters, and fragments.

In practice (Python example):

import re

html_content = """
<p>Check out <a href="https://example.com/page1">this link</a></p>
<script>var url = "https://example.com/api/data";</script>
Some text with https://example.com/page2 embedded.
"""

# Extract all URLs
urls = re.findall(r'https?://[^\s<>"]+', html_content)

for url in urls:
    print(url)

Output:

https://example.com/page1
https://example.com/api/data
https://example.com/page2

JavaScript version (browser console or Node.js):

const text = `Your HTML or text here`;
const urlPattern = /https?:\/\/[^\s<>"]+/g;
const urls = text.match(urlPattern);
console.log(urls);

Extracting from specific HTML attributes

If you specifically want URLs from href or src attributes rather than all URLs in the document:

Grep approach (Linux/Mac terminal):

grep -oP '(?<=href=")[^"]+' page.html

This extracts everything between href=" and the closing ".

For src attributes (images, scripts):

grep -oP '(?<=src=")[^"]+' page.html

Python with BeautifulSoup (cleaner for complex HTML):

from bs4 import BeautifulSoup

html = open('page.html').read()
soup = BeautifulSoup(html, 'html.parser')

# Extract all href URLs
hrefs = [a.get('href') for a in soup.find_all('a') if a.get('href')]

# Extract all src URLs (images, scripts, iframes)
srcs = [tag.get('src') for tag in soup.find_all(['img', 'script', 'iframe']) if tag.get('src')]

print("Links:", hrefs)
print("Resources:", srcs)

BeautifulSoup properly parses HTML, handles malformed markup gracefully, and lets you filter by tag type, much more reliable than regex when dealing with messy real-world HTML.

Handling relative URLs

One gotcha: extracted URLs might be relative (/about, ../contact.html) rather than absolute (https://example.com/about). If you need full URLs, you'll need to resolve them against a base URL.

from urllib.parse import urljoin

base_url = "https://example.com/blog/"
relative_url = "../about.html"

absolute_url = urljoin(base_url, relative_url)
print(absolute_url)  # https://example.com/about.html

JavaScript (browser):

const base = new URL('https://example.com/blog/');
const relative = '../about.html';
const absolute = new URL(relative, base).href;
console.log(absolute);  // https://example.com/about.html

This is crucial when you're extracting links from exported HTML that assumes a particular domain context.

How to extract video links (YouTube, Vimeo, MP4s, etc.)

Video URL extraction is its own beast because video platforms don't just hand you direct file URLs, they embed videos through players, use streaming protocols, and often obfuscate direct links to prevent downloading.

Why video URLs are tricky

When you view a YouTube video, the URL in your browser (https://www.youtube.com/watch?v=dQw4w9WgXcQ) isn't the actual video file—it's a webpage that loads a player, which then fetches video chunks from CDNs using adaptive bitrate streaming protocols. Getting the actual video stream URL requires parsing player data or using specialized tools.

Extracting YouTube video URLs from pages

If you just want to find which YouTube videos are embedded on a page (not download them), that's straightforward:

Regex pattern:

(?:https?:)?(?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:watch\?v=|embed\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})

This matches all common YouTube URL formats and captures the video ID.

import re

html = """
<iframe src="https://www.youtube.com/embed/dQw4w9WgXcQ"></iframe>
<a href="https://youtu.be/jNQXAC9IVRw">Watch this</a>
"""

pattern = r'(?:https?:)?(?:\/\/)?(?:www\.)?(?:youtube\.com\/(?:watch\?v=|embed\/)|youtu\.be\/)([a-zA-Z0-9_-]{11})'
video_ids = re.findall(pattern, html)

for vid_id in video_ids:
    print(f"https://www.youtube.com/watch?v={vid_id}")

Finding direct video file URLs (MP4, WebM)

Some sites serve videos as direct file links—you'll see URLs ending in .mp4, .webm, .mov, etc. These are much simpler to extract:

https?://[^\s<>"]+\.(?:mp4|webm|mov|avi|mkv|flv)

Use case: Media libraries, older video hosting platforms, direct file servers. If you inspect network requests (browser DevTools → Network tab → filter by "media"), you can often find these direct URLs when videos play.

Extracting Vimeo video IDs

Vimeo uses numeric IDs in their URLs:

(?:https?:)?(?:\/\/)?(?:www\.)?vimeo\.com\/(\d+)

Example:

vimeo_url = "https://vimeo.com/123456789"
video_id = re.search(r'vimeo\.com\/(\d+)', vimeo_url).group(1)

Tools for extracting streamable URLs

If you need actual downloadable video URLs (not just video page URLs), you'll need tools that reverse-engineer player APIs:

youtube-dl / yt-dlp (command line):

yt-dlp --get-url "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

This returns the actual streaming URLs that the video player uses. yt-dlp supports hundreds of sites—YouTube, Vimeo, Twitch, Twitter, Reddit, you name it.

For extracting videos from entire sites:

yt-dlp --get-urls --flat-playlist "https://www.youtube.com/c/ChannelName/videos"

This lists all video URLs from a channel without downloading them.

Legal and ethical considerations

I need to be straight with you here: extracting video URLs, especially for downloading, sits in legally murky territory.

• Copyright: Most videos are copyrighted. Extracting URLs for personal archival might fall under fair use (jurisdiction-dependent), but redistributing or commercial use definitely doesn't.
• Terms of Service: YouTube, Vimeo, and most platforms explicitly prohibit downloading videos in their TOS. Violating TOS can get your IP blocked or result in legal nastiness.
• Ethical use: If you're extracting video URLs for analysis (checking which videos are embedded where), that's generally fine. If you're building a system to rip and rehost content, you're asking for trouble.

Legitimate use cases:

• Archiving your own content from platforms before account closure
• Academic research analyzing video distribution patterns
• Accessibility purposes (adding captions to videos you have rights to)
• Media monitoring (tracking which videos competitors embed)

When in doubt, assume you don't have the right to download or redistribute content unless you created it or have explicit permission.

Free vs paid URL extraction tools (comparison table)

When people search for “URL extractor,” they’re often deciding between a quick free tool and a more robust, paid crawler. Here’s a simple comparison table you can embed directly in the article.

If your goal is to crawl an entire site reliably, especially one that’s dynamic, paginated, or has heavy anti-bot measures, paid, cloud-based crawlers tend to be the only option that won’t break mid-crawl.

Common issues when extracting URLs (and how to fix them)

Even simple websites can hide links in ways that confuse extractors. Here are the issues I most often run into when extracting URLs, and how I typically fix them.

Pagination not detected

Problem: The crawler only grabs page 1 and misses “Next,” “Load more,” or infinite scroll links.
Fix:

• Use a tool that supports autorules or heuristics for next-page detection.
• Manually map pagination selectors if needed.

How DataHen helps:
DataHen automatically discovers pagination patterns and continues crawling until no new URLs remain.

JavaScript-rendered links

Problem: Many modern websites load links through JavaScript frameworks (React, Vue, Angular), so simple HTML extractors return incomplete lists.
Fix:

• Use a headless browser (Puppeteer or Playwright).
• Switch to tools that support full JS rendering.

Disallowed robots.txt

Problem: Your crawler can’t access certain paths.
Fix:

• Check the site’s /robots.txt.
• Respect restrictions, or if you own the domain, update the file.

Infinite loops (calendar pages, faceted navigation)

Problem: The crawler loops endlessly through URLs like ?page=1, ?page=2, or ?date=previousMonth.
Fix:

• Set URL filters, maximum depth, or crawl limits.
• Block repetitive query parameters.

Session-based URLs

Problem: Pages generate unique URLs per session (?sessionid=xxxx), causing huge duplicate lists.
Fix:

• Strip query parameters.
• Use canonical URLs.

Geofenced or IP-restricted pages

Problem: Links only appear for visitors from certain countries/regions.
Fix:

• Use proxies or servers in targeted geolocations.
• Log in if required.

Frequently asked questions (FAQs)