Search Shelf

Trial. A test of how a curated site works, not a finished site. An AI assistant chose these videos from their titles and descriptions, and nobody has watched them. What this means

Teaching or conference talk · for beginners

Challenges in Web Crawling

Venkatesh Vinayakarao 3 min Published Oct 2020

This video is about 6 years old. Search advice changes quickly, so check this against current guidance.

What it covers

Gives a brief summary of web crawling challenges such as politeness, freshness, coverage, scale and avoiding traps.

Watch for: Very short overview

Why it is here

Three minutes listing the problems a web crawler has to solve: politeness, freshness, coverage, scale and traps. A quick way in before the longer lecture.

Selected by an AI assistant, from the title and description only.

On the same topics