Search Shelf

Trial. A test of how a curated site works, not a finished site. An AI assistant chose these videos from their titles and descriptions, and nobody has watched them. What this means

Topic

crawling

13 videos. Ordered by the type of source, then by how recent the video is.

  2. Official source

    Do sitemaps still matter?

    Google Search Central

    Asks whether XML sitemaps are still needed now crawlers and AI bots are more capable, and discusses the Couldn't fetch warning in Search Console.

  5. Official source

    How Google Search crawls pages

    Google Search Central

    Explains the steps of how Google Search crawls pages, from URL discovery to fetching and rendering.

    Older: check current guidance

  6. Official source

    Curing crawling chaos

    Google Search Central

    Tells a debugging story about an indexing challenge on a site that crawling tools could not crawl in full.

    Older: check current guidance

  8. Teaching or conference talk

    IR Lecture 10: Crawlers

    Venkatesh Vinayakarao

    Covers how web crawlers work, the challenges of crawling, and a brief overview of the Nutch crawler.

    Older: check current guidance

  9. Teaching or conference talk

    Challenges in Web Crawling

    Venkatesh Vinayakarao

    Gives a brief summary of web crawling challenges such as politeness, freshness, coverage, scale and avoiding traps.

    Older: check current guidance

  12. Sells in this field

    How Much Content is NOT Indexed in Google?

    Onely ‎

    Asks how much content Google leaves out of its index and what the answer means for JavaScript sites, in a 2019 conference session.

    Older: check current guidance