Repositories list

• w3lib

  Public

  scrapy/w3lib’s past year of commit activity
  Python library of web-related functions

  pythonhacktoberfest

  pythonhacktoberfest

  HTML

  •

  BSD 3-Clause "New" or "Revised" License

  •143143 forks in scrapy/w3lib•419419 stars in scrapy/w3lib•44 issues in scrapy/w3lib•1111 pull requests in scrapy/w3lib•Updated Nov 2, 2026Nov 2, 2026

• scrapy

  Public
  Scrapy, a fast high-level web crawling & scraping framework for Python.

  pythoncrawlerframework+ 5

  pythoncrawlerframeworkscrapingcrawlingweb-scrapinghacktoberfestweb-scraping-python

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •12k12k forks in scrapy/scrapy•65k65k stars in scrapy/scrapy•109109 issues in scrapy/scrapy•7979 pull requests in scrapy/scrapy•Updated Nov 2, 2026Nov 2, 2026

• scrapy-lint

  Public

  scrapy/scrapy-lint’s past year of commit activity
  A linter for Scrapy projects.

  linterlinter-pluginscrapy+ 2

  linterlinter-pluginscrapyflake8flake8-plugin

  Python

  •

  MIT License

  •99 forks in scrapy/scrapy-lint•2222 stars in scrapy/scrapy-lint•2020 issues in scrapy/scrapy-lint•2222 pull requests in scrapy/scrapy-lint•Updated Nov 2, 2026Nov 2, 2026

• queuelib

  Public
  Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

  pythonpython3persistent+ 3

  pythonpython3persistenthacktoberfestqueuesnon-persistent

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •5959 forks in scrapy/queuelib•300300 stars in scrapy/queuelib•00 issues in scrapy/queuelib•00 pull requests in scrapy/queuelib•Updated Nov 2, 2026Nov 2, 2026

• frostwork

  Public
  Python

  •

  Apache License 2.0

  •11 fork in scrapy/frostwork•1414 stars in scrapy/frostwork•00 issues in scrapy/frostwork•22 pull requests in scrapy/frostwork•Updated Nov 1, 2026Nov 1, 2026

• itemadapter

  Public
  Common interface for data container classes

  pythonmetadatapython3+ 4

  pythonmetadatapython3scrapyhacktoberfestpython-dataclassespython-attrs

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •2424 forks in scrapy/itemadapter•7070 stars in scrapy/itemadapter•33 issues in scrapy/itemadapter•33 pull requests in scrapy/itemadapter•Updated Nov 1, 2026Nov 1, 2026

• protego

  Public

  scrapy/protego’s past year of commit activity
  A pure-Python robots.txt parser with support for modern conventions.

  pythonrobots-txthacktoberfest+ 1

  pythonrobots-txthacktoberfestrobots-parser

  DIGITAL Command Language

  •

  BSD 3-Clause "New" or "Revised" License

  •3939 forks in scrapy/protego•9595 stars in scrapy/protego•00 issues in scrapy/protego•11 pull request in scrapy/protego•Updated Nov 1, 2026Nov 1, 2026

• unattended-pr-guard

  Public
  A GitHub action to close pull requests that an LLM is opening unattended.

  JavaScript

  •

  Apache License 2.0

  •11 fork in scrapy/unattended-pr-guard•22 stars in scrapy/unattended-pr-guard•00 issues in scrapy/unattended-pr-guard•00 pull requests in scrapy/unattended-pr-guard•Updated Nov 1, 2026Nov 1, 2026

• sphinx-scrapy

  Public
  Sphinx extension for documentation in the Scrapy ecosystem

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •33 forks in scrapy/sphinx-scrapy•33 stars in scrapy/sphinx-scrapy•00 issues in scrapy/sphinx-scrapy•00 pull requests in scrapy/sphinx-scrapy•Updated Nov 1, 2026Nov 1, 2026

• itemloaders

  Public
  Library to populate items using XPath and CSS with a convenient API

  hacktoberfest

  hacktoberfest

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •2525 forks in scrapy/itemloaders•4949 stars in scrapy/itemloaders•88 issues in scrapy/itemloaders•22 pull requests in scrapy/itemloaders•Updated Nov 1, 2026Nov 1, 2026

• form2request

  Public
  Python library to build HTTP requests out of HTML forms

  Python

  •

  Apache License 2.0

  •55 forks in scrapy/form2request•55 stars in scrapy/form2request•22 issues in scrapy/form2request•22 pull requests in scrapy/form2request•Updated Nov 1, 2026Nov 1, 2026

• parsel

  Public
  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  csspythonxml+ 5

  csspythonxmlscrapingselectorsxpathlxmlhacktoberfest

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •181181 forks in scrapy/parsel•1.4k1.4k stars in scrapy/parsel•77 issues in scrapy/parsel•66 pull requests in scrapy/parsel•Updated Nov 1, 2026Nov 1, 2026

• sphinx-llm-friendly

  Public
  Sphinx extension to make documentation LLM-friendly

  Python

  •

  MIT License

  •11 fork in scrapy/sphinx-llm-friendly•22 stars in scrapy/sphinx-llm-friendly•11 issue in scrapy/sphinx-llm-friendly•00 pull requests in scrapy/sphinx-llm-friendly•Updated Nov 1, 2026Nov 1, 2026

• cssselect

  Public
  CSS Selectors for Python

  csspythonselectors+ 1

  csspythonselectorshacktoberfest

  Python

  •

  Other

  •7474 forks in scrapy/cssselect•311311 stars in scrapy/cssselect•55 issues in scrapy/cssselect•99 pull requests in scrapy/cssselect•Updated Nov 1, 2026Nov 1, 2026

• scrapy-mcp-official

  Public
  Scrapy MCP server

  Python

  •

  Apache License 2.0

  •11 fork in scrapy/scrapy-mcp-official•1414 stars in scrapy/scrapy-mcp-official•00 issues in scrapy/scrapy-mcp-official•00 pull requests in scrapy/scrapy-mcp-official•Updated Nov 1, 2026Nov 1, 2026

• formerly

  Public
  Deprecate a class that users inherit from

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •11 fork in scrapy/formerly•22 stars in scrapy/formerly•00 issues in scrapy/formerly•00 pull requests in scrapy/formerly•Updated Nov 1, 2026Nov 1, 2026

• scrapyd

  Public

  scrapy/scrapyd’s past year of commit activity
  A service daemon to run Scrapy spiders

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •581581 forks in scrapy/scrapyd•3.1k3.1k stars in scrapy/scrapyd•66 issues in scrapy/scrapyd•11 pull request in scrapy/scrapyd•Updated Oct 23, 2026Oct 23, 2026

• xtractmime

  Public
  https://mimesniff.spec.whatwg.org/ implementation for Python

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •22 forks in scrapy/xtractmime•1313 stars in scrapy/xtractmime•00 issues in scrapy/xtractmime•00 pull requests in scrapy/xtractmime•Updated Sep 26, 2026Sep 26, 2026

• scrapy-agent-plugin

  Public
  Scrapy plugin for agents

  Apache License 2.0

  •22 forks in scrapy/scrapy-agent-plugin•66 stars in scrapy/scrapy-agent-plugin•00 issues in scrapy/scrapy-agent-plugin•00 pull requests in scrapy/scrapy-agent-plugin•Updated Sep 10, 2026Sep 10, 2026

• scurl

  Public
  Performance-focused replacement for Python urllib

  pythoncythonchromium+ 2

  pythoncythonchromiumgurlurlparse

  Python

  •

  Apache License 2.0

  •66 forks in scrapy/scurl•2121 stars in scrapy/scurl•1010 issues in scrapy/scurl•11 pull request in scrapy/scurl•Updated Apr 13, 2026Apr 13, 2026

• scrapyd-client

  Public
  Command line client for Scrapyd server

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •143143 forks in scrapy/scrapyd-client•771771 stars in scrapy/scrapyd-client•55 issues in scrapy/scrapyd-client•00 pull requests in scrapy/scrapyd-client•Updated Feb 27, 2026Feb 27, 2026

• scrapy-bench

  Public
  A CLI for benchmarking Scrapy.

  pythonweb-crawlerscrapy+ 3

  pythonweb-crawlerscrapycommand-line-toolbenchmark-suitescrapy-bench

  Python

  •

  MIT License

  •1515 forks in scrapy/scrapy-bench•3232 stars in scrapy/scrapy-bench•66 issues in scrapy/scrapy-bench•00 pull requests in scrapy/scrapy-bench•Updated Jun 28, 2025Jun 28, 2025

• scrapy.org

  Public archive
  The scrapy.org website (old code)

  htmlhacktoberfest

  htmlhacktoberfest

  HTML

  •142142 forks in scrapy/scrapy.org•6666 stars in scrapy/scrapy.org•11 issue in scrapy/scrapy.org•11 pull request in scrapy/scrapy.org•Updated May 8, 2025May 8, 2025

• loginform

  Public
  Fill HTML login forms automatically

  Python

  •8484 forks in scrapy/loginform•278278 stars in scrapy/loginform•1111 issues in scrapy/loginform•33 pull requests in scrapy/loginform•Updated Apr 24, 2024Apr 24, 2024

• quotesbot

  Public
  This is a sample Scrapy project for educational purposes

  Python

  •

  MIT License

  •783783 forks in scrapy/quotesbot•1.4k1.4k stars in scrapy/quotesbot•22 issues in scrapy/quotesbot•88 pull requests in scrapy/quotesbot•Updated Nov 29, 2023Nov 29, 2023

• booksbot

  Public
  A crawler for http://books.toscrape.com

  Python

  •1k1k forks in scrapy/booksbot•4242 stars in scrapy/booksbot•00 issues in scrapy/booksbot•44 pull requests in scrapy/booksbot•Updated Aug 8, 2023Aug 8, 2023

• scrapely

  Public

  scrapy/scrapely’s past year of commit activity
  A pure-python HTML screen-scraping library

  HTML

  •272272 forks in scrapy/scrapely•1.9k1.9k stars in scrapy/scrapely•2626 issues in scrapy/scrapely•66 pull requests in scrapy/scrapely•Updated Apr 4, 2022Apr 4, 2022

• scrapy-itemloader

  Public archive
  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API

  Python

  •

  BSD 3-Clause "New" or "Revised" License

  •77 forks in scrapy/scrapy-itemloader•77 stars in scrapy/scrapy-itemloader•22 issues in scrapy/scrapy-itemloader•00 pull requests in scrapy/scrapy-itemloader•Updated May 5, 2020May 5, 2020

• url-chromium

  Public
  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromiumgurl

  chromiumgurl

  C++

  •

  BSD 3-Clause "New" or "Revised" License

  •22 forks in scrapy/url-chromium•44 stars in scrapy/url-chromium•00 issues in scrapy/url-chromium•00 pull requests in scrapy/url-chromium•Updated Aug 7, 2018Aug 7, 2018

• base-chromium

  Public
  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/

  C++

  •

  BSD 3-Clause "New" or "Revised" License

  •33 forks in scrapy/base-chromium•88 stars in scrapy/base-chromium•00 issues in scrapy/base-chromium•00 pull requests in scrapy/base-chromium•Updated Jul 31, 2018Jul 31, 2018