github.com
Crawlee is a Node.js library for web scraping and automation, enabling efficient data extraction and crawler management.
relay
Tech Stack
Summary
Crawlee is a web scraping and browser automation library for Node.js designed to help developers build reliable crawlers. It supports both JavaScript and TypeScript and is capable of extracting data for various applications, including AI and LLMs.
Key features:
- Single interface for both HTTP and headless browser crawling
- Persistent queue for managing URLs to crawl
- Integrated proxy rotation and session management
- Automatic scaling based on system resources
- CLI tools for project bootstrapping
- Zero-config HTTP2 support and automatic browser management
- Support for Playwright and Puppeteer
- Open-source and compatible with the Apify platform
Crawlee is ideal for developers looking to automate web data extraction while maintaining compliance with modern bot protections.
Comments
No comments yet. Sign in to add the first comment!