Implementing a Lightweight Outbound Rate Limiter in Bun for Web Scraping Efficiency

Web scraping is a powerful technique for extracting data from websites, but it comes with its own set of challenges—chief among them being the risk of IP bans due to excessive request rates. To mitigate this, developers often resort to rate limiting strategies that control the outbound request flow. Recently, I developed a minimal, high-performance outbound rate limiter in Bun to address this very issue, ensuring that my scrapers can operate smoothly without overwhelming target servers.

Why a Custom Rate Limiter?

While there are established solutions and libraries for rate limiting, I needed a lightweight, dependency-free approach tailored to the specific needs of my scraping tasks. My primary goal was simplicity and speed—creating a tool that integrates seamlessly with my existing codebase and can process tens of thousands of requests per second with minimal latency.

Technical Overview

• Platform & Language: Built with TypeScript, running on Bun—an ultra-fast JavaScript runtime known for its performance.
• Algorithm: Implements the classic Token Bucket algorithm, providing an intuitive and efficient way to regulate outgoing requests.
• Dependencies: Zero external dependencies, ensuring ease of deployment and maintenance.

Performance Benchmarks

On my MacBook, the limiter comfortably handles roughly 75,000 requests per second, maintaining sub-millisecond latency per request when tested with the wrk benchmarking tool. This high throughput makes it suitable for large-scale scraping tasks without compromising on performance.

Implementation Highlights

The solution is straightforward: it tracks token availability and refills tokens periodically based on the specified rate, delaying requests when tokens are exhausted. This approach ensures that requests are throttled appropriately, reducing the risk of IP bans from target servers.

Code & Resources

The complete implementation is available as an open-source project on GitHub: Throttl. Whether you’re refining your scraping pipeline or building custom rate limiting solutions, this tool can serve as a reliable foundation.

For a more detailed explanation of the design decisions, implementation details, and usage instructions, visit my blog post: buildbreakrepeat.dev/posts/rate-limiter-for-web-scraping/.

Conclusion

If you’re dealing with high-volume web scraping and need a simple yet effective way to manage outbound request rates, consider incorporating this lightweight rate limiter into your