Skip to main content

Reddit Scraper Tool

The Reddit Scraper Tool is a custom tool that allows the Newsletter AI Agent to gather discussions from Reddit using an Apify actor. It provides a way to collect community insights and discussions related to the specified topic.

Overview

The Reddit Scraper Tool is primarily used by the Researcher Agent to gather community discussions about the specified topic. It provides a flexible interface for searching Reddit and extracting structured data from posts and comments.

Implementation

The Reddit Scraper Tool is implemented as a CrewAI BaseTool that interacts with an Apify Reddit scraper actor. Here’s the implementation:

Parameters

The Reddit Scraper Tool accepts the following parameters:

Usage

The Reddit Scraper Tool is used by the Researcher Agent to gather community discussions about the specified topic:

Return Value

The tool returns a list of Reddit posts and comments, where each item is a dictionary containing information about a post or comment, including:
  • title: The title of the post (for posts only)
  • text: The text content of the post or comment
  • url: The URL of the post or comment
  • author: The username of the author
  • score: The score (upvotes - downvotes) of the post or comment
  • created: The creation date of the post or comment
  • Additional metadata about the post or comment

Apify Integration

The tool uses an Apify Reddit scraper actor, which provides several advantages:
  1. Scalability: The actor can handle large numbers of Reddit searches efficiently
  2. Reliability: The actor is designed to handle rate limiting and other issues that can arise when scraping Reddit
  3. Structured Data: The actor returns Reddit posts and comments in a structured format that is easy to process

Configuration

To use the Reddit Scraper Tool, you need to set up the following environment variables:

Next Steps