Twitter Scraper Tool
Learn about the Twitter Scraper Tool used by the Newsletter AI Agent
Twitter Scraper Tool
The Twitter Scraper Tool is a custom tool that allows the Newsletter AI Agent to collect tweets related to a specified topic using the Apify Twitter Scraper Lite actor.
Overview
The Twitter Scraper Tool is primarily used by the Researcher Agent to gather social media insights about the specified topic. It provides a flexible interface for searching Twitter and extracting structured data from tweets.
Implementation
The Twitter Scraper Tool is implemented as a CrewAI BaseTool
that interacts with the Apify Twitter Scraper Lite actor. Here’s the implementation:
Parameters
The Twitter Scraper Tool accepts the following parameters:
Parameter | Type | Description | Default |
---|---|---|---|
searchTerms | List[str] | Search terms to find tweets containing these terms | None |
sort | str | How to sort the returned tweets (“Top” or “Latest”) | “Top” |
start | str | Scrape tweets starting from this date (YYYY-MM-DD) | None |
end | str | Scrape tweets until this date (YYYY-MM-DD) | None |
Usage
The Twitter Scraper Tool is used by the Researcher Agent to gather social media insights about the specified topic:
Return Value
The tool returns a list of tweets, where each tweet is a dictionary containing information about the tweet, including:
text
: The text content of the tweeturl
: The URL of the tweetusername
: The username of the tweet authortimestamp
: The timestamp when the tweet was postedlikes
: The number of likes the tweet receivedretweets
: The number of retweets the tweet received- Additional metadata about the tweet
Apify Integration
The tool uses the Apify Twitter Scraper Lite actor, which provides several advantages:
- Scalability: The actor can handle large numbers of Twitter searches efficiently
- Reliability: The actor is designed to handle rate limiting and other issues that can arise when scraping Twitter
- Structured Data: The actor returns tweets in a structured format that is easy to process
- No API Key Required: Unlike the official Twitter API, the actor doesn’t require API keys for basic functionality
Configuration
To use the Twitter Scraper Tool, you need to set up the following environment variables:
Next Steps
- Learn about the YouTube Scraper Tool
- Explore the Researcher Agent that uses this tool
- See how this tool contributes to the newsletter generation process