YouTube Scraper Tool
The YouTube Scraper Tool is a custom tool that allows the Newsletter AI Agent to find relevant video content on YouTube using an Apify actor. It provides a way to gather video-based information and insights related to the specified topic.Overview
The YouTube Scraper Tool is primarily used by the Researcher Agent to gather video content about the specified topic. It provides a flexible interface for searching YouTube and extracting structured data from videos, channels, and playlists.Implementation
The YouTube Scraper Tool is implemented as a CrewAIBaseTool
that interacts with an Apify YouTube scraper actor. Here’s the implementation:
Parameters
The YouTube Scraper Tool accepts the following parameters:Parameter | Type | Description | Default |
---|---|---|---|
searchQueries | List[str] | Search terms for YouTube’s search bar | Required |
maxResultsShorts | int | Limit the number of Shorts videos to crawl | 0 |
maxResultStreams | int | Limit the number of Stream videos to crawl | 0 |
startUrls | List[str] | Direct URLs to YouTube videos, channels, playlists | [] |
downloadSubtitles | bool | Download subtitles for videos | False |
saveSubsToKVS | bool | Save downloaded subtitles to key-value store | False |
subtitlesLanguage | str | Language for subtitles download | ”any” |
preferAutoGeneratedSubtitles | bool | Prefer auto-generated subtitles | False |
subtitlesFormat | str | Format for subtitle downloads | ”srt” |
sortingOrder | str | How to sort the results | None |
dateFilter | str | Filter results by date | None |
videoType | str | Filter by video type | None |
lengthFilter | str | Filter by video length | None |
isHD | bool | Filter for HD videos | None |
hasSubtitles | bool | Filter for videos with subtitles | None |
Usage
The YouTube Scraper Tool is used by the Researcher Agent to gather video content about the specified topic:Return Value
The tool returns a list of YouTube videos, where each video is a dictionary containing information about the video, including:title
: The title of the videourl
: The URL of the videodescription
: The description of the videochannelName
: The name of the channel that uploaded the videochannelUrl
: The URL of the channelviewCount
: The number of views the video haspublishedAt
: The date the video was publishedduration
: The duration of the video- Additional metadata about the video
Apify Integration
The tool uses an Apify YouTube scraper actor, which provides several advantages:- Scalability: The actor can handle large numbers of YouTube searches efficiently
- Reliability: The actor is designed to handle rate limiting and other issues that can arise when scraping YouTube
- Structured Data: The actor returns YouTube videos in a structured format that is easy to process
- Advanced Filtering: The actor supports advanced filtering options to narrow down search results
Configuration
To use the YouTube Scraper Tool, you need to set up the following environment variables:Next Steps
- Learn about the Google News Scraper Tool
- Explore the Researcher Agent that uses this tool
- See how this tool contributes to the newsletter generation process