YouTube Scraper Tool
Learn about the YouTube Scraper Tool used by the Newsletter AI Agent
YouTube Scraper Tool
The YouTube Scraper Tool is a custom tool that allows the Newsletter AI Agent to find relevant video content on YouTube using an Apify actor. It provides a way to gather video-based information and insights related to the specified topic.
Overview
The YouTube Scraper Tool is primarily used by the Researcher Agent to gather video content about the specified topic. It provides a flexible interface for searching YouTube and extracting structured data from videos, channels, and playlists.
Implementation
The YouTube Scraper Tool is implemented as a CrewAI BaseTool
that interacts with an Apify YouTube scraper actor. Here’s the implementation:
Parameters
The YouTube Scraper Tool accepts the following parameters:
Parameter | Type | Description | Default |
---|---|---|---|
searchQueries | List[str] | Search terms for YouTube’s search bar | Required |
maxResultsShorts | int | Limit the number of Shorts videos to crawl | 0 |
maxResultStreams | int | Limit the number of Stream videos to crawl | 0 |
startUrls | List[str] | Direct URLs to YouTube videos, channels, playlists | [] |
downloadSubtitles | bool | Download subtitles for videos | False |
saveSubsToKVS | bool | Save downloaded subtitles to key-value store | False |
subtitlesLanguage | str | Language for subtitles download | ”any” |
preferAutoGeneratedSubtitles | bool | Prefer auto-generated subtitles | False |
subtitlesFormat | str | Format for subtitle downloads | ”srt” |
sortingOrder | str | How to sort the results | None |
dateFilter | str | Filter results by date | None |
videoType | str | Filter by video type | None |
lengthFilter | str | Filter by video length | None |
isHD | bool | Filter for HD videos | None |
hasSubtitles | bool | Filter for videos with subtitles | None |
Usage
The YouTube Scraper Tool is used by the Researcher Agent to gather video content about the specified topic:
Return Value
The tool returns a list of YouTube videos, where each video is a dictionary containing information about the video, including:
title
: The title of the videourl
: The URL of the videodescription
: The description of the videochannelName
: The name of the channel that uploaded the videochannelUrl
: The URL of the channelviewCount
: The number of views the video haspublishedAt
: The date the video was publishedduration
: The duration of the video- Additional metadata about the video
Apify Integration
The tool uses an Apify YouTube scraper actor, which provides several advantages:
- Scalability: The actor can handle large numbers of YouTube searches efficiently
- Reliability: The actor is designed to handle rate limiting and other issues that can arise when scraping YouTube
- Structured Data: The actor returns YouTube videos in a structured format that is easy to process
- Advanced Filtering: The actor supports advanced filtering options to narrow down search results
Configuration
To use the YouTube Scraper Tool, you need to set up the following environment variables:
Next Steps
- Learn about the Google News Scraper Tool
- Explore the Researcher Agent that uses this tool
- See how this tool contributes to the newsletter generation process