Google News Scraper Tool
Learn about the Google News Scraper Tool used by the Newsletter AI Agent
Google News Scraper Tool
The Google News Scraper Tool is a custom tool that allows the Newsletter AI Agent to gather the latest news articles related to a specified topic using the Apify Super Fast Google News Scraper actor.
Overview
The Google News Scraper Tool is primarily used by the Researcher Agent to gather the latest news and developments about the specified topic. It provides a flexible interface for searching Google News and extracting structured data from news articles.
Implementation
The Google News Scraper Tool is implemented as a CrewAI BaseTool
that interacts with the Apify Super Fast Google News Scraper actor. Here’s the implementation:
Parameters
The Google News Scraper Tool accepts the following parameters:
Parameter | Type | Description | Default |
---|---|---|---|
keywords | List[str] | The keywords used to search for news articles | Required |
language | str | Language and country code for the search (e.g., “US:en”) | “US:en” |
maxItems | int | Maximum number of items to scrape for each keyword | 20 |
Usage
The Google News Scraper Tool is used by the Researcher Agent to gather the latest news about the specified topic:
Return Value
The tool returns a list of news articles, where each article is a dictionary containing information about the article, including:
title
: The title of the news articlelink
: The URL of the news articlesource
: The source of the news article (e.g., “CNN”, “BBC”)publishedAt
: The date the article was publishedsnippet
: A brief snippet or summary of the article- Additional metadata about the article
Apify Integration
The tool uses the Apify Super Fast Google News Scraper actor, which provides several advantages:
- Scalability: The actor can handle large numbers of news searches efficiently
- Reliability: The actor is designed to handle rate limiting and other issues that can arise when scraping Google News
- Structured Data: The actor returns news articles in a structured format that is easy to process
- Freshness: The actor focuses on retrieving the latest news articles, ensuring that the information is up-to-date
Configuration
To use the Google News Scraper Tool, you need to set up the following environment variables:
Next Steps
- Explore the Researcher Agent that uses this tool
- Learn about the other tools used by the Newsletter AI Agent in the Tools Overview
- See how this tool contributes to the newsletter generation process