# Editor Agent
Source: https://newsletter-ai-agent.pratikdani.com/agents/editor

Learn about the Editor Agent in the Newsletter AI Agent

# Editor Agent

The Editor Agent is the final agent in the newsletter generation process, responsible for reviewing, refining, and ensuring the quality of the newsletter content. It takes the draft created by the Writer Agent and transforms it into a polished, publication-ready newsletter.

## Role and Responsibilities

The Editor Agent is defined with the following attributes:

* **Role**: Newsletter Editor
* **Goal**: Review, refine, and ensure the quality of the newsletter content
* **Backstory**: An experienced editor with expertise in technology publications, ensuring content is accurate, engaging, well-structured, and maintains a consistent style throughout.

## Implementation

The Editor Agent is implemented in `src/agents/editor.py` using CrewAI's Agent class:

```python theme={null}
@staticmethod
def create(llm) -> Agent:
    return Agent(
        role='Newsletter Editor',
        goal='Review, refine, and ensure the quality of the newsletter content',
        backstory="""You are an experienced editor with expertise in technology 
        publications. You ensure content is accurate, engaging, well-structured, 
        and maintains a consistent style throughout. You are also responsible for 
        ensuring the content is up to date and relevant to the latest trends in the 
        technology industry. Include all the links to the sources in the content.""",
        tools=[],  # No additional tools needed for editing
        verbose=True,
        allow_delegation=False,
        llm=llm
    )
```

Like the Writer Agent, the Editor Agent doesn't use external tools, relying instead on its language model capabilities to review and improve the content.

## Editing Process

When assigned a task, the Editor Agent follows this process:

1. **Review Content**: Examines the draft newsletter provided by the Writer Agent
2. **Check for Issues**: Identifies problems with formatting, structure, or content
3. **Improve Content**: Makes changes to enhance readability and engagement
4. **Ensure Consistency**: Maintains a consistent style throughout the newsletter
5. **Finalize Newsletter**: Adds metadata and finalizes the newsletter for publication

## Content Review

The Editor Agent reviews the content for several common issues:

* **Length**: Ensures the content is substantial enough
* **Markdown Formatting**: Checks for proper headers, links, and spacing
* **Header Hierarchy**: Ensures consistent and logical header levels
* **Link Integrity**: Verifies that all links are properly formatted
* **List Formatting**: Ensures proper formatting of list items

## Newsletter Finalization

The Editor Agent finalizes the newsletter by adding metadata and a consistent structure:

```markdown theme={null}
# AI Technology Newsletter
*Issue Date: March 7, 2025*

**Focus Topic:** AI Agents

## Executive Summary
Brief summary of the newsletter content...

[Main content sections...]

---
*This newsletter is automatically generated using AI technology.*
*For more information, please contact us.*
```

## Quality Assessment

The Editor Agent provides a quality score for the newsletter based on various factors:

* **Content Structure**: Proper organization and flow
* **Markdown Formatting**: Correct use of markdown elements
* **Link Integrity**: Properly formatted and working links
* **Spacing and Readability**: Appropriate spacing and formatting for readability

The quality score ranges from 0.0 to 1.0, with higher scores indicating better quality.

## Configuration

The Editor Agent can be configured by modifying its creation parameters in `src/agents/editor.py`:

* **LLM**: Change the language model used by the agent
* **Verbosity**: Set `verbose` to `True` or `False` to control the amount of output

## Next Steps

* Learn about the [Writer Agent](/agents/writer) that creates the draft newsletter
* Explore the [Researcher Agent](/agents/researcher) that gathers the information
* Understand the [overall agent architecture](/agents/overview) of the Newsletter AI Agent


# Agents Overview
Source: https://newsletter-ai-agent.pratikdani.com/agents/overview

Learn about the agent system used by the Newsletter AI Agent

# Agents Overview

The Newsletter AI Agent uses a multi-agent system powered by [CrewAI](https://github.com/joaomdmoura/crewAI) to generate newsletters. This system consists of three specialized agents that work together to research, write, and edit the newsletter content.

## Agent Architecture

The agent system is implemented using CrewAI, which provides a framework for creating and coordinating multiple agents. Each agent has a specific role, goal, and set of tools that it can use to accomplish its tasks.

### Integration with Apify

The agents use tools that leverage [Apify](https://apify.com) actors for web scraping and data collection. Apify provides a platform for running web scraping and automation tasks at scale, which allows the Newsletter AI Agent to gather information from various sources efficiently.

The integration with Apify is implemented through a base class in `src/tools/base.py` that provides a standardized way to call Apify actors and process their results:

```python theme={null}
class RunApifyActor:
    """Run an Apify actor and return the results."""
    def __init__(self, actor):
        self.actor = actor

    def _run(self, actor_name, run_input):
        # Implementation to run the Apify actor and return results
        # ...
```

## Agent Roles

The Newsletter AI Agent consists of three specialized agents:

### 1. Researcher Agent

The [Researcher Agent](/agents/researcher) is responsible for gathering information about the specified topic. It uses a set of tools to search the web, collect social media posts, find relevant videos, and gather news articles.

**Tools:**

* [Google Scraper Tool](/tools/google-search): Uses the Apify Google Search Scraper actor to search the web
* [Reddit Scraper Tool](/tools/reddit): Uses an Apify Reddit scraper actor to gather discussions
* [Twitter Scraper Tool](/tools/twitter): Uses the Apify Twitter Scraper Lite actor to collect tweets
* [YouTube Scraper Tool](/tools/youtube): Uses an Apify YouTube scraper actor to find videos
* [Google News Scraper Tool](/tools/google-news): Uses the Apify Super Fast Google News Scraper actor to gather news articles

### 2. Writer Agent

The [Writer Agent](/agents/writer) is responsible for transforming the research data into engaging newsletter content. It takes the information gathered by the Researcher Agent and creates a well-structured newsletter draft.

**Capabilities:**

* Content creation from research data
* Structuring the newsletter into sections
* Formatting the content in markdown
* Adding links to sources

### 3. Editor Agent

The [Editor Agent](/agents/editor) is responsible for reviewing and improving the newsletter draft. It ensures that the content is polished, error-free, and ready for distribution.

**Capabilities:**

* Grammar and spelling correction
* Style and tone consistency
* Fact-checking
* Formatting and layout improvement

## Agent Workflow

The agents work together in a sequential workflow to generate the newsletter:

1. The **Researcher Agent** gathers information about the specified topic
2. The **Writer Agent** transforms the research data into a newsletter draft
3. The **Editor Agent** reviews and improves the draft to create the final newsletter

This workflow is orchestrated by the `NewsletterCrew` class in `src/newsletter_crew.py`, which creates the agents, defines their tasks, and manages the flow of information between them.

## Agent Implementation

Each agent is implemented as a class in the `src/agents` directory:

* `src/agents/researcher.py`: Implements the Researcher Agent
* `src/agents/writer.py`: Implements the Writer Agent
* `src/agents/editor.py`: Implements the Editor Agent

The agents are created using CrewAI's `Agent` class, which provides a framework for defining an agent's role, goal, backstory, and tools.

## Configuration

The agent system can be configured through environment variables and configuration files:

* **Environment Variables**: Set API keys and other credentials in the `.env` file
* **Configuration Files**: Modify agent behavior in the `src/config` directory

## Next Steps

* Learn more about the [Researcher Agent](/agents/researcher)
* Explore the [Writer Agent](/agents/writer)
* Discover the [Editor Agent](/agents/editor)
* See how the agents contribute to the [newsletter generation process](/features/newsletter-generation)


# Researcher Agent
Source: https://newsletter-ai-agent.pratikdani.com/agents/researcher

Learn about the Researcher Agent in the Newsletter AI Agent

# Researcher Agent

The Researcher Agent is responsible for gathering comprehensive and accurate information about the specified topic from various sources. It serves as the foundation for the newsletter generation process, providing the raw material that the Writer Agent will transform into engaging content.

## Role and Responsibilities

The Researcher Agent is defined with the following attributes:

* **Role**: Research Specialist
* **Goal**: Gather comprehensive and accurate information about specified topics
* **Backstory**: An expert research specialist with a keen eye for detail and the ability to find the most relevant and up-to-date information, specializing in AI technology, industry trends, and market analysis.

## Tools

The Researcher Agent is equipped with several tools to gather information from different sources:

* **[Google Scraper Tool](/tools/google-search)**: Searches the web for relevant information
* **[Reddit Scraper Tool](/tools/reddit)**: Gathers discussions from relevant subreddits
* **[Twitter Scraper Tool](/tools/twitter)**: Collects tweets related to the topic
* **[YouTube Scraper Tool](/tools/youtube)**: Finds relevant video content
* **[Google News Scraper Tool](/tools/google-news)**: Gathers the latest news articles

These tools allow the agent to collect information from a wide range of sources, ensuring comprehensive coverage of the topic.

## Implementation

The Researcher Agent is implemented in `src/agents/researcher.py` using CrewAI's Agent class:

```python theme={null}
@staticmethod
def create(llm, actor) -> Agent:
    return Agent(
        role='Research Specialist',
        goal='Gather comprehensive and accurate information about specified topics',
        backstory="""You are an expert research specialist with a keen eye for detail 
        and the ability to find the most relevant and up-to-date information. You 
        specialize in AI technology, industry trends, and market analysis.""",
        tools=[
            GoogleScraperTool(actor=actor),
            RedditScraperTool(actor=actor),
            TwitterScraperTool(actor=actor),
            YouTubeScraperTool(actor=actor),
            GoogleNewsScraperTool(actor=actor)
        ],
        verbose=True,
        allow_delegation=False,
        llm=llm
    )
```

## Research Process

When assigned a task, the Researcher Agent follows this process:

1. **Search for Information**: Uses its tools to search for information related to the topic
2. **Filter and Organize**: Filters the information based on relevance, recency, and credibility
3. **Structure the Data**: Organizes the information into categories for easier processing by the Writer Agent
4. **Provide Sources**: Includes links to the original sources for reference

The agent's output is structured as a JSON object with sections for different types of content:

```json theme={null}
{
  "summary": "Brief overview of the topic",
  "key_points": ["Key point 1", "Key point 2", "..."],
  "sources": ["URL1", "URL2", "..."],
  "sections": {
    "Latest News": [{"title": "News Title", "description": "News Description", "url": "URL"}],
    "Community Discussions": [{"title": "Discussion Title", "text": "Discussion Text", "url": "URL"}],
    "Social Media Insights": [{"text": "Tweet Text", "author": "Author", "url": "URL"}],
    "Video Content": [{"title": "Video Title", "description": "Video Description", "url": "URL"}]
  }
}
```

## Configuration

The Researcher Agent can be configured by modifying its creation parameters in `src/agents/researcher.py`:

* **LLM**: Change the language model used by the agent
* **Tools**: Add or remove tools to change the sources of information
* **Verbosity**: Set `verbose` to `True` or `False` to control the amount of output

## Next Steps

* Learn about the [Writer Agent](/agents/writer) that transforms the research data into engaging content
* Explore the [tools](/tools/overview) that the Researcher Agent uses to gather information


# Writer Agent
Source: https://newsletter-ai-agent.pratikdani.com/agents/writer

Learn about the Writer Agent in the Newsletter AI Agent

# Writer Agent

The Writer Agent is responsible for transforming research data into engaging and informative newsletter content. It takes the raw information gathered by the Researcher Agent and crafts it into well-structured, readable sections that form the draft newsletter.

## Role and Responsibilities

The Writer Agent is defined with the following attributes:

* **Role**: Content Writer
* **Goal**: Create engaging and informative newsletter content from research materials
* **Backstory**: A skilled content writer specializing in technology and AI topics, excelling at transforming complex information into clear, engaging content that resonates with both technical and non-technical readers.

## Implementation

The Writer Agent is implemented in `src/agents/writer.py` using CrewAI's Agent class:

```python theme={null}
@staticmethod
def create(llm) -> Agent:
    return Agent(
        role='Content Writer',
        goal='Create engaging and informative newsletter content from research materials',
        backstory="""You are a skilled content writer specializing in technology 
        and AI topics. You excel at transforming complex information into clear, 
        engaging content that resonates with both technical and non-technical readers. 
        You are also responsible for ensuring the content is up to date and relevant to 
        the latest trends in the technology industry. Include all the links to the 
        sources in the content.""",
        tools=[],  # No additional tools needed for content writing
        verbose=True,
        allow_delegation=False,
        llm=llm
    )
```

Unlike the Researcher Agent, the Writer Agent doesn't use external tools. Instead, it relies on its language model capabilities to transform the research data into engaging content.

## Content Creation Process

When assigned a task, the Writer Agent follows this process:

1. **Analyze Research Data**: Reviews the information provided by the Researcher Agent
2. **Structure Content**: Organizes the content into logical sections based on the research data
3. **Write Engaging Sections**: Creates well-written, engaging content for each section
4. **Format in Markdown**: Ensures proper markdown formatting for headings, links, and other elements
5. **Include Sources**: Incorporates links to original sources throughout the content

## Section Formatting

The Writer Agent formats different types of content according to their nature:

### News Section

```markdown theme={null}
### [News Title](URL)
*Published: Date*

News description text...
```

### Community Discussions

```markdown theme={null}
### [Discussion Title](URL)
*Posted by Author*

Discussion text...
```

### Social Media Insights

```markdown theme={null}
> Tweet text
*— Author*
[View on Twitter](URL)
```

### Video Content

```markdown theme={null}
### [Video Title](URL)
*By Channel*

Video description...
```

## Markdown Formatting

The Writer Agent ensures proper markdown formatting through several helper methods:

* **format\_markdown**: Cleans up the content and ensures proper spacing
* **\_format\_news\_section**: Formats news items into markdown content
* **\_format\_community\_section**: Formats community discussions into markdown content
* **\_format\_social\_section**: Formats social media content into markdown content
* **\_format\_video\_section**: Formats video content into markdown content
* **\_format\_general\_section**: Formats general content into markdown content

## Configuration

The Writer Agent can be configured by modifying its creation parameters in `src/agents/writer.py`:

* **LLM**: Change the language model used by the agent
* **Verbosity**: Set `verbose` to `True` or `False` to control the amount of output

## Next Steps

* Learn about the [Editor Agent](/agents/editor) that reviews and finalizes the newsletter
* Explore the [Researcher Agent](/agents/researcher) that provides the raw material for the Writer Agent


# Actor API
Source: https://newsletter-ai-agent.pratikdani.com/api-reference/actor-api

Technical details of the Newsletter AI Agent Apify Actor API

# Newsletter AI Agent Actor API

This document provides technical details about the Newsletter AI Agent Apify Actor API, including input schema, output format, and usage examples.

## Actor Input Schema

The Newsletter AI Agent Actor accepts the following input parameters:

```json theme={null}
{
  "topic": "string"
}
```

### Parameters

| Parameter | Type   | Description                  | Required | Default                                                                                                                       |
| --------- | ------ | ---------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `topic`   | string | The topic for the newsletter | No       | "I want to know everything about AI agents – current news, AI agentic platforms and frameworks, and companies in this field." |

## Actor Output Schema

The actor outputs the generated newsletter in the following format:

```json theme={null}
{
  "topic": "string",
  "content": "string",
  "status": "string",
  "timestamp": "string"
}
```

### Output Fields

| Field       | Type   | Description                                                       |
| ----------- | ------ | ----------------------------------------------------------------- |
| `topic`     | string | The topic that was used to generate the newsletter                |
| `content`   | string | The generated newsletter content in Markdown format               |
| `status`    | string | The status of the generation process ("success" or "error")       |
| `timestamp` | string | ISO 8601 formatted timestamp of when the newsletter was generated |

## Error Output Schema

If an error occurs during newsletter generation, the actor will output:

```json theme={null}
{
  "topic": "string",
  "error": "string",
  "status": "error",
  "timestamp": "string"
}
```

### Error Fields

| Field       | Type   | Description                                             |
| ----------- | ------ | ------------------------------------------------------- |
| `topic`     | string | The topic that was used (if available)                  |
| `error`     | string | Error message describing what went wrong                |
| `status`    | string | Always "error" for error outputs                        |
| `timestamp` | string | ISO 8601 formatted timestamp of when the error occurred |

## Usage Examples

### Running the Actor via Apify Console

1. Navigate to the Newsletter AI Agent on the Apify platform
2. Set the `topic` input parameter to your desired topic
3. Click "Run" to start the actor
4. Wait for the actor to complete
5. View the results in the "Dataset" tab

### Running the Actor via Apify API Client (Python)

```python theme={null}
import os
from apify_client import ApifyClient

# Initialize the ApifyClient with your API token
client = ApifyClient(os.getenv('APIFY_API_KEY'))

# Start the actor and wait for it to finish
run = client.actor('your-username/newsletter-ai-agent').call({
    'topic': 'Latest advancements in quantum computing'
})

# Fetch the actor's output
output = client.dataset(run['defaultDatasetId']).list_items().items[0]
newsletter_content = output['content']

# Do something with the newsletter content
print(newsletter_content)
```

### Running the Actor via Apify API Client (JavaScript)

```javascript theme={null}
const { ApifyClient } = require('apify-client');

// Initialize the ApifyClient with your API token
const client = new ApifyClient({
    token: process.env.APIFY_API_KEY,
});

// Start the actor and wait for it to finish
const run = await client.actor('your-username/newsletter-ai-agent').call({
    topic: 'Latest advancements in quantum computing',
});

// Fetch the actor's output
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const newsletterContent = items[0].content;

// Do something with the newsletter content
console.log(newsletterContent);
```

## Actor Lifecycle

1. The actor initializes and loads environment variables
2. It processes the input parameters (using default if none provided)
3. It creates a NewsletterCrew instance with the necessary agents
4. The Researcher Agent gathers information about the topic
5. The Writer Agent transforms the research into a newsletter draft
6. The Editor Agent reviews and improves the draft
7. The final newsletter is returned as output

## Resource Usage

The Newsletter AI Agent performs web scraping and uses large language models, which can be resource-intensive. The actor typically takes 3-5 minutes to generate a newsletter, depending on the complexity of the topic and the amount of information available.

## Next Steps

* Learn about the [agents](/agents/overview) that power the Newsletter AI Agent
* Explore the [tools](/tools/overview) used by the agents
* See how the Newsletter AI Agent [generates newsletters](/features/newsletter-generation)


# Apify Integration
Source: https://newsletter-ai-agent.pratikdani.com/api-reference/apify-integration

Learn how to use the Newsletter AI Agent with Apify

# Apify Integration

The Newsletter AI Agent is implemented as an [Apify Actor](https://apify.com/docs/actors), which means it runs on the Apify platform and can be accessed through Apify's API.

## Setting Up Apify

To use the Newsletter AI Agent, you'll need an Apify account and API key:

1. [Sign up](https://console.apify.com/sign-up) for an Apify account if you don't have one
2. Navigate to your [Account Settings](https://console.apify.com/account#/integrations) to find your API key
3. Store your API key securely

## Required Environment Variables

The Newsletter AI Agent requires the following environment variables:

```bash theme={null}
# In your .env file
APIFY_API_KEY=your_apify_api_key_here
GOOGLE_API_KEY=your_google_api_key_here
```

## Running the Actor

You can run the Newsletter AI Agent in several ways:

### Using the Apify Console

1. Navigate to the Newsletter AI Agent on the Apify platform
2. Configure the input parameters (topic for the newsletter)
3. Click "Run" to start the actor

### Using the Apify API

You can also run the actor programmatically using the Apify API:

```python theme={null}
import os
from apify_client import ApifyClient

# Initialize the ApifyClient with your API token
client = ApifyClient(os.getenv('APIFY_API_KEY'))

# Start the actor and wait for it to finish
run = client.actor('your-username/newsletter-ai-agent').call({
    'topic': 'AI advancements in healthcare'
})

# Fetch the actor's output
output = client.dataset(run['defaultDatasetId']).list_items().items[0]
newsletter_content = output['content']
print(newsletter_content)
```

## Input Parameters

The Newsletter AI Agent accepts the following input parameters:

| Parameter | Type   | Description                  | Required |
| --------- | ------ | ---------------------------- | -------- |
| `topic`   | string | The topic for the newsletter | Yes      |

## Output Format

The actor outputs the generated newsletter in Markdown format, along with metadata:

```json theme={null}
{
  "topic": "AI advancements in healthcare",
  "content": "# AI in Healthcare Newsletter\n\n...",
  "status": "success",
  "timestamp": "2023-01-01T12:00:00Z"
}
```

## Error Handling

If an error occurs during newsletter generation, the actor will output an error message:

```json theme={null}
{
  "topic": "AI advancements in healthcare",
  "error": "Error in newsletter generation: ...",
  "status": "error",
  "timestamp": "2023-01-01T12:00:00Z"
}
```

## Rate Limiting

Usage of the Newsletter AI Agent is subject to Apify's rate limiting and pricing. Please refer to the [Apify pricing page](https://apify.com/pricing) for more information.

## Next Steps

* Learn about the [agents](/agents/overview) that power the Newsletter AI Agent
* Explore the [tools](/tools/overview) used by the agents
* See how the Newsletter AI Agent [generates newsletters](/features/newsletter-generation)


# Newsletter Generation
Source: https://newsletter-ai-agent.pratikdani.com/features/newsletter-generation

Learn how the Newsletter AI Agent generates newsletters

# Newsletter Generation

The Newsletter AI Agent uses a sophisticated process powered by CrewAI to generate high-quality newsletters about specific topics. This page explains the newsletter generation process in detail.

## Generation Process

The newsletter generation process follows these steps:

1. **Topic Specification**: The user specifies a topic of interest
2. **Research**: The [Researcher Agent](/agents/researcher) gathers comprehensive information about the topic
3. **Writing**: The [Writer Agent](/agents/writer) transforms the research data into engaging newsletter content
4. **Editing**: The [Editor Agent](/agents/editor) reviews and finalizes the newsletter
5. **Output**: The final newsletter is returned to the user

## CrewAI Workflow

The newsletter generation process is implemented using CrewAI, which orchestrates the agents and their tasks. The workflow is defined in the `NewsletterCrew` class:

```python theme={null}
def generate_newsletter(self, topic: str) -> str:
    # Create tasks for each agent
    research_task = Task(
        description=f"Research the topic '{topic}' and gather comprehensive information.",
        expected_output="Detailed research data in JSON format.",
        agent=self.researcher
    )
    
    writing_task = Task(
        description="Transform the research data into engaging newsletter content.",
        expected_output="Draft newsletter in markdown format.",
        agent=self.writer,
        context=[research_task]
    )
    
    editing_task = Task(
        description="Review and finalize the newsletter content.",
        expected_output="Final newsletter in markdown format.",
        agent=self.editor,
        context=[writing_task]
    )
    
    # Execute the tasks
    result = self.crew.kickoff(tasks=[research_task, writing_task, editing_task])
    
    return result
```

This workflow ensures that each agent builds upon the work of the previous one, creating a cohesive and high-quality newsletter.

## Agent Interactions

The agents interact with each other through the tasks' context. Each task has access to the output of its context tasks, allowing agents to build upon each other's work:

1. The Researcher Agent performs its task independently, gathering information about the topic
2. The Writer Agent receives the Researcher Agent's output as context for its task
3. The Editor Agent receives the Writer Agent's output as context for its task

This sequential process ensures that each agent has the information it needs to perform its task effectively.

## Customization Options

The newsletter generation process can be customized in several ways:

### Topic Customization

The most basic customization is specifying the topic of interest:

```python theme={null}
from src.newsletter_crew import NewsletterCrew

crew = NewsletterCrew()
newsletter = crew.generate_newsletter("Artificial Intelligence")
```

### Section Customization

You can customize the sections included in the newsletter by modifying the `DEFAULT_NEWSLETTER_SECTIONS` in `src/config/config.py`:

```python theme={null}
DEFAULT_NEWSLETTER_SECTIONS = [
    "Latest News",
    "Industry Updates",
    "Tools & Frameworks",
    "Companies & Startups",
    "Research & Development",
    "Community Discussions",
    "Video Content",
    "Podcasts",
]
```

### LLM Customization

You can customize the language model used by the agents by modifying the `LLM` initialization in `src/newsletter_crew.py`:

```python theme={null}
self.llm = LLM(
    model="gemini/gemini-2.0-flash-lite",  # Change to another supported model
    temperature=0.7,  # Adjust for more or less creativity
    api_key=os.getenv("GOOGLE_API_KEY"),
    verbose=False  # Set to True for more detailed output
)
```

## Output Format

The newsletter is generated in markdown format, which can be easily converted to HTML, PDF, or other formats. The markdown format includes:

* **Headers**: For section titles and article titles
* **Links**: For references to sources
* **Formatting**: For emphasis, lists, and other styling
* **Images**: For thumbnails and other visual elements

Here's an example of the output format:

```markdown theme={null}
# AI Technology Newsletter
*Issue Date: March 7, 2025*

**Focus Topic:** Artificial Intelligence

## Executive Summary
A brief summary of the latest developments in AI technology...

## Latest News
### [Google Announces New AI Model](https://example.com/news/1)
*Published: March 5, 2025*

Google has announced a new AI model that achieves state-of-the-art results on several benchmarks...

## Industry Updates
### [AI Adoption in Healthcare](https://example.com/news/2)
*Published: March 3, 2025*

Healthcare organizations are increasingly adopting AI technologies to improve patient care...

## Tools & Frameworks
### [New Version of TensorFlow Released](https://example.com/news/3)
*Published: March 1, 2025*

Google has released a new version of TensorFlow with improved performance and new features...

---
*This newsletter is automatically generated using AI technology.*
*For more information, please contact us.*
```


# Features Overview
Source: https://newsletter-ai-agent.pratikdani.com/features/overview

Explore the features of the Newsletter AI Agent

# Features Overview

The Newsletter AI Agent offers a range of features powered by CrewAI to help you generate high-quality newsletters about specific topics.

## Core Features

### Agent-Based Architecture

The Newsletter AI Agent uses a multi-agent architecture powered by CrewAI:

* **[Researcher Agent](/agents/researcher)**: Gathers comprehensive information about the specified topic
* **[Writer Agent](/agents/writer)**: Transforms research data into engaging newsletter content
* **[Editor Agent](/agents/editor)**: Reviews and finalizes the newsletter

This agent-based approach allows each agent to specialize in a specific aspect of the newsletter creation process, resulting in better overall quality.

### Custom Tools

The agents use custom tools to gather information from various sources:

* **[Google Search Tool](/tools/google-search)**: Searches the web for relevant information
* **[Reddit Tool](/tools/reddit)**: Gathers discussions from relevant subreddits
* **[Twitter Tool](/tools/twitter)**: Collects tweets related to the topic
* **[YouTube Tool](/tools/youtube)**: Finds relevant video content
* **[Google News Tool](/tools/google-news)**: Gathers the latest news articles

### Automated Newsletter Generation

The Newsletter AI Agent automates the entire newsletter generation process:

1. **Topic Specification**: You specify a topic of interest
2. **Research**: The Researcher Agent gathers comprehensive information
3. **Writing**: The Writer Agent transforms the research data into engaging content
4. **Editing**: The Editor Agent reviews and finalizes the newsletter
5. **Output**: The final newsletter is returned to you

### Markdown Formatting

The Newsletter AI Agent generates newsletters in markdown format, which can be easily converted to HTML, PDF, or other formats. The markdown format includes:

* **Headers**: For section titles and article titles
* **Links**: For references to sources
* **Formatting**: For emphasis, lists, and other styling
* **Images**: For thumbnails and other visual elements

## Advanced Features

### Customizable Sections

You can customize the sections included in your newsletters by modifying the `DEFAULT_NEWSLETTER_SECTIONS` in `src/config/config.py`:

```python theme={null}
DEFAULT_NEWSLETTER_SECTIONS = [
    "Latest News",
    "Industry Updates",
    "Tools & Frameworks",
    "Companies & Startups",
    "Research & Development",
    "Community Discussions",
    "Video Content",
    "Podcasts",
]
```

### LLM Customization

You can customize the language model used by the agents by modifying the `LLM` initialization in `src/newsletter_crew.py`:

```python theme={null}
self.llm = LLM(
    model="gemini/gemini-2.0-flash-lite",  # Change to another supported model
    temperature=0.7,  # Adjust for more or less creativity
    api_key=os.getenv("GOOGLE_API_KEY"),
    verbose=False  # Set to True for more detailed output
)
```

### CrewAI Integration

The Newsletter AI Agent is built on top of CrewAI, a framework for orchestrating role-playing AI agents. This integration provides:

* **Task Management**: CrewAI manages the tasks assigned to each agent
* **Agent Coordination**: CrewAI coordinates the interactions between agents
* **Context Sharing**: CrewAI enables agents to share context and build upon each other's work

### Extensibility

The Newsletter AI Agent is designed to be extensible:

* **Add New Agents**: You can add new agents to the crew to handle additional tasks
* **Add New Tools**: You can create new tools to gather information from additional sources
* **Customize Workflows**: You can modify the workflow to include additional steps or change the order of existing steps

## Next Steps

* Learn more about the [Newsletter Generation](/features/newsletter-generation) process
* Explore the [Agents](/agents/overview) that power the Newsletter AI Agent
* Check out the [Tools](/tools/overview) used to gather information


# Getting Started
Source: https://newsletter-ai-agent.pratikdani.com/getting-started

Learn how to use the Newsletter AI Agent

# Getting Started

This guide will help you set up and start using the Newsletter AI Agent powered by CrewAI.

## Installation

To use the Newsletter AI Agent, you need to:

1. Clone the repository:

```bash theme={null}
git clone https://github.com/pratik-dani/newsletter-ai-agent.git
cd newsletter-ai-agent
```

2. Install the required dependencies:

```bash theme={null}
pip install -r requirements.txt
```

3. Set up your environment variables:

```bash theme={null}
cp .env.example .env
```

Edit the `.env` file to include your API keys and other configuration options:

```
# Required API keys
GOOGLE_API_KEY=your_google_api_key_here  # Required for the LLM and search tools
APIFY_API_KEY=your_apify_api_key_here    # Required for web scraping tools
```

## Basic Usage

1. Define your topic of interest in the `input.json` file:

```json theme={null}
{
    "topic": "Your topic of interest"
}
```

2. Run the agent:

```bash theme={null}
python src/main.py
```

3. Find your generated newsletter in the output directory.

## CrewAI Configuration

The Newsletter AI Agent uses CrewAI to orchestrate a team of specialized agents. The main configuration is in `src/newsletter_crew.py`:

```python theme={null}
# Initialize the LLM
self.llm = LLM(
    model="gemini/gemini-2.0-flash-lite",  # You can change this to another supported model
    temperature=0.7,
    api_key=os.getenv("GOOGLE_API_KEY"),
    verbose=False
)

# Create the crew with agents
self.crew = Crew(
    agents=[self.researcher, self.writer, self.editor],
    tasks=[],  # Tasks are added dynamically when generating a newsletter
    verbose=True  # Set to False to reduce console output
)
```

You can customize the following aspects:

* **LLM Model**: Change the `model` parameter to use a different language model
* **Temperature**: Adjust the `temperature` parameter to control creativity vs. determinism
* **Verbosity**: Set `verbose` to `True` or `False` to control the amount of output

## Customizing Newsletter Content

You can customize the content of your newsletters by modifying the `DEFAULT_NEWSLETTER_SECTIONS` in `src/config/config.py`:

```python theme={null}
DEFAULT_NEWSLETTER_SECTIONS = [
    "Latest News",
    "Industry Updates",
    "Tools & Frameworks",
    "Companies & Startups",
    "Research & Development",
    "Community Discussions",
    "Video Content",
    "Podcasts",
]
```

## Next Steps

* Learn more about the [Agents](/agents/overview) that power the Newsletter AI Agent
* Explore the [Custom Tools](/tools/overview) used to gather information
* Check out the [Features](/features/overview) section to learn more about what the Newsletter AI Agent can do


# Introduction
Source: https://newsletter-ai-agent.pratikdani.com/introduction

Welcome to the Newsletter AI Agent documentation

# Newsletter AI Agent

The Newsletter AI Agent is a powerful tool designed to automatically generate newsletters about specific topics. It leverages CrewAI to orchestrate a team of specialized AI agents that work together to gather relevant information, organize it, and create a well-structured newsletter ready for distribution.

## Overview

This agent helps you stay informed about your areas of interest by:

1. Collecting the latest information on your specified topics
2. Organizing the content in a readable format
3. Generating a complete newsletter ready for distribution

## Key Features

* **Topic-based content gathering**: Specify your interests and let the agent find relevant information
* **Automated newsletter generation**: Transform raw information into well-structured newsletters
* **Customizable outputs**: Adjust the format and style of your newsletters
* **Agent-based architecture**: Leverages CrewAI to coordinate specialized agents for research, writing, and editing

## Codebase Architecture

The Newsletter AI Agent is built as an Apify Actor with a multi-agent system powered by CrewAI. Below is a diagram of the codebase architecture:

```mermaid theme={null}
graph TD
    subgraph "User Interface"
        A[Apify Actor Input] --> B[main.py]
    end

    subgraph "Core System"
        B --> C[NewsletterCrew]
        C --> D[Agent Orchestration]
    end

    subgraph "Agents"
        D --> E[Researcher Agent]
        D --> F[Writer Agent]
        D --> G[Editor Agent]
    end

    subgraph "Tools"
        E --> H[Google Search Tool]
        E --> I[Reddit Tool]
        E --> J[Twitter Tool]
        E --> K[YouTube Tool]
        E --> L[Google News Tool]
    end

    subgraph "Apify Integration"
        H --> M[Apify Actors]
        I --> M
        J --> M
        K --> M
        L --> M
        M --> N[Web Data]
    end

    subgraph "Output"
        G --> O[Newsletter Content]
        O --> P[Apify Actor Output]
    end

    classDef default fill:#F3F4F6,stroke:#D1D5DB,color:#1F2937
    classDef core fill:#E5E7EB,stroke:#9CA3AF,color:#111827
    classDef highlight fill:#DBEAFE,stroke:#93C5FD,color:#1E40AF
    classDef agents fill:#FCE7F3,stroke:#F9A8D4,color:#9D174D
    classDef tools fill:#D1FAE5,stroke:#6EE7B7,color:#065F46
    classDef apify fill:#E0F2FE,stroke:#7DD3FC,color:#0C4A6E

    class B,C,D core;
    class E,F,G agents;
    class H,I,J,K,L tools;
    class M,N apify;
```

### Key Components

1. **Entry Point**: `main.py` serves as the entry point for the Apify Actor, handling input and initializing the system.

2. **Newsletter Crew**: The `NewsletterCrew` class orchestrates the agents and manages the workflow.

3. **Agents**:
   * `ResearcherAgent`: Gathers information using various tools
   * `WriterAgent`: Transforms research into newsletter content
   * `EditorAgent`: Reviews and improves the final output

4. **Tools**: Each tool is implemented as a CrewAI `BaseTool` that interacts with Apify actors:
   * `GoogleSearchTool`: Searches the web using Apify's Google Search Scraper
   * `RedditTool`: Gathers discussions using a Reddit scraper
   * `TwitterTool`: Collects tweets using Twitter Scraper Lite
   * `YouTubeTool`: Finds videos using a YouTube scraper
   * `GoogleNewsTool`: Gathers news using Google News Scraper

5. **Apify Integration**: The `RunApifyActor` base class provides a standardized way to call Apify actors and process their results.

## CrewAI Implementation

The Newsletter AI Agent uses [CrewAI](https://docs.crewai.com), a framework for orchestrating role-playing AI agents. Our implementation includes:

* **Researcher Agent**: Gathers comprehensive information about the specified topic from various sources
* **Writer Agent**: Transforms research data into engaging newsletter content
* **Editor Agent**: Reviews, improves, and finalizes the newsletter for publication

These agents work together in a sequential process, with each agent building upon the work of the previous one to create a high-quality newsletter.

## Custom Tools

The agents use custom tools to interact with external data sources:

* **Google Search Tool**: Searches the web for relevant information
* **Reddit Tool**: Gathers discussions from relevant subreddits
* **Twitter Tool**: Collects tweets related to the topic
* **YouTube Tool**: Finds relevant video content
* **Google News Tool**: Gathers the latest news articles

## Getting Started

Check out the [Getting Started](/getting-started) guide to begin using the Newsletter AI Agent.


# Google News Scraper Tool
Source: https://newsletter-ai-agent.pratikdani.com/tools/google-news

Learn about the Google News Scraper Tool used by the Newsletter AI Agent

# Google News Scraper Tool

The Google News Scraper Tool is a custom tool that allows the Newsletter AI Agent to gather the latest news articles related to a specified topic using the [Apify Super Fast Google News Scraper](https://apify.com/aymorato/super-fast-google-news-scraper-pay-per-result) actor.

## Overview

The Google News Scraper Tool is primarily used by the [Researcher Agent](/agents/researcher) to gather the latest news and developments about the specified topic. It provides a flexible interface for searching Google News and extracting structured data from news articles.

## Implementation

The Google News Scraper Tool is implemented as a CrewAI `BaseTool` that interacts with the Apify Super Fast Google News Scraper actor. Here's the implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional, Literal
from apify import Actor
from src.tools.base import RunApifyActor

class GoogleNewsScraperInput(BaseModel):
    """Input schema for GoogleNewsScraper tool."""
    keywords: List[str] = Field(
        description="The keywords used to search for news articles"
    )

    maxItems: Optional[int] = Field(
        description="Set the maximum number of items you want to scrape for each keyword. If left unset, the actor will extract all available news.",
        default=20
    )

class GoogleNewsScraperTool(BaseTool):
    name: str = "Google News Scraper"
    description: str = "Tool for scraping Google News articles with configurable parameters"
    args_schema: type[BaseModel] = GoogleNewsScraperInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)

    def _run(
        self,
        keywords: List[str],
        language: Optional[str] = "US:en",
        maxItems: Optional[int] = None
    ) -> str:
        run_inputs = {
            "keywords": keywords,
            "language": language
        }
        
        if maxItems:
            run_inputs["maxItems"] = maxItems

        proxy = {
            "useApifyProxy": True,
            "apifyProxyGroups": [
                "RESIDENTIAL"
            ]
        }
        run_inputs["proxy"] = proxy
        
        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("aymorato/super-fast-google-news-scraper-pay-per-result", run_inputs)
        return dataset
```

## Parameters

The Google News Scraper Tool accepts the following parameters:

| Parameter  | Type       | Description                                              | Default  |
| ---------- | ---------- | -------------------------------------------------------- | -------- |
| `keywords` | List\[str] | The keywords used to search for news articles            | Required |
| `language` | str        | Language and country code for the search (e.g., "US:en") | "US:en"  |
| `maxItems` | int        | Maximum number of items to scrape for each keyword       | 20       |

## Usage

The Google News Scraper Tool is used by the Researcher Agent to gather the latest news about the specified topic:

```python theme={null}
# Initialize the tool
news_tool = GoogleNewsScraperTool(actor=actor)

# Use the tool
news_results = news_tool._run(
    keywords=[topic],
    language="US:en",
    maxItems=20
)
```

## Return Value

The tool returns a list of news articles, where each article is a dictionary containing information about the article, including:

* `title`: The title of the news article
* `link`: The URL of the news article
* `source`: The source of the news article (e.g., "CNN", "BBC")
* `publishedAt`: The date the article was published
* `snippet`: A brief snippet or summary of the article
* Additional metadata about the article

## Apify Integration

The tool uses the Apify Super Fast Google News Scraper actor, which provides several advantages:

1. **Scalability**: The actor can handle large numbers of news searches efficiently
2. **Reliability**: The actor is designed to handle rate limiting and other issues that can arise when scraping Google News
3. **Structured Data**: The actor returns news articles in a structured format that is easy to process
4. **Freshness**: The actor focuses on retrieving the latest news articles, ensuring that the information is up-to-date

## Configuration

To use the Google News Scraper Tool, you need to set up the following environment variables:

```
APIFY_API_KEY=your_apify_api_key_here
```

## Next Steps

* Explore the [Researcher Agent](/agents/researcher) that uses this tool
* Learn about the other tools used by the Newsletter AI Agent in the [Tools Overview](/tools/overview)
* See how this tool contributes to the [newsletter generation process](/features/newsletter-generation)


# Google Search Tool
Source: https://newsletter-ai-agent.pratikdani.com/tools/google-search

Learn about the Google Search Tool used by the Newsletter AI Agent

# Google Search Tool

The Google Search Tool is a custom tool that allows the Newsletter AI Agent to search the web for relevant information using the [Apify Google Search Scraper](https://apify.com/apify/google-search-scraper) actor.

## Overview

The Google Search Tool is primarily used by the [Researcher Agent](/agents/researcher) to gather general information about the specified topic. It provides a flexible interface for searching Google and extracting structured data from search results.

## Implementation

The Google Search Tool is implemented as a CrewAI `BaseTool` that interacts with the Apify Google Search Scraper actor. Here's the implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional
from apify import Actor
from src.tools.base import RunApifyActor

class GoogleScraperInput(BaseModel):
    """Input schema for GoogleScraper tool."""
    queries: List[str] = Field(
        description="Search terms or Google Search URLs. Can use advanced techniques like 'AI site:twitter.com'. Limit 32 words per query."
    )

    resultsPerPage: Optional[int] = Field(
        description="Number of results to return per page",
        default=10
    )

    languageCode: Optional[str] = Field(
        default="en",
        description="Language for search results (passed as hl parameter)"
    )
    
    # Additional parameters...

class GoogleScraperTool(BaseTool):
    name: str = "Google Scraper"
    description: str = "Tool for scraping Google search results with configurable parameters"
    args_schema: type[BaseModel] = GoogleScraperInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)
    
    def _run(
        self,
        queries: List[str],
        resultsPerPage: Optional[int] = 10,
        languageCode: Optional[str] = "en",
        # Additional parameters...
    ) -> str:
        run_inputs = {
            "queries": "\n".join(queries)
        }
        
        # Set additional parameters...
        
        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("apify/google-search-scraper", run_inputs)
        return dataset
```

## Parameters

The Google Search Tool accepts the following parameters:

| Parameter                  | Type       | Description                                           | Default  |
| -------------------------- | ---------- | ----------------------------------------------------- | -------- |
| `queries`                  | List\[str] | Search terms or Google Search URLs                    | Required |
| `resultsPerPage`           | int        | Number of results to return per page                  | 10       |
| `languageCode`             | str        | Language for search results                           | "en"     |
| `forceExactMatch`          | bool       | Wrap query in quotes for exact phrase matching        | False    |
| `site`                     | str        | Limit search to specific site (e.g. site:example.com) | None     |
| `relatedToSite`            | str        | Filter pages related to specific site                 | None     |
| `wordsInTitle`             | List\[str] | Filter pages with specific words in title             | \[]      |
| `wordsInText`              | List\[str] | Filter pages with specific words in text              | \[]      |
| `wordsInUrl`               | List\[str] | Filter pages with specific words in URL               | \[]      |
| `quickDateRange`           | str        | Filter by date range (e.g. d10, w2, m6, y1)           | "d30"    |
| `beforeDate`               | str        | Filter results before date (YYYY-MM-DD)               | None     |
| `afterDate`                | str        | Filter results after date (YYYY-MM-DD)                | None     |
| `fileTypes`                | List\[str] | Filter by file types                                  | \[]      |
| `mobileResults`            | bool       | Return mobile version of search results               | False    |
| `includeUnfilteredResults` | bool       | Include lower quality results                         | False    |

## Usage

The Google Search Tool is used by the Researcher Agent to gather information about the specified topic:

```python theme={null}
# Initialize the tool
google_tool = GoogleScraperTool(actor=actor)

# Use the tool
search_params = {
    "queries": [topic],
    "resultsPerPage": 5,
    "maxPagesPerQuery": 2,
    "languageCode": "en",
    "quickDateRange": "m1"  # Last month
}

web_results = google_tool._run(**search_params)
```

## Return Value

The tool returns a list of search results, where each result is a dictionary containing information about a search result, including:

* `title`: The title of the search result
* `url`: The URL of the search result
* `description`: A snippet of text from the search result
* `position`: The position of the result in the search results
* Additional metadata about the search result

## Apify Integration

The tool uses the Apify Google Search Scraper actor, which provides several advantages:

1. **Scalability**: The actor can handle large numbers of search queries efficiently
2. **Reliability**: The actor is designed to handle rate limiting and other issues that can arise when scraping search results
3. **Structured Data**: The actor returns search results in a structured format that is easy to process

## Configuration

To use the Google Search Tool, you need to set up the following environment variables:

```
APIFY_API_KEY=your_apify_api_key_here
```

## Next Steps

* Learn about the [Reddit Scraper Tool](/tools/reddit)
* Explore the [Researcher Agent](/agents/researcher) that uses this tool
* See how this tool contributes to the [newsletter generation process](/features/newsletter-generation)


# Tools Overview
Source: https://newsletter-ai-agent.pratikdani.com/tools/overview

Learn about the custom tools used by the Newsletter AI Agent

# Tools Overview

The Newsletter AI Agent uses a set of custom tools powered by [Apify](https://docs.apify.com) actors to gather information from various sources. These tools are primarily used by the [Researcher Agent](/agents/researcher) to collect comprehensive data about the specified topic.

## Tool Architecture

The tools are implemented using CrewAI's `BaseTool` class and Apify's actor system. Each tool is designed to interact with a specific Apify actor to gather information from a particular source, such as Google Search, Reddit, Twitter, YouTube, or Google News.

### Base Implementation

All tools share a common base implementation in `src/tools/base.py` that handles the interaction with Apify actors:

```python theme={null}
class RunApifyActor:
    """Run an Apify actor and return the results."""
    def __init__(self, actor):
        self.actor = actor

    def _run(self, actor_name, run_input):
        # Implementation to run the Apify actor and return results
        # ...
```

This base class provides a standardized way to call Apify actors and process their results.

## Available Tools

The Newsletter AI Agent includes the following tools:

1. **[Google Scraper Tool](/tools/google-search)**: Uses the `apify/google-search-scraper` actor to search the web for relevant information
2. **[Reddit Scraper Tool](/tools/reddit)**: Uses a Reddit scraper actor to gather discussions from relevant subreddits
3. **[Twitter Scraper Tool](/tools/twitter)**: Uses the `apidojo/twitter-scraper-lite` actor to collect tweets related to the topic
4. **[YouTube Scraper Tool](/tools/youtube)**: Uses a YouTube scraper actor to find relevant video content
5. **[Google News Scraper Tool](/tools/google-news)**: Uses the `aymorato/super-fast-google-news-scraper-pay-per-result` actor to gather the latest news articles

## Tool Implementation

Each tool follows a similar pattern:

1. Define an input schema using Pydantic models
2. Create a tool class that inherits from `BaseTool`
3. Implement the `_run` method to call the appropriate Apify actor

Here's a general pattern for tool implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional
from apify import Actor
from src.tools.base import RunApifyActor

class CustomToolInput(BaseModel):
    """Input schema for the tool."""
    # Define input parameters with descriptions
    param1: List[str] = Field(description="Description of parameter 1")
    param2: Optional[int] = Field(default=10, description="Description of parameter 2")

class CustomTool(BaseTool):
    name: str = "Tool Name"
    description: str = "Tool description"
    args_schema: type[BaseModel] = CustomToolInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)

    def _run(self, param1, param2=10, **kwargs):
        # Prepare input for the Apify actor
        run_inputs = {
            "param1": param1,
            "param2": param2
        }
        
        # Run the Apify actor
        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("apify/actor-name", run_inputs)
        return dataset
```

## Apify Integration

The tools use the Apify Python SDK to interact with Apify actors. This requires an Apify API key, which should be set in the `.env` file:

```
APIFY_API_KEY=your_apify_api_key_here
```

The Apify actors provide powerful web scraping and data extraction capabilities without requiring complex infrastructure setup.

## Tool Usage

Tools are assigned to the Researcher Agent during agent creation:

```python theme={null}
@staticmethod
def create(llm, actor) -> Agent:
    return Agent(
        role='Research Specialist',
        goal='Gather comprehensive and accurate information about specified topics',
        backstory="...",
        tools=[
            GoogleScraperTool(actor=actor),
            RedditScraperTool(actor=actor),
            TwitterScraperTool(actor=actor),
            YouTubeScraperTool(actor=actor),
            GoogleNewsScraperTool(actor=actor)
        ],
        verbose=True,
        allow_delegation=False,
        llm=llm
    )
```

## Next Steps

Explore each tool in detail:

* [Google Scraper Tool](/tools/google-search)
* [Reddit Scraper Tool](/tools/reddit)
* [Twitter Scraper Tool](/tools/twitter)
* [YouTube Scraper Tool](/tools/youtube)
* [Google News Scraper Tool](/tools/google-news)

Or learn about the [agents](/agents/overview) that use these tools to generate newsletters.


# Reddit Scraper Tool
Source: https://newsletter-ai-agent.pratikdani.com/tools/reddit

Learn about the Reddit Scraper Tool used by the Newsletter AI Agent

# Reddit Scraper Tool

The Reddit Scraper Tool is a custom tool that allows the Newsletter AI Agent to gather discussions from Reddit using an Apify actor. It provides a way to collect community insights and discussions related to the specified topic.

## Overview

The Reddit Scraper Tool is primarily used by the [Researcher Agent](/agents/researcher) to gather community discussions about the specified topic. It provides a flexible interface for searching Reddit and extracting structured data from posts and comments.

## Implementation

The Reddit Scraper Tool is implemented as a CrewAI `BaseTool` that interacts with an Apify Reddit scraper actor. Here's the implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional, Literal
from apify import Actor
from src.tools.base import RunApifyActor

class RedditScraperInput(BaseModel):
    """Input schema for RedditScraper tool."""
    searches: List[str] = Field(
        description="Here you can provide a search query which will be used to search Reddit's topics."
    )

    startUrls: Optional[List[str]] = Field(
        description="If you already have URL(s) of page(s) you wish to scrape, you can set them here. If you want to use the search field below, remove all startUrls here.",
        default=None
    )
    
    skipComments: Optional[bool] = Field(
        default=False,
        description="This will skip scrapping comments when going through posts"
    )
    
    # Additional parameters...

class RedditScraperTool(BaseTool):
    name: str = "Reddit Scraper"
    description: str = "Tool for scraping Reddit content with configurable parameters"
    args_schema: type[BaseModel] = RedditScraperInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)
    
    def _run(
        self,
        searches: List[str],
        startUrls: Optional[List[str]] = None,
        skipComments: Optional[bool] = False,
        # Additional parameters...
    ) -> str:
        run_inputs = {}
        
        if searches:
            run_inputs["searches"] = searches
        if startUrls:
            run_inputs["startUrls"] = startUrls
        if skipComments:
            run_inputs["skipComments"] = skipComments
        # Set additional parameters...
        
        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("reddit-scraper-actor-name", run_inputs)
        return dataset
```

## Parameters

The Reddit Scraper Tool accepts the following parameters:

| Parameter             | Type       | Description                                                | Default  |
| --------------------- | ---------- | ---------------------------------------------------------- | -------- |
| `searches`            | List\[str] | Search queries for Reddit topics                           | Required |
| `startUrls`           | List\[str] | Direct URLs to Reddit pages to scrape                      | None     |
| `skipComments`        | bool       | Skip scraping comments when processing posts               | False    |
| `skipUserPosts`       | bool       | Skip scraping user posts when processing user activity     | False    |
| `skipCommunity`       | bool       | Skip scraping community info but still get community posts | False    |
| `searchPosts`         | bool       | Search for posts with the provided search                  | True     |
| `searchComments`      | bool       | Search for comments with the provided search               | False    |
| `searchCommunities`   | bool       | Search for communities with the provided search            | False    |
| `searchUsers`         | bool       | Search for users with the provided search                  | False    |
| `sort`                | str        | How to sort the results (e.g., "new", "top", "hot")        | "new"    |
| `time`                | str        | Time filter for results                                    | None     |
| `includeNSFW`         | bool       | Include NSFW content in results                            | True     |
| `maxPostCount`        | int        | Maximum number of posts to retrieve                        | 20       |
| `maxComments`         | int        | Maximum number of comments to retrieve per post            | 20       |
| `maxCommunitiesCount` | int        | Maximum number of communities to retrieve                  | 2        |
| `maxUserCount`        | int        | Maximum number of users to retrieve                        | 2        |

## Usage

The Reddit Scraper Tool is used by the Researcher Agent to gather community discussions about the specified topic:

```python theme={null}
# Initialize the tool
reddit_tool = RedditScraperTool(actor=actor)

# Use the tool
reddit_results = reddit_tool._run(
    searches=[topic],
    searchPosts=True,
    searchComments=False,
    sort="relevance",
    maxPostCount=10
)
```

## Return Value

The tool returns a list of Reddit posts and comments, where each item is a dictionary containing information about a post or comment, including:

* `title`: The title of the post (for posts only)
* `text`: The text content of the post or comment
* `url`: The URL of the post or comment
* `author`: The username of the author
* `score`: The score (upvotes - downvotes) of the post or comment
* `created`: The creation date of the post or comment
* Additional metadata about the post or comment

## Apify Integration

The tool uses an Apify Reddit scraper actor, which provides several advantages:

1. **Scalability**: The actor can handle large numbers of Reddit searches efficiently
2. **Reliability**: The actor is designed to handle rate limiting and other issues that can arise when scraping Reddit
3. **Structured Data**: The actor returns Reddit posts and comments in a structured format that is easy to process

## Configuration

To use the Reddit Scraper Tool, you need to set up the following environment variables:

```
APIFY_API_KEY=your_apify_api_key_here
```

## Next Steps

* Learn about the [Twitter Scraper Tool](/tools/twitter)
* Explore the [Researcher Agent](/agents/researcher) that uses this tool
* See how this tool contributes to the [newsletter generation process](/features/newsletter-generation)


# Twitter Scraper Tool
Source: https://newsletter-ai-agent.pratikdani.com/tools/twitter

Learn about the Twitter Scraper Tool used by the Newsletter AI Agent

# Twitter Scraper Tool

The Twitter Scraper Tool is a custom tool that allows the Newsletter AI Agent to collect tweets related to a specified topic using the [Apify Twitter Scraper Lite](https://apify.com/apidojo/twitter-scraper-lite) actor.

## Overview

The Twitter Scraper Tool is primarily used by the [Researcher Agent](/agents/researcher) to gather social media insights about the specified topic. It provides a flexible interface for searching Twitter and extracting structured data from tweets.

## Implementation

The Twitter Scraper Tool is implemented as a CrewAI `BaseTool` that interacts with the Apify Twitter Scraper Lite actor. Here's the implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional
from apify import Actor
from src.tools.base import RunApifyActor

class TwitterScraperInput(BaseModel):
    """Input schema for TwitterScraper tool."""
    searchTerms: Optional[List[str]] = Field(
        description="Search terms to find tweets containing these terms. Alternative to using Twitter URLs.",
        default=None
    )

    sort: Optional[str] = Field(
        description="How to sort the returned tweets. Setting to 'Latest' yields more results.",
        default="Top",
        enum=["Top", "Latest"]
    )

    start: Optional[str] = Field(
        description="Scrape tweets starting from this date (format: YYYY-MM-DD)",
        default=None
    )

    end: Optional[str] = Field(
        description="Scrape tweets until this date (format: YYYY-MM-DD)", 
        default=None
    )

class TwitterScraperTool(BaseTool):
    name: str = "Twitter Scraper"
    description: str = "Tool for scraping Twitter content with configurable parameters"
    args_schema: type[BaseModel] = TwitterScraperInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)

    def _run(
        self,
        searchTerms: Optional[List[str]] = None,
        sort: Optional[str] = "latest",
        start: Optional[str] = None,
        end: Optional[str] = None
    ) -> str:
        run_inputs = {}
        
        if searchTerms:
            run_inputs["searchTerms"] = searchTerms
        if sort:
            run_inputs["sort"] = sort
        run_inputs["maxItems"] = 5
        if start:
            run_inputs["start"] = start
        if end:
            run_inputs["end"] = end

        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("apidojo/twitter-scraper-lite", run_inputs)
        return dataset
```

## Parameters

The Twitter Scraper Tool accepts the following parameters:

| Parameter     | Type       | Description                                         | Default |
| ------------- | ---------- | --------------------------------------------------- | ------- |
| `searchTerms` | List\[str] | Search terms to find tweets containing these terms  | None    |
| `sort`        | str        | How to sort the returned tweets ("Top" or "Latest") | "Top"   |
| `start`       | str        | Scrape tweets starting from this date (YYYY-MM-DD)  | None    |
| `end`         | str        | Scrape tweets until this date (YYYY-MM-DD)          | None    |

## Usage

The Twitter Scraper Tool is used by the Researcher Agent to gather social media insights about the specified topic:

```python theme={null}
# Initialize the tool
twitter_tool = TwitterScraperTool(actor=actor)

# Use the tool
twitter_results = twitter_tool._run(
    searchTerms=[topic],
    sort="latest",
    start="2023-01-01",  # Optional: start date
    end="2023-12-31"     # Optional: end date
)
```

## Return Value

The tool returns a list of tweets, where each tweet is a dictionary containing information about the tweet, including:

* `text`: The text content of the tweet
* `url`: The URL of the tweet
* `username`: The username of the tweet author
* `timestamp`: The timestamp when the tweet was posted
* `likes`: The number of likes the tweet received
* `retweets`: The number of retweets the tweet received
* Additional metadata about the tweet

## Apify Integration

The tool uses the Apify Twitter Scraper Lite actor, which provides several advantages:

1. **Scalability**: The actor can handle large numbers of Twitter searches efficiently
2. **Reliability**: The actor is designed to handle rate limiting and other issues that can arise when scraping Twitter
3. **Structured Data**: The actor returns tweets in a structured format that is easy to process
4. **No API Key Required**: Unlike the official Twitter API, the actor doesn't require API keys for basic functionality

## Configuration

To use the Twitter Scraper Tool, you need to set up the following environment variables:

```
APIFY_API_KEY=your_apify_api_key_here
```

## Next Steps

* Learn about the [YouTube Scraper Tool](/tools/youtube)
* Explore the [Researcher Agent](/agents/researcher) that uses this tool
* See how this tool contributes to the [newsletter generation process](/features/newsletter-generation)


# YouTube Scraper Tool
Source: https://newsletter-ai-agent.pratikdani.com/tools/youtube

Learn about the YouTube Scraper Tool used by the Newsletter AI Agent

# YouTube Scraper Tool

The YouTube Scraper Tool is a custom tool that allows the Newsletter AI Agent to find relevant video content on YouTube using an Apify actor. It provides a way to gather video-based information and insights related to the specified topic.

## Overview

The YouTube Scraper Tool is primarily used by the [Researcher Agent](/agents/researcher) to gather video content about the specified topic. It provides a flexible interface for searching YouTube and extracting structured data from videos, channels, and playlists.

## Implementation

The YouTube Scraper Tool is implemented as a CrewAI `BaseTool` that interacts with an Apify YouTube scraper actor. Here's the implementation:

```python theme={null}
from crewai.tools import BaseTool
from pydantic import BaseModel, Field, ConfigDict
from typing import List, Optional
from apify import Actor
from src.tools.base import RunApifyActor

class YouTubeScraperInput(BaseModel):
    """Input schema for YouTubeScraper tool."""
    searchQueries: Optional[List[str]] = Field(
        description="Search terms just like you would enter in YouTube's search bar"
    )
    
    maxResultsShorts: Optional[int] = Field(
        default=0,
        description="Limit the number of Shorts videos to crawl"
    )
    
    maxResultStreams: Optional[int] = Field(
        default=0,
        description="Limit the number of Stream videos to crawl"
    )
    
    startUrls: Optional[List[str]] = Field(
        default=[],
        description="Direct URLs to YouTube videos, channels, playlists, hashtags or search results"
    )
    
    # Additional parameters...

class YouTubeScraperTool(BaseTool):
    name: str = "YouTube Scraper"
    description: str = "Tool for scraping YouTube videos, channels, playlists with configurable parameters"
    args_schema: type[BaseModel] = YouTubeScraperInput
    actor: Actor = Field(description="Apify Actor instance")
    model_config = ConfigDict(arbitrary_types_allowed=True)
    
    def _run(
        self,
        searchQueries: Optional[List[str]] = None,
        maxResultsShorts: Optional[int] = 0,
        maxResultStreams: Optional[int] = 0,
        startUrls: Optional[List[str]] = [],
        # Additional parameters...
    ) -> str:
        run_inputs = {}
        
        if searchQueries:
            run_inputs["searchQueries"] = searchQueries
        if maxResultsShorts:
            run_inputs["maxResultsShorts"] = maxResultsShorts
        if maxResultStreams:
            run_inputs["maxResultStreams"] = maxResultStreams
        if startUrls:
            run_inputs["startUrls"] = startUrls
        # Set additional parameters...
        
        run_actor = RunApifyActor(self.actor)
        dataset = run_actor._run("youtube-scraper-actor-name", run_inputs)
        return dataset
```

## Parameters

The YouTube Scraper Tool accepts the following parameters:

| Parameter                      | Type       | Description                                        | Default  |
| ------------------------------ | ---------- | -------------------------------------------------- | -------- |
| `searchQueries`                | List\[str] | Search terms for YouTube's search bar              | Required |
| `maxResultsShorts`             | int        | Limit the number of Shorts videos to crawl         | 0        |
| `maxResultStreams`             | int        | Limit the number of Stream videos to crawl         | 0        |
| `startUrls`                    | List\[str] | Direct URLs to YouTube videos, channels, playlists | \[]      |
| `downloadSubtitles`            | bool       | Download subtitles for videos                      | False    |
| `saveSubsToKVS`                | bool       | Save downloaded subtitles to key-value store       | False    |
| `subtitlesLanguage`            | str        | Language for subtitles download                    | "any"    |
| `preferAutoGeneratedSubtitles` | bool       | Prefer auto-generated subtitles                    | False    |
| `subtitlesFormat`              | str        | Format for subtitle downloads                      | "srt"    |
| `sortingOrder`                 | str        | How to sort the results                            | None     |
| `dateFilter`                   | str        | Filter results by date                             | None     |
| `videoType`                    | str        | Filter by video type                               | None     |
| `lengthFilter`                 | str        | Filter by video length                             | None     |
| `isHD`                         | bool       | Filter for HD videos                               | None     |
| `hasSubtitles`                 | bool       | Filter for videos with subtitles                   | None     |

## Usage

The YouTube Scraper Tool is used by the Researcher Agent to gather video content about the specified topic:

```python theme={null}
# Initialize the tool
youtube_tool = YouTubeScraperTool(actor=actor)

# Use the tool
youtube_results = youtube_tool._run(
    searchQueries=[topic],
    maxResultsShorts=0,
    maxResultStreams=0,
    sortingOrder="relevance",
    dateFilter="last_month"
)
```

## Return Value

The tool returns a list of YouTube videos, where each video is a dictionary containing information about the video, including:

* `title`: The title of the video
* `url`: The URL of the video
* `description`: The description of the video
* `channelName`: The name of the channel that uploaded the video
* `channelUrl`: The URL of the channel
* `viewCount`: The number of views the video has
* `publishedAt`: The date the video was published
* `duration`: The duration of the video
* Additional metadata about the video

## Apify Integration

The tool uses an Apify YouTube scraper actor, which provides several advantages:

1. **Scalability**: The actor can handle large numbers of YouTube searches efficiently
2. **Reliability**: The actor is designed to handle rate limiting and other issues that can arise when scraping YouTube
3. **Structured Data**: The actor returns YouTube videos in a structured format that is easy to process
4. **Advanced Filtering**: The actor supports advanced filtering options to narrow down search results

## Configuration

To use the YouTube Scraper Tool, you need to set up the following environment variables:

```
APIFY_API_KEY=your_apify_api_key_here
```

## Next Steps

* Learn about the [Google News Scraper Tool](/tools/google-news)
* Explore the [Researcher Agent](/agents/researcher) that uses this tool
* See how this tool contributes to the [newsletter generation process](/features/newsletter-generation)