A Practical Guide

Introduction

In today’s fast-paced financial markets, information is power. Every news article, analyst report, and social media post has the potential to influence investor sentiment and, in turn, impact stock prices. For investors and financial analysts, understanding these subtle shifts in market sentiment is critical for making informed decisions. However, with the overwhelming volume of information available, manually gauging sentiment is both impractical and prone to bias.

This is where machine learning steps in, offering a data-driven approach to sentiment analysis. By leveraging sophisticated algorithms, investors can now analyze the tone and sentiment of financial news in real time, gaining insights into how these factors might affect stock values. This blog post explores a practical application of sentiment analysis, where we build a tool that interprets news sentiment to inform buy, hold, or sell decisions. Whether you’re a seasoned investor or a curious technologist, this guide will demonstrate how you can harness the power of AI to navigate the complexities of the financial markets with greater confidence.

Understanding Sentiment Analysis in Finance

What is Sentiment Analysis?

Sentiment analysis is a natural language processing (NLP) technique used to determine the emotional tone behind a body of text. In the context of finance, sentiment analysis can help investors gauge the overall mood or sentiment of the market by analyzing news articles, social media posts, earnings reports, and other textual data sources. By quantifying sentiment, investors can make more informed decisions about whether to buy, hold, or sell a particular stock.

Importance of Sentiment in Financial Markets

Sentiment can be a powerful driver of stock prices. For example, positive news about a company’s earnings or a favorable analyst rating can boost investor confidence, leading to a rise in stock price. Conversely, negative news, such as a product recall or a lawsuit, can lead to a decline in stock price. By analyzing the sentiment of these news pieces, investors can anticipate market movements and adjust their portfolios accordingly.

In the code below, we use the TextBlob library to perform basic sentiment analysis on three sentences, each representing a different sentiment:

from textblob import TextBlob

# Define two sentences
positive_sentence = "This is the perfect book!"
neutral_sentence = "The book is meh."
negative_sentence = "That book is awful."

# Create TextBlob objects
blob1 = TextBlob(positive_sentence)
blob2 = TextBlob(neutral_sentence)
blob3 = TextBlob(negative_sentence)

# Print the polarity of each sentence
print(f"The polarity of positive_sentence is: {blob1.sentiment.polarity}")
print(f"The polarity of neutral_sentence is: {blob2.sentiment.polarity}")
print(f"The polarity of negative_sentence is: {blob3.sentiment.polarity}")

Output:

The polarity of positive_sentence is: 1.0
The polarity of neutral_sentence is: 0.0
The polarity of negative_sentence is: -1.0

Here, TextBlob assigns a polarity score between -1 and 1 to each sentence, with -1 indicating a very negative sentiment, 1 indicating a very positive sentiment, and 0 indicating a neutral sentiment. This simple example demonstrates how sentiment analysis can transform qualitative text into quantitative data, which can then be used for further analysis.

How Sentiment Influences Investment Decisions

To apply sentiment analysis in a financial context, we need to analyze the sentiment of news articles related to a specific stock and determine how this sentiment might influence stock price movements. For example, consider a scenario where you want to assess the impact of news sentiment on Apple Inc. (AAPL) stock.

The following code snippet demonstrates how to retrieve historical news data for AAPL, perform sentiment analysis on the news content, and then aggregate the sentiment scores:

import requests
import pandas as pd

LIMIT = 1000
FROM = "2024-02-01"
TO = "2024-05-01"

url = "https://eodhd.com/api/news?s=AAPL.US&offset=0&limit={}&api_token=demo&fmt=json&from={}&to={}".format(
    LIMIT, FROM, TO)

response = requests.get(url)
data = response.json()

# Convert to a pandas DataFrame
data = pd.DataFrame(data)
data['date'] = data['date'].str.slice(0, 10)
data = data.dropna()

# Extract sentiment components
data['polarity'] = data['sentiment'].apply(lambda x: x['polarity'])
data['pos'] = data['sentiment'].apply(lambda x: x['pos'])
data['neg'] = data['sentiment'].apply(lambda x: x['neg'])
data['neu'] = data['sentiment'].apply(lambda x: x['neu'])

# Aggregate the data by date
aggreate_data = data.groupby('date').agg({'polarity': 'mean', 'pos': 'max', 'neg': 'max', 'neu': 'mean'}).reset_index()

In this code:

• We use an API to fetch historical news data related to AAPL.
• The news articles are processed to extract sentiment components, including polarity, positivity, negativity, and neutrality.
• The sentiment data is then aggregated by date, allowing us to analyze how daily sentiment might correlate with stock price movements.

By applying sentiment analysis in this way, investors can gain a deeper understanding of how market sentiment influences stock prices and make more informed, data-driven investment decisions.

Building a Sentiment Analysis Tool for Financial News

Now that we have a basic understanding of sentiment analysis and its importance in finance, let’s dive into building a practical tool that leverages this technique to inform investment decisions. This tool will analyze historical financial news related to a specific stock and backtest a trading strategy based on the sentiment of that news.

Overview of the Tool

The tool we are building has two main components:

1. Historical Sentiment Analysis and Backtesting: This part of the tool analyzes historical news data, performs sentiment analysis, and tests a simple trading strategy based on the results.
2. Real-Time Sentiment Analysis: This component fetches live news articles, analyzes the sentiment in real time, and provides buy, hold, or sell recommendations.

Technical Setup

For this tool, we will use the following libraries and tools:

• TextBlob: A Python library for processing textual data and performing sentiment analysis.
• yfinance: A Python wrapper for the Yahoo Finance API, used to fetch historical stock price data.
• requests: A library to make HTTP requests to fetch financial news data.
• pandas: A powerful data manipulation library to process and analyze the news and financial data.
• backtesting: A Python library for testing trading strategies on historical data.

Historical Sentiment Analysis and Backtesting

The first step is to analyze historical financial news and backtest a trading strategy based on the sentiment of that news.

Step 1: Fetching and Processing Historical News Data

We begin by fetching historical news articles related to Apple (AAPL) and processing them to extract sentiment components. The following code snippet shows how this is done:

import requests
import pandas as pd
from textblob import TextBlob

LIMIT = 1000
FROM = "2024-02-01"
TO = "2024-05-01"

url = "https://eodhd.com/api/news?s=AAPL.US&offset=0&limit={}&api_token=demo&fmt=json&from={}&to={}".format(
    LIMIT, FROM, TO)

response = requests.get(url)
data = response.json()

# Convert to a pandas DataFrame
data = pd.DataFrame(data)
data['date'] = data['date'].str.slice(0, 10)
data = data.dropna()

# Extract sentiment components
data['polarity'] = data['sentiment'].apply(lambda x: x['polarity'])
data['pos'] = data['sentiment'].apply(lambda x: x['pos'])
data['neg'] = data['sentiment'].apply(lambda x: x['neg'])
data['neu'] = data['sentiment'].apply(lambda x: x['neu'])

# Apply TextBlob sentiment analysis
data['SpacyPolarityMax'] = data['content'].apply(lambda text: TextBlob(text).sentiment.polarity)
data['SpacyPolarityMin'] = data['SpacyPolarityMax']

Here’s what this code does:

• Fetches Historical News Data: The code uses the requests library to pull historical news articles related to AAPL.
• Processes Sentiment: It then processes the news articles to extract sentiment scores, including polarity, positivity, negativity, and neutrality.
• Applies Custom Sentiment Analysis: Additionally, the code applies custom sentiment analysis using TextBlob to further analyze the content of each news article.

Step 2: Aggregating Sentiment Data

Once the sentiment analysis is complete, we aggregate the sentiment data by date, which allows us to examine how daily sentiment might correlate with stock price movements:

# Aggregate the data by date
aggregate_data = data.groupby('date').agg({'polarity': 'mean', 'pos': 'max', 'neg': 'max', 'neu': 'mean',
                                          'SpacyPolarityMax': 'max', 'SpacyPolarityMin': 'min'}).reset_index()

This aggregation step helps in simplifying the analysis by summarizing the sentiment data for each day, which will be crucial when backtesting our trading strategy.

Step 3: Fetching Historical Stock Data

Next, we fetch the historical stock price data for AAPL using yfinance. This data will be merged with our sentiment data to create a comprehensive dataset for backtesting:

import yfinance as yf

# Get data on this ticker
tickerData = yf.Ticker('AAPL')

# Get the historical prices for this ticker
tickerDf = tickerData.history(period='1d', start=FROM, end=TO)

# Reset the index of tickerDf
tickerDf = tickerDf.reset_index()

# Convert the 'Date' column in tickerDf to string
tickerDf['Date'] = tickerDf['Date'].astype(str).str.slice(0, 10)

# Perform the left join
merged_data = pd.merge(aggreate_data, tickerDf, how='left', left_on='date', right_on='Date')

# Drop any rows with NaN values and set 'date' as index
merged_data = merged_data.dropna()
merged_data['date'] = pd.to_datetime(merged_data['date'])
merged_data = merged_data.sort_values('date', ascending=True)
merged_data.set_index('date', inplace=True)

This snippet accomplishes the following:

• Fetches Historical Stock Prices: We use yfinance to retrieve the historical prices for AAPL within the same date range as our news data.
• Merges Data: The sentiment data and stock prices are merged, resulting in a dataset that includes both sentiment scores and corresponding stock prices for each date.

Step 4: Implementing and Backtesting the Trading Strategy

With our combined dataset, we can now implement a simple trading strategy: buy when the sentiment is positive, sell when it’s negative. This strategy will be backtested using the backtesting library:

from backtesting import Backtest, Strategy

class SpacyStrategy(Strategy):
    polBuy = 0
    polSell = 0
    
    def init(self):
        pass

    def next(self):
        self.polarityMax = self.data.df['SpacyPolarityMax'].iloc[-1]
        self.polarityMin = self.data.df['SpacyPolarityMin'].iloc[-1]
        if self.polarityMax > self.polBuy/100:
            self.position.close()
            self.buy()
        elif self.polarityMin < self.polSell/100:
            self.position.close()
            self.sell()

bt = Backtest(merged_data, SpacyStrategy, cash=10000, commission=0)
stats_skopt, _, _ = bt.optimize(
    polBuy=(0, 100), polSell=(-100, 0),
    maximize='Equity Final [$]', method='skopt', max_tries=300,
    return_heatmap=True, return_optimization=True)

# Get the best parameters
best_params = stats_skopt._strategy
print("Best Parameters: ", best_params)

# Print the final equity
print("Final Equity: ", stats_skopt['Equity Final [$]'])

Here's what this code does:

• Defines the Trading Strategy: A simple strategy is defined where the tool buys when the sentiment is positive (above a certain threshold) and sells when it's negative (below a certain threshold).
• Optimizes the Strategy: The strategy parameters (polBuy and polSell) are optimized using skopt to maximize the final equity.
• Backtests the Strategy: The backtesting library simulates the trading strategy over the historical data to evaluate its performance.

This section provides a solid foundation for building a sentiment analysis tool that can be used to backtest trading strategies, giving investors valuable insights into how market sentiment can affect stock prices.

Real-Time Sentiment Analysis with LangChain

While historical sentiment analysis and backtesting provide valuable insights, the true power of sentiment analysis lies in its ability to inform real-time investment decisions. In this section, we'll explore how to extend our tool to analyze live news data and provide up-to-the-minute buy, hold, or sell recommendations.

Why Real-Time Analysis Matters

In financial markets, timing is everything. Market conditions can change rapidly based on new information, making it crucial for investors to stay informed and act quickly. Real-time sentiment analysis allows investors to react to news as it happens, giving them an edge in making timely and informed decisions. By incorporating real-time data, our tool can provide dynamic recommendations that adapt to current market conditions.

Live Sentiment Analysis Setup

To analyze live news data, we use LangChain, an open-source framework designed to build applications powered by language models. This allows us to retrieve the latest news articles, perform sentiment analysis on the content, and generate actionable investment recommendations.

Step 1: Setting Up the News Retriever

We start by defining a custom retriever that fetches the latest news articles for a specific stock ticker (e.g., Apple - AAPL). The following code snippet demonstrates how to set up the retriever using LangChain:

import requests
from bs4 import BeautifulSoup
from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever
from datetime import datetime

class FinanceAPIRetriever(BaseRetriever):
    """Finance API retriever."""

    def _get_relevant_documents(self, query: str) -> List[Document]:
        symbol = query
        date = datetime.now().strftime('%Y-%m-%d')
        baseurl = (
            "https://api.marketaux.com/v1/news/all?symbols={}&published_on={}&"
            "api_token=IPmJJQ0cn0s5dP1pQFbp7lQvBnEqYqWax6ogJnAr&"
            "domains=finance.yahoo.com"
        ).format(symbol, date)
        response = requests.get(baseurl)
        data = response.json()
        docs = data['data']

        documents = []
        for doc in docs:
            url = doc['url']
            response = requests.get(url)
            soup = BeautifulSoup(response.text, 'html.parser')
            text = soup.get_text()
            document = Document(
                page_content=text,
                metadata={
                    "title": doc.get("title", ""),
                    "source": url,
                },
            )
            documents.append(document)

        return documents

This code does the following:

• Fetches Live News Data: The FinanceAPIRetriever class retrieves the latest news articles for a given stock ticker from an API.
• Extracts Content: It uses BeautifulSoup to parse the HTML content of each article and extract the text.
• Creates Document Objects: The content is wrapped in Document objects, which include metadata such as the article's title and source URL.

Step 2: Performing Sentiment Analysis on Live Data

Once the live news articles are retrieved, we can perform sentiment analysis on their content using the TextBlob library, just as we did with the historical data. We define a tool for sentiment analysis and integrate it with LangChain:

from textblob import TextBlob
from langchain.tools import BaseTool

class SentimentAnalysisTool(BaseTool):
    def invoke(self, input: str):
        blob = TextBlob(input)
        return blob.sentiment_assessments.polarity

@tool
def SpacyPolarity(text):
    """Tool for analyzing sentiment of text using TextBlob"""
    blob = TextBlob(text)
    return blob.sentiment_assessments.polarity

This code sets up a SentimentAnalysisTool that:

• Analyzes Sentiment: Uses TextBlob to analyze the sentiment of the retrieved news articles.
• Calculates Polarity: Returns the polarity score, which will later be used to make investment decisions.

Step 3: Running the Live Sentiment Analysis

Finally, we bring everything together by setting up an agent with LangChain that fetches the latest news, analyzes sentiment, and provides real-time recommendations:

from langchain.agents import initialize_agent, AgentType
from langchain.memory import ConversationBufferMemory
from langchain_community.llms.bedrock import Bedrock
from langchain.tools.retriever import create_retriever_tool

# Setup LangChain agent
def setup_agent(llm, tools):
    memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
    agent_executor = initialize_agent(
        tools,
        llm,
        agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
        memory=memory,
        verbose=False,
    )
    return agent_executor

REGION_NAME = 'us-west-2'
LLM_MODEL = "anthropic.claude-v2:1"
TEMPERATURE = 0

# Initialize tools and agent
finRetriever = FinanceAPIRetriever()
retriever_tool = create_retriever_tool(finRetriever, 'news_retriever', 'Tool for searching the web, input should be stock symbol.')

llm = Bedrock(model_id=LLM_MODEL, region_name=REGION_NAME, model_kwargs={"temperature": TEMPERATURE})
tools = [retriever_tool, SpacyPolarity]
agent_executor = setup_agent(llm, tools)

# Execute the agent to get sentiment analysis
COMPANY_NAME = "Apple"
news = list(agent_executor.stream({"input": f"Find the latest news about {COMPANY_NAME}."}))
results = news[-1]['output']

sentiment = list(agent_executor.stream({"input": f"Analyze the sentiment of the following summary: {results}"}))
print(sentiment[-1]['output'].strip())

Here's what this code accomplishes:

• Sets Up an Agent: The LangChain agent is configured with tools for retrieving live news and performing sentiment analysis.
• Executes Sentiment Analysis: The agent fetches the latest news about Apple, analyzes the sentiment, and provides an investment recommendation based on the sentiment score (e.g., buy, hold, or sell).

This real-time sentiment analysis capability allows investors to stay informed and react to the latest news as it happens, potentially improving their decision-making process.

Practical Application

With the live sentiment analysis tool in place, investors can now receive real-time recommendations based on the sentiment of the latest news articles. For example, if the sentiment score for the most recent news is positive and exceeds a certain threshold, the tool might suggest buying the stock. Conversely, if the sentiment is negative, it might recommend selling.

This approach offers a dynamic and data-driven way to navigate the complexities of financial markets, empowering investors with actionable insights as they unfold.

Conclusion

In the ever-changing world of finance, staying ahead of the curve requires not only access to the latest information but also the ability to interpret and act on that information effectively. By integrating sentiment analysis into your investment strategy, you can harness the power of machine learning to gain deeper insights into market sentiment and make more informed, data-driven decisions.

In this blog post, we've explored how to build a practical sentiment analysis tool that analyzes both historical and real-time financial news to inform buy, hold, or sell decisions. From setting up the necessary tools and libraries to implementing a trading strategy and optimizing it through backtesting, we've demonstrated how this approach can provide valuable insights into the relationship between market sentiment and stock price movements.

Moreover, the addition of real-time sentiment analysis using LangChain enables you to react swiftly to the latest market developments, providing a competitive edge in today's fast-paced financial environment. Whether you're a seasoned investor looking to refine your strategies or a technologist interested in applying machine learning to finance, this tool offers a flexible and powerful solution.