In this blog post, we’ll show you a different approach to searching through a large database of blog posts. The previous approach involved creating a single embedding for the entire article and storing it in a vector database. The new approach is much more effective, and in this post, we’ll explain why and how to implement it.
The new approach involves the following steps:
- Chunk the article into pieces of about 400 tokens using LangChain
- Create an embedding for each chunk
- Store each embedding, along with its metadata such as the URL and the original text, in Pinecone
- Store the original text in Pinecone, but not indexed
- To search the blog posts, find the 5 best matching chunks and add them to the ChatCompletion prompt
We’ll explain each step in more detail below, but first, let’s start with a brief overview of the previous approach.
The previous approach used OpenAI’s embeddings API to vectorize the blog post articles and Pinecone, a vector database, to store and query the vectors. The article was vectorized as a whole, and the resulting vector was stored in Pinecone. To search the blog posts, cosine similarity was used to find the closest matching article, and the contents of the article were retrieved using the Python requests library and the BeautifulSoup library. Finally, a prompt was created for the ChatCompletion API, including the retrieved article.
The problem with this approach was that the entire article was vectorized as one piece. This meant that if the article was long, the vector might not represent the article accurately, as it would be too general. Moreover, if the article was too long, the ChatCompletion API call might fail because too many tokens were used.
The new approach solves these problems by chunking the article into smaller pieces, creating an embedding for each chunk, and storing each embedding in Pinecone. This way, we have a much more accurate representation of the article, as each chunk represents a smaller, more specific part of the article. And because each chunk is smaller, there is less risk of using too many tokens in the ChatCompletion API call.
To implement the new approach, we’ll use LangChain to chunk the article into pieces of about 400 tokens. LangChain is a library aimed at assisting in the development of applications that use LLMs, or large language models.
Next, we’ll create an embedding for each chunk using OpenAI’s embeddings API. As before, we will use the text-embedding-ada-002 model. And once we have the embeddings, we’ll store each one, along with its metadata, in Pinecone. The key for each embedding will be a hash of the URL, combined with the chunk number.
The original text will also be stored in Pinecone, but not indexed, so that it can be retrieved later. With this approach, we do not need to retrieve a blog article from the web. Instead, we just get the text from Pinecone directly.
To search the blog posts, we’ll use cosine similarity to find the 5 best-matching chunks. The 5 best matching chunks will be added to the ChatCompletion prompt, allowing us to ask questions based on the article’s contents.
Uploading the embeddings
The code to upload the embeddings is shown below. You will need to set the following environment variables:
export OPENAI_API_KEY=your_openai_api_key export PINECONE_API_KEY=your_pinecone_api_key export PINECONE_ENVIRONMENT=your_pinecone_environment
import feedparser
import os
import pinecone
import openai
import requests
from bs4 import BeautifulSoup
from retrying import retry
from langchain.text_splitter import RecursiveCharacterTextSplitter
import tiktoken
import hashlib
# use cl100k_base tokenizer for gpt-3.5-turbo and gpt-4
tokenizer = tiktoken.get_encoding('cl100k_base')
# create the length function used by the RecursiveCharacterTextSplitter
def tiktoken_len(text):
tokens = tokenizer.encode(
text,
disallowed_special=()
)
return len(tokens)
@retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)
def create_embedding(article):
# vectorize with OpenAI text-emebdding-ada-002
embedding = openai.Embedding.create(
input=article,
model="text-embedding-ada-002"
)
return embedding["data"][0]["embedding"]
# OpenAI API key
openai.api_key = os.getenv('OPENAI_API_KEY')
# get the Pinecone API key and environment
pinecone_api = os.getenv('PINECONE_API_KEY')
pinecone_env = os.getenv('PINECONE_ENVIRONMENT')
pinecone.init(api_key=pinecone_api, environment=pinecone_env)
if "blog-index" not in pinecone.list_indexes():
print("Index does not exist. Creating...")
pinecone.create_index("blog-index", 1536, metadata_config= {"indexed": ["url", "chunk-id"]})
else:
print("Index already exists. Deleting...")
pinecone.delete_index("blog-index")
print("Creating new index...")
pinecone.create_index("blog-index", 1536, metadata_config= {"indexed": ["url", "chunk-id"]})
# set index; must exist
index = pinecone.Index('blog-index')
# URL of the RSS feed to parse
url = 'https://blog.baeke.info/feed/'
# Parse the RSS feed with feedparser
print("Parsing RSS feed: ", url)
feed = feedparser.parse(url)
# get number of entries in feed
entries = len(feed.entries)
print("Number of entries: ", entries)
# create recursive text splitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=400,
chunk_overlap=20, # number of tokens overlap between chunks
length_function=tiktoken_len,
separators=['\n\n', '\n', ' ', '']
)
pinecone_vectors = []
for i, entry in enumerate(feed.entries[:50]):
# report progress
print("Create embeddings for entry ", i, " of ", entries, " (", entry.link, ")")
r = requests.get(entry.link)
soup = BeautifulSoup(r.text, 'html.parser')
article = soup.find('div', {'class': 'entry-content'}).text
# create chunks
chunks = text_splitter.split_text(article)
# create md5 hash of entry.link
url = entry.link
url_hash = hashlib.md5(url.encode("utf-8"))
url_hash = url_hash.hexdigest()
# create embeddings for each chunk
for j, chunk in enumerate(chunks):
print("\tCreating embedding for chunk ", j, " of ", len(chunks))
vector = create_embedding(chunk)
# concatenate hash and j
hash_j = url_hash + str(j)
# add vector to pinecone_vectors list
print("\tAdding vector to pinecone_vectors list for chunk ", j, " of ", len(chunks))
pinecone_vectors.append((hash_j, vector, {"url": entry.link, "chunk-id": j, "text": chunk}))
# upsert every 100 vectors
if len(pinecone_vectors) % 100 == 0:
print("Upserting batch of 100 vectors...")
upsert_response = index.upsert(vectors=pinecone_vectors)
pinecone_vectors = []
# if there are any vectors left, upsert them
if len(pinecone_vectors) > 0:
print("Upserting remaining vectors...")
upsert_response = index.upsert(vectors=pinecone_vectors)
pinecone_vectors = []
print("Vector upload complete.")
Searching for blog posts
The code below is used to search blog posts:
import os
import pinecone
import openai
import tiktoken
# use cl100k_base tokenizer for gpt-3.5-turbo and gpt-4
tokenizer = tiktoken.get_encoding('cl100k_base')
def tiktoken_len(text):
tokens = tokenizer.encode(
text,
disallowed_special=()
)
return len(tokens)
# get the Pinecone API key and environment
pinecone_api = os.getenv('PINECONE_API_KEY')
pinecone_env = os.getenv('PINECONE_ENVIRONMENT')
pinecone.init(api_key=pinecone_api, environment=pinecone_env)
# set index
index = pinecone.Index('blog-index')
while True:
# set query
your_query = input("\nWhat would you like to know? ")
# vectorize your query with openai
try:
query_vector = openai.Embedding.create(
input=your_query,
model="text-embedding-ada-002"
)["data"][0]["embedding"]
except Exception as e:
print("Error calling OpenAI Embedding API: ", e)
continue
# search for the most similar vector in Pinecone
search_response = index.query(
top_k=5,
vector=query_vector,
include_metadata=True)
# create a list of urls from search_response['matches']['metadata']['url']
urls = [item["metadata"]['url'] for item in search_response['matches']]
# make urls unique
urls = list(set(urls))
# create a list of texts from search_response['matches']['metadata']['text']
chunks = [item["metadata"]['text'] for item in search_response['matches']]
# combine texts into one string to insert in prompt
all_chunks = "\n".join(chunks)
# print urls of the chunks
print("URLs:\n\n", urls)
# print the text number and first 50 characters of each text
print("\nChunks:\n")
for i, t in enumerate(chunks):
print(f"\nChunk {i}: {t[:50]}...")
try:
# openai chatgpt with article as context
# chat api is cheaper than gpt: 0.002 / 1000 tokens
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{ "role": "system", "content": "You are a thruthful assistant!" },
{ "role": "user", "content": f"""Answer the following query based on the context below ---: {your_query}
Do not answer beyond this context!
---
{all_chunks}""" }
],
temperature=0,
max_tokens=750
)
print(f"\n{response.choices[0]['message']['content']}")
except Exception as e:
print(f"Error with OpenAI Completion: {e}")
In Action
Below, we ask if Redis supports storing vectors and what version of Redis we need in Azure. The Pinecone vector search found 5 chunks, all from the same blog post (there is only one URL). The five chunks are combined and sent to ChatGPT, together with the original question. The response from the ChatCompletion API is clear!
Conclusion
In conclusion, the “chunked” approach to searching through a database of blog posts is much more effective and solves many of the problems associated with the previous approach. We hope you found this post helpful, and we encourage you to try out the new approach in your own projects!
baeke.info 