11-hyde-rag.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HyDE - Hypothetical Document Embeddings\n",
    "    -> Instead of performing the RAG on user query, here we put INTERMEDAITE LLM to generate an answer (hypothetical)\n",
    "    -> End user can't see this answer. this answer will compare (similarity using embeddings) with source documents.\n",
    "    -> earlier we compared the user query with the documents in the rag (query-to-answer RAG)\n",
    "       now we compare hypothetical answer with the documents in the rag (answer-to-answer RAG)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms import OpenAI\n",
    "from langchain.vectorstores import Chroma\n",
    "import yaml, os, openai, textwrap, langchain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.embeddings import HuggingFaceBgeEmbeddings\n",
    "from langchain.chat_models import ChatOpenAI, AzureChatOpenAI\n",
    "from langchain.chains import LLMChain, HypotheticalDocumentEmbedder, RetrievalQA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('cadentials.yaml') as f:\n",
    "    credentials = yaml.load(f, Loader=yaml.FullLoader)\n",
    "\n",
    "os.environ[\"OPENAI_API_TYPE\"] = credentials['AD_OPENAI_API_TYPE']\n",
    "os.environ[\"OPENAI_API_VERSION\"] = credentials['AD_OPENAI_API_VERSION']\n",
    "os.environ[\"OPENAI_API_BASE\"] = credentials['AD_OPENAI_API_BASE']\n",
    "os.environ[\"OPENAI_API_KEY\"] = credentials['AD_OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Method 01 - Single Hypothetical Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/1zuu/anaconda3/envs/llm/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.\n",
      "  warn(\"The installed version of bitsandbytes was compiled without GPU support. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'NoneType' object has no attribute 'cadam32bit_grad_fp32'\n"
     ]
    }
   ],
   "source": [
    "bge_embeddings = HuggingFaceBgeEmbeddings(\n",
    "                                        model_name=\"BAAI/bge-small-en-v1.5\",\n",
    "                                        model_kwargs={'device': 'mps'},\n",
    "                                        encode_kwargs={'normalize_embeddings': True}\n",
    "                                        )\n",
    "\n",
    "llm = AzureChatOpenAI(\n",
    "                    deployment_name=credentials['AD_DEPLOYMENT_ID'],\n",
    "                    model_name=credentials['AD_ENGINE'],\n",
    "                    temperature=0.9, \n",
    "                    max_tokens = 256\n",
    "                    )\n",
    "\n",
    "embeddings = HypotheticalDocumentEmbedder.from_llm(\n",
    "                                                    llm,\n",
    "                                                    bge_embeddings,\n",
    "                                                    prompt_key=\"web_search\"\n",
    "                                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please write a passage to answer the question \n",
      "Question: {QUESTION}\n",
      "Passage:\n"
     ]
    }
   ],
   "source": [
    "print(embeddings.llm_chain.prompt.template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "langchain.debug = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"Human: Please write a passage to answer the question \\nQuestion: What items does McDonalds make?\\nPassage:\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] [3.53s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"McDonald's is a fast-food restaurant chain that is famous for its burgers, fries, and milkshakes. However, their menu offers much more than just those few items. Some of the most popular items that McDonald's makes include chicken sandwiches, breakfast sandwiches, salads, and wraps. In addition to these, McDonald's also offers a wide range of side items such as hash browns, mozzarella sticks, and apple slices. The restaurant also offers various beverages, including sodas, juices, and coffee. Moreover, McDonald's has special menus for kids, which include items like Happy Meals, which are designed to appeal to younger audiences. Overall, McDonald's is known for providing its customers with a diverse range of food options that cater to different tastes and preferences.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"McDonald's is a fast-food restaurant chain that is famous for its burgers, fries, and milkshakes. However, their menu offers much more than just those few items. Some of the most popular items that McDonald's makes include chicken sandwiches, breakfast sandwiches, salads, and wraps. In addition to these, McDonald's also offers a wide range of side items such as hash browns, mozzarella sticks, and apple slices. The restaurant also offers various beverages, including sodas, juices, and coffee. Moreover, McDonald's has special menus for kids, which include items like Happy Meals, which are designed to appeal to younger audiences. Overall, McDonald's is known for providing its customers with a diverse range of food options that cater to different tastes and preferences.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 29,\n",
      "      \"completion_tokens\": 154,\n",
      "      \"total_tokens\": 183\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = embeddings.embed_query(\"What items does McDonalds make?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Method 02 - Multiple Hypothetical Generation\n",
    "\n",
    "generate multiple hypothetical answers and aggreate all of their embeddings to get a single embedding for the overall hypothetical answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "bge_embeddings = HuggingFaceBgeEmbeddings(\n",
    "                                        model_name=\"BAAI/bge-small-en-v1.5\",\n",
    "                                        model_kwargs={'device': 'mps'},\n",
    "                                        encode_kwargs={'normalize_embeddings': True}\n",
    "                                        )\n",
    "\n",
    "llm = AzureChatOpenAI(\n",
    "                    deployment_name=credentials['AD_DEPLOYMENT_ID'],\n",
    "                    model_name=credentials['AD_ENGINE'],\n",
    "                    temperature=0.9, \n",
    "                    max_tokens = 256,\n",
    "                    n=4,      ########################################### Check this\n",
    "                    )\n",
    "\n",
    "embeddings = HypotheticalDocumentEmbedder.from_llm(\n",
    "                                                    llm,\n",
    "                                                    bge_embeddings,\n",
    "                                                    prompt_key=\"web_search\"\n",
    "                                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"Human: Please write a passage to answer the question \\nQuestion: What is McDonalds best selling item?\\nPassage:\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] [2.06s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"McDonald's best selling item is none other than their iconic Big Mac sandwich. The Big Mac has been a fan favorite since its creation in 1967 and continues to dominate sales at McDonald's restaurants worldwide. The sandwich consists of two beef patties, special sauce, lettuce, cheese, pickles, and onions, all tucked between a sesame seed bun. It's no surprise that this classic sandwich has remained a top seller for over five decades, as it satisfies cravings for both savory meat and sweet sauce in one delicious bite. Despite McDonald's offering a vast menu of burgers, chicken sandwiches, and breakfast items, the Big Mac remains the king of the fast food chain, continuing to be a customer favorite.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"McDonald's best selling item is none other than their iconic Big Mac sandwich. The Big Mac has been a fan favorite since its creation in 1967 and continues to dominate sales at McDonald's restaurants worldwide. The sandwich consists of two beef patties, special sauce, lettuce, cheese, pickles, and onions, all tucked between a sesame seed bun. It's no surprise that this classic sandwich has remained a top seller for over five decades, as it satisfies cravings for both savory meat and sweet sauce in one delicious bite. Despite McDonald's offering a vast menu of burgers, chicken sandwiches, and breakfast items, the Big Mac remains the king of the fast food chain, continuing to be a customer favorite.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"McDonald's best selling item is the Big Mac. This iconic sandwich was first introduced in 1968 and has been a fan favorite ever since. The Big Mac consists of two beef patties, special sauce, lettuce, cheese, pickles, and onions, all sandwiched between a sesame seed bun. It has become a staple of fast food culture and is recognized all over the world. In addition to the Big Mac, McDonald's also has a variety of other popular menu items, such as the Quarter Pounder with Cheese, Chicken McNuggets, and their famous french fries. However, the Big Mac remains the most famous and beloved item on the McDonald's menu.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"McDonald's best selling item is the Big Mac. This iconic sandwich was first introduced in 1968 and has been a fan favorite ever since. The Big Mac consists of two beef patties, special sauce, lettuce, cheese, pickles, and onions, all sandwiched between a sesame seed bun. It has become a staple of fast food culture and is recognized all over the world. In addition to the Big Mac, McDonald's also has a variety of other popular menu items, such as the Quarter Pounder with Cheese, Chicken McNuggets, and their famous french fries. However, the Big Mac remains the most famous and beloved item on the McDonald's menu.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"McDonald's best selling item is the Big Mac. This iconic burger, first introduced in 1968, features two beef patties, special sauce, lettuce, cheese, pickles, and onions, all on a sesame seed bun. It has remained popular over the years and has become a staple on the fast food chain's menu. In fact, the Big Mac has been such a success that it has even inspired its own jingle and has been the subject of numerous marketing campaigns. Despite the introduction of new menu items, the Big Mac continues to be a top seller for McDonald's.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"McDonald's best selling item is the Big Mac. This iconic burger, first introduced in 1968, features two beef patties, special sauce, lettuce, cheese, pickles, and onions, all on a sesame seed bun. It has remained popular over the years and has become a staple on the fast food chain's menu. In fact, the Big Mac has been such a success that it has even inspired its own jingle and has been the subject of numerous marketing campaigns. Despite the introduction of new menu items, the Big Mac continues to be a top seller for McDonald's.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"McDonalds' best selling item is the classic Big Mac burger. This iconic burger is made of two beef patties, special sauce, lettuce, cheese, pickles, onions, all served on a sesame seed bun. It has been a staple on the McDonald's menu since it was first introduced in 1968. The Big Mac has remained a fan favorite for decades due to its delicious taste and filling portions. In addition to the Big Mac, McDonald's also sells a variety of other popular items such as the McChicken sandwich, McNuggets, and their famous fries.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"McDonalds' best selling item is the classic Big Mac burger. This iconic burger is made of two beef patties, special sauce, lettuce, cheese, pickles, onions, all served on a sesame seed bun. It has been a staple on the McDonald's menu since it was first introduced in 1968. The Big Mac has remained a fan favorite for decades due to its delicious taste and filling portions. In addition to the Big Mac, McDonald's also sells a variety of other popular items such as the McChicken sandwich, McNuggets, and their famous fries.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 30,\n",
      "      \"completion_tokens\": 514,\n",
      "      \"total_tokens\": 544\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = embeddings.embed_query(\"What is McDonalds best selling item?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Method 03 - Custom Prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"Please answer the user's question as a single food item\n",
    "Question: {question}\n",
    "Answer:\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(input_variables=[\"question\"], template=prompt_template)\n",
    "\n",
    "llm_chain = LLMChain(llm=llm, prompt=prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = HypotheticalDocumentEmbedder(\n",
    "                                            llm_chain=llm_chain,\n",
    "                                            base_embeddings=bge_embeddings\n",
    "                                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"Human: Please answer the user's question as a single food item\\nQuestion: What is is McDonalds best selling item?\\nAnswer:\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] [834ms] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"Big Mac\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Big Mac\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"Big Mac\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Big Mac\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"Big Mac.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Big Mac.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      },\n",
      "      {\n",
      "        \"text\": \"Big Mac\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Big Mac\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 33,\n",
      "      \"completion_tokens\": 9,\n",
      "      \"total_tokens\": 42\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = embeddings.embed_query(\n",
    "    \"What is is McDonalds best selling item?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use HyDE for RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "loaders = [\n",
    "    TextLoader('data/langchain_blog_posts/blog.langchain.dev_announcing-langsmith_.txt'),\n",
    "    TextLoader('data/langchain_blog_posts/blog.langchain.dev_benchmarking-question-answering-over-csv-data_.txt'),\n",
    "    TextLoader('data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt'),\n",
    "]\n",
    "docs = []\n",
    "for l in loaders:\n",
    "    docs.extend(l.load())\n",
    "\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "texts = text_splitter.split_documents(docs) #split_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_template = \"\"\"Please answer the user's question as related to Large Language Models\n",
    "Question: {question}\n",
    "Answer:\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(\n",
    "                        input_variables=[\"question\"], \n",
    "                        template=prompt_template\n",
    "                        )\n",
    "\n",
    "llm = AzureChatOpenAI(\n",
    "                    deployment_name=credentials['AD_DEPLOYMENT_ID'],\n",
    "                    model_name=credentials['AD_ENGINE'],\n",
    "                    temperature=0.9, \n",
    "                    max_tokens = 256\n",
    "                    )\n",
    "\n",
    "llm_chain = LLMChain(\n",
    "                    llm=llm,  ####################################### for Hypothetical Answer Generation\n",
    "                    prompt=prompt\n",
    "                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = HypotheticalDocumentEmbedder(\n",
    "                                        llm_chain=llm_chain,\n",
    "                                        base_embeddings=bge_embeddings\n",
    "                                        )\n",
    "\n",
    "docsearch = Chroma.from_documents(texts, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"Human: Please answer the user's question as related to Large Language Models\\nQuestion: What are chat loaders?\\nAnswer:\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] [1.52s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"Chat loaders are software components that are used to load, initialize and configure large language models that are used in chatbots or other conversational AI applications. These loaders typically perform a number of functions, such as setting up the model's parameters and hyperparameters, initializing the internal state of the model, and loading its weights and other associated data. Chat loaders are an important part of building and deploying large language models in natural language processing and conversational AI.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Chat loaders are software components that are used to load, initialize and configure large language models that are used in chatbots or other conversational AI applications. These loaders typically perform a number of functions, such as setting up the model's parameters and hyperparameters, initializing the internal state of the model, and loading its weights and other associated data. Chat loaders are an important part of building and deploying large language models in natural language processing and conversational AI.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 30,\n",
      "      \"completion_tokens\": 90,\n",
      "      \"total_tokens\": 120\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "query = \"What are chat loaders?\"\n",
    "docs = docsearch.similarity_search(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(page_content='URL: https://blog.langchain.dev/chat-loaders-finetune-a-chatmodel-in-your-voice/\\nTitle: Chat Loaders: Fine-tune a ChatModel in your Voice\\n\\nSummary\\n\\nWe are adding a new integration type, ChatLoaders, to make it easier to fine-tune models on your own unique writing style. These utilities help convert data from popular messaging platforms to chat messages compatible with fine-tuning formats like that supported by OpenAI.\\n\\nThank you to Greg Kamradt for Misbah Syed for their thought leadership on this.\\n\\nImportant Links:\\n\\nContext\\n\\nOn Tuesday, OpenAI announced improved fine-tuning support, extending the service to larger chat models like GPT-3.5-turbo. This enables anyone to customize these larger, more capable models for their own use cases. They also teased support for fine-tuning GPT-4 later this year.\\n\\nWhile fine-tuning is typically not advised for teaching an LLM substantially new knowledge or for factual recall; it is good for style transfer.', metadata={'source': 'data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt'}),\n",
       " Document(page_content=\"These utilities take data exported from popular messaging platforms and convert them to LangChain message objects, which you can then easily convert platform-agnostic message formats, such as OpenAI, Llama 2, and others. This training data can be used directly for fine-tuning a model.\\n\\nWe've added loaders for the following popular messaging platforms so far:\\n\\nFacebook Messenger\\n\\nSlack\\n\\nTelegram\\n\\nWhatsApp\\n\\nWe have also added a recipe on how to do so for Discord and Twitter (using Apify) and plan to integrate additional chat loaders in the near future. If you have a favorite messaging platform you'd like to support, we'd love to help you land a PR!\\n\\nTo get you started, we've added an end-to-end example notebook to the LangChain documentation showing how to fine-tune gpt-3.5-turbo (the model behind ChatGPT) on an example set of Facebook messages.\\n\\n❗ Please ensure all participants of your conversations support the decision to train a model on the chat data before proceeding.\", metadata={'source': 'data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt'}),\n",
       " Document(page_content=\"We're excited to see all the creative applications fine-tuning unlocks. We have implemented a few ChatLoaders already, but we need your help to make it easier to create your own personalized model. Help us create more ChatLoaders!\", metadata={'source': 'data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt'}),\n",
       " Document(page_content='Once you have your fine-tuned model, you can use the model name directly in LangChain\\'s ChatOpenAI class:\\n\\nfrom langchain.chat_models import ChatOpenAI llm = ChatOpenAI(model=\"ft:gpt-3.5-turbo-0613:{openaiOrg}::{modelId}\") llm.predict(\"What classes are you taking this year?\")\\n\\nThen you can plug this into any other LangChain component!\\n\\nEnd-to-End Example\\n\\nWe\\'ve also created an end-to-end example of finetuning a model based on Elon Musk\\'s tweets. This uses Apify to load data. Note that it\\'s less than 100 examples so results may not be the most amazing they could be.\\n\\nWe open-sourced this example at the GitHub repo here. We also hosted it on Streamlit app so you can easily play around with it here.\\n\\nWebinar\\n\\nThere is a lot more to discuss on this topic. What types of messages are best for finetuning? What others sources of data exist for this? How many points do you need?\\n\\nWe\\'ll be discussing this and more next week in a webinar with Greg Kamradt. Come join!\\n\\nConclusion', metadata={'source': 'data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt'})]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "                                        llm=llm, ####################################### for RAG\n",
    "                                        chain_type=\"stuff\",\n",
    "                                        retriever=docsearch.as_retriever(search_kwargs={\"k\": 3}),\n",
    "                                        return_source_documents=True\n",
    "                                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def wrap_text_preserve_newlines(text, width=110):\n",
    "    lines = text.split('\\n')\n",
    "    wrapped_lines = [textwrap.fill(line, width=width) for line in lines]\n",
    "    wrapped_text = '\\n'.join(wrapped_lines)\n",
    "\n",
    "    return wrapped_text\n",
    "\n",
    "def process_llm_response(llm_response):\n",
    "    print(wrap_text_preserve_newlines(llm_response['result']))\n",
    "    print('\\n\\nSources:')\n",
    "    for source in llm_response[\"source_documents\"]:\n",
    "        print(source.metadata['source'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"query\": \"What is Flash attention?\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"Human: Please answer the user's question as related to Large Language Models\\nQuestion: What is Flash attention?\\nAnswer:\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:llm:AzureChatOpenAI] [1.90s] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"Flash attention is a type of attention mechanism used in large language models. It allows the model to selectively focus on certain parts of the input sequence, rather than processing the entire sequence at once. The attention is computed using a small set of key-value pairs, which are learned jointly with the rest of the model parameters. Flash attention is particularly useful for processing long sequences, as it allows the model to attend to the most relevant parts of the sequence while ignoring irrelevant details.\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"Flash attention is a type of attention mechanism used in large language models. It allows the model to selectively focus on certain parts of the input sequence, rather than processing the entire sequence at once. The attention is computed using a small set of key-value pairs, which are learned jointly with the rest of the model parameters. Flash attention is particularly useful for processing long sequences, as it allows the model to attend to the most relevant parts of the sequence while ignoring irrelevant details.\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 30,\n",
      "      \"completion_tokens\": 93,\n",
      "      \"total_tokens\": 123\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain] Entering Chain run with input:\n",
      "\u001b[0m[inputs]\n",
      "\u001b[32;1m\u001b[1;3m[chain/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain > 4:chain:LLMChain] Entering Chain run with input:\n",
      "\u001b[0m{\n",
      "  \"question\": \"What is Flash attention?\",\n",
      "  \"context\": \"Why is this better than direct instructions? Style and tone can be hard to describe! Most of us don't write like ChatGPT, and it can sometimes be frustratingly difficult to get the LLM to consistently respond in a particular voice (especially over longer conversations).\\n\\nWhy is this better than few-shot examples? It can be challenging to capture your voice in only a few concise snippets! Fine-tuning lets you provide a larger number of examples the model can learn from without having to see them every time you want to query the model.\\n\\nChatLoaders\\n\\nAt LangChain, we want to make it as easy as possible for you to take advantage of this improved fine-tuning support. To make it simple to adapt a model to your voice, we're adding a new integration type: ChatLoaders .\\n\\nWe’ve also made it easy to change and rerun examples from the UI. We added this feature after seeing teams take logs of bad examples and copy-paste into the OpenAI playground to tweak the prompt until they got a good result. We wanted to eliminate that friction, and now with the click of a button, you can go from a log to a playground where you can actively edit. This is currently supported for both OpenAI and Anthropic models, with support for more coming soon. We’re also working on supporting this for chains in general.\\n\\nThis deep visibility into model performance has been particularly helpful for teams developing complex applications. LangSmith helped Streamlit and Snowflake implement agents that could intelligently and reliably answer questions about their data.\\n\\nIn another example of debugging in action, we partnered with DeepLearningAI to equip learners in the recently-released LangChain courses with access to LangSmith. This allowed students to easily visualize the exact sequence of calls, and the inputs and outputs at each step in the chain with precision. Students can understand exactly what the chains, prompts, and LLMs were doing, which helps build intuition as they learn to create new and more sophisticated applications.\\n\\nTesting\\n\\nOne of the main questions we see developers grapple with is: “If I change this chain/prompt, how does that affect my outputs?” The most effective way to answer this question is to curate a dataset of examples that you care about, and then run any changed prompts/chains over this dataset. LangSmith first makes it easy to create these datasets from traces or by uploading datasets you’ve curated manually. You can then easily run chains and prompts over those data sets.\"\n",
      "}\n",
      "\u001b[32;1m\u001b[1;3m[llm/start]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain > 4:chain:LLMChain > 5:llm:AzureChatOpenAI] Entering LLM run with input:\n",
      "\u001b[0m{\n",
      "  \"prompts\": [\n",
      "    \"System: Use the following pieces of context to answer the user's question. \\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\\n----------------\\nWhy is this better than direct instructions? Style and tone can be hard to describe! Most of us don't write like ChatGPT, and it can sometimes be frustratingly difficult to get the LLM to consistently respond in a particular voice (especially over longer conversations).\\n\\nWhy is this better than few-shot examples? It can be challenging to capture your voice in only a few concise snippets! Fine-tuning lets you provide a larger number of examples the model can learn from without having to see them every time you want to query the model.\\n\\nChatLoaders\\n\\nAt LangChain, we want to make it as easy as possible for you to take advantage of this improved fine-tuning support. To make it simple to adapt a model to your voice, we're adding a new integration type: ChatLoaders .\\n\\nWe’ve also made it easy to change and rerun examples from the UI. We added this feature after seeing teams take logs of bad examples and copy-paste into the OpenAI playground to tweak the prompt until they got a good result. We wanted to eliminate that friction, and now with the click of a button, you can go from a log to a playground where you can actively edit. This is currently supported for both OpenAI and Anthropic models, with support for more coming soon. We’re also working on supporting this for chains in general.\\n\\nThis deep visibility into model performance has been particularly helpful for teams developing complex applications. LangSmith helped Streamlit and Snowflake implement agents that could intelligently and reliably answer questions about their data.\\n\\nIn another example of debugging in action, we partnered with DeepLearningAI to equip learners in the recently-released LangChain courses with access to LangSmith. This allowed students to easily visualize the exact sequence of calls, and the inputs and outputs at each step in the chain with precision. Students can understand exactly what the chains, prompts, and LLMs were doing, which helps build intuition as they learn to create new and more sophisticated applications.\\n\\nTesting\\n\\nOne of the main questions we see developers grapple with is: “If I change this chain/prompt, how does that affect my outputs?” The most effective way to answer this question is to curate a dataset of examples that you care about, and then run any changed prompts/chains over this dataset. LangSmith first makes it easy to create these datasets from traces or by uploading datasets you’ve curated manually. You can then easily run chains and prompts over those data sets.\\nHuman: What is Flash attention?\"\n",
      "  ]\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[llm/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain > 4:chain:LLMChain > 5:llm:AzureChatOpenAI] [963ms] Exiting LLM run with output:\n",
      "\u001b[0m{\n",
      "  \"generations\": [\n",
      "    [\n",
      "      {\n",
      "        \"text\": \"I'm sorry, but there is no context provided that could help me determine what Flash attention is. Could you please provide more information or context about where you heard or read about it?\",\n",
      "        \"generation_info\": {\n",
      "          \"finish_reason\": \"stop\"\n",
      "        },\n",
      "        \"type\": \"ChatGeneration\",\n",
      "        \"message\": {\n",
      "          \"lc\": 1,\n",
      "          \"type\": \"constructor\",\n",
      "          \"id\": [\n",
      "            \"langchain\",\n",
      "            \"schema\",\n",
      "            \"messages\",\n",
      "            \"AIMessage\"\n",
      "          ],\n",
      "          \"kwargs\": {\n",
      "            \"content\": \"I'm sorry, but there is no context provided that could help me determine what Flash attention is. Could you please provide more information or context about where you heard or read about it?\",\n",
      "            \"additional_kwargs\": {}\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    ]\n",
      "  ],\n",
      "  \"llm_output\": {\n",
      "    \"token_usage\": {\n",
      "      \"prompt_tokens\": 557,\n",
      "      \"completion_tokens\": 37,\n",
      "      \"total_tokens\": 594\n",
      "    },\n",
      "    \"model_name\": \"gpt-35-turbo\",\n",
      "    \"system_fingerprint\": \"\"\n",
      "  },\n",
      "  \"run\": null\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain > 4:chain:LLMChain] [964ms] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"text\": \"I'm sorry, but there is no context provided that could help me determine what Flash attention is. Could you please provide more information or context about where you heard or read about it?\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA > 3:chain:StuffDocumentsChain] [964ms] Exiting Chain run with output:\n",
      "\u001b[0m{\n",
      "  \"output_text\": \"I'm sorry, but there is no context provided that could help me determine what Flash attention is. Could you please provide more information or context about where you heard or read about it?\"\n",
      "}\n",
      "\u001b[36;1m\u001b[1;3m[chain/end]\u001b[0m \u001b[1m[1:chain:RetrievalQA] [2.94s] Exiting Chain run with output:\n",
      "\u001b[0m[outputs]\n",
      "I'm sorry, but there is no context provided that could help me determine what Flash attention is. Could you\n",
      "please provide more information or context about where you heard or read about it?\n",
      "\n",
      "\n",
      "Sources:\n",
      "data/langchain_blog_posts/blog.langchain.dev_chat-loaders-finetune-a-chatmodel-in-your-voice_.txt\n",
      "data/langchain_blog_posts/blog.langchain.dev_announcing-langsmith_.txt\n",
      "data/langchain_blog_posts/blog.langchain.dev_announcing-langsmith_.txt\n"
     ]
    }
   ],
   "source": [
    "query = \"What is Flash attention?\"\n",
    "llm_response = qa_chain(query)\n",
    "process_llm_response(llm_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}