Category: Others

The world of Generative AI and Large Language Models (LLMs) is booming, offering groundbreaking possibilities for creative text formats, intelligent chatbots, and more. But for those new to AI development, the technical hurdles can be daunting. Setting up complex environments with libraries and frameworks can slow down the learning process.

CloudxLab is here to break down those barriers. We offer a unique platform where you can build Generative AI applications entirely within our cloud lab. This means you can:

• Focus on Creativity, Not Configuration: No more wrestling with installations or environment setups. Our cloud lab provides everything you need to start building right away.
• Seamless Learning Experience: Dive straight into the exciting world of Generative AI and LLMs. Our platform streamlines the process, letting you concentrate on understanding and applying these powerful technologies.
• Accessible for All: Whether you’re a seasoned developer or a curious beginner, CloudxLab’s cloud environment makes Gen AI and LLM development approachable.

Continue reading “Building Generative AI and LLMs with CloudxLab”

Imagine a tireless assistant on your website, ready to answer customer questions 24/7. That’s the power of a chatbot! In this post, we’ll guide you through building a custom chatbot specifically trained on your website’s data using OpenAI and Langchain. Let’s dive in and create this helpful conversational AI!

If you want to perform the steps along with the project in parallel, rather than just reading, check out our project on the same at Building a RAG Chatbot from Your Website Data using OpenAI and Langchain. You will also receive a project completion certificate which you can use to showcase your Generative AI skills.

Step 1: Grabbing Valuable Content from Your Website

We first need the gold mine of information – the content from your website! To achieve this, we’ll build a web crawler using Python’s requests library and Beautiful Soup. This script will act like a smart visitor, fetching the text content from each webpage on your website.

Here’s what our web_crawler.py script will do:

1. Fetch the Webpage: It’ll send a request to retrieve the HTML content of a given website URL.
2. Check for Success: The script will ensure the server responds positively (think status code 200) before proceeding.
3. Parse the HTML Structure: Using Beautiful Soup, it will analyze the downloaded HTML to understand how the webpage is built.
4. Clean Up the Mess: It will discard unnecessary elements like scripts and styles that don’t contribute to the core content you want for the chatbot.
5. Extract the Text: After that, it will convert the cleaned HTML into plain text format, making it easier to process later.
6. Grab Extra Info (Optional): The script can optionally extract metadata like page titles and descriptions for better organization.

Imagine this script as a virtual visitor browsing your website and collecting the text content, leaving behind the fancy formatting for now.

Let’s code!

import requests
from bs4 import BeautifulSoup
import html2text


def get_data_from_website(url):
    """
    Retrieve text content and metadata from a given URL.

    Args:
        url (str): The URL to fetch content from.

    Returns:
        tuple: A tuple containing the text content (str) and metadata (dict).
    """
    # Get response from the server
    response = requests.get(url)
    if response.status_code == 500:
        print("Server error")
        return
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.content, 'html.parser')

    # Removing js and css code
    for script in soup(["script", "style"]):
        script.extract()

    # Extract text in markdown format
    html = str(soup)
    html2text_instance = html2text.HTML2Text()
    html2text_instance.images_to_alt = True
    html2text_instance.body_width = 0
    html2text_instance.single_line_break = True
    text = html2text_instance.handle(html)

    # Extract page metadata
    try:
        page_title = soup.title.string.strip()
    except:
        page_title = url.path[1:].replace("/", "-")
    meta_description = soup.find("meta", attrs={"name": "description"})
    meta_keywords = soup.find("meta", attrs={"name": "keywords"})
    if meta_description:
        description = meta_description.get("content")
    else:
        description = page_title
    if meta_keywords:
        meta_keywords = meta_description.get("content")
    else:
        meta_keywords = ""

    metadata = {'title': page_title,
                'url': url,
                'description': description,
                'keywords': meta_keywords}

    return text, metadata

Explanation:

The get_data_from_website function takes a website URL and returns the extracted text content along with any optional metadata. Explore the code further to see how it performs each step mentioned!

Step 2: Cleaning Up the Raw Text

Continue reading “Building a RAG Chatbot from Your Website Data using OpenAI and Langchain (Hands-On)”