Main Dish (Idea):
A delicious and informative web-based dish that answers your specific questions based on the content of the URLs you provide, without relying on pre-existing knowledge. It’s like a personalized, URL-focused encyclopedia at your fingertips.
Ingredients (Concepts & Components):
- Fresh URLs: One or more web addresses, the source of our information. Think of these as the raw vegetables.
- Beautiful Soup (bs4): A web scraping tool to extract the textual content from the HTML structure of the URLs.
- Requests Library: Fetches the HTML content from the specified URLs. Like a net for catching the fish (web pages).
- Gemini API: A generative language model to provide answers to questions, given the extracted content.
- Streamlit: Creates the interactive web interface where users can enter URLs and ask questions. It’s the plate that presents our dish nicely.
- FAISS (Facebook AI Similarity Search): A library for efficient similarity search and clustering of dense vectors. It allows you to quickly find the most relevant parts of the scraped data for a given question.
- Sentence Transformers: A Python framework for state-of-the-art sentence, text and image embeddings.
Cooking Process (How It Works):
- Harvesting the Ingredients: The user provides a list of URLs. The app ensures that URLs are valid and separated by commas.
- Cleaning and Chopping: The “Requests” library fetches the HTML content from each URL. Then, “Beautiful Soup” skillfully extracts the main textual information, discarding irrelevant markups and tags.
- The Secret Sauce: The extracted text from all URLs is combined into a single, large document, ready to be processed by the language model.
- Text splitting The documents are chunked into small sizes
- Encode into embeddings Split documents are now transformed into embeddings which are numerical presentations of the text
- FAISS Indexing FAISS indexes the embeddings
- Flavoring with Knowledge: The user enters a question. The question is used to search embeddings
- Baking the Response: The question and the extracted text (the context) are sent to the Gemini API, along with the user’s API key. Gemini analyzes the context and generates an answer tailored to the question.
- Presentation is Key: The Streamlit interface neatly displays the question and the generated answer to the user.
- History Preservation: The question and answer are saved in the chat history sidebar, letting you review past interactions.
Serving Suggestion (Outcome):
A concise, accurate, and context-aware answer to your question, derived solely from the content of the specified URLs, all served up in a user-friendly web interface. The dish gives you the power to ask questions to the content of the websites without needing to have the website be pre-indexed.
