Knowledgebase RAG System - Intelligent Chatbot Integration
RAG-based Question-Answering with Document, FAQ and News Integration
Project Description
This project implements a comprehensive knowledgebase system using Retrieval-Augmented Generation (RAG) technology for intelligent question-answering in the chatbot. The system integrates documents, FAQs, and news articles to provide context-based answers using OpenAI GPT-4o-mini. It features vector search for document similarity, TOON format for efficient context formatting, automatic storage of unanswered questions, and seamless integration with the document management system. The chatbot prioritizes FAQ matches, then uses the knowledgebase for detailed answers, and falls back to news articles as an additional information source.
Key Features
RAG-based Answer Generation
OpenAI GPT-4o-mini with structured JSON responses
Vector Search
Embedding-based document similarity search
Document Integration
Automatic indexing and vectorization of documents
FAQ Integration
Keyword-based FAQ search with scoring
News Integration
News article search as fallback information source
Question Management
Automatic storage and management of unanswered questions
Technology Stack
Backend Framework
AI & ML
Frontend
Data Management
Workflow
- User Query: User asks a question in the chatbot
- FAQ Search: System searches in FAQ files with keyword matching and scoring
- Knowledgebase Search: If FAQ score is low, system searches in vectorized documents
- Context Formatting: Search results are formatted as TOON (Token-Oriented Object Notation)
- Answer Generation: OpenAI GPT-4o-mini generates answer based on context with JSON schema
- Source Display: Relevant document sources are displayed with chunk information
- Question Storage: If no answer found, question is automatically stored for review
- News Fallback: If no document match, system searches in news articles