Development of A Legal Information Question Answering System Using Openai, RAG, and Langchain Technology

Abstract

This research aims to develop a QA system based on Retrieval-Augmented Generation (RAG) as a solution to improve public accessibility and understanding of legal information regarding public services. The background for this study is the difficulty often faced by the public in understanding legal procedures and their rights, as well as the limitations of the existing information service system at the Lembaga Ombudsman Daerah Istimewa Yogyakarta (LO DIY). The literature review in this study covers several main pillars: the profile and function of the Regional Ombudsman Institution of the Special Region of Yogyakarta (LO DIY), the basic concepts of chatbot technology, Generative AI, and Large Language Models (LLM). The main focus is on the Retrieval-Augmented Generation (RAG) architecture, which combines external information retrieval capabilities with text generation to enhance the accuracy and relevance of answers. A review of previous research indicates the successful implementation of RAG in various domains. Based on the literature study, four main indicators were established for chatbot evaluation: Perceived Usefulness, Accuracy, Relevance, and Clarity. This research methodology is systematically structured through four main stages: requirements analysis, design, implementation, and evaluation. The requirements analysis stage was conducted through interviews with experts from LO DIY and the collection of relevant legal documents. The design stage included the creation of use case diagrams, activity diagrams, and an intuitive user interface design. The implementation stage involved developing the system using the LangChain framework, the GPT-4 model, and the Chroma vector database to apply the RAG architecture. The evaluation stage was conducted by involving five legal experts from LO DIY to compare the responses of the RAG chatbot with a non-RAG one and to fill out a questionnaire based on the established indicators. The research results show that the chatbot developed with the RAG architecture is significantly superior to the non-RAG chatbot. The evaluation by experts confirmed these findings with very positive scores, with the average for Perceived Usefulness (12), Relevance (8), and Clarity (8.4) indicators falling into the "High" category, and Accuracy (18.6) in the "Very High" category. Therefore, it is concluded that the implementation of the RAG architecture successfully created an effective, accurate, and user-friendly chatbot for public service legal guidance.