Diagnosis and Q&A of multi-crop diseases based on computer vision and large language model

Plant diseases have caused over 30% of crop yield losses worldwide each year. Accurate identification of plant diseases is often required for global food security in sustainable agriculture. However, existing research has focused on a limited model plants or cash crops. Furthermore, current end-to-end Visual Large Language Models (VLLMs) cannot detect subtle or early-stage disease symptoms under complex field backgrounds, particularly due to excessive computational costs, low accuracy, and severe "hallucination" issues. In this study, a decoupled framework was proposed for diagnostic and question-answering of plant diseases. Advanced computer vision was also integrated with lightweight large language models. A multimodal dataset was constructed with 138 499 annotated images, including 30 common crop species and 157 disease categories (including healthy states), alongside 54 297 dialogue samples for instruction tuning. Specifically, a semantic vector database was constructed to retrieve peer-reviewed 2016–2025 literature abstracts on 30 major crop diseases. 75 156 peer-reviewed literature records were valid from the retrieval of 85 997 documents. The image recognition module, named AgriExpert, was developed using the Swin-Transformer Tiny (Swin-Tiny) architecture. Multi-Scale Fusion (MSF) block was incorporated to capture fine-grained lesions and textural variations. Shared Projection (SP) layer aligned semantic feature spaces using multi-species expert classification heads (multiHead). The expert routing mechanism effectively alleviated gradient conflicts among visually similar diseases across different host plants, thus reducing the negative impacts of long-tail data distributions. Four lightweight LLMs were evaluated and finely tuned in the intelligent question-answering module, including LLaMA-2-7B, TinyLlama-1.1B, Mistral-7B, and Qwen2.5-7B. According to the Low-Rank Adaptation (LoRA) instruction fine-tuning strategy, the complex agricultural semantics were aligned with disease symptoms, prevention measures, and pathogenic mechanisms, indicating high efficiency and leveraging pre-trained backbone weights. Furthermore, a Retrieval-Augmented Generation (RAG) mechanism was introduced to dynamically retrieve highly relevant peer-reviewed literature fragments using SentenceTransformer and FAISS indexing. The response generation was integrated to significantly improve knowledge reliability, factual accuracy, and diagnostic traceability. Experimental results demonstrated that the superiority of the framework was achieved in visual recognition, with an overall accuracy of 98.74% on the 157-category classification task. Remarkable robustness was also maintained with high accuracies of 89.52% and 86.82%, respectively, under real-world complex background and simulated extreme noise conditions. The performance significantly outperformed conventional CNN and standard Transformer baselines. In the question-answering module, instruction fine-tuning substantially improved the evaluation metrics. Specifically, the LoRA-fine-tuned Mistral-7B-4bit achieved BLEU-1, ROUGE-1, and ROUGE-L scores of 0.314 2, 0.433 2, and 0.274 7, respectively, indicating enhanced lexical precision and semantic coverage. Similarly, Qwen2.5-7B-4bit improved to 0.244 2, 0.383 5, and 0.231 7 on the same metrics. In external benchmark evaluations using 50 multimodal question-answer pairs, manual scoring by domain experts over five dimensions—correctness, clarity, reasoning depth, usefulness, and completeness—demonstrated that the fine-tuned models generated highly professional and structured responses, thus exceeding current massive proprietary models. Overall, the framework was integrated to decouple visual perception from language reasoning. Retrieval knowledge grounding was reinforced to effectively reduce hallucinated content for verifiable literature evidence. Notably, the lightweight architecture required less than 7 GB of Peak VRAM for inference, thus reducing hardware reliance for seamless deployment on agricultural edge computing devices. The findings can provide an efficient, scalable, and reliable solution for intelligent diagnosis in modern agriculture. Substantial practical value can also offer real-time detection of plant diseases in the field.