Abstract: Accurate identification and effective management of plant diseases are fundamental to ensuring global food security, agricultural sustainability, and public health. Annually, plant diseases cause over 30% of crop yield losses worldwide. However, existing research predominantly focuses on limited model plants or cash crops. Furthermore, current end-to-end Visual Large Language Models (VLLMs) encounter significant bottlenecks in practical agricultural applications, including prohibitive computational costs, low accuracy in detecting subtle or early-stage disease symptoms against complex field backgrounds, and severe "hallucination" issues leading to insufficient knowledge reliability. To comprehensively address these challenges, this study proposed an innovative, decoupled plant disease diagnostic and question-answering framework that synergistically integrates advanced computer vision with lightweight large language models. To establish a robust foundation, the study constructed a comprehensive multimodal dataset comprising 138 499 rigorously annotated images covering 30 common crop species and 157 disease categories (including healthy states), alongside 54 297 high-quality instruction-tuning dialogue samples. Additionally, we built a specialized semantic vector database via retrieving peer-reviewed 2016–2025 agricultural literature abstracts on 30 major crop diseases: initial retrieval reached 85 997 documents, and after filtering out entries with missing abstracts, non-academic editorials and retracted corrections, 75 156 high-value valid peer-reviewed literature records were finally retained. The image recognition module, named AgriExpert, was developed based on the Swin-Transformer Tiny (Swin-Tiny) architecture. It innovatively incorporated a Multi-Scale Fusion (MSF) block to capture fine-grained lesions and textural variations, a Shared Projection (SP) layer to align semantic feature spaces, and multi-species expert classification heads (multiHead). This decoupled expert routing mechanism effectively alleviated gradient conflicts caused by visually similar diseases across different host plants and mitigated the negative impacts of long-tail data distributions. For the intelligent question-answering module, four mainstream lightweight large language models (LLMs)—including LLaMA-2-7B, TinyLlama-1.1B, Mistral-7B, and Qwen2.5-7B—were evaluated and fine-tuned. Utilizing the Low-Rank Adaptation (LoRA) instruction fine-tuning strategy, the models were deeply aligned with complex agricultural semantics regarding disease symptoms, prevention measures, and pathogenic mechanisms, achieving high efficiency while freezing pre-trained backbone weights. Furthermore, a Retrieval-Augmented Generation (RAG) mechanism powered by SentenceTransformer and FAISS indexing was introduced. This mechanism dynamically retrieved highly relevant peer-reviewed literature fragments and integrated them into the response generation process, thereby significantly improving knowledge reliability, factual accuracy, and diagnostic traceability. Extensive experimental results demonstrated the superiority of the proposed framework. The visual recognition model achieved an outstanding overall accuracy of 98.74% on the 157-category classification task. It also exhibited remarkable robustness, maintaining a high accuracy of 89.52% under real-world complex background conditions, significantly outperforming conventional CNN and standard Transformer baselines. For the question-answering module, instruction fine-tuning led to substantial quantitative improvements across automatic evaluation metrics. Specifically, the LoRA-fine-tuned Mistral-7B-4bit achieved BLEU-1, ROUGE-1, and ROUGE-L scores of 0.314 2, 0.433 2, and 0.274 7, respectively, indicating enhanced lexical precision and semantic coverage. Similarly, Qwen2.5-7B-4bit improved to 0.244 2, 0.383 5, and 0.231 7 on the same metrics. In external benchmark evaluations using 50 multimodal question-answer pairs, comprehensive manual scoring by domain experts across five dimensions—correctness, clarity, reasoning depth, usefulness, and completeness—further demonstrated that the fine-tuned models generated highly professional and structural responses comparable to or exceeding current massive proprietary models. Overall, the integrated framework successfully decoupled visual perception from language reasoning. Reinforced by retrieval-based knowledge grounding, it effectively reduced hallucinated content and provided verifiable literature evidence. Notably, the lightweight architecture requires less than 7 GB of Peak VRAM for inference, drastically reducing hardware reliance and facilitating seamless deployment on agricultural edge computing devices. The results suggested that this decoupled paradigm provides an efficient, scalable, and highly reliable solution for intelligent agricultural diagnosis, offering substantial practical value for real-time field disease management.