Abstract: Inorder to achieve the classified automatic picking, this system was designed based on infrared image recognition and machine vision.The RGB and HSI grayscale analysis were done in visible band by detached tomato image.The result showed that grayscale of hue H could classify the Maturity of harvesting-tomato except semi mature and mature period. There was a obviously different in grayscale of infrared image in 810 nm between semi mature period and mature period. The model for distinguish the semi mature period and mature period was achieved by grayscale of G, R, H and NIR, the standard deviation of grayscale of G, R, H and NIR. Based on hue H, the core and radius of tomato were calculated by clustering algorithm. Tests for maturity differentiation and radius of tomato were taken, the result showed that accuracy for the maturity differentiation was above 94.8%, and radius error of tomato was less than 6%.