In this paper, the calibration method with nonlinear optimization introduced in camera system is researched. The mathematical model of pinhole perspective transformation of the camera is discussed, and the distortion coefficient of camera imaging is figured out. The traditional linear calibration method of camera is briefly introduced. The planar calibration method with nonlinear optimization introduced is emphatically analyzed, based on which, the camera parameters are combining with LM optimization algorithm. Finally, the experimental simulation proves that this method can describe the imaging situation of the camera better, and thus proves the feasibility of the calibration method with nonlinear optimization introduced in missile guidance system. In order to identify the precision of the camera system, a new approach is proposed to realize the accurate identification of the target.