Based on the energy conservation relationship, nonlinear thermo-acoustic effects of Rijke tube including instability range, saturation processes and higher harmonics modes were investigated. With coupling between the external flow and the inner space of a Rijke tube, the acoustic characteristics of self-excited oscillation were simulated. The experimental study was also carried out and the results were compared with those from simulation. The nonlinear factors which distort the acoustic waveform distortion were analyzed. From the results, it is seen that varying size of the nozzle outlet changes the acoustic impedance in the boundary, and leads to reduction of the nonlinear effects. The results show that the modes of self-excited oscillation could be influenced by the position of higher harmonics. In the large amplitude oscillation, the distortion of pressure wave within Rijke tube could be induced by the acoustic losses due to vortices on nozzle. It is found that the waveform distortion could be avoided by the shrinkage of nozzle.