This paper presents a novel hybrid model for classifying time series of heart sound signal using time-growing neural network. The proposed hybrid model takes segmental behaviour of heart sound signal into account by combining two different deep learning methods, the Static and the Moving Time-Growing Neural Network, which we call STGNN and MTGNN, respectively. Flexibility of the model in learning both deterministic and stochastic segments of signal allows it to learn those complicated characteristics of heart sound signal caused by any obstruction on semilunar heart valve. The model is trained to distinguish between a patient group and a reference group. The patient group is comprised of the subjects with the semilunar heart valve abnormalities including aortic stenosis, pulmonary stenosis and bicuspid aortic valve, whereas the reference group which is composed of the individuals with the heart abnormalities other than those of the reference group or the healthy ones. The model is validated using two different databases: one comprised of 140 children with various heart conditions, and the other one constituted of 50 elderly patients with aortic stenosis. Both the datasets were collected from the referrals to the University hospitals. The overall accuracy and sensitivity of the model are estimated to be 84.2% and 82.8%, respectively. The results show that the model exhibits sufficient capability to distinguish between the patient and the reference group in such a demanding clinical application.