Mesial temporal lobe epilepsy (TLE) is characterized by stereotyped origination and spread pattern of epileptogenic activity, which is reflected in stereotyped topographic distribution of neuronal atrophy on magnetic resonance imaging (MRI). Both epileptogenic activity and atrophy spread appear to follow white matter connections. We model the networked spread of activity and atrophy in TLE from first principles via two simple first order network diffusion models. Atrophy distribution is modeled as a simple consequence of the propagation of epileptogenic activity in one model, and as a progressive degenerative process in the other. We show that the network models closely reproduce the regional volumetric gray matter atrophy distribution of two epilepsy cohorts: 29 TLE subjects with medial temporal sclerosis (TLE-MTS), and 50 TLE subjects with normal appearance on MRI (TLE-no). Statistical validation at the group level suggests high correlation with measured atrophy (R = 0.586 for TLE-MTS, R = 0.283 for TLE-no). We conclude that atrophy spread model out-performs the hyperactivity spread model. These results pave the way for future clinical application of the proposed model on individual patients, including estimating future spread of atrophy, identification of seizure onset zones and surgical planning. 