I think its due to a strain that they put on the mind. Maybe it is a price you must pay in order for them to lend you their power. The memories dont even seem to be lost, rather they are suppressed, as Squall and the others eventually remember their time at the orpahnage after going back there. What do you think?