Well, you can show me skeleton being called "remains of Napoleon.
Does human turn into bare bones after death? Sure!
But I would rather see person
alive AND dying than seeing those remains. THEN I would know, that the date of this persons death is 100% accurate and such event happened.
But get back to memories:
You do have some memories, right? Feelings, situations etc. But
imagine, that is possible to create human looking for 30 y.o. with brain memorizing events that never happened. This human will be like occupant from
"The lost room". Noone knows his history, and he's the only one being concerned about some "other" past.