There is an idea or saying that elephants never forget or that they have very good memories First article in google about this.

So to say you have a memory of an elephant could mean your memory is very good.

The twist is that this can also mean you remember seeing an elephant, which is what the second line implies.