Considering the fact that we can get fossil records on single-celled organisms up to 3.46 billion years ago, it's unlikely we would completely miss the evolution and existence of a sapient species similar to man. Earlier than 3.5 billion years earth was a real mess with lots of tectonic activity and murderous weather patterns so it would be a shock to discover even extremophilic bacteria from that period or earlier.
All that said, I love this idea popping up in science fiction. The idea of an ancient race wiser than man is cool beans, but the existential dread that rises from the question "why did they disappear?" is a great plot hook. If they could not stop the interstellar destroyer race or giant asteroid or whatever, then humans have no chance of saving themselves the same fate.
On the other end of the scale, there's post-human evolution in the future. When we disappear, who will take our place? Land squids is my personal favorite, but there is a great (cheesy) book by Dougal Dixon that theorizes our future and what shall descend from our lineage. You can read the full book (it's short) with scanned images legally on this site:
http://www.sivatherium.narod.ru/library/Dixon_3/01_en.htmIt gets really goofy sometimes, but I enjoyed the whole thing. Dougal Dixon illustrated some of my favorite dinosaur books from when I was a kid - his illustrations are awesome.