Scientists have discovered a new species of human ancestor deep in a South African cave, adding a baffling new branch to the family tree.