Why do we have leap years every four years?

What would happen to our calendar and seasons over time if we did not add an extra day to February every four years? How does Earth's actual orbit map to days?