The short answer

It's easier for our brains. By looking away from our conversation partner's face, we are reducing the cognitive load of processing external stimuli so that we can shift our attention to internal stimuli of problem solving, creative thinking, retrieving memory, etc.

The long answer

Looking at another person's face is a very intensive exercise for your brain. You are constantly monitoring the other person's facial expression changes and body language to make inferences on their emotional state. In short, you are absorbing a ton of external stimuli that needs to be processed instantaneously.

If you don't break eye contact, your brain's ability to think is hampered and you might find it difficult to think of what to say in response to your conversation partner. A study conducted by Kajimura and Nomura (2016) demonstrated this effect. The researchers set up a word association game with participants who were asked to immediately respond with a verb that they associated with a provided noun. For example, if the word "book" was given, the participants would reply with "read" or "study." Additionally, participants were asked to play the game while maintaining eye contact with a face on a computer screen. The findings showed that participants who broke eye contact when faced with a challenging noun were able to respond more quickly than those who maintained eye contact.

Breaking eye contact, "looking at nothing", and increased blinking are all associated with your brain trying to process internal stimuli. As the artist Paul Gauguin once said, "I shut my eyes in order to see."

Sources

Kajimura, S., &; Nomura, M. (2016). When we cannot speak: Eye contact disrupts resources available to cognitive control processes during verb generation. Cognition, 157, 352–357. https://doi.org/10.1016/j.cognition.2016.10.002

Salvi, C., &; Bowden, E. M. (2016). Looking for Creativity: Where Do We Look When We Look for New Ideas? Frontiers in Psychology, 7. https://doi.org/10.3389/fpsyg.2016.00161