Metaphors or Sparse Prose?

Sometimes I have the burning desire to erase every single metaphor and simile I use in writing. It’s not that I don’t like them, but I wonder if I could improve my story telling by not relying on them at all. In Dwellers, I’ve tried to keep them to a minimum. To my own minimum, I mean. In some cases, I erased a newly forming metaphor in mid-sentence and looked for a more direct description so as not to distract the reader from the story. But is a metaphor a distraction?
If written well, a metaphor brilliantly enriches a story. If it’s a cliché (with some exceptions) it’s at the very least a distraction if not a downright depletion of an entire paragraph. I think my metaphors and similes generally serve the story well, but sometimes during the revision process I find that they really don’t add anything of value. When I write a metaphor, I rather like the idea of readers gliding upon them. I would rather my readers linger on a powerful, sparse sentence than on a metaphor. But writing a killer sentence is easier said than done.
Dwellers is a story using a single, close third person POV. The story’s strength is dialogue, both the inner dialogue and the dialogue between characters. I think I’ll cut out a few more metaphors.