"All But"

That little phrase really annoys me. In the English language, the phrase "all but" is commonly used to say something is "almost". Example: something that is "all but impossible" is interpreted as "near impossible", or "practically impossible". This annoys me. Why? Because when you say all, you mean everything. When you say but, you mean contrary, except, that, than, or only. So really, what this should come out to is everything but, or everything except. Substituting, "all but impossible" becomes "everything except impossible". See what happened there? Using substitution -and not invalid substitution- we have changed the meaning of "all but impossible" from "near impossible" to "everything except impossible", which is to say, not impossible, or possible. And this really throws my brain for a loop, because I know that both interpretations are right, but no-one will interpret "all but impossible" as "possible", simply because it is not commonly used that way. And that makes me angry, because I'd like to use it to mean "something isn't", as opposed to "something is". Would you agree?