If you're around someone and you begin muttering under your breath, the natural assumption would be that you're complaining to yourself about them. Otherwise, you'd speak up and let them know how you feel. So do your muttering to yourself when you're alone or just be bold enough and kind enough to speak your mind without hurting anyone's feelings.