Microphones are extremely sensitive to direction.
To prevent excessive background noise most microphones will reject sound from certain directions and only accept frequencies from a particular area.
Distance is key.
Speaking too close to your microphone will cause distortion, speaking too far away will sound very distant and thin. Understanding the correct proximity is vital.
Mics don't just pick up your voice!
As well as hearing your voice, your mic will also hear creaking chairs, tapping on tables, bumps on the mic stand etc. Getting yourself and the mic in a comfortable and secure position will avoid these things.
How Can I Improve?
Understand your microphone's polar pattern (where it picks up sound most cleanly) and speak into it from this area. If you are using a Blue Yeti then see here for the correct placement.
Experiment with speaking from different distances from the mic. Most cardioid microphones such as the Blue Yeti work best when speaking from around 15cm away.
Make sure your microphone is in a secure position where it can't be knocked and there are no objects in the way that could be rustling, creaking, or brushing against the cables.
Get comfy! Before recording make sure you have a good posture which you can maintain for the duration of the recording. Ensure you won't be tempted to slouch back in your chair and slip further from the microphone. Also ensure you won't be fidgeting.
Even expensive microphones will sound terrible with bad placement;
Microphones don't just pick up your voice;
If in doubt, always feel free to check in with me for a mic test.