Recording environmental sounds like singing birds, wild animal calls, water running through a streambed, or the chirp of baby birds is one of the more fascinating (and challenging) aspects of audio recording. Some of the same audio recording techniques apply when recording things like car noises, lawn mower engines and other man made sounds, but you don't always get a chance to do a second take with nature like you do with machines.
Getting started in recording nature sounds requires three basic things:
1) A recording device, like a MiniDisc recorder, Compact Flash recorder, or other digital audio recorder.
2) A stereo microphone or stereo microphone pair
3) Headphones for listening to what you're recording
The big reason to use a digital recorder is to avoid the hiss that comes with using a tape recording medium. The stereo microphone or pair of microphones provide the listener with something much closer to what you'd hear in the real world than a single microphone accomplishes on its own. A microphone zeppelin windscreen is also useful for cutting noise from wind while recording.
You also need a good dose of luck and patience in order to find the specific species of bird or animal you're hoping to record. For a great overview of what's required for recording sounds from the wilderness, the Wildebeat Podcast has a great interview on recording the sounds nature with Kurt Fristurp (of the National Parks Service Natural Sounds Program) and Dan Dugan (of the Nature Sounds Society). Two great tips from the podcast are how to find places free from unnatural sounds and some of the basic techniques for getting a good sample of the sound you hope to record.
One of the best places to get a quick foundation in how to record nature sounds is the Nature Sounds Society. Their Question Bird service provides a solid foundation of answers for people looking for detailed information on the types of audio gear you need for recording various kinds of natural sounds.