Amazon didn't officially include podcasting support when they shipped the first Kindle devices, but that doesn't mean you can't listen to your favorite podcasts on the Kindle (or your favorite songs for that matter). There are actually two interesting ways to make this work:
1) You can play any MP3 audio file as background music on the Kindle. You transfer MP3 files to the Kindle using the USB connection, to either store audio on the device or on an SD card. Then from the Home screen do the following:
Scroll to the Menu and select Experimental
Select Play Music on the screen shown
This will continue to play until you click the Stop Music option in Experimental (or the Alt+P keyboard shortcut). You can jump to the next track using Alt+F.
2) The more useful way to listen to podcasts on your Kindle is by adding them to the Audible folder either on your device or on your SD card. This is actually designed specifically for listening to Audible Audiobooks, but also happens to work with MP3s. Once you have the MP3 files in the Audible folder, the Kindle treats them just like other audiobook files allowing you to jump back 30 seconds, skip ahead 30 seconds, play and pause audio.
While you can't subscribe to podcasts directly on the Kindle yet, these are two ways to use it as a podcast listening device.
See also: A giant list of Kindle compatible Audiobooks.