There are a few different settings that you can go with on this; the default setting is set so you can remove Firefox's download history manually one file at a time, but you can also set the Download Manager to erase download history of files every time you exit Firefox.
I have my Firefox Download Manager set to erase downloaded files every time they successfully download. You typically do not need to re-download files if they have indeed been successfully downloaded, and this will keep your download history nice and clear.
