Is there anything specific to open source about this question? If you're a software developer, you might have to decide whether you want to work for a shady company, or whether you want your smaller company to contract with a larger shady company. Those are I think harder decisions to make, because it could be your job on the line.
In the open source world, at least you don't know for sure what people are going to do with your work.
But we do know that if a company is looking to be evil, it's probably going to find a way, whether or not it uses your library.