I would be a lot more wary of a blacksmith who could knock up a whole load of pretty damn lethal weapons in short order with his bare hands than a physicist who would have to be a billionaire with a super secret head quarters and thousands of specialised minions to make one basic atomic bomb.
By the logic of "do we want people who know how to make dangerous weapons" we would't have any chemists, biologists, physicists, doctors, politicians/public figures (a mob is a dangerous weapon), carpenters, blacksmiths, toolsmiths...the list goes on and on and on.
On the other (rational) hand more scientists allow society to conduct more research and grow both our knowledge and understanding of the world. This allows us to develop better technologies and techniques to ultimately improve both the quality and longevity of our lives. A fraction of that scientific understanding is devoted towards military matters because ultimately human social interaction is interwoven with violent tendencies. But that isn't going to change if you stop doing research, what will change it is more research in social sciences and more application of better economic and social theories towards reducing conflict in the world right through from neighbourhoods to nations.