Dismiss Notice
Join Physics Forums Today!
The friendliest, high quality science and math community on the planet! Everyone who loves science is here!

How can scientists trust closed source programs?

  1. Mar 11, 2016 #1

    fluidistic

    User Avatar
    Gold Member

    I wonder how can scientists trust closed source programs/softwares.
    How can they be sure there aren't bugs that return a wrong output every now and then? Assuming they use some kind of extensive tests that figures out whether the program behaves as it should, how can they be sure that the programs aren't going to suffer from bugs and stuff like that (malicious code included) in further releases? Are there any kind of extensive tests performed on software that are generally used in branches of physics or any other science involving data analysis? Blindly trusting a closed source program seems to go against scientific mindset to me.
     
  2. jcsd
  3. Mar 11, 2016 #2
    You can't, but then open source software is not immune to bugs either.
     
  4. Mar 11, 2016 #3

    Choppy

    User Avatar
    Science Advisor
    Education Advisor

    Unfortunately I think there is a lot of blind trust in both closed source programs (and open ones for that matter).

    That said, proper operation of a code is verified through (i) benchmarking and (ii) routine quality assurance testing, and (iii) independent checking. In my field, Medical Physics, for example, we often use commercial software for planning radiation therapy treatments in the clinic. They determine where the radiation dose goes in the patient and what parameters to set on the linear accelerator to deliver the intended treatment. It's very important that these codes get these calculations correct every time.

    So before implementing clinically, we first have to run through a set of basic tests to confirm that the code accurately reproduces measurements under given conditions. Of course even before this, we go through the literature, where these tests have been performed by others. This is how we can establish how reliable the given algorithm is and conditions under which any assumptions break down. This also lets us know what a reasonable tolerance is - how close to measurement values can we expect to get. Then we run through a set of our own tests confirming that our version performs as advertised. Of course, you can't test everything, but you can try to approximate both commonly encountered situations and extreme situations where the code may not perform so well.

    Once you've effectively benchmarked your code, it's also important to put it through routine quality assurance testing. So, for example, you may want to repeat a subset of your benchmarking calculations once a month, or after a software version upgrade, or after a patch installation, to assure yourself that your code is still performing as you expect.

    Finally, when it comes to something critical like clinical calculations, we confirm the results through redundant, and independent checks or measurements. This can be as simple as performing a hand calculation or using a completely different planning system to redo the calculation. When independent systems arrive at the same answer, you have some increased confidence that the answer is correct. It's still possible they can both arrive at the wrong answer - GIGO and all that - but this serves to increase confidence that at least your black box is working as expected.

    On a research front, it's important to be doing the same things.
     
  5. Mar 11, 2016 #4

    PeroK

    User Avatar
    Science Advisor
    Homework Helper
    Gold Member

    Why trust anything? How do you know the brakes on your car won't fail after 1000 miles? And, when you fly, do you do a personal check of the aircraft's engine, guidance systems and everything?

    Or, if you buy copper sulphate from someone, how do you know it really is copper sulphate? Or, if you buy a box of drugs, how do you know some of them aren't a placebo? Or, a different drug altogether?

    In order to fully check the source code for a system, you would need to be relatively expert in the software technologies. Many systems may be an integration of several technolgies, so no one person (even in the software development company) would be able to check it: you would need a team of people. Even then, source code may be inscrutable without all the software development facilities that were used to create it. In fact, from a software engineering perspective, starting with the source code would be a very inefficient way to verify a system.
     
  6. Mar 11, 2016 #5

    FactChecker

    User Avatar
    Science Advisor
    Gold Member

    For use in applications where safety is involved, there is usually a certification process that must be done before the software can be used. @Choppy 's example of radiation therapy is a good example of that. But then the computer operating system and compilers must also be certified. They are not just trusted blindly.

    In the case of scientific research that does not have safety consequences, there is no formal certification process. You should not be reckless about the software you pick to use. Don't use experimental versions unless you want to do a lot of testing that has nothing to do with your research. Unless you are doing something really unusual, there are probably well tested versions of software that you can use.
     
  7. Mar 11, 2016 #6

    FactChecker

    User Avatar
    Science Advisor
    Gold Member

    There is no "magic bullet" to guarantee valid code. Making bug-free software is an example of "defense-in-depth". There are software development process that should be followed, unit testing, integrated testing, code standards, code review processes, etc., etc., etc. There is a set of code standards, MISRA-C that tells you what you should do or not do in your code. There are several code analysis tools that examine code for risky practices, test coverage, etc. Even when all the processes and rules are followed, some bugs escape to the released software. Then it is up to the public and the developer to spot and fix the mistakes.
     
  8. Mar 11, 2016 #7

    fluidistic

    User Avatar
    Gold Member

    When you safely land with the plane, you know that if there was a problem it did not matter at all for you, unlike the case of having the output of a closed source program where you have no intuition on whether the results are fine or whether they are too low/high by say 0.8%. You don't necessarily get the feedback you'd get with a drug or plane or coppers sulphate.

    Of course checking the full source code is most of the time impossible. But one does not generally use all the functionalities of a complicated software either. Say I use a program that fits X ray diffractograms and the program claims to be using "name_of_algorithm"'s algorithm and I want to check out how exactly it's implemented. Or say the program claims to use the Scherrer equation to give out the crystallite's size but does not specify the value it uses for "K", the shape factor. In both cases it gets complicated to figure out what the program is really doing.

    I realize that when publishing a paper a scientist could specify that the values were calculated via "name_of_software_name_of_version". So that if one day someone realize something was faulty with that software, then one should either fix the results of the published scientist or discard them.
     
  9. Mar 11, 2016 #8

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    And if you crash, you know there was a problem. Which is certainly a much worse outcome than a value in a publication that is off a bit (with a few exceptions, like studies related to safety of systems like aircrafts...).

    But unlike the aircraft you use, you can check the software. Run it on test cases where you know the expected outcome. Sure, they don't cover everything, but if the software passes all test cases you can be quite confident that it works with your actual data as well.
    This is routinely done for basically all software packages.
     
  10. Mar 11, 2016 #9

    fluidistic

    User Avatar
    Gold Member

    That's what I wanted to know and that's reassuring. If the information of which tests have been done for which program and which version were available to the public, that would be great.
     
  11. Mar 11, 2016 #10

    jedishrfu

    Staff: Mentor

    One way to test it is to develop a test suite. However, even then its possible that a bug would slip through.

    If you recall there was the famous Pentium bug that occurred under specific circumstances:

    https://en.wikipedia.org/wiki/Pentium_FDIV_bug

    It would appear as a software bug but was in fact hardware related.
     
  12. Mar 12, 2016 #11

    mfb

    User Avatar
    2016 Award

    Staff: Mentor

    Publications often have a limited size, you cannot write up every little detail.
     
  13. Mar 12, 2016 #12

    FactChecker

    User Avatar
    Science Advisor
    Gold Member

    Popular software products often have web sites where bugs are reported and discussed.
     
  14. Mar 12, 2016 #13

    Vanadium 50

    User Avatar
    Staff Emeritus
    Science Advisor
    Education Advisor

    Ah yes. They call it "floating" point for a reason. :)
     
  15. Mar 12, 2016 #14

    stevendaryl

    User Avatar
    Staff Emeritus
    Science Advisor

    I don't want to be paranoid, but there is a huge difference between testing for accidental programming errors and testing for intentional malicious code. Intentionally malicious code can be programmed to only show up under certain circumstances that may not ever occur during testing. To me, that's a big difference between open source and closed source. In the case of open source, you can actually study the code to see if there is peculiar logic that would only show up in certain circumstances.
     
  16. Mar 12, 2016 #15

    stevendaryl

    User Avatar
    Staff Emeritus
    Science Advisor

    Paranoia may be justified in the case of software where there might be a motivation to sometimes giving the wrong answer (for example, the code that counts electronic votes in an election).
     
  17. Mar 12, 2016 #16
    In the end software development is just another engineering discipline.
    As with any other, the first version of something usually does have unexpected bugs, even though there may have been a lot of time dedicated to testing and QA.
    As the product matures later versions become more reliable until there are no longer are a significant number of bug reports, and those that there are frequently are not actually bugs but operator or input data errors.
    Even those get ironed eventually by 'defensive' programming adjustments which detect and report improper input and so on before the program will proceed.
     
  18. Mar 12, 2016 #17

    anorlunda

    User Avatar
    Science Advisor
    Gold Member

    This is a worthy topic. I think perhaps the focus on errors is too narrow. It is even too narrow to focus on closed software, or to focus on software at all. There are many ways for things to go wrong or right. Machines add some new risks and reduce other risks.

    May I recommend the book "Computer Related Risks" by Peter G. Neumann. Using numerous case histories, the book illustrates the nature and number of risks involved when humans and machines interact. It was written way back in 1994, but it is not at all dated. Many of the mistakes committed in decades past will be repeated in decades future. If you do read it, I expect that you will see that a much broader view of risks is appropriate.
     
  19. Mar 12, 2016 #18

    jedishrfu

    Staff: Mentor

    The VW experience is another example of software giving maliciously incorrect answers during pollution tests.
     
  20. Mar 12, 2016 #19

    PeroK

    User Avatar
    Science Advisor
    Homework Helper
    Gold Member

    How would you know that an election system was actually running the open source code that someone claimed it was?
     
  21. Mar 12, 2016 #20
    An individual voter may not know that, but if it happened it would be fraud and it would have to be perpetrated on a massive scale to be effective.
    Such a conspirarcy theory falls down at the first hurdle, like the 'moon landing hoax' theory.
    The conspiracy would need to involve many people, hundreds at least, keeping silent about something they knew about.
    That it's realistically not possible,
     
Know someone interested in this topic? Share this thread via Reddit, Google+, Twitter, or Facebook

Have something to add?
Draft saved Draft deleted