zsmith.co

Is there any science in computer science or software engineering?

Revision 14
© 2013-2018 by Zack Smith. All rights reserved.

Question assumptions

Assumptions are beliefs that usually go unquestioned, especially when large numbers of people believe them to be valid. But it is a logical fallacy to decide something is true just because many people believe it. In fact, the wisdom of the crowd is a misnomer; a crowd's collective IQ is often (but not always) quite a bit lower than any individual member's for a number of reasons.

There is a great deal of pro-high-tech hype in the profit-minded media and in profit-minded academic institutions and this hype is supported by assumptions. One of these assumptions is that CS is a science, which lends a dignifying shine and air of authority to the subject.

Is computer science a science?

No, for the most part, it is not. In order for a discipline to be a science in any genuine or meaningful sense, it has to employ the scientific method. This is what qualifies something as a hard science. A soft science by comparison is one where, like in anthropology or astronomy, experimentation is not possible so observation and analysis are the focus.

Computer scientists largely do not use the scientific method and surely some could not describe it. Computer science is also not primarily about observation. It is therefore not a science in either the hard or soft sense. Better would be to replace the term computer science with computing studies or even use the common European term informatics.

CS as it is practiced is probably about as scientific as economics or political science. And even though software does much to enable scientists to do new kinds of experiments, that does not make CS a science.

A Stanford lecturer Andy Freeman joked that computer scientists firmly believe in the one true way, which varies based on beliefs and circumstances. While he was joking, there is truth in this. Any student of science knows that such thinking is the opposite of scientific reasoning. However to extend the joke, I will leave it as a homework assignment for the reader to identify some ways in which computer science is religion-like. For instance, have you ever heard someone refer to the beauty of object oriented programming?

What is the Scientific Method?

It is a brave approach to learning the truth that embraces uncertainty as a challenge rather than a threat, relies on evidence rather than wishes or dogma, questions its own conclusions and rejects unprovable claims.

What are the steps of the Scientific Method?

  1. Observe. Formulate a useful question.
  2. Hypothesize.
  3. Predict. If the hypothesis were true, what will result?
  4. Experiment to test the hypothesis.
  5. Analysis. Form a conclusion.

If a hypothesis is proven false, then think hard, question yourself, devise a new hypothesis and run a new experiment to test it. Rinse and repeat.

Science by association

Academics ought to resist the urge to include under the heading of CS more sciencey disciplines such as Human-Computer Interaction simply for the purpose of giving CS more respect or scientific street cred.

Academics should also resist importing terminology from hard science into CS for the purpose of adding a dint of scientificness. For instance, the term polymorphism was imported from biology. Polymorphism technically means many shapes, although in biology it refers to multiple stable coexisting traits within a species, like blonde hair and black hair in humans. In CS, polymorphism has come to mean several things, but its most commonly understood meaning is that of many methods or many implementations. This would be better expressed as polymethodoi as the Ancient Greek term for method is methodos, the plural being methodoi.

Searching for the science in CS

While CS as a whole is not a science, some bits of it may use the Scientific Method. Other bits may use it informally. In this way it is a bit like the study of psychology.

Examples of science within CS may include the development of voice recognition and genetic algorithms, and penetration testing. These required experimentation, potentially involving clear and explicit hypotheses and controlled experiments. Did researchers follow a rigorous scientific approach? Only they know. And are these not more like engineering endeavors than pure research? Perhaps.

A list of computing-related experiments

This is a section where I hope to list a great many found examples of scientific experiments related to computing, as they become known to me.

  • An experiment to assess what job ads are shown to women.
    • Automated Experiments on Ad Privacy Settings by Datta, Tschantz and Datta.
    • Women less likely to be shown ads for high-paid jobs on Google, The Guardian July 2015.

Does software engineering employ the scientific method?

Software engineering is sometimes described as being an offshoot of Psychology or Sociology, because it is as much about dealing with solving problems to do with people as it is about solving technological problems. Nevertheless there is a handful of situations wherein software engineers do something akin to the Scientific Method.

Experimentation on opaqueware

Some inherited software is so convoluted and hard to understand, you might as well call it opaqueware rather than the usual software debt. In the face of absurd complexity, over-engineered code, and/or bad or missing documentation a programmer may have to resort to something rather like the Scientific Method just to make sense of the code.

Observation: Input X leads to output Y.
Hypothesize: There is an internal set of rules Z causes this behavior.
Prediction: Given those rules, other inputs should cause specific outputs.
Experiment: Test the hypothesis using a variety of inputs that conform or break the rule(s).
Analysis: Hypothesis confirmed or dispelled.

Algorithm design

The act of creating a novel algorithm is sometimes done in such a way that the design process involves a succession of micro-experiments.

The engineer first contemplates the problem, and then asks If I write the code this way will it work?. This implies the hypothesis: Doing it this way will produce the correct performance or behavior.

Then the engineer runs experimental code that implements the algorithm on some input to test whether it works. Or he walks through the code mentally and writes down the program output.

If the code does not work (the hypothesis that it will work when written a certain way was false) he must question his assumptions, then tweak the code or rewrite it and run the new micro-experiment.

These micro-experiments are more like creating furniture however than doing proper science.

Observation: The algorithm is not working right. The output is wrong or the running time is excessive.
Hypothesis: It is missing a key feature that I can add.
Prediction: If I change this like so, all the tests will run correctly.
Experiment: Run the code, measure results.
Analysis: It worked or didn't work.

Debugging

While some debugging does not require experimentation but rather analysis, when an engineer is faced with a mystifying bug sometimes it is in fact better to treat the code like a black box and then make and test hypotheses. Don't assume you can guess the problem. Assuming may actually delay you longer than not assuming.

I am wise because I know I know nothing.
   - Socrates

When you ASSUME, you make an ASS out of U and ME!
   - Benny Hill

For instance, someone claimed the code will recognize a sensor that comes into range. That's an assumption, not a proven fact. A simple experiment may prove or disprove it, and additional hypotheses and experiments can flesh out the range of circumstances where the basic claim holds.

Reproducible builds

In order to establish reliably that the software a user receives is solely based on the software that the programmers wrote, and does not include any spyware or other malware that was introduced by a middleman it is possible to leverage the crowd to verify software by regenerating builds on many machines to be compared with the software builds that project maintainers actually distributed.

However in order for builds to be reproducible, software has to be readily buildable, unlike something like e.g. the Tor Browser.

Observation: We need reproducible builds.
Hypothesis: Our software does not have any spyware added to it e.g. on corrupted build computers.
Prediction: When other people build the software from the same source code, they will get the same checksums as we did on the build computer.
Experiment: Have many people rebuild the software and compare checksums to that of the original.
Analysis: The build computer is corrupted or not.

Regression testing

The Q/A personnel have a simple question to answer along the lines of: Did anything break when the engineers made changes to module X of the software at 2am? The experiment is to simply run through a list of program behaviors for the affected code. This proves or disproves the hypothesis.

Observation: Software changed in module X.
Hypothesis: The software did get broken.
Prediction: When we do these 20 different tests, some will definitely show incorrect results.
Experiment: Run the tests.
Analysis: It's broken or not.

A/B testing

These are UI experiments run by UI/UX designers, and simply implemented by engineers. Certainly A/B testing is an example of actual science if done properly. They contemplate their users' psychology, they design an experiment and they run the experiment. Hypotheses like This background color will spur more online sales are proven or disproven. Conclusions are drawn, such as warm colors may increase sales.

Observation: People respond to color emotionally.
Hypothesis: A blue background on the checkout cart webpage will help sell more products.
Prediction: Assigning a blue background to half of the users' checkout pages instead of the usual white background will show greater sales for users who get the blue background.
Experiment: Randomly change 50% of users' backgrounds to blue, leaving the rest red.
Analysis: Sales to blue-background users were higher by 5%, so it appears the hypothesis is proven correct.