The coming data explosion and what this means for employable skills

[Simfish]

http://www.nytimes.com/external/rea...writeweb-the-coming-data-explosion-13154.html

An interesting book:
http://research.microsoft.com/en-us/collaboration/fourthparadigm/

So basically, the two main paradigms used to be experiment and theory. Then in the 1950s came simulations, and now we have data-intensive scientific discovery. Some people have recently written programs that can derive physical formulas from massive amounts of data. Such methods can produce true results without an a priori basis for scientific discovery, which runs counter to the scientific method.

==

Anyways, so I'm seeing that there are several skillsets that will become valuable quite soon. (a) working with better sensors that have additional dimensions of physical data, (b) data mining/pattern recognition, (c) finding ways to efficiently analyze mass amounts of physical data, (d) intuition with respect to finding patterns out of massive datasets (or finding algorithms that find the best patterns out of them)

So the question here, is, do you see these skillsets as extremely employable in the near future (perhaps more employable than many other skillsets)? And what would people look for if they look for people with such skillsets?

For instance, I would like to go for a PhD in astrophysics. Astrophysics, of course, is one beneficiary of this revolution, as we get better sensors (telescopes/CCDs) and massive amounts of data to analyze. But I have many scientific interests, and I'm especially interested in other applications of this upcoming revolution (especially as it applies to the biological sciences, which are also in the process of an upcoming revolution - this revolution may depend on training different from the types of training biologists have traditionally received). Anyways, would people in other fields be convinced that astrophysics would provide me with the skills to go into this?

 

[twofish-quant]

So the question here, is, do you see these skillsets as extremely employable in the near future (perhaps more employable than many other skillsets)? And what would people look for if they look for people with such skillsets?

Learn to problem. If you are good at programming computers, it's like being about to read English. Also study history and philosophy. Technology changes quickly, but humans change rather slowly, and in looking at patterns, it's a good idea to look at human patterns.

Also, it's not a "coming revolution" it's a current one.

For instance, I would like to go for a PhD in astrophysics. Astrophysics, of course, is one beneficiary of this revolution, as we get better sensors (telescopes/CCDs) and massive amounts of data to analyze.

One thing about the massive amounts of data is that it's much too much for anyone human being to understand, so a lot of dealing with complex problems involves having cross-disciplinary teams. Just find a subject that you like and go with it.

The other thing is to develop basic communications and education skills. One key skill is to be able to take several exabytes of data and summarize it all in two sentences. You need a human to do that.

Anyways, would people in other fields be convinced that astrophysics would provide me with the skills to go into this?

A lot of what matters is to be able to give someone the key google term that they need. The word you are looking for is "bioinformatics." In any event, because computers are touching everything, what field you go into isn't that important since they are all getting hit by cheap computer power, and a lot of the basic techniques are field independent.

 

[DaveC426913]

So basically, the two main paradigms used to be experiment and theory. Then in the 1950s came simulations, and now we have data-intensive scientific discovery. Some people have recently written programs that can derive physical formulas from massive amounts of data. Such methods can produce true results without an a priori basis for scientific discovery, which runs counter to the scientific method.

How is this qualitatively different from, say, Kepler's Laws of planetary motion? His laws were derived from observation, without regard to theory, model or explanation.

 

[twofish-quant]

How is this qualitatively different from, say, Kepler's Laws of planetary motion? His laws were derived from observation, without regard to theory, model or explanation.

It's really not, except that now we have power tools rather than hand tools. Kepler took 19 years to figure out his three laws. What he did could be done by modern computers in about an hour.

The amount of data and hardware out there is incredibly but the bottle necks are the software and the social systems. Data is useless without a social context to make sense out of it.

 

[DaveC426913]

The amount of data and hardware out there is incredibly but the bottle necks are the software and the social systems. Data is useless without a social context to make sense out of it.

Yep. That is the central theme of Web 2.0 the Semantic Web initiative, and why HTML5 has been released with all sorts of new features to enable semantic interpretation.