Data Analysis career prospects

[boomtrain]

Hi physics forums.

I'm a 4th year student in Applied Physics. Originally I had no idea what I wanted to do with my education, or what useful skills I was learning. This year I've got a lab course and a computational physics course that have piqued my interest. I'm really enjoying computational data analysis, and was wondering if there's any way to make a career out of it.

I'm planning on going to graduate school in physics to get a bit more experience and take some relevant courses. Does anyone out there work in Data Mining? If you did grad school, was it helpful? Do any of your colleagues have a physics background, or are they mostly from math/stats/compsci? Are fundamental programming skills really important (things learned in an algorithms class), or is the programming involved something a physics major might be able to tackle? Is there reasonable demand for this sort of work (versus other fields for physicists to pursue? (e.g. engineering))

Thanks in advance!

 

[Locrian]

There is significant demand for data mining and other data analysis work. However, it is a broad field and the job requirements vary. A few tools you might encounter include SQL, SAS, R, and staples like Excel and Access. There are many others.

In most non-scientific industries those with advanced physics degrees are assumed to be very smart people with no useful skills. These assumptions tend to be made in the absence of actual data. Depending on what you study in grad school they could be right about both or wrong about one or the other (I knew some dense PhD students with no useful skills). Just ask yourself which of those tools I listed above that your physics studies are going to make you an expert in . . .

If you’re looking for a fall back plan, then you might be on the right track. However, if you have any primary interest in working in the area of data management, data mining, data analysis, statistics etc., consider getting another degree.

 

[ParticleGrl]

I got a physics phd and eventually moved on to data analytics at a large insurance company.

I can say with certainty-

1. no specific skill from graduate school has been useful in even the slightest.
2. Some things I DID while in graduate school have helped (pretty much technical writing and that's it)- i.e. you do quite a bit of technical writing in graduate school and even while you are never TAUGHT to write, you improve through practice.

I think the broad skills in 2 you could get with any phd program, and doing specifically physics is 0 value added.

There are other phds I work with, from a range of fields (biology, math,stats), but I would say that I feel like the first year of work I was (and still am) mostly catching up to the stats guys in terms of useful algorithm knowledge. Both stats phds were hired into a higher pay grade straight out of grad school (they skipped the 'junior data scientist' position), probably because they had more relevant skills.

I do learn new programming packages and database tools very fast in comparison though. I've quickly become the SQL/data pre-processing guru for our office (this is in part because I saw a niche and filled it). I think some of this might be the physics training, but it might be intrinsic to me. Hard to generalize from my own experience.

But yea, if you want to do data mining- get a phd in stats or informatics of some kind. It'll be way more useful, and probably a significant salary bump over the physics phd.

 

[boomtrain]

Thanks a lot for the replies!

A few tools you might encounter include SQL, SAS, R, and staples like Excel and Access. There are many others...
Just ask yourself which of those tools I listed above that your physics studies are going to make you an expert in . . .

I had the impression that you'd use those and similar tools in grad school to analyse data. Is that not the case?

There are other phds I work with, from a range of fields (biology, math,stats), but I would say that I feel like the first year of work I was (and still am) mostly catching up to the stats guys in terms of useful algorithm knowledge.

But yea, if you want to do data mining- get a phd in stats or informatics of some kind. It'll be way more useful, and probably a significant salary bump over the physics phd.

thanks for the tip. Do you mind if I ask some specifics about the job? What sorts of skills were helpful (that you learned in undergrad/independently)? What does the work involve (querying data from a database? Applying statistical functions from a pre-built software package? writing your own functions/algorithms?)

 

[ParticleGrl]

I had the impression that you'd use those and similar tools in grad school to analyse data. Is that not the case?

It depends on what you do, but in high energy, mostly no. High energy is loaded with purpose built, grad student written code. From talking to friends from grad school, other fields are similar. i.e. software written by earlier grad students, extended by current grad students rather than standard statistical packages, but I only have first hand experience with high energy. Also, some labs might use R because its free but I doubt they have the money to waste on SAS or other products you have to license.

High energy experimentalists will learn a ton of statistical analysis, but theorists not so much. Theorists can read and interpret experimental plots, but that's not the focus.

Do you mind if I ask some specifics about the job? What sorts of skills were helpful (that you learned in undergrad/independently)? What does the work involve (querying data from a database? Applying statistical functions from a pre-built software package? writing your own functions/algorithms?)

So our focus is making models, which is a little bit of all of the above. I would say the absolute most useful material for what I'm doing I picked up in undergrad- technical writing, CS 101, CS-data base principles, and stats 101. If I could go back in time to undergrad, I'd take less engineering electives (I took engineering thermo, statics, signal analysis, machine shop certifications, etc mostly to help with undergrad research and because I wanted (and still desperately want but can't find) a job in engineering or science) and more statistics and CS courses.

No classes for grad school (particle physics, field theory,etc) are at all useful for what I'm doing. The largest part of my grad school experience (pen and paper physics and light fortran coding modifying existing HEP tools) has as much relevance to my work now as an english lit phd.

The useful part of graduate school was writing papers, writing responses to reviewers (telling someone you think they are dead wrong, or that they didn't read page X of your paper without making them defensive is a tricky and important skill), writing the thesis. I write a lot of technical documentation now (insurance industry is riddled with both regulatory documentation and internal documents describing models), and I've found gradschool has sharpened that skill quite a bit. But practice makes perfect- I could probably have gotten better at it anywhere I had to do a lot of writing.

The work involves a lot of exploring data and looking for useful variables for models, which is a lot of writing SQL queries to slice-and-dice the data. We use a variety of standard statistical techniques, but we slightly modify standard algorithms to directly fit what we are doing much of the time. Its not quite algorithm design, but we also don't treat the packages we use as magic black boxes either.

 

[Mépris]

ParticleGrl, would one be well equipped for an engineering job (in this case, I'm guessing R&D), if one's bachelor's degree was in physics but one received one's PhD from an engineering department?

 

[ParticleGrl]

ParticleGrl, would one be well equipped for an engineering job (in this case, I'm guessing R&D), if one's bachelor's degree was in physics but one received one's PhD from an engineering department?

I have no idea. It SEEMS plausible, but at the same time, it always seemed plausible to me (until I was applying for jobs) that a physics phd would be more likely to get a job in an engineering-related-field than in an insurance-related-field, given the subject overlap.

Best thing I can suggest is to find some engineering phds who work in research and ask them not only about their coworkers but also about the people they went to grad school with.

 

[gbeagle]

I have a PhD in experimental high energy physics, and I now work as a software engineer in a SaaS (Software as a Service) company that does analytics. I'm on the software implementation side of things. I don't really do any data analysis myself, just creating the software infrastructure that actually does it.

Was grad school helpful?

The physics parts of my PhD were not helpful at all. I currently use none of my physics skills. The CS classes I took in undergrad were far more useful than any of the physics classes I took. I did gain software development skills during my PhD, but I would have gained far more had I spent the same length of time working or in school in a more relevant subject.

Do any of your colleagues have a physics background?

The vast majority of the software developers where I work have a CS, CE, or EE background. There are some other physics people in some of the more data analyst-like roles.

Are fundamental programming skills really important?

I'm in a mostly programming role, so yes fundamental programming skills are essential. Software engineering skills are also very important (what I mean by this is skills with building maintainable and flexible systems and knowledge of standard practices and tools). They're what got me my job. I wouldn't have one without them. The physicists on the data analyst side also have quiet strong software dev skills in general.

Is there reasonable demand for this sort of work?

Seems to be at least for now. At least in my case I encountered far more interest from analytics and software related jobs than any sort of 'traditional' engineering jobs (not counting software jobs here obviously). Basically, I had no real chance of landing an engineering job. Why hire a physicist who would need training when there are more than enough EE/CE grads with training in the specific skills you need?

Tools

All the data analysis tools that I used in graduate school were custom to the field solutions. Basically it was a mix of custom (to the experiment I was working on) tools and ROOT plus it's ecosystem that I used for all my analyses. I used no "industry standard" tools for the stats and data analysis related parts. On the other hand a lot of the software development tools I used are industry standard ones.

 

[Locrian]

Do you mind if I ask some specifics about the job? What sorts of skills were helpful (that you learned in undergrad/independently)? What does the work involve (querying data from a database? Applying statistical functions from a pre-built software package? writing your own functions/algorithms?)

I have a very hard time briefly describing what I do. I promised someone else on a different thread I would do so and never did, which I feel bad about. So this time I’m not going to make promises and try to write anything big up, I’m just going to throw down some thoughts and hope that’s sufficient.

I’m an actuarial analyst* at an insurance company. Most of what I do can be boiled down to working with models. Many of these models are meant to project the impact of various business decisions or environmental/governmental changes on our financial condition. Some models are used to estimate future contingent liabilities for use in financial reporting. Most of these models are not complicated, and I build most of them from scratch in Excel, Access or similar tools. A few are complicated – some of these were built in house, some were contracted for through consultants. Sometimes the timeframe for producing the model, gathering the data, generating results and packaging the results can be very short.

So some of my work is routine (e.g. monthly reserving), but a substantial portion is not. Much of my work stems from a question someone important had somewhere – “what happens if we do X?” To answer it I have to understand the business well enough to include all important factors required to generate an estimate. I have to know where the data that could produce it is located, and have an understanding of what the data represents, how it was generated, and what limitations there are in using it. (There are at least a dozen small, medium or ginormous databases I beat up on a regular basis.) In answering their question I have to be able to clearly communicate the results, and yet still convey the sources of uncertainty. People don’t like uncertainty, so this can be difficult to do.

Actuaries have both a code of conduct and a set of standards of practice that I adhere to.

My job is probably less technical than other’s in this thread. Actuarial work is business work that requires knowledge of some peculiar mathematical areas. It is not mathematical work that takes place in a business. The farther you go in the field, the less technical your daily work becomes (mine is already much less than when I began). While this concerned me initially, I’ve come to appreciate the non-technical challenges – they’re more difficult, and both the penalty for failure and the reward for success are greater. There’s still a part of me that often wishes I could just code in a dark corner where no one bothered me.

Finally, this is my job. Other actuarial types can have very different jobs. There are some actuaries who, apparently, just plug-and-chug into software that is sometimes new and sometimes as old as their parents. There are others who do what sounds to me like more complicated statistical work. The tools actuaries use differ greatly depending on the type of business (insurance, government, consulting, etc.) and the area of specialty (property & casualty, health, life, etc.). Everything I’ve told you is just one data point amongst a wide distribution of work duties.

...

Well, that ended up suitably long-winded – it seems I failed again at briefly describing what I do, but at least I have something down this time. I need to remember this post so I don’t ever have to type it again.

*An actuarial analyst is a pre-actuary – it’s a nice way of saying student. With some luck, I may be a credentialed actuary within the next couple of months. Here’s hoping.

 

[pi-r8]

So how do you go about getting an entry level job in Data Analysis? Is it just networking?

 

[boomtrain]

Great replies!

I'm a bit surprised that no one's said that the skills they got in grad school would have been more valuable than job experience/a different PhD, so this has been an eye opener. I guess I was half expecting there to be a Twofish_quant of the stats/data analysis world on the forums.

I'm going to take some extra courses and shoot for a more relevant PhD (stats/compsci) or try and get a job straight out of undergrad.