Mathematics needed to be a researcher in machine learning?

[marlon29]

Hi everyone,

My name is Marlon and I'm a maths major from New Zealand. Recently, I've been very interested in machine learning. The research and potential sounds very exciting to me and I would definitely love to learn more about the subject.

So I have two questions and would be very appreciative if anybody could share their knowledge with me.

Question 1: Do researchers in ML use more of Analysis or more of Algebra, or is it a bit of both?
I know that probability is one of, if not, the main area/s of maths used, so I'm thinking that this would imply that I should lean towards Analysis right?

and finally

Question 2: Would anybody have any idea with regards to my chances of getting into a grad programme with a focus on ML with a maths degree? I've taken the more theoretical statistics papers like statistical inference and stochastic processes, but a part of me is thinking that this might not be enough to qualify? If need be, I'll start from scratch and pick up another major in computer science. The obvious downfall here is time and money, but I'm willing to do it as the subject matter has really got me now.

Thank you to any helpers in advance.

Regards

 

[chiro]

I'm not an expert on the subject so take this with a grain of salt.

From what I'm aware in regards to ML, statistics is very very important (as you probably know) which include the Bayesian approach.

On top of that things like neural networks and signal processing would be great tools to be using in this area.

Things like analyzing noise (which you can do with something Fourier analysis) and trying to get rid of that noise is important in real world applications since the real world is full of noise and crap, and getting the data that you want is usually harder than would be thought. So in that sense some analysis with emphasis on different decompositions on different spaces (like L^2(R) or L^2(-pi,pi)) and maybe even integral transforms in general might be beneficial. Anything that corresponds to analysis of noise and getting rid of it would be, dare I say, paramount.

Another important thing I think in ML would be the use of language. When I mean language I don't mean things like say your written languages, or even symbols, but basically a framework for analyzing structure. Basically language has structure. Structure has inherent probabilities associated with it. For example the English language has specific structure where strings have markovian properties that would not exist if you were dealing with a signal that was i.i.d Gaussian noise. So if you were to to do some ML where it could basically classify signals appropriately, knowing some theory of language in a mathematical sense (think mathematical linguistics) would go a very long way.

So yeah in a nutshell, noise, language, and statistics are my suggestions. Good luck!

 

[marlon29]

Hi there Chiro!

Man, thanks for such an awesome reply. You've definitely given me very useful information. In fact, I'll be in the uni library tomorrow searching for relevant material like the things you've highlighted. I've already begun searching for papers involving signal processing and I've found one I can hopefully (it's an engineering science paper) do later (digital signal processing).

I had never thought about structures like that so I'm very very intrigued and even more keen to learn. There is a paper called 'MATHS 320: ALGEBRAIC STRUCTURES' which has the following description:

"This is a framework for a unified treatment of many different mathematical structures. It concentrates on the fundamental notions of groups, rings and fields. The abstract descriptions are accompanied by numerous concrete examples. Applications include symmetries, geometry, coding theory, cryptography and many more. This course is recommended for those planning graduate study in pure mathematics."

so I'm not sure if it'd be anything like mathematical linguistics, but I like the sound of it , I'll be taking that paper. I'll be sure to look out for all the info you've given me too.

Once again, thanks a lot for your time and advice. Really took it on board and appreciate the help.

Regards

 

[diggy]

I think you have it right -- its all statistics and applied math and computers. Very little actual analysis and even less abstract/modern algebra.

Math wise focus on:
Statistics, Bayesian Statistics, Linear Algebra, Optimization (both global and local), and general applied math such as understanding how to decompose a function/signal into various basis functions.

The other item big help is programming. Matlab/Octave or R is a good. Python is popular also, or C/C++. Just pick one and go with it to start.

Also, like you say machine learning groups are usually in the CS department ala Artificial Intelligence and Vision, which means having CS classes/degree under your belt would boost your chances in getting into a program that does ML.

Btw, are you more interested in AI type stuff or data/stats?

 

[marlon29]

Hi diggy,

Thank you for you help.

Yes, I'll do my best to cover as much of the mathematics you've pointed out for me. I really enjoy analysis and algebra so I'll have to get over that. Machine learning looks extremely interesting anyway so I'm sure that won't be too hard to do.

Also, thank you for your advice on the CS degree. I've adjusted my degree to allow for another major. Turns out that I can still do it in the same amount of time if I plan it well :)

I've only recently become interested in computer science so I'm not really sure what is out there to explore and discover. I thought that I always wanted to become a mathematics guy (the idea of working on something like the Riemannn Hypothesis always encouraged that goal...dreamer, I know haha), so I'm still 'looking around' I guess. AI looks very cool though. I watched a few of Andrew Ng's lectures online and loved them. He genuinely seems passionate about the topic and I think it rubbed off. But I've also read some stuff on the use of machine learning in fields like biology, amazing applications used by groups like D.E.Shaw research group working on cancer research.

So to try and give you a proper answer, I think I'm interested in both AI and data/stats.
I bet I'll have to choose only one though, and soon.

I was wondering (totally naive question), do you know if there is any overlap between machine learning and algorithmic information theory? I tried looking online myself but I couldn't really find anything.

 

[chiro]

Hi diggy,
I was wondering (totally naive question), do you know if there is any overlap between machine learning and algorithmic information theory? I tried looking online myself but I couldn't really find anything.

Again, I'm not an expert on this so take it with a grain of salt.

One thing about information (especially language) is that most language is structured and not random. For example if you read about written languages and spoken languages there are markovian properties associated with them, whereas if you are talking about random noise, the next value of your noise will be completely independent of your whole past history of noise.

If you want something more complete and concise, look at the work of Shannon. He covers examples of this (to do with language), and I'm sure that if you go further into information theory, you'll find deeper discussions and results.

But yeah one of the key things that distinguishes noise from most natural languages is structure, and mathematically that is represented in probability information, and in a formal context through markovian and other conditional probabilities.

I'll give you a very simple example. Say you are analyzing a signal which gives out characters in the english language that corresponds to some text (be it from a human mouth, a book, etc). Say you get the word "the ". Now your expected next word would have restrictions. You would not expect another "the" to be said. In fact you could eliminate entire classes of words based on grammatical rules if you assume these rules.

In a situation where the signal is just random noise (ie random characters), there is no reason to think that you could not get the string "the" after your string. It is entirely possible with equal probability of getting any other three letter signal.

Now data mining has to deal with these kind of situations. You can get structured or semi-structured data, or you can get completely unstructured data.

It really does depend on what you are trying to do, but hopefully this example has given some insight into how information theory can play a role in machine learning.

 

[diggy]

Just to add to what Chiro said, there is a semi-popular book on information theory and machine learning by Mackay you can check out.

If you can pull it off adding a CS minor (or major) won't hurt you. But just to kind of reiterate, much of machine learning isn't really "machine learning", its ideas and techniques borrowed/stolen from other fields -- for example many of the most popular ML texts are written by physicists. So a solid foundation in math, stats, linear algebra, etc, is more important in my opinion.

Best of luck and have fun.