Final year University Project, Music Visualisation question

[ellipsoid]

TL;DR Summary

Would generating an image from a given piece of music be too difficult and too much work for a final year project and what languages and software would be best for it?

Hi there. I start my final year of my computer science degree this September and, alongside other modules, will do an independent final year project. I have not had very much CS experience outside my degree so I am a bit unsure what would be feasible for me while not being too trivial such that I receive a poor grade.

I was looking to do something related to my interests and one of my initial ideas that my tutor seemed to like was generating an image from a given piece of music. I am not yet fully sure about what metrics I would use to do this, but I was thinking a combination of rhythm, lyrics, velocity and harmonics.

For a general example, an aggressive sounding piece of music would result in an aggressive feeling image, a calming song, a calm feeling image and so on. The project must have some success criteria that can be evaluated and I was thinking I could survey participants and ask whether the generated image fits with the song.

In the past two years in my degree, I have programmed a fair bit in Java, Haskell and a tiny bit in C. Would you say that this idea for a project would be two difficult to do in the roughly 6 months of time given considering my perhaps more limited experience? Also could you tell what language(s) you would recommend something like this be done in as well as existing packages and/or software that could be useful for this?

I was thinking that with a given audio file, I could do some web retrieval of existing images and combine it with some abstract procedurally generated images however if you think doing both would be too much or if one of these would be too difficult then I can simplify?

Also, my initial plan was for this to be done in real time with visual elements coming in as a piece is playing but once again, if this would be too difficult to achieve good results with, I can rethink that? I guess I have a few questions here but I would appreciate any thoughts on any of this and am happy to provide more info where I can. Thank you very much.

 

[.Scott]

Don't tackle any kind of "live" image.

So minimally, you need to read the audio file and extract your basic input: rhythm, velocity, harmonics. Don't try to do anything with lyrics from the audio source.

I would suggest doing this very, very incrementally. Your main requirement it to get something done and demonstrable in 6 months.

I don't know what development tools are available to you. If you have C, you know how to compile, build, and execute - that's probably the best.

So <with C notes>...
Step 1: Read a wave file <fopen fread> one 1024-sample frame at a time are report out the average amplitude each frame <printf>. No GUI. No stereo. If your favorite music is MP3, convert it to wav. Simple text output, one text output line per frame.

Step 2: If you want rhythm and velocity, you're going to have to play with the data and look for it. Perhaps isolating the base with a simple low pass filter and then looking for amplitude variations will give you the beat. Hopefully, you'll find that fun.

Step 3: If you want harmonics, you need FFT. As an example, if your 1024-sample frames are numbered 1 through 25, then I would try taking them in overlapping two's (Frames 1 and 2, frames 2 and 3, etc) and then doing an FFT on each 4096-sample block and find the peaks - which will correspond to the audio frequencies.

After each step above, if you find yourself with less than 3 months left to go, stop and start the image part.

Since I have no idea where your talents lie, I have no idea if this is doable for you - or if it is too simple.

 

[Jarvis323]

what languages and software would be best for it?

I would look into processing. It can do the graphics, has utilities for audio, and it's based on java.

https://processing.org/reference/libraries/sound/index.html

 

[Vanadium 50]

Would generating an image from a given piece of music be too difficult

Not intrinsically.

and what languages and software would be best for it?

Now I am more worried. If you are asking this basic a question, I think another project might be wiser.

 

[DrJohn]

You should use whatever language you are most familiar with.

Basically you are trying to assign values to rhythm, loudness, frequency and changes in these to generate a pattern of dots, possibly a line at a time based on a time interval. Perhaps with the commonest frequencies shown as wave forms. The choice of programming language is almost irrelevant. Unless one language has an established way to extract data from an audio file.

And as stated above, do little things one at a time then combine them, then expand to more than a few seconds.

 

[berkeman]

Welcome to PF.

I was looking to do something related to my interests and one of my initial ideas that my tutor seemed to like was generating an image from a given piece of music. I am not yet fully sure about what metrics I would use to do this, but I was thinking a combination of rhythm, lyrics, velocity and harmonics.

Is there a possibility that you can team up with an art major in a combined project? Ideally if he/she needs to do a senior project as well, this could be a good project to collaborate on. For example, in your initial brainstorming with your collaborator, you could ask them to sketch what they feeling as they listen to some parts of some musical pieces, and together you may be able to start to put some rules/themes in place. Without the help of an artistic person on this project, I have a feeling that it will not go well for you.

I was thinking that with a given audio file, I could do some web retrieval of existing images

As an extension of the above, once you start to get some agreement on the resulting inspired skeches/drawings from the content of the audio files, you could start to do some machine learning where you program associations between the music and the strength of the matches in what you retrieve from the web. At first you could use simple Google Images searches for images that contain "fast" or " intense" or "rhythmic", etc., and then you may be able to use your collaboration to get more sophisticated combined searches that give better image hits.

Also, my initial plan was for this to be done in real time with visual elements coming in as a piece is playing

So that is an important thing to decide going in, IMO. It seems more natural for the images that you display to be updated every 10-15 seconds or so, based on the current mood of the music. Most pieces move around a bit over the course of a complete piece, I think, so just distilling one image for the whole piece seems unsatisfying for me.

OTOH, you may need to have the ability to detect sudden changes in the music and adjust quickly to a new image. That would also imply that this is not done in real time, but is done more like a compiler processes the input program. Think of the piece "In the air tonight" by Phil Collins at 2:38 and of course at 3:15.

 

[berkeman]

Also, there may be some value in getting the associated lyrics somehow (subtitles? Are synchronized lyrics routinely available for musical pieces?) and making those words and phrases also Google search terms to find images that illustrate those words and phrases. So when Phil sings "I can hear it coming in the air tonight...", that could pull up images of forests at night, or similar...

 

[berkeman]

Sorry, just one more post for now. It reminds me of the time back in high school when I was listening to a lot of Santana Abraxas on an old reel-to-reel tape player, and my best friend Andy came over. I put the headphones on him and played "I hope you're feeling better" at high volume, and told him to watch when I opened a motocross magazine (to a picture of MX champion Joel Robert on a high speed MX jump) at 0:09 in the song... He was pleased.

https://en.wikipedia.org/wiki/Joël_Robert

 

[Vanadium 50]

Santana Abraxas

The third album was better.

Everybody's Everything is spectacular (trumpets Mic Gillette and Gref Adams) as is Everything's Coming Our Way (especially the organ solo - Gregg Rolie?)

 

[DrJohn]

Also, there may be some value in getting the associated lyrics somehow (subtitles? Are synchronized lyrics routinely available for musical pieces?) and making those words and phrases also Google search terms to find images that illustrate those words and phrases. So when Phil sings "I can hear it coming in the air tonight...", that could pull up images of forests at night, or similar...

So I just googled rhythm and frequency and switched to images. Lots of interesting ideas there, especially under the stock image collections' websites. You could even try generating an individual image for rhythm, loudness, frequency, changes in each of these for each piece of music. That gives you options in your project description to have, say, two stated goals (say rhythm and frequency) and then follow-up goals (say loudness vs frequency or rhythm vs changes in rhythm).

It's always a good idea to avoid too wide a range of stated goals in a student project and end up leaving several with no evidence of attempts at getting them to work. But extra features are better, as if time runs out, you just don't worry. And if you get the first goals completed to a reasonable standard, you can afford to switch to an optional goal even if you do not complete it, but can still include it in the report.

IE Must have, Should have, Could have, Wont have (google moscow rules for programming)
Having follow-up goals also shows you don't have limited ideas, just limited time (the usual problem for all student projects ;) )

 

[robphy]

I would look into processing. It can do the graphics, has utilities for audio, and it's based on java.

https://processing.org/reference/libraries/sound/index.html

https://www.google.com/search?q=audio+visualizer+processing

Uses ellipses: