Are programming interviews effective these days?

[kolleamm]

From what I've come to understand is that, in order to get a programming position, you have to be able to answer white board questions. Basically random things such as Fibonacci sequence methods etc...

My question is : is this really the best way to asses whether someone is a good programmer or not?
These questions seem far too diverse and random to have any good meaning.
A person could have 20 years of good programming experience and get stuck on one of these questions and not get the job.

What are your thoughts?
Thanks in advance

 

[rcgldr]

This might be true for a programmer seeking their very first job, but even in that case, usually the questions will be related to the type of work that the potential employee is expected to be doing. A long time ago (1960's, 1970's), some companies did use something similar to programming IQ type tests, but I haven't seen this done since the 1980's. For an experienced programmer, generally the questions will be about the projects the programmer has worked on, and what role the programmer had in the development team (such as being a lead).

 

[jedishrfu]

In the sixties they used to give Aptitude tests to applicants unless you graduated from college recently. At IBM, they started to hire English majors as programmers because so much of programming at the time involved writing specifications and user manuals. IBM found that English majors had pretty good reasoning skills and could write better than the average natural sciences or engineering college graduates.

Later, it became a fad to ask technical questions that only someone versed in the field would know. In one case, I had an interview where I was asked some arcane feature of C++ which I answered and then asked the interviewer if he had read the recent Dr Dobbs journal article on C++ interview questions. He hadn’t, so I shared one with him and he couldn’t answer it but to avoid any embarrassment on his part I said something humorous and then answered it too.

They wanted to hire me but as a project manager and I knew in this company it would be an onerous job. I told them I’d prefer a programming position but they tried to entice me with more money and doing the project manager shot for a year. I knew that wouldn’t work and I just knew they would renege on their promise so I walked away.

Now it seems the questions are quite fanciful where they try to get the interviewee out of his/her comfort zone to see how they’ll react. This guarantees that they'll hire a certain kind of frat boy as I consider this type of interview demeaning.

Another kind of interview is the team interview where you’re invited to have lunch with the team and they pound you with questions so you can’t eat a thing. My strategy here was to ask questions back about the work environment setting up a dialog and getting one person to talk about how it was working for the hiring manager. Some embarrassment resulted but you can tell how well the team works.

On a different team interview, I thought it was odd that each person in round robin fashion asked a question. It turns out everything was carefully scripted with each team member reading the next question in line on their interview script. I didn’t like that interview much as I felt they were too stiff. Basically, you need to be prepared to just say no to the job when the vibes are wrong.

Back to the fanciful questions, you need to develop a sense of out of the box thinking. During the interview, think out loud and ask questions about what you can assume. It’s not expected that you solve the question rather they want to know how you pull the problem apart and develop a solution. Of course, there are always some folks who gloat a bit as you work, providing little guidance and hope you’ll really sweat. However, that gives you a clue of who not to work with once you get hired.

My candidate interview strategy has always been to ask questions based on what’s in the resume provided and to explore what I find. I also encourage a dialog telling them about the work environment and the tools we use and find out what tools they have experience with. It’s always great to hire someone with alternative experience to bring in new ideas for the team to consider.

College students are particularly fun to interview because they don’t always put down important stuff in their resume and you have to tease it out during the interview. In one case, I asked the student if they had any graphics programming experience and they said no but on the resume they had done a gaming project so I’d ask about that and lo and behold they did some graphics there.

My nephew lost a job once because he said he had no C programming experience when he actually did in some college courses. I asked him about it afterward and he said he didn’t feel he knew it well enough. A second interview fixed that issue and he got hired.

Often they feel if they don’t really know something then they won’t say which is a shame since you need to sell yourself and tell the interviewer what they want to hear particularly in how you can help their business.

My apologies for this long winded post. You got me to remembering the poor interviewing practices we now have in the industry.

 

[kolleamm]

In the sixties they used to give Aptitude tests to applicants unless you graduated from college recently. At IBM, they started to hire English majors as programmers because so much of programming at the time involved writing specifications and user manuals. IBM found that English majors had pretty good reasoning skills and could write better than the average natural sciences or engineering college graduates.

Later, it became a fad to ask technical questions that only someone versed in the field would know. In one case, I had an interview where I was asked some arcane feature of C++ which I answered and then asked the interviewer if he had read the recent Dr Dobbs journal article on C++ interview questions. He hadn’t, so I shared one with him and he couldn’t answer it but to avoid any embarrassment on his part I said something humorous and then answered it too.

They wanted to hire me but as a project manager and I knew in this company it would be an onerous job. I told them I’d prefer a programming position but they tried to entice me with more money and doing the project manager shot for a year. I knew that wouldn’t work and I just knew they would renege on their promise so I walked away.

Now it seems the questions are quite fanciful where they try to get the interviewee out of his/her comfort zone to see how they’ll react. This guarantees that they'll hire a certain kind of frat boy as I consider this type of interview demeaning.

Another kind of interview is the team interview where you’re invited to have lunch with the team and they pound you with questions so you can’t eat a thing. My strategy here was to ask questions back about the work environment setting up a dialog and getting one person to talk about how it was working for the hiring manager. Some embarrassment resulted but you can tell how well the team works.

On a different team interview, I thought it was odd that each person in round robin fashion asked a question. It turns out everything was carefully scripted with each team member reading the next question in line on their interview script. I didn’t like that interview much as I felt they were too stiff. Basically, you need to be prepared to just say no to the job when the vibes are wrong.

Back to the fanciful questions, you need to develop a sense of out of the box thinking. During the interview, think out loud and ask questions about what you can assume. It’s not expected that you solve the question rather they want to know how you pull the problem apart and develop a solution. Of course, there are always some folks who gloat a bit as you work, providing little guidance and hope you’ll really sweat. However, that gives you a clue of who not to work with once you get hired.

My candidate interview strategy has always been to ask questions based on what’s in the resume provided and to explore what I find. I also encourage a dialog telling them about the work environment and the tools we use and find out what tools they have experience with. It’s always great to hire someone with alternative experience to bring in new ideas for the team to consider.

College students are particularly fun to interview because they don’t always put down important stuff in their resume and you have to tease it out during the interview. In one case, I asked the student if they had any graphics programming experience and they said no but on the resume they had done a gaming project so I’d ask about that and lo and behold they did some graphics there.

My nephew lost a job once because he said he had no C programming experience when he actually did in some college courses. I asked him about it afterward and he said he didn’t feel he knew it well enough. A second interview fixed that issue and he got hired.

Often they feel if they don’t really know something then they won’t say which is a shame since you need to sell yourself and tell the interviewer what they want to hear particularly in how you can help their business.

My apologies for this long winded post. You got me to remembering the poor interviewing practices we now have in the industry.

Not to worry! I really enjoyed reading it.

 

[sysprog]

At one company in the late '80s, they gave a battery of on-paper tests to prospective hires. I had already been hired based on interview, as the boss took it for granted that I would pass the tests, but I still had to take the tests. When I took the tests, I was the only one taking them who had already been hired.

The other test-takers, consequently, had more incentive to get a maximally favorable score than I did. I knew that I needed to perform well, but I didn't have an especial incentive to perform off-the-charts well, as that might increase the performance level expected of me, without necessarily bringing about a commensurate increase in compensation.

Some of the the tests were designed to measure reasoning abilities, and others sought to identify psychological characteristics. One of the psych tests asked the respondent to reply with yes, no, or can't answer, and advised resorting to the third option as infrequently as possible.

That test included e.g. 'I like to measure the temperature in various areas of the room', apparently intended to find OCD responders.

Anyway, while we test-takers were on break, clustered around the coffee station, nervously speaking quietly and politely to one another, I said aloud "you know that question where it says 'sometimes I think everyone's out to get me'?" (everyone went silent and waited and looked at me for a moment) -- after a brief pause, I said "well I put 'no' because I didn't want anyone to know that I knew", and everyone laughed -- it broke the tension, and the boss, hearing the laughter, came out of his nearby office, and quietly advised me that he had tests with questions that were a lot harder.

 

[jedishrfu]

I sure hope you distanced yourself from that boss. He sounds somewhat vindictive. Sometimes there are people in power at your workplace who crack jokes like that, that are veiled threats.

 

[newjerseyrunner]

I think you may be misunderstanding the purpose of the whiteboard exercise. I often ask white board questions that I know are very difficult to answer. We choose our questions primarily because we want to see how you approach the problem, not that you can actually solve it on your feet.

 

[Born2bwire]

I've found that programming questions are still good at weeding out people with poor skills or not enough experience. We still come across the occasional person that cannot put together a basic algorithm. Beyond that, I've stopped asking programming questions like those. Most applicants are spending huge amounts of time on hard technical skills, particularly now that there are so many resources like LeetCode that have practically every kind of programming question imaginable.

I'm more interested now in a candidate's soft skills because those are becoming more of the limiting factors in team performance. In particular, I want candidates that understand computer science concepts and skills like writing maintainable code. I don't need someone that can come up with an log N algorithm to replace an N-squared. That's not where we are losing time, it's in the time it takes to add features and maintain the code. It's getting very hard to find people that actually take the time to write clean and maintainable code.

That's my viewpoint, but I still see a very heavy reliance on programming questions in interviews. I find it increasingly frustrating because they're still emphasizing them with mid-career candidates.

 

[sysprog]

I sure hope you distanced yourself from that boss. He sounds somewhat vindictive. Sometimes there are people in power at your workplace who crack jokes like that, that are veiled threats.

To be fair to him: I was poking fun at the company's approved tests for candidate mental aptitude and attitude measurement, and he was merely letting me know that he had more demanding tests and could raise the admission bar if he deemed that to be appropriate.

 

[Locrian]

It's not great, and it shouldn't be the only thing, but I do think it adds some value.

 

[EngWiPy]

Are interviews effective in general? Is there a strong correlation between interview performance and job performance?

 

[Vanadium 50]

Is there a strong correlation between interview performance and job performance?

Given that people who interview poorly don't get jobs, I would say "yes", almost by construction.

 

[symbolipoint]

Are interviews effective in general? Is there a strong correlation between interview performance and job performance?

Some common human understanding: People exchange information using spoken language communication. The interviewer wants to know if candidate knows what interviewer wants him to know, and in many cases, if candidate has the kind of experience that interviewer is looking for. This works well unless one or more of the participants is lying.

Here is an absolutely fascinating fictional example of a job interview from a movie. The relevant part begins at time 1:10 through the clip:

 

[rkr]

From what I've come to understand is that, in order to get a programming position, you have to be able to answer white board questions. Basically random things such as Fibonacci sequence methods etc...

My question is : is this really the best way to asses whether someone is a good programmer or not?
These questions seem far too diverse and random to have any good meaning.
A person could have 20 years of good programming experience and get stuck on one of these questions and not get the job.

What are your thoughts?
Thanks in advance

Context: I've hired dozens of programmers and conducted a few hundred technical interviews.

The interview process is definitely imperfect. The way I see it is that there's 3 categories of things: known knowns (A), known unknowns (B), and unknown unknowns (C).

We've definitely had situations where a candidate turns out to be amazing outside the interview setting because of things we did not assess (i.e. C) and we regret not hiring that person, but hiring someone is an expensive bet and it's usually better not to take on that uncertainty.

And in all cases, the set of items in A is much smaller (I say 2 orders of magnitude smaller) than the set of items in B, since there's just a limit to how much you can know about someone with just 10-20 hours of interaction. So even if a candidate turns out to be amazing based on what you've evaluated (in set A), and you hire said person, you'll most often find that there's a lot of issues working with that person that weren't discovered in the interview process (in set B).

I can however say we've almost never regretted rejecting a candidate if we did find a problem with the candidate during the interview process (in set A), so we've almost never had a situation where something in set C was discovered after-the-fact that would've made us change our minds about decision and regret that decision.

 

[DaveC426913]

...random things such as Fibonacci sequence methods etc...

My question is : is this really the best way to asses whether someone is a good programmer or not?

It gets them talking. You get insight into their thought processes.

See NewJerseyRunner's post above.

 

[harborsparrow]

Well if you're wanting to prepare for this ordeal or rite of passage or whatever it is, there is no better way that this:

https://www.amazon.com/dp/0984782850/?tag=pfamazon01-20

The author, Gayle Laakmann, has said that the important thing is not to be perfect in coding, or even with the algorithm, but to keep working at the problem and showing how you can think about an issue to improve what is there so far.

 

[Zap]

They still give these tests. I have been asked to take programming aptitude tests and battery exams for a job. So far, the tests have been stupid easy. So, I don't understand what the point of them are. Are they to weed out the mentally handicapped? I don't have a PhD. So, they should at least know I'm not retarded.

 

[symbolipoint]

...I have been asked to take programming aptitude tests and battery exams for a job. So far, the tests have been stupid easy. So, I don't understand what the point of them are. Are they to weed out the mentally handicapped? ...

So imagine someone passes this or these examinations. Maybe this candidate is competent enough to do the work needed for the job.

Then imagine if someone does not pass this or these "stupid easy" test. This must mean that the candidate is absolutely incompetent for the job.

 

[sysprog]

They still give these tests. I have been asked to take programming aptitude tests and battery exams for a job. So far, the tests have been stupid easy. So, I don't understand what the point of them are. Are they to weed out the mentally handicapped? I don't have a PhD. So, they should at least know I'm not retarded.

Can you provide any examples of what you think were among the least easy questions on the tests, so that we can have at least some basis for arriving at our own assessments of how easy the tests were?

 

[Zap]

I only took the CPAB and the Wonderlic test. I'm not exactly sure what those tests are suppose to measure. The Wonderlic is suppose to be similar to an IQ test, but it's really more like a measure of your test taking strategy. It's like a game where you're suppose to beat the clock instead of solve all the problems. If you don't know that ahead of time, you will most likely score low. So, it's easy in that you're suppose to skip all the difficult problems in order to achieve a high score. Maybe, it's suppose to measure your performance under pressure. Idk. I suppose it is an easy way to weed out incompetent people, or those who simply didn't bother to study what it was and weren't prepared for the small time limit.

 

[symbolipoint]

Looking over the original post again:

you have to be able to answer white board questions. Basically random things such as Fibonacci sequence methods etc...

My question is : is this really the best way to asses whether someone is a good programmer or not?
These questions seem far too diverse and random to have any good meaning.

Did that mean, the questions asked are not about programming or that the questions are either about programming or in an inferred way, or what?

 

[symbolipoint]

I only took the CPAB and the Wonderlic test. I'm not exactly sure what those tests are suppose to measure. The Wonderlic is suppose to be similar to an IQ test, but it's really more like a measure of your test taking strategy. It's like a game where you're suppose to beat the clock instead of solve all the problems. If you don't know that ahead of time, you will most likely score low. So, it's easy in that you're suppose to skip all the difficult problems in order to achieve a high score. Maybe, it's suppose to measure your performance under pressure. Idk. I suppose it is an easy way to weed out incompetent people, or those who simply didn't bother to study what it was and weren't prepared for the small time limit.

What would be nice is, some members who have had recent interviews for programming jobs, tell us what the assessment questioning was like. Did the assessment questions target programming knowledge or something else?

 

[Zap]

Wonderlic is not about programming. The CPAB is the computer programming aptitude battery exam, which is actually not about programming, either.

 

[symbolipoint]

What would be nice is, some members who have had recent interviews for programming jobs, tell us what the assessment questioning was like. Did the assessment questions target programming knowledge or something else?

Wonderlic is not about programming. The CPAB is the computer programming aptitude battery exam, which is actually not about programming, either.

My later posting placement was maybe not the best.

My question quoted above is regarding parts of post #1.

kolleamm said this:

My question is : is this really the best way to asses whether someone is a good programmer or not?
These questions seem far too diverse and random to have any good meaning.
A person could have 20 years of good programming experience and get stuck on one of these questions and not get the job.

My question is hopefully clearer now. I wonder what any forum members who have been given recent job interviews for programmer positions can tell us about the programming knowledge and skills assessments done in their interviews. Company should be able to do an interactive or some kind of precise assessment of the candidate, somehow. Like, "Show me/us what you can do"; or, "Show us that you can do this."

 

[JJminusI]

I think you may be misunderstanding the purpose of the whiteboard exercise. I often ask white board questions that I know are very difficult to answer. We choose our questions primarily because we want to see how you approach the problem, not that you can actually solve it on your feet.

The author, Gayle Laakmann, has said that the important thing is not to be perfect in coding, or even with the algorithm, but to keep working at the problem and showing how you can think about an issue to improve what is there so far.

100% of the white board (or write code down in a text document) interviews I have done required a compilable answer at the end that was of maximal efficiency. I failed one where I wrote a modular, functioning answer with unit tests because it wasn't the "preferred answer." I failed one where the prompt changed partway through and then didn't have enough time to fire out the answer because I don't memorize algorithms.

Whether or not whiteboard questions *should* be evaluated as you say, they are not. If you want to do well on these tests, in general, memorize the answers to a wide range of questions so you can spit them out as quickly as possible.

 

[jedishrfu]

Another approach is to not bother with jobs that use these testing methods as ways to hire programmers. Tests are used to screen folks out, it's always better to have person-to-person interviews.

If you want tests then get certified beforehand and add those to your resume for everyone to see. Certification tests are more comprehensive and it means you can skip this on-the-fly testing nonsense where you are forced to dance when you are already anxiously nervous to begin with.

 

[symbolipoint]

jedishrfu, I wonder about this part:

Certification tests are more comprehensive and it means you can skip this on-the-fly testing nonsense where you are forced to dance when you are already anxiously nervous to begin with.

What you say about certification tests seems very logical. But, really comprehensive? Many many skills and topics tested in enough detail? For how much time will the certified person hold his certified skills and knowledge? Maybe not all of them long enough after the testing; and this can be why an employer may directly test a job-candidate. Employer may want to know, "Do you have the skills and knowledge needed right now for the job we want to fill?" Hey, I am just asking and commenting.

 

[jedishrfu]

Employers used to do testing like that in the sixties and then realized that your education level sufficed. More recently it been claimed that this technique finds exceptional people but I'm sure they are missing some real talent who don't do tests well.

I look at the companies today that do this kind of testing like i do a fraternity. The tests are as demeaning as hazing and just give the company a reason to reject you on arbitrary grounds as they troll the employment pool and there is no way of knowing whether they even scored the test they gave.

Trick questions are a similar tactic designed to unnerve you during an interview so that you won't ask the questions you need to ask about the company and its work environment

Ive had my share of crappy interviews from lunchdates with programmer teams where you can't even eat, round the table interviews where each person asks you an HR scripted question, trick question interviews where they either want you to swap two variables or implement some algorithm on a white board or solve some ridiculous puzzle none of which is relevant to the work youd be doing.

In each case, having passed the interview, i rejected their offer for the poor treatment i got deciding i didn't want to work for those bozos.

Interviews are like reading tea leaves where you can discern a lot about a prospective employer. Its important to understand the environment, see the people and tools used before deciding. One friend of mine got a promise of a pay hike in six months no strings when hired but when the time came they reneged. Thats an indicator of a bad company and of a good time to leave.

 

[WWGD]

<snip>
My question is : is this really the best way to asses

What are your thoughts?
Thanks in advance

I assume you meant 'assess' , instead of 'asses'? ;).

 

[DaveC426913]

Works either way...

 

[WWGD]

I assume you meant 'assess' , instead of 'asses'? ;).

Don't worry, I have screwed up a few. Trying to make conversation with the interviewer, I asked, about a portrait on his table:
" How come you have a portrait of Bill Clinton wearing such an ugly dress?"
A: " That's my mom."
Me: " She looks really strong" (??)

Didn't know what else after screwing up so badly.

 

[jedishrfu]

Jokes aside that kind of thing is a bad idea in an interview as the interviewer will think you’ve lost interest in the interview or worse have an attention deficit disorder. My manager told this happened once when he was interviewing someone and out of the blue they asked about a picture on the wall and then got back to the interview. Later he commented on the zone out oddity and wondered if other interviewers noticed it too.

 

[jedishrfu]

jedishrfu, I wonder about this part:

What you say about certification tests seems very logical. But, really comprehensive? Many many skills and topics tested in enough detail? For how much time will the certified person hold his certified skills and knowledge? Maybe not all of them long enough after the testing; and this can be why an employer may directly test a job-candidate. Employer may want to know, "Do you have the skills and knowledge needed right now for the job we want to fill?" Hey, I am just asking and commenting.

Most certification tests revolve around specific vendor products and while someone can pass the test and still be bad for your company, most who do pass are quite qualified to do the work required. It’s most common for network management, and database management. However, I’m sure that for really big implementations these certifications are just a starting point and in-house training will take over with policies and procedures and how things are really coordinated for big data networks.

 

[.Scott]

I have been on both the interviewer and interviewee side of those "puzzle" questions.

As an interviewer, the first thing I do is compare the resume to what we need to fill the position.
Then, assuming there is a reasonable overlap, I will focus on the specific position-relevant experiences described in the resume.
For those experiences, I will quiz them on the application requirements and some of the specifics of their participation in the coding, testing, and documentation. This is a kind of "audit". There should be all kinds of "evidence" that someone really did participate in a development as they are describing.

In some cases, the interviewee will enthusiastically described parts of the process in such detail that there will be no doubt. In other cases, I need to keep up the search. I will attempt to focus of some of the things they were likely to have run into during their efforts - perhaps a HW interface, a communications protocol, data structures, resource arbitration, etc. And this commonly results in whiteboard use.

So all of that said, there have been times when I have formally "tested" applicants - although in all such cases, the tests were intended to be very rudimentary. In one case, the State Employment service decided that it was better to send us Veterans then programmers. So I created a programming test that was super-simple, specific to the type of coding they we would be expecting, and could be completed is minutes. When a candidate arrived, they were provided a desk with ample reference material (this was pre-internet), paper, pen, the instructions in writing, as much time as they wanted, solitude, and access to me should there be a problem. They were also told that bugs were fine. "Actual programmers" would complete the assignment in a few minutes and would move on to the interview process. The others would excuse themselves - sometimes after spending over an hour at the desk.

In another case, I was teaching a 1-week course for Fortran programmers that introduced them to everything they needed to know to develop their specialized code on the companies CAD/CAM system. Being a Fortran programmer was a prerequisite, so the test was given at the start of the course. Most of the "students" were actually managers - but this was fine. At the start of the course I alerted the marketing department to what I had in the way of non-programming students and those students were given the option of being taken out to dinner each day. I remember one guy who took this as being "caught". He said he was a Cobol coder. I told him that he could take the test in Cobol. When that didn't work for him, I told him to simply draw a flowchart. Needless to say, he ended up with our Marketing people.

In that test, I provided a function that when provided two line segments (Xa,Ya to Xb,Yb) and (Xc,Yc to Xd, Yd) would return the number of times that the line segments intersect (0 or 1). They were provided two triangles (Xe,Ye; Xf,Yf; Xg,Yg) and (Xh,Yh; Xi,Yi; Xj,Yj), and asked to write a function that would return the total number of times those triangles intersected - presumably using the function provided. I tested this test of several coworkers and decided I should allow 30 minutes. Ultimately, I allowed an hour.

 

[jedishrfu]

I am opposed on moral grounds to these kinds of tests. When I conducted interviews I realized that only a small portion of the job was actual coding so I looked for logical thinking more, experience, and out of the box thinking.

I never posed trick questions although I knew folks who did. Many of our applicants were college students who hadn’t yet graduated. Often they were very nervous and didnt know what they knew. One student when asked if he’d done any graphics program said no, but on his resume had done game programming. When asked about that, he got quite descriptive and I knew then that he had the skills to do interactive graphics.

At the time of the interview he was a borderline student because he had majored in engineering, got poor grades then switched to programming getting decidedly better grades. Ultimately though upper level management said no because we had a rule of minimum B average even though his CS coursework was A’s but his engineering work. More recently, I saw that he’s now a successful highly paid security programmer. One we missed.

Another time, I interviewed a student with little to no CS background who boldly presented her math proofs as evidence that she could do programming. She was an amazing programmer in the short time she worked for us and moved on to a major company upon graduation.

I myself once had the trick question sprung on me in an interview and immediately decidedly not to work for this company. The question was specific to C++ and had been described in a recent Dr Dobbs article on the top ten C++ interview questions. I answered it quickly and asked the interviewer if he had read the article and he said no so we discussed the article for the remainder of the interview.

In yet another interviewfor a senior position at a startup company, I got interviewed by a junior programmer who asked the variable swap question. I was dumbfounded like really you’re looking for a senior architect and you ask your candidates this kind of question.

Candidates will be interviewing you even as you interview them and the smart ones may well decide that your work environment sucks. I look at these kinds of tests as a form of hazing that should be dropped as they create an atmosphere of fear that will scare talented programmers away.