Bringing a programmer in for an interview and a coding test can lead to some interesting experiences, both for the interviewer and the interviewee. Most end up with the hiring manager telling them that they’ll “be in touch,” but sometimes a candidate just nails it. That’s when you consider extending a job offer before they get a chance to leave the building.
At TimeTrade we run a coding test during interviews that, for the majority of programmers, should take about 2 hours in total to complete. The whole test is comprised of a number of small problems to solve, each harder than the one before. That gives us a good initial gauge of performance based purely on completion time: if everything has been solved in under an hour, we’ll be smiling. But if two hours pass and even the first problem still hasn’t been solved, the candidate will most likely just be shown the door.
Above and beyond just solving test problems quickly, here are some signs that a programmer is truly awesome and should be handed a job offer before they leave your building:
1. They present multiple solutions
I recently interviewed a programmer who solved an entire set of tests twice: once with iterative solutions, and again recursively. I quickly made him an offer. Finding multiple solutions to a problem is a skill that engineers will need to use every day.
2. They write full documentation
Last year I interviewed someone who was so diligent, so detailed and so professional about his work that he created full Javadoc and comments for his code before he considered the solution complete. He even wrote fully automated unit tests and checked their coverage percentage. When I came back into the room at the 2-hour mark and found him typing furiously I initially thought he was having trouble with the test, but he was actually in the process of adding HTML formatting to his Javadoc. Engineers who do this intuitively are the kind you’ll want on your team.
3. They improve the test
We deliberately create tests that have some minor issues lurking within them, purely to see if the candidate (a) spots them and (b) is willing to fix them. It might be an inconsistent usage of quotation marks for strings, misleading variable names or anything along those lines. Candidates that look at all of the provided code as the test — not just the pieces we’ve asked them to write — are the ones who will do the same in our real product once they join our team.
An engineer who is willing to tell a potential employer that the supplied test contains problems shows that they consider the quality of their work to be more important than just agreeing to do what they’re told. Hire them and they’ll likely work wonders for your product, going above and beyond their assigned areas to make improvements where they are needed.
4. They refactor smartly
Most candidates like to get a solution working, then sit back and breathe a sigh of relief that they finished it successfully. That’s good, but rarely good enough to justify an on-the-spot job offer. The candidates that solve the problem but then jump right back in to refactor it are in a different category entirely. Their choice of algorithm doesn’t feel right, and they can’t ignore the feeling that it could be more efficient. Their code has some duplication in it, and that burns them up inside. These are the candidates who refactor, rewrite and improve their solution until it’s been crafted.
This can be a double-edged sword, though. If the candidate just keeps rewriting because they’re not happy until they reach a mythical point of “perfection”, there’s a chance they are one of those programmers who doesn’t know when to stop (and similarly, ship). However if they watch the clock carefully and are able to both solve the problem and refactor their solution before their time runs out, that’s a really good sign that you should consider an offer.
5. All other signs point to “hire”
Sometimes there are plenty of non-technical signs that you’ve found the right candidate. Your other team members take you aside and tell you, “We have to hire this lady.” Their personality feels like a great fit for the team. They have relevant and recent experience in what they’ll need to do. You know some people who have worked with them before and they tell you they are wonderful to have on a team (and that they’d hire them again in a second). The candidate is excited about the company and the opportunity and is hungry to start contributing.
If the candidate passes technical muster and all other signs point to “hire,” why wait? If you do, you may lose the candidate to another employer who knows how to read the same signs faster than you can. Instead, be decisive and make the offer fast, thereby telling the candidate how much the company wants them on board. It will help start the whole relationship off on the right foot, for both parties.
So the next time you’ve got a wonderful candidate in your building, don’t assume someone even better will arrive the next day. Make them an offer and get yourself – and the candidate – back to work.
This blog entry has since been translated into Spanish (by Maria Ramos/Web Hosting Hub) and French (by Hinault Romaric/Developpez.com).
More Than Coding 
What kind of coding test do you guys give? I’d like to see if I can do a quality job in 2 hours.
Send us your resume and we might bring you in to do so 🙂
Sounds tempting, but I’m in deep south Texas and Massachusetts is too far. 🙂
Maybe just a hint of what the test consists of? Is the test geared towards algorithms or actual code design?
I’d also love to take a peak not because I’m looking for an offer but because your test has me really curious. I’d love to see any of it you’re willing to share.
Perhaps posting some question analogous to those on the “test” might be a good idea! It will attract the attention of the people you want, discourage the people you don’t, and gain you and TimeTrade some valuable mind share in the software development community.
We tailor the tests to the position for which we’re hiring, because a one-size-fits-all test wouldn’t tell us much. If it’s a web developer position, it’s going to involve adding new features to an existing web app (and require knowledge of HTML5, CSS3, AJAX, jQuery, PHP, etc). If the position is focused more on server-side stuff then the problem will likely touch on multithreading, scalability, architecture, implementation patterns, etc.
One thing I didn’t make clear in the article above is that we bring engineers in to interview only after they’ve passed a technical phone screen. That’s where we cover CS basics like algorithm complexity and data structures, and then have them code online to solve the usual quickie problems that should take 5-10 minutes each to code up. The phone screen is a pretty effective filter, and you wouldn’t believe how many people self-select out of the process once they’re asked to actually prove they can code.
The interview itself is as close to a simulation as we can get of what the work environment would be for them, should they be hired. We’re looking to find out if they can operate effectively in our environment and within our organization’s culture. In the few hours that interviews typically provide, we’ve found it to be a pretty good mechanism for hiring the right people.
Brian can you tell me know how do you conduct that technical phone screen, to me it seems like quite a time consumer. I mean what if you have cca 50 candidates, do you phone screen each candidate? That doesn’t seem efficient at all and the worst of all that phone screen needs to be conducted by some developer because you cannot pass this task to non-technical staff member. That seems like taking quite some developer’s time from the production.
In the previous company I worked they would tailor the tests for the required position, similar as you do, but they would invite participants to solve them online. True this does not have the benefits of testing how the candidates behave in your environment and in collaboration with other developers, but it is quite a time saver because you filter out very large percentage of candidates without interaction.
If I remember correctly this was the recruitment procedure; first they would send you an online test (I believe it was from these coding test site: TestDome.com), after that you get invited on follow-up interview in which you discuss about the test code, about their previous work, etc. but most of the discuss is non-technical related
Mario – we filter candidates pretty aggressively before they get a phone screen (which as you’ve correctly noted, takes a lot of time from developers to conduct). We have also recently added a new stage to our hiring process that filters them even further: candidates cannot even submit a resume to us unless they solve some technical puzzles within the job description itself. That screens out the “tire kickers”, gives us some confidence in their technical abilities, and saves us a lot of time phone screening candidates that are great at crafting resumes but are terrible at programming.
I see, well in that case I must admit you have a great recruitment strategy, I like it :).
“candidates that are great at crafting resumes but are terrible at programming”
It’s sad but the truth is that there are quite a lot of them …
I can provide you a copy of the test, I was so diligent, detailed and professional that I did everything twice.
You can contact me at jenny_b@msn.com
Thats the right thing to do but can be done faster and much more effectively with Interview Mocha.
Try using Interview Mocha that can help you hire the programmer on the spot as Interview Mocha’s assessments can help you assess candidates skill according to the job profile.
I’ve done a lot of these sort of code tests and while your points are good, I’d add another. Always ask the prospective new code what blogs they follow and if they start to elaborate enthusiastically it’s a really good sign that they actually care about their craft.
I like your first point especially, though it really depends on what sort of questions you pose and how difficult they are.
I hate those types of questions. Do you really want some code monkey who lives to code? While those guys are usually awesome coders, coding is only part the job. The other half involves understanding and appreciating the business as well as interacting with the non-technical people who generate the revenue in most companies.
We don’t hire code monkeys at our firm as we really need people who can work with the business. If I need a code monkey for his ninja skills, I will call in a consultant for a few weeks and then show him the door when the project is done.
I agree with your comment.
Totally agree here
“If I need a code monkey for his ninja skills, I will call in a consultant for a few weeks….” Teams definitely needs skills balance to be healthy, but candidly, I believe that your philosophy here means you’ll _never_ have a world-class team, or even have a realistic hope of working towards one. Superstars advance the state of everyone’s art, when their skills are leveraged appropriately.
Being in tune with the community is so much more than coding. If you are following blogs it usually means you also are keeping up with methodology, best practices, recruiting and requirement analysis. All of which aren’t about code but vital in understanding what actually makes both software and users tick. All blogs aren’t about ninja hardcore coding tech.
“Always ask the prospective new code what blogs they follow….” I agree. I use a similar question that’s a little broader, “What do you read?” The answer says a lot about how well the candidate stays abreast of new technologies, and in what areas they’ve been working to advance the state of their skillset. It’s even more telling when they can’t answer the question right away. 😉
As a potential customer, this gives me more confidence in your company — you guys have a great product and obviously a great culture of excellence
I’m curious as to the contents of your interview questions as well. Especially since it doesn’t sound like someone’s in the room with these candidates, hearing their thought process, solving hard problems together, or traditional white-boarding techniques.
Have you thought about this in-person factor? Did you intentionally trade that for another benefit (like ability to screen a huge number of candidates and still get work done) while developing this method?
Hi Jonathan,
Actually, someone is always in the room with them during the test. It’s a collaborative experience where the candidate gets to ask clarifying questions if they wish, because that’s they way they’d be working with the rest of the team if they were hired.
– Brian
Have you ever followed up on candidates you didn’t hire? In other words, have you evaluated your method.
What kind of follow-up are you referring to, Martin? Are you asking if we contact them a month after rejecting them to find out if they’re happily employed, or something else?
You partition the set of candidates in two; those you hire and those you don’t. You assert that your method is effective, at least in your context. You seem to conflate/correlate effectiveness of your method with the situation of your product and/or company.
Unfortunately, you probably lack a control group, hires hired using some other method. You could evaluate your hiring method in several ways. Here is one way. A year later, arbitrary: rate your hires; compare them to each other, developers in other companies, to those you rejected. It might not be easy but attempt to survey rejects about their current job, salary, etc.
Just a thought.
I understand your point of view here, and can’t deny that someone that can handle those tasks in two hours under interview conditions is exceptional. However I do want to caution against using this as a sole judgement. I’ve been developing for over 12 years, and I have to tell you this process is so abused that messes with the average developers head. I recently took a coding challenge for an up and coming software company that asked me to complete a coding challenge. I did my best work. I used the latest technologies, fixed problems in their sample code, and so on, as you mentioned above. I falied the test. Why? Because they expected me to work within the parameters that I have been given. They wanted to know if I could work with poorly named variables, code from 10 years ago, provide solutions without resorting to libraries, etc.
OK, so at another proespective company two weeks later – another code challenge. This time I took what I learned from the last test, worked within the parameters I was given, provided custom solutions within the framework I was given – but in the comments noted how I would fix it if not given those parameters. Failed the test. They were disappointed that I didn’t go outside the box, scrap what they gave me and do it “right” using new technologies and levering common libraries. UGH!
At the end of the day, you need to give a LITTLE direction to your potential hire, and if he fails the “test”, then ask him why before showing him the door. If he just didn’t know to do something, then OK, maybe he is not the right guy. But if had a different vision or maybe does perform well in an interview climate, ask for code samples, documents, screen shots, whatever from his previous work.
Good points, and I agree. A couple of things: we don’t use a binary pass/fail system with the test, and the 5 points aren’t our sole criteria for instant hiring (just a few of the ones we’ve used in the past).
I also wouldn’t classify our tests as coding challenges, rather they’re more like simulations of what it’s like to work with the team. You collaborate with an engineer and build something together, and there’s plenty of direction and feedback throughout.
Thanks, you made me laugh at 5:40 AM, before I started pounding on some code-retooling my coding, languages and skills “repertoire” :O)
Todd, this article isn’t about average developers… it’s about how to identify exceptional developers. The article is lacking, in that it only provides one possible method of doing that – but it’s a good method. I’m fond of saying I could walk into any kindergarten class in America and pick out the potential good programmers – but how do I do that? I can identify smart kids by the way they play. I could also give them a test, of course, and that would be effective, but other methods are effective as well. Average programmers may be flummoxed by the test, but exceptional programmers won’t be, and there are other methods of identifying exceptional programmers.
I like this… I wish I got hired like this. In my current job, I had to explain my past work, tough situations and how I solved them and all. That’s normal stuff you’re always prepared for. When it came to technical specifics, they asked really simply questions like basic, foundational stuff we learn in high school and I was applying for a senior level developer position. I felt bad during the interview and they could notice the slight changes in my facial expressions due to shock and awe at the type of questions (what’s a class, abstract class, interface, sql joins etc). Then I actually jutted in and told “these are actually basic questions” and then they showed me a pile of resumes of people claiming senior level experience who couldn’t even answer these and at that point they just wanted to hire the next person who could explain their past experience and knew the basics.
They hired me. Well, I consider myself average when I compared myself to most people at stackoverflow and I think my boss was lucky that I happened to be as good as they required someone for their position but they hired someone else using the same technique and he’s a total snob who can’t even google properly or clear his own compile errors not to mention that he can’t even refactor his own code. They’re thinking of firing him now.
Seriously… I consider myself as average but when I see more people in the market like him, my ego gets a boost and the people that are involved with hiring me or managing me treat me more special. If that’s how they treat me then boy, have standards come down.
I don’t believe in rubrics for hiring anyone on the spot. Programming tasks during interviews allow you to observe behavior, but not its motivation. Is the candidate testing, documenting, and refactoring, because that’s how s/he usually operates, or is it because s/he is trying to pass an interview and knows that’s what some interviewers expect?
A far greater predictor of engineering behavior is, IMHO, is the development culture in the workplace. An engineer who’s spent the last ten years refactoring, unit-testing, and documenting will either discontinue these practices or become resentful in an environment that does not value and encourage these practices. On the other hand, an engineer who has not been sufficiently exposed to these practices will be under social (and managerial) pressure to adopt them once immersed in a healthy engineering culture. If the candidate demonstrates the ability and willingness to learn from mistakes and grow beyond his or her established preferences, then that candidate may still be a fit for a culture that will challenge him or her to do better.
So the only reason why anyone should hire an engineer “on the spot”, is because that engineer would fit into and thrive in the culture of the workplace. Tragically, there are many workplaces, where the hypothetical engineer described in this post, would be grievously out of place.
Great points, Yev. On the motivation question: that’s for the interviewer to figure out. In my experience it’s pretty obvious to tell if someone is pandering to the interviewer or if they’re naturally inclined to just write things like unit tests because of their past experiences.
Another challenge for an interviewer is the detection of raw talent, and whether a candidate exhibiting innate awesomeness (but who might not yet gravitate to things like automated tests, etc) could be allocated the time and mentoring effort while on the job to develop those skills.
I totally agree with your last point. There’s no way I would hire someone for reasons #1-#4 if they obviously wouldn’t fit in with the team or the culture. I don’t think it’s tragic that there are many workplaces where the hypothetical engineer would be out of place, though – it would be tragic to hire them into one of those places and have them doomed to failure from their first day on the job.
Hi Brian
I would be very interested to see if I could pass your test (not that I am looking for a job, but because I consider myself very good one-man-company developer – as I have several business applications that are commercially used (www.zpas.hr))
Andrew
This is an excellent point…I have been a Senior Engineer for years and have been through these types of test and as a previous comment stated…Failed. But companies that hired me found I picked up the culture, was a team player and once learned my place and how to best contribute became the best employee they had on staff. Tests and pre-screening apart from culture fit and personality will not gain anything but a robot coder with zero personality
Code tests are a complete waste of both parties time and effort as its more around a pissing contest then it is on quality of a candidate.
Phsycologically speaking your doing more harm to unearth potential than good, as some folks may have deep insights into problem solving skills that often require more thought but in most cases code tests put an emphasis on fire drilling code checking / mathematic hurdles.
If you have two hours to solve a customers problem and you need to grab a random developer from your resource pool – sure you can now define that list.
Ultimately engineers need to stop finding mathematical benchmarks to rationalize a human to human metric. I’d hire a engineer who enjoys learning but failed a pop quiz vs am engineer who nailed pop quizzes but can’t interact socially or is highly introverted – worse – isn’t passionate about the role on offer
Establishing whether a candidate is a good technical fit is why you have probationary periods, as now you have them on real problems that need real solutions.. But more importantly you have these candidates also fitting on socially and culturally within your teams..
You can spot a fail within the first 3weeks quite easily.
I blame Joel on software for this lunacy around “every programmer should be tested before hiring”. It’s pandering to elitism that assumes your hiring the top 5%
News flash : top. 5% don’t sit tests and aren’t looking for jobs.
If our coding tests were administered blindly (IOW, without regard for personality fit, approach, nervousness of the candidate, etc) then I would agree with you. However, that’s just not what we do.
We make a concerted effort to simulate the candidate’s potential future work environment for the in-person interview so that both parties can figure out if it’s a good fit. It’s not an exam, nor is it a check-box of pass/fail on some super-high nerd-bar of difficulty. It’s an experiment conducted by both sides to see if the working relationship would succeed. It’s better to hire the right person who is a great fit than to let someone go after 3 weeks because you didn’t vet them correctly. Sure, that can still happen – but we prefer to drastically reduce the chance of that happening through a solid technical interview.
I would consider myself in the top 5% and I like tests. What I can’t stand is a company who clearly doesn’t care if I have the skills I said I have and doesn’t want me to prove it. That indicates a management problem, and may indicate that I’ll be working with people whose skills aren’t up to par, on the assumption that they were hired with the same non-objective process. I agree with Joel Spolsky on this, because I don’t like working on teams with people who schmoozed the interview and were never tested on their skills, and who drag the team down with their lack of ability. I am a good developer, and I’m ready to prove it, and if you’re good enough, you should be willing to prove it too.
People in positions they are not qualified for is a rampant problem in the software industry. Testing addresses that issue in an objective way. Of course, the test has to be good, and I think this article describes one good testing method. There are bad testing methods as well, such as the Microsoft Certifications – many of the certified folks I’ve worked with obviously passed the test, but they couldn’t do the job.
very true. Durning our interviews we have a senior and a “lower” level developer in the interview process. I’m the “lower” level dev I’m basically highering my team lead.. Most of the interviews I’ve done are with people who have 10+ years while I have 5. (Side note I haven’t been impressed with most about 90%) During the interview I tell them my experience level and I ask them questions that someone at my level should answer. Depending on the response I’ll either ask them more generic questions or lead them to something thats more complex that a “newer” developer might need an explanation and see how they respond.
Also on a side note that I really haven’t seen too many responses indicate that is very important the the person that being interviewed is someone who you’d enjoy to work with everyday. Don’t get me wrong that a developer who aces everything is great, but if she/he can’t hold a conversation about the weather is a big flag in my book.
I think that an objective process is definitely key… but making a ‘good’ test is really tricky and it should be geared towards evaluating the type of skills that are the MOST important (or at least somewhat important… cmon) for the job that you are applying for. The tests should have well-defined constraints and instructions, and once the tests are complete the code should be the basis for a discussion about why certain choices were made.
I’ve basically received the same type of coding test from 10 different companies for the Sr. DevOps roles that I’m looking at… and it’s really sad honestly to see what they’re asking me to do. It’s basically high-school CS where I’m supposed to open one file and pull out specific parts of that file and then compare it with a second file and output only certain lines from the second file based on a comparison with the lines from the first file.
The instructions are poor, and I’m expected to clarify each time with these kinds of questions: “Ok do I need to worry about memory?” “Does this need to work for files of arbitrary sizes?” “Does this need to handle error checking?” “Is it important that the code is readable & documented?” “How long do I have to complete this test?” etc…
But in DevOps, I am given a specific situation that I know a lot about… and based on the situation I am used to answering these questions myself. I should be given a specific and relevant situation… and then create a solution to solve it based on my experience and knowledge of what the constraints are when dealing with that kind of situation…
If I’m given a situation that compares the /etc/passwd file with the /etc/group file, I’m not assuming that it’s reasonable/possible in even a system set up by the most bumbling sysadmin that those files would be millions of lines long each… And if they’re blank or corrupt we definitely have other issues… If this script needs to run unattended for a long period of time it should have error checking and logging. But if it’s just run once I wouldn’t need to do everything in software as long as I:
1) make backups of the files
2) ‘less’ into the files and take a quick look to make sure that there aren’t any surprises.
3) Run a test on copies of the files…
Wayyy back in CS 101 did it ever really matter if a one-time piece of code executes in 0.0000000001 seconds versus 0.000000000099999 ? If that is important to you, then please give me a real-world example that requires optimization. Or just specify that the solution needs to be as fast as possible. My recollection was that highly optimized code often needed to be rewritten when future assignments built on the code written in the earlier assignments.
Also fyi for interviewers… if I’m using a scripting language there might be some behind the optimizations that alters the typical CS understanding of the performance characteristics of sets, hash maps/hash tables, and arrays. (See php and python for example) It seems that any question that *CAN* use a hash map probably should use one in a programming interview… Scripting languages can also handle errors in a pretty amazing way that interviewers may not be familiar with. For example, splitting a string a=”a:b:c::::::::::” in python by using b=a.split(‘:’) works just fine and results in b=[‘a’, ‘b’, ‘c’, ”, ”, ”, ”, ”, ”, ”, ”].
Which brings up a final point… in the *real-world*, writing a first version of your code in a more generic and unoptimized way leads to code that you can more easily adapt in the future when you rewrite or refactor due to changing requirements. And less optimized code is generally more readable and for other coders who may actually need to read your code and make future changes it can be easier to spot problems or make changes many months or years after you finish your version. But in a coding interview, always assume that they want the most optimized version of the code… but make sure to ask anyhow just in case. Some people may actually care to see if you can code in a way that balances between speed and maintainability.
Testing is important, but it is extremely important to design tests in a way that helps reveal the characteristics that are important to the job you are hiring for.
True.
I would be interested to hear what points you look for to see “good potential”. There are cases where technically one doesn’t know the answer, but learning and finding a solution would then come in. In an interview that’s generally difficult.
I understand there are most likely generic tests that any developer should know. But say for example they fresh out of collage or is that a case of “some one else can train them up”?
Nice Article 🙂
“We’re looking to find out if they can operate effectively in our environment and within our organization’s culture.”
But there’s an extra filter you didn’t mention, i.e. what you get is people who can solve problems quick under observation, in the position of a test subject with “many eyes staring at them”. Not necessarily the best overall.
“In the few hours that interviews typically provide, we’ve found it to be a pretty good mechanism for hiring the right people.”
You can’t know, since you don’t know how good the people you didn’t hire because of above mentioned problem actually are.
I’ve been given some bigger tests to do in a few days, at home, and found that to be much better.
As for simulating the actual working environment to see whether one “fits in”, I doubt that you do this effectively – in an actual working environment you’d also ask colleagues something you don’t know/remember off the cuff, or just ask for a better idea here and there, to get forward more quickly.
But productivity is a measure of what you got done per unit of time. So if the test isn’t limited in time, how can you measure if they will be productive? I have worked with guys who have to put in 60 hour weeks to keep up with everyone else – that is BAD for them and BAD for the company.
Yes, but productivity on one set of problems, especially well known [core] problems, does not automatically equate to productivity on esoteric scenarios. I know many coders that would probably ace those tests but will take 10 times longer on what I would call “simple” issues in a real and changing environment because they’ve been too busy writing HTML documentation for their code and tests…
That should be where previous experience, job history, and references are suppose to come in. Testing should NEVER be the only criteria on hiring.
And just because someone has aced a test or shown any of the other exceptional abilities should not cancel or overlook their history.
As far as the tests go, I’m okay with it… but, only believe 2 hours may be too long to begin with and would deter many applicants. Not that 2 hours of work is long; but, that 2 hours of heavy work may signify what the 8-10 hours in the office will be like in the long run.
As far as what the test really gauge should be more up in the air than based on performance. Real problems are never simple or as easy as building a stack… if that where the case we wouldn’t have a job; because the computers already know that stuff.
I don’t believe the person writing the blog really wanted everyone to just concentrate on the test or the few exceptional traits that everyone should be taught or learn anyway. Most schools are trying to teach a better bread of coders and commenters for the code. Comments are hard to find and often neglected … as a result, readability and changes later really suffer. I still find myself coming back to some projects after many years and saying “what was I thinking when I wrote this?” Not that it didn’t work… but, we all learn things as we develop that make us surprised when we look back at some of our earlier works.
I had a company (well there were other partners as well) developing both hardware and software for industrial solutions (which we eventually sold off) and at every interview I did with the software developers (primarily with an SQL background) I tended to go for those who were not afraid to have an opinion, just as the article stated, had a natural drive towards improvement, robustness, efficiency etc… now at the peak there was a large open plan office with 11 people (men and women) punching out stored procedures, queries, C and Assembler based embedded code etc … (sorry to waffle on there) but could I ever get them to agree on anything as a unity group they would argue and debate everything … hahahahahahaha so I suppose its just “be careful what you wish for”!
Oh and one other thing we never tested the Software guys we hired we only employed through background checks … hmmmm maybe tests are a better way!
Scott
Is it just me or is anyone else flashing back to a certain sadistic Monty Python job interview skit? At least it does sound like you intended to hire someone. You know, maybe if you were not such a nerd you would acutally have talked to /worked with enough colleagues that you wouldn’t have to find someone you didn’t know from Adam.
lol.
1. Multiple solutions means he wastes precious time over nothing. He/she should think multiple solutions but write the best only, and btw iteration can be simulated with recursion while the solution is the same alghorithm wise bor both implementations. Not to hire!
2. Its a plus, but not a deal maker.
3. True
4. If Refactoring is the new word for legible code, ok, that is a must. Actually the oposite makes more sense: code duplication = fail. Tbh, a good programer hardly writes duplicate code from the start.
5. Agree, but only with a fortuneteller proffessional services :)))
I am a software developer and I would be the best one you have, however I don’t care too much for your test because its missing my best strength. I am an innovator, so asking me to do a good job of changing a tire does not interest me much but if you asked me to design a new car or ask me how I would solve a problem then I will show you where I shine.
Hi. I’ve never really been to a job interview, I’m not old enough, but some people have discussed hiring me. I’m just interested in what type of problems can I expect to be given. It would be really great if you could send some examples to me. My email is: yannbane@gmail.com.
@William – You are the kind of developer who can start things but when the boring part of finishing it comes up, you’re less motivated. In a job interview, you need to show you can get things done, and not just the cool fresh stuff, you need be able to get maintenance tasks done, even when they are boring. Your attitude is very common, and almost all good engineers share it – but that passion for new exciting creative things doesn’t get our address validation database updated every month – which I will be doing tonight, and I honestly have better things to do, like work on my new Android app – but I’m being paid to do a job, and I need to be ok with the fact that I won’t always like it and I won’t always be working on something interesting. In fact, a good 90% of the job is working on boring stuff, so if you can’t shine there, you won’t be very useful, and you will hate every minute of it.
Good article. But, being on the receiving end of some recent interviews I’ve also learned to look for a few signs you should run away from the interview.
(1) They give you a tour and show you the place (or more specifically room) they have shoved all the programmers into. There was one empty desk in the room and 8 other sardines in the can (so to speak)… sorry, I don’t do well in cans. This one was also located in a heavy industrial park with constant moving of heavy equipment, not exactly perfect.
(2) Another, I interviewed in a business that was mainly manufacturing with only 2 programmers (or engineers)… I wasn’t told which they were or if there were any more. This company was working on a problem with a processor I could have helped with…. This one really wasn’t what I’d call a run away opportunity, but more of a cautious one. Where they hiring just for the problem, and let you go afterwards or would you end up being the only one left or only programmer in the group?
I’m sure others may have more horrifying stories to share.
Words to live by.
I would have been so happy if the last interview i had here in India with one of the biggest American gaming companies which has the name “Bally” on it.
It was also a 2 hour coding test where it contained one paper in C++ and another one in MFC. I have done very well in C++. Infact that was the best coding test i have ever written, but the other paper i have not done that good because i always do not memorize the method/function names that the MFC provides because its a framework and you can get the method names from the IDE itself. But all of the questions from the MFC were about the function names!
HR told me that i was not selected because i have not done well in the MFC part :(. It was really a very sad experience. I really do not know why we need to memorize something that is readily available.
I thought the coding test is to test the coding skills of the programmer and not memorizing a framework?
Can somebody correct me on this, is it necessary to memorize the methods that framework provides?
Why, disable L1/L2 cache on your CPU and compare performance. That’s what you get with your supposedly helpful IDE.
Overall: nice article and great discussion, made me smile gleefully way more than once recalling different occasions (including getting interviewed once back then). Thank you, nice folks over there.
I’d like to contribute a reference to the excellent Johanna Rothman’s “Manage It!” book which had very similar effects on myself while reading it.
I have only gone through the whole cold interview process to get a job twice – every other job I’ve gotten by word of mouth or already knowing the potential employer. In the first case I was freshly dropped out of college. I was given a programming test, although it was based on several systems of pseudo-code. More of an aptitude test than a test of knowledge or experience on any particular language or platform. Several friends had applied for this job and said I’d probably walk out on the test. I opened up the test to the first page, flipped through it, and knew I had the job. The second time came 12 years later, and it wasn’t a programming test but a four-hour psychological profile – which I later found out to be illegal. I got both jobs. Interestingly enough another 23+ years later, I might have trouble passing the pseudo-code test just because it was a perfect fit for my youthful ability (and willingness) to learn rapidly as well as my general test-taking skills. Today I have more resistance to learning something just for its own sake. I don’t study languages and systems that aren’t needed in my work – it’s not a hobby. I believe I’m capable of learning new skills, and learning them rapidly. But odds are I wouldn’t have the immediate knowledge necessary to pass your test in the allotted time.
As to the other aspect of fixing problems in the existing code, I’ve mellowed with age there too. Used to be my motto was, I’ll clean up the mess but you can’t pay me enough to sit in the mess. But greed has replaced pride and sometimes the mess is a cash cow. This is where it’s clear that different companies have vastly differing practices. Some would appreciate a guy like me with a passion for rewriting, cleaning up, improving (even my own code), others wouldn’t give me the time of day if I didn’t embrace the sheer genius of all that came before me.
I want to add one more thing that I personally look for in a candidate: willing to admit they could be wrong, or that they don’t know everything. I’ve worked with some very good technical people who will refuse to do something that makes all kinds of sense for the product and end user but doesn’t fit their technical view of how it should be. This trait tends to show up in recent graduates (especially with masters) but can also appear in people who have primarily internal team experience and little or no end user contact or feedback. Most of my life I’ve had more contact with end users than any 10 programmers ever experience, and I always give them the benefit of the doubt because in many cases when it comes to software, I’m an end user myself. So it’s easy for me to take their side in any debate over UI or just why a feature might be needed.
Good points Winston. I have been coding on Wall St for 15 years and I have similar opinions about code tests and the latest technologies. On interviews I have been asked what I read, I answer the NY Times. What non work projects I am working on, my answer is the lawn and my kids, I don’t code for fun or adventure. Quite honestly, I have never done well on a technical interview but I have been financially successful in our craft.
At some point I think that we all mature out of the code monkey phase and accept that our skill set is suited for the job, not a lifestyle. These kinds of interview techniques and code tests only serve to find the most suitable code monkey in the hiring managers own image.
When interviewing programmers, my opinion is that you can teach almost any decent programmer to code the way you want them to. It is harder to teach them to understand and appreciate the business that the company is in. It’s even harder to get an introvert to interact with and understand the business users, and that is a skill set that is invaluable in our line of work.
I would add one to the list:
– Create detailed estimates: Estimates are estimates – but there are many programmers out there who just give a ballpark estimate of the costs, without any justification. Detailed estimates are appreciated by the client who wants to know how his money is spent.
Additionally detailed estimates prove that you have fully understood the requirements.
If you handed me a two hour test during an interview, I would show myself the door.
Would you audition a violin player without making her play?
Instead, I have asked people to bring in a copy of some source code they have written. Their whole personality will be in that code, the disciplined and consistent formatting or lack of it, inefficient coding, and maturity to name some off the top of my head.
Working alone is not quite the same thing as working in a team, interacting with other developers, providing and accepting feedback, and, yes, dealing with some time pressure – all facts of life in real-world engineering. While the prospect of being tested on these skills, which can never be fully perfected, is legitimate grounds for anxiety, it does not make for an unfair interview practice.
In your opinion, musicians are analogous to programmers? I disagree with that premise entirely. Programming is not an art form; it is just a technical skill.
You cannot detect if a person can create beautiful music by questioning their knowledge of music theory. On the other hand, you can quickly figure out if a programmer knows what they’re doing with a short conversation about problem solving with heavy use of technical jargon. You can also figure out a lot about their personality, interests and suitability for your environment at the same time.
No one said testing was unfair. In my opinion, anything above and beyond testing for some problem solving skills and basic technically literacy is overkill and you are risking losing quality candidates. Honestly, my initial objection to this article was that the test took two hours. That is excessive and a bad omen with regard to how the company values the time of its associates.
@Chris,
You are right in the sense that a 2-hour test seems a little long. It is sorta like asking the musician to play for 8-hours straight. You want to test the applicant so you can attempt to determine their skill level; but, at the same time giving them an overly long test can be a large discouragement to the applicant and they may think twice before accepting the position.
Weather programming is an art form or a technical skill is a topic still open for debate. I’ll say it may be a bit of both, in reality.
Would you audition violin player on Piece of music that she/he never played before ?
You can get the basic. And actually the only way to know she/he is advanced in her/his craft is for the interviewer to know the subject that the performer play.
To make it little more clear (cause my English is not so good 🙂 ) if i should not write some code you want for limited hours to show my craft. You should look at my code to decide am i good in my craft or not. There can be assignment but the time limit is stressing.
I have like 3-4 jobs all with some code test and none of it prove anything about the actual work. On my jobs no one ask me to code him something only for 2 hour or i fail. They are delivery crunches but if this is you code practice i better not work for you 🙂
You are right about the art form debate James, and I fully expected to get called on that comment. And yes, coding can be an art form in the eyes of many, but after 15 years I have lost that loving feeling. What I have learned is that to most non-coders, and by that I mean the people who pay us, time is money and that they don’t care about art. I guess I have just become jaded by my reality.
@simmol you ask “Would you audition violin player on Piece of music that she/he never played before ?” The answer is yes. Every music audition I ever took part in involved sight reading. It wasn’t the only part of the process, but an essential one. Whether a placement audition or a whole band competition, how you perform a piece of music you’ve never seen before is something highly valued in a musician. Many musicians make their living playing one-night jobs, traveling shows, and have very little opportunity to rehearse – they are expected to show up, blow through one dress rehearsal (or none) and go.
IMO anyone who has both played music and programmed clearly understands how closely analogous they are. And is also well aware of the roles egos play in both crafts 🙂
My pet peeve is interviewers that expect you to write a program on a white board. White board programming isn’t readable by humans or machines. All the variable names are x,y,z and i,j,k because it takes to long to write proper understandable names for function. Then play computer writing output on the board to see how it runs. Stepping up to a computer to write a program and run it can be done as simply as using an editor and browser or more elaborate with an IDE–and it shows a lot more about a programmers capabilities. Two hours is enough to develop a good piece of work.
If I encountered someone like that, I would also have to evaluate their interpersonal skills too. That kind of attention to detail is almost OCD.
Ah, here’s a blatant mistake in your article. “Refactoring” implies a machine-based, algorithmic reimplementation of code, not human-based. There is no such term for a human-based refactoring. You should be explicit in your metaphor!
So no humans engineers have ever done refactoring? Machines are sentient enough to detect code smells and commit changes to fix them? Your definition seems academic at best, and removed from reality at worst.
Um, like I said. “You should be explicit in your metaphor”. The author should say something like “human-based refactoring” or “human-like refactoring” to be explicit and proper. Please work on your reading interpretation.
I should have said “reading comprehension”.
“Machines are sentient enough to detect code smells and commit changes to fix them?”
Also, are you so far withdrawn from reality to note the existence of refactoring tools in modern IDEs? That is where the buzzword is derived from.
No, of course not, I’ve used them for years. But I’m still the driving force behind the initiation of the refactoring process within my IDE. My point was that your initial comment seemed to imply that it was a 100% automated process that required no human involvement even to initiate it, and I wanted to clarify that point.
Now, boys, let’s not fight. “Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior.” — Martin Fowler (http://www.refactoring.com/). No mention of humans and/or machines.
Have you confirmed this with other sources? Can you tell me what Wikipedia even says? Rafactoring.com is merely “a simple portal”, and cannot be considered an authoritative source.
One other thing I’ll add, and this may be slightly off-topic, however just as important is for the employer to know exactly what they are looking for, and tailor the interview to that position/duty.
For example, my background is largely in “front-end” technologies, and my primary skills would involve things such as HTML, CSS, AJAX, Javascript, as well UX design and workflow, usability, etc. I simply cannot tell you how many interviews I’ve been on where the company has the technical interview conducted by a “back-end” developer, who then proceeds to ask me all about back-end technologies and methodologies. While there are certainly arguments to be made that a good front-end guy would have some knowledge of back-end, it should not be considered the sole basis of the technical interview in that case. You interview based on the core competencies of the position.
I mention only in regards to those who expect to setup a perfect test that fits every hiring situation they have, which in my experience, is a LOT of companies. Each hire is unique. As many have said, a person who is passionate about their work and is a good fit personally and professionally may be a much better employee in the long run than a “rock star” developer who can code anything thrown at him, but expects the world to kiss his… ya know.
That’s funny you should see that in “rock star” developers. My image of one is of the Russian twenty-something whiz kid from an Ivy League University who develops algorithms for winning at gambling… and succeeding back in the ’90s, like in one movie I saw (can’t remember the name). It is unlikely that such a developer would be working at such a level presented here.
Some coders are slower than others because they are 4 finger typists. They choose short names because they can’t type very fast It doesn’t mean they are bad programmers: just that they aren’t fast typists. They’d have difficulty finishing a 2 hour test, just because of their typing speeds. I’d watch how they coded rather than how quickly they came up with a solution: quick problem solvers aren’t necessarily good coders: especially when they cannot understand anyone else’s code other than their own.
Presenting multiple solutions to a problem: I’ve actually been rejected for a job for doing that. They reckoned I was overqualified and they didn’t need someone who was that good. That was 15 years ago.
Writing full documentation and creating test harnesses: I’ve actually been rejected for a job for doing just that 10 years ago. They reckoned that since I did more than was required, that I’d get bored and leave after a short while.
Maybe I was just looking at the wrong jobs.
Excellent article – having been taken for a ride by a programmer who talk a good game and had an impeccable CV, this is putting your money where your mouth is.
A balance of good programming skills, communications, and common sense is generally the ideal candidate. Focusing too much on how quickly a person can build a working stack of functions and not focusing on the other areas can be a recipe for disaster. There’s a very good chance you’ll end up with a lot of bright people, but a very handicapped team of know-it-alls who think they should write everything from scratch (i.e. reinvent the wheel). This will typically result in very long development cycles and a code base that doesn’t take advantage of proven standards (making it difficult to maintain). I’ve been in the business for years and have seen this happen quite a bit. I’d much rather have a team of good programmers who can work together than a team of superstars who would rather work in a silo.. This causes way too much management and will eventually lead to a non-technical manager making all the decisions rather than the team moving the project forward. If that’s the kind of team being built, then you’re taking a huge risk that will mostly likely result in a failure.
Biggest criteria from an employee is honesty. Willing to to a good days work and not steal anything (including time). Then ability to get on in a group and ability to learn. Where in your silly test are you looking for the most important attributes. Lots of people get shaken down. Remember you never get ripped off by someone you didn’t like. Laslty have you ever employed someone who did NOT pass the test to see if the test was valid?
very good ….
Really nice artlcle and so true. We have been into hiring mode recently and believe me its very tough to find a candidate who is good on all these parameters. We interviewed around 40 candidates for a single position and could manage to get only 2-3 good candidates for the final round.
We are trying to hire 60 developers this year, 5 a month. We are in NYC and its hard to find 2-3 people that might do ok. So if you’re reading this let me know.
I don’t look at developers from the code level, don’t ask them to code for a lot of reasons, and I definitely don’t judge developers on their level of insistence on refactoring ( I’d rather have them think ‘this isn’t the most elegant solution, but I know it works, is readable, and meets requirements, so I’ll move on to the next problem and see how much it annoys me later before I spend time refactoring it)
I want to talk to them about projects, and *how* they talk about them, projects they’ve done, problems they’ve had, how it could have been avoided next time. in most cases, I expect them to not talk about code, but the process.
when a developer keeps wanting to talk about code, it says to me he still considers that lower level to be a problem he still obsesses with. higher-level programmers don’t worry about that stuff, they know they will find a solution for any problem at the code level. they worry about the external factors of how the process is managed to insure success.
That said, there are a breed of great developers who still love to talk about code and are great to have, but In startups especially I want developers who see more broadly and can elevate to lead others later. in other words, developers who see code as a tool and delivering software as their skill, not those that see code as their skill and projects this annoying thing that sits on top.
Very good points.
Another good trait is being able to think outside the BOX.
I mean this for all those that haven’t started as well. You have to think differently when programming a computer; realizing that they do exactly as they are told and don’t improvise (that doesn’t come until they fully perfect the super-computer). It also relies on talking to the hardware, regardless of what anyone says, you always have to touch at least one piece of hardware in most circumstances or you really aren’t having fun (so to speak).
The trick with programming is finding the correct sequence of 1’s and 0’s that produce the desired results and doesn’t cause problems in the long run.
@stationstops – This is exactly the best comment on the entire blog…Process, Process Process… The Process should be focused on. The “Why” not the “How”. The “How” can be Googled by any high school student but the Process is what you are being paid for. This coupled with personality is the two ingredients that should be the catalyst for hiring someone on the spot. The blog is really saying that they want walking robot encyclopedias of code. When you hire a life-less person that is exactly what you will get….No creativity, No Real Problem solving, Only Code focused, and a No Thinking out of the box mentality employee that probably has inter-personal skills issues.
These interview tests reveal more important stuff about the people giving the test than the tested. Personally as a matter of principle I will refuse any offer of work from a company that employs formal tests on interviewees. This has had quite amusing results at times over my 30+years. But I _always_ stick to it.
Why do you not want to demonstrate your skills? If you’re good, you should be able to do that, right? In my 30 years, the places with testing have ended up being good places to work. I have rejected an offer in the past based on the content of the test (I applied as a Senior developer and they tested my algebra skills, which I found insulting). However, it’s not an insult to ask a person to demonstrate their skills – they don’t know you, and a glowing past can be fabricated. If you have the chops, I don’t see any problem with proving it. So, could you please explain your position? Are you famous or something? (I would not give a coding test to Steve Wozniak, but I would give you one)
In today’s world, able to provide multiple solutions for a single problem is a valuable skill set.
It seems that the current brutish employment market allows employers to be extraordinarily picky with their hires. I guess programming jobs will be only attainable for the top 10% of coders who are so obsessed with the latest k3wl technology that they code at work, and then code at home. I’ll bet they really have a great understanding about how the world works …
In my experience, many companies are extremely proud of, and confident in, their particular interview processes despite the fact that studies of interviews have shown them to have little ability to predict performance. I have no idea whether the test process you describe is effective or not. Obviously, it’s very difficult to study the people you reject. However, it would seem that you could assess your process by looking at the relationship between your assessment of a candidate at interview time and how he or she actually performed when hired (e. g. after they leave). It might also be worth considering how different interview processes affect the acceptance rates of candidates to whom you do extend an offer. Prediction of something like job performance is extremely difficult, so I don’t put much weight on speculation about how each feature of an interview may or may not determine some specific aspect of a candidate’s ability. I’d be far more convinced by a few actual measurements.
While I like to use a simple logic code test in my interviews, I rely more heavily on a more open dialog with the candidates that follows a more natural path through their programming experience and history poking into those awkward moments we’ve all experienced and uncovering how they acted, felt, thought, reacted, learned etc. A lot of the examples you have shown in this blog highlight people who may suffer from “perfectionism”. Running a project loaded with perfectionists in, you’ll find that the quality of work is key and you can merely wave byebye at the deadlines as they go sailing past.
Personally I appreciate it when companies show their hand early in the piece. I for one would most definitely show myself the door as well. I finished all my pressure testing at uni (High Distinctions) before anyone says cant handle the pressure. What guarentee does the employee get that your worth working for? This sounds like a shop full of geeky programmers trying to out geek the next poor interviewee by putting them under pressure with their cleverly crafted tests….in their opinion. Very biased way of doing business. IMHO. You can take a punt on me just like I have to take a punt on you. Your probably devoid of personality and criticise people at the drop of the hat while you stand around laughing at programming jokes.
Superb Article. I appreciate your work. I wish i got the programmer who has all the above qualities. And I also keep in mind all the points while hiring a programmer. Thanks for sharing this article.
You have stated very good points , I read all the comments , martin , I think you have never been an interviewer !!
Thanks , UOBabylon
The “interview as a game” approach inherently flawed: http://www.youtube.com/watch?v=cHiGVJeuPbs&feature=relmfu
As a T-SQL developer I’d want to know if during the coding test I had all my normal tools available such as my personal “tool kit” of scripts, my snippet manager, my RedGate tools, rights to run scripts against the master db, rights to view execution plans and create indexes, etc.,…not to mention simple internet access to Google. Would you expect an employee to work without his tools? Why not tell a carpenter to build a house with a handsaw and toy hammer?
Another thing coding tests won’t tell you is whether or not your new hire is willing to work until 3am on the night before that important project goes live…and then come back in at 9am the next day to make sure everything is working OK. We recently lost a good employee who worked with us only a few months because he was basically an 8-5 kind of guy and his wife wouldn’t put up with unpaid overtime (we are all on salary) and didn’t like it when he had to work from home either.
And one more thing these tests can’t determine is a candidate’s inherent business sense. Does the super-star coder know what a bill of lading is? Or the difference between nominal interest and APR? Or even the difference between revenue and gross margin (probably the most important thing in any business)? Without knowing what to ask of clients up-front and getting the requirements of a project correct, the super-coder will just do the wrong things very fast, then do them over again very fast, and then after a few more rounds of such real-time “refactoring” watch the profits sail off when the client bails or the hours expended wind up 3 times over the original estimate.
Efficiency and speed is great as long as it’s focused on the correct things. In my experience the consumers of my services rarely see the big picture during the planning stages and it’s my job to give them a finished product that meets their NEED–and more–even if they didn’t think of it at the start of the project.
There are different kinds of people, and I use to work fine with both 9-6 folks and overtimers like me given they know their job (still usually they know way more) — but *demanding* overtime is demanding to kill off productivity in the long run, I wouldn’t invest that in any business but my own (and overtime is one of the worst things I can do to my own one).
My Point exactly…Excellent comment
The comments are all good. In fairness to the original post, though, the article didn’t say that “These are the only indications of a keeper” but “these are 5″… and they are. If viewed as “the acid test” then, no, they will select for certain desirable attributes. I’m reminded of the “Moneyball” effect.
Personally, I would “fall out” on a test like that. Many people do poorly on tests. A bright young math teacher who “gets” *why* so many bright people don’t succeed in our schools has put together a great presentation on a better approach to harnessing his student’s energy, and I think the same approach will apply to the workplace:
I’ve had the good fortune to put my *strengths* to work and it has made me very valuable.
Reblogged this on 7heaven11’s Blog.
hello, i have a question, i need to hire 1 developer, i do not have any skill in developing anything, still i will group 10 developers and 1 will be hired, i’m researching for programs and things they can develop and test them like that, make them work for the position, any suggestions?
A word of warning: project failure chance is much higher if the group leader/manager has no on-hands experience in what a group has to do. For one, I am carefully avoiding such situations.
If you are determined that this is the way to go, then:
– ask those 10 (or 9, or how much?) developers what they would suggest you regarding hiring — a reasonable recommendation is a good thing;
– be prepared to check with your customer every week or two regarding the prototype/early code being developed and what’s to be accepted as a delivered product (can’t imagine a product company where clear understanding of the result and the process are IMHO even more critical).
A good reading for a software dev team manager: http://pragprog.com/book/jrpm/manage-it
Jorge, are you wanting a test for the new hire or for the whole team?
@Michael Shigorin… thank you for the answer and yes it is much more difficult for me since I have really really basic experience, but i’m also the customer, this project is for me to use. I will read what you send me.
@WoundedEgo, no I’m only hiring one, my idea is to give them an assignment test and whoever does it better, gets the spot.
@Jorge, I’m curious who decided you needed a 10th developer and why, since you are just assembling the team and presumably have not even started the project and you seem unclear about the job requirements. Maybe you should hold off hiring until you have a clearer picture of what you need. I heard on NPR today that decisions made prematurely are often disasters. “Ready! Fire! Aim!” kind of things!
@WoundedEgo, Regarding the project I really do have very clear what do I want and what do I need, i dont need a 10 people developing I need one, i’ll be interviewing 10 developers, and i will only hire one. The idea is to give them an assignment test and whoever does it better, gets the spot.
What my real question is if there are any suggestions regarding Assignment Test.
This clarifies the issue…have you read ALL the replies to this article? The majority of respondents have taken issue with this article. Developer assignments are NOT a proper way to asertain if you should hire or not. In fact the real test would be the verbal discussion, their team working ability, their project history types and how they handled previous issues that arose during the project. Also, and overridding most of the other items listed is the ability to work with and personality of propective employee. To test knowledge of material, a walking encyclopedia a good developer does not make, instead do solid reference checks, and give a probation period of a few weeks with part of the actual work in the environment that the employee will be working in…this will test the prospect properly!
@Michael Shigorin… thank you for the answer and yes it is much more difficult for me since I have really really basic experience, but i’m also the customer, this project is for me to use. I will read what you send me.
@WoundedEgo, no I’m only hiring one, my idea is to give them an assignment test and whoever does it better, gets the spot.
@Jorge – Oh, I see, I misunderstood! Then maybe:
* relevant resume
* references
* rapport
Ah, that’s a different game of course.
Might also be worth looking at what they’ve done already and ask for recommendations regarding the projects that are more or less similar to yours. Someone’s experience in the problem domain helps a lot too.
That book is a huge overkill for a single man project (but even then it does show the common pitfalls like leaving things going on their way for too long only to find out that they went astray — e.g. into scratching our technical itches instead of getting something actually useful done).
Another good idea would be to give the test and ask them what they would improve & what language they can use to improve the test. (For example if the challenge you gave them can be done better in Ruby on Rails or Java versus PHP)
The Microsoft approach: http://www.microsoft.com/learning/en/us/certification/bethenext.aspx
I read somewhere that you need to spend at least an hour with someone (or any proposition) before making up your mind. An offer that sounds like a no-brainer at first might start looking like a problem after 45 minutes and vice versa. I’m not talking about a test, just about maybe doing a long lunch and chatting about some the things that characterize your company culture and see what comes up: http://www.youtube.com/watch?v=rMRqZV-P6Ok
Good points to review before hiring a developer! Actually, I’d rather do a series of interviews to decipher a person whether he’s adept at programming or not. In most cases that’s how recruiters differentiate an applicant among others when it comes to skills and attitude. When you are hiring a programmer I would prefer outsourcing it to a trusted site especially if it focuses on full time work. We could name a lot of freelance sites today but I have more trust with http://www.staff.com. They have a pool of talented programmers and it’s always easy to contact these contractors and interview them for as low as $1.
ya true and really good
Indeed these are better than lot of those tricky Java questions
Very cool 🙂 Check out http://www.jobstock.com/blog/how-to-hire-a-programmer/ for a really great perspective on the basics of hiring a programmer when you’re not a programmer. Also very handy read.
Nice blog post. I have one comment, point 3: “We deliberately create tests that have some minor issues lurking within them”… If I took a coding test and I saw that, I’d probably think that the team were sloppy with their work and be less likely to want to work there. I’ve worked at a couple of places where the work was, frankly, sloppy, and it doesn’t make for a happy place to work (when the server goes down 10 minutes before you were due to leave because of an easily preventable error, you start to wonder why you work where you do).
It may be something you’d want to mention after the test to the candidate, or you might be leaving the wrong impression. Interviews aren’t really interviews, they’re business meetings, they work both ways.
Coding interviews. Program challenges. Nothing but a scam and an insult. I own a software company, and I only hire the best people. It’s very clear to me, after many years as a programmer, who is good and who is not. I don’t need any tests. That is just so insulting and unprofessional. As for interviews, all I need is to chat with someone for five minutes and I know if I want them. In general, the work they have done speaks for itself. What I really need is to find out if they are compatible with our company ethos. By the way, it never ceases to amaze me how utterly rude, arrogant, antisocial, and generally clueless most programming nerds are. I wouldn’t trust most of you to give me the time of day. There are truckloads of smart programmers, but the real challenge is finding someone who is not only very smart, but a decent person who knows how to treat other human beings. Someone who is not a primadonna and knows how to work as a team player. Something most of you lack in spades. But I’m lucky because I love computers and this allows me to put up with the few occasions when I absolutely have to have contact with computer nerds I despise. Fortunately I can work with my own people who I select, and I like them all. Have a nice day and try not to be such snot-nosed pricks, if you please. (I’m sure not of you think this applies to you, just to your manager or someone else.) The funny thing is, you all think you are so amazingly smart, just because you work with computers. Yeah, right. At university I took classes from a few Nobel Prize winners. They were pretty smart but even they admitted they were not that smart compared to the greats. Humility is part of being truly smart. By the way, the way some of you nerds like to trick people with your oh-so-clever “tests” just shows what jerks you are. You disgust me. Now back to work and make some good software, please. Go ahead and reply, have fun. I’m not even going to bookmark this site, I just was in the mood for a nice rant.
A smart programmer will never take an in-house coding test. In-house tests are usually written by a non-certified programmer who has a personal and limited area of knowledge. You should only be willing take standardized tests written by a certified programmer from an accredited group or company i.e. W3C, Brainbench, Pearson-Vue, etc.
Razor, that’s, probably, is not completely correct. Try to think from the perspective of employer. He wants to be sure that he is hiring the right person meeting to the exact standards of his company. Giving in-house coding test seems to be correct approach to check that. And I think, that if the person is really smart he will accept that fact and pass the test.
I absolutely agree that the test and conditions of testing should be correctly and strictly defined. As briefly described here http://skillbox.io/blog/coding-interview-done-right
coding tasks should be error-free, have precise definition, should not require specific academic math knowledge, etc.
If done right the coding test practice could be beneficial for the employer. It also should create positive experience of the interviewee.
This list is stupid. if you are looking for a developer who can build an IT infrastructure from scratch to build a corporate website, you end up hiring a guy who can code and solve your coding problem, that doesn’t mean he’s capable of building an IT infrastructure. I know my boss hired that guy. He was a star at solving coding problems at the interview, but after my boss hired him, he doesn’t know jack about anything else. The application he built was riddled with errors, my boss had to hire 2 QA engineers just to find his bugs.
Coding interviews are beneficial for people who like to solve puzzles, thats all there is to it. Not a lot of people like to solve puzzles that doesn’t mean they are not good at what they do. I work with senior engineers who can’t solve these types of puzzles but they build concrete bugfree production applications that are used by the whole company.
And the guy my boss hired, he couldn’t work under pressure. Everything he built was amateurish. But he had no problem solving coding problems if you ask him.
I am the CTO at a software company, with nearly two decades of IT experience. I personally don’t pay much importance to coding while hiring. Coding on the spot puts unusual stress on the candidate, and usually does not give reliable results. Many candidates who fail coding tests are observed to do well on coding jobs where there is no pressure as in an interview. The world of programming changes so quickly that within two or three years, many skills get outdated. So we need people who are quick to adapt rather than people who can do a lot of outdated coding but cannot learn something new quickly. I like to test how adaptable the person is, in terms of ability to solve new problems, strong ability in high school/college mathematics (these guys are usually good at whatever job you assign them, even if they have less or no experience doing these jobs), ability to interact with a team, has knowledge of algorithms and software engineering, and knowledge of the business side of programming. Yes, I do have people write code, but that is not the ultimate test of overall programming ability. Some times I challenge the candidates to ask me questions that should make me think hard – this usually gives quick results. Most importantly, I like to assess how enthusiastic the candidate is – some times this is all I look for selection in junior programmers.
I’m reading all this because I cannot sleep before yet another technical interview. Just one extra bullshit layer added over all the rh non-sense and filters.
Just this morning I had a surprise technical test where they sat me on a computer to type a web page with two blinking css words. Simple, insulting, I was in panic.
I typed <stylesheet.. (wrong). I was remembering "rel". What's rel? relative? Relative to what. I was remembering "type". A css is necessarly text, why do we still have to define type? They give me permission to use what I wanted with someone watching me over the shoulder. It would have taken me 2 seconds to any web page, say goodle, open the inspector and check the declaration of a basic css link, not stylesheet (would be too logical considering the script tag). Confirming the impression of me being a fraud. Because, if I can't even do that… what is even true in my CV?
The last time I had to type a css link from memory was ages ago. I'm working with cms that all have their different take on css management. Some declare them in templates, other add them through generator code… you want to add one (at the right place as it could be in the core, the theme or pluggins files) you copy-paste the line and change the ref (making sure you use the right variable likely defining the root path). If I need the full spec of a link tag, it's just a few seconds away.
The guy was not pleased to realise that I was not able to use "vi". Even more amazed that I did'nt found the web server path… I was looking in /srv, it was obviously not there and I knew it. I was just stressed to do this simple exercice quickly to even take the time to recompose myself and go into /var/www. I was asked at the previous interview the command to restart a service.. I was only seing /etc and .d. /etc/init.d was blanked out.
Trivialities that killed in a instant my confidence and erased years of experience and craft that is more than a assemblage of computer facts and data.
The stress of the interview itself (added to the stress of not having a job), having someone over the shoulder seems to kill my brain and I get huges memory blanks. You are not sensible to that reality when you are in the other seat.
I'm a good developper but I'm thinking more than ever to leave this stupid career. Special thanks to the guy who posted "I will no do your technical interview" on the internet… will try.
There’s a big difference between employers that test for aptitude and ability compared to those that test for trivia retention. Employers that focus on the latter tend to also focus too much on their pet tools and favorite frameworks, to the detriment of their hiring process.
Technical evaluation is an important part of a hiring process for an engineer, but it should be a realistic one that gives the candidate insight into what it would be like to work there. If people aren’t going to stand over your shoulder when you’re on the job, them doing so during the interview is unfair and not representative of what it would be like to work there.
So my recommendation would be in future to turn to the person looking over your shoulder and to ask them if they would be there every day on the job. When they inevitably say, “of course not!”, then ask them why they’re doing it now. Their answer will likely be revealing, and tell you a lot about their culture, their ability to trust their employees to do the right thing, and the caliber of the other members that they’ve recently hired with the same process.
“We tailor the tests to the position for which we’re hiring, because a one-size-fits-all test wouldn’t tell us much. If it’s a web developer position, it’s going to involve adding new features to an existing web app (and require knowledge of HTML5, CSS3, AJAX, jQuery, PHP, etc).”
Haha. Brian, you would never hire me since you are hiring for experience, and not for ability. I’m one of the most exceptional programmers you’ll ever get to interview, and I’ll be adding more value that ten other good developers who work at your company. But your hiring process is broken and you’ll never hire me.
Also, completing N number of problems in two hours is a joke. I’d rather find the most optimal solution to one problem in that time, instead of finding N hacked up solutions.
Honestly, with this interview process, I believe that you are ending with some good developers, but no exceptional developers. And an exceptional developer is miles apart from a good one.
Great article, thank you!
We at Alconost translated this post into Russian: http://habrahabr.ru/company/alconost/blog/264735/
Thanks for translating it!
I’m a veteran c++ developer with many AAA projects in credit. Though being happily employed, I have to move to Germany asap for family reason, so an easy way is to find a senior level developer position there, and that has got me to go an interview recently, after a long time being away for this! I am probably one of those who doesn’t do well on on spot code tests, sometime even forgetting things I had thought others before in my classrooms! So I understand the frustration of those against the practice of written tests. This puts in the camp those against it, at least for senior (at least well experienced) candidates, though I know I am biased I guess!
My main arguments are the followings:
1)This type of candidates have already accumulated lots of track record, so examining that should be a good substitute for a written code test, if you’re obliged to. How to do that? Well, that depends on what are you trying to find out and what the two sides can agree upon.
2)It is definitely insulting if you ask them entry level or tricky question, because the first one is simply disrespectful (questioning their basic programming abilities), and the second one (asking tricky question) is plain stupid! If you don’t remember a peculiar trick algorithm, which is easily available with a quick search, that it really says anything about you?! In this interview I went, I was asked to do width-first parsing of a binary tree. Well, I remembered from old school days that I need a container for it, but not the full solution, so after I mentioned it to the interviewer I embarked on inventing a new solution myself! Sure I wouldn’t be able to finish that in a few minutes!
3) People have used the analogy of hiring plumber in previous posts above, but probably a better analogy is like hiring a doctor in a hospital or in a clinic. The same as an software developer, med doctors have also spent years studying and also acquiring experience in residency. Now, are they actually asked to perform an operation on the spot, or even put them through some biology or anatomy question drills for interview?! I very much doubt it, though love to be corrected if it’s the contrary!
So, for the next application that I sent out just a couple of days ago and sure I received a response next day I cordially refused to do pre-screening automated robotic programming test online! I emailed them it is simply not acceptable! They came back in hour, apologizing that although they totally understand and agree with me, but it is the head office policy and applied to everyone in the company!
cheers
The analogy to a doctor is not quite apt, because in medicine there are strict certification and continuing education requirements in each specialty. In software engineering, it is possible to be a veteran and produce atrocious code. I have seen and maintained plenty of such code. I would rather work with developers whose code the interviewers have seen. Yes, that may weed out some good people who produce quality code under normal working condition, but it’s far better to eliminate a good candidate than hire a bad one.
Wouldn’t want to work at a company that hired this way. They have no clue what a truly proficient and quality candidate looks like.
I am the CTO at a software company, with nearly two decades of IT experience. I personally don’t pay much importance to coding while hiring. Coding on the spot puts unusual stress on the candidate, and usually does not give reliable results. Many candidates who fail coding tests are observed to do well on coding jobs where there is no pressure as in an interview.
The problem I am continually running into, especially since the increasing popularity of agile methodologies, is that you test for skills that probably, based on my own experience, do not manifest in your actual code and processes. The industry standard in code quality today is crap. Very low level, poorly structured, non-commented procedural crap. Four times larger than need be and at least twice as slow.
Most organizations today are only interested in speed of development – all other factors being considered non-valuable. So you may test and look for top notch skills, but do you allow top notch developers in your organization to do their jobs at the level of quality they are capable of, or do you whip them into and out of each iteration?
Are there any developers out there that feels like the code base they are working on represents high quality software – or are you, like me, finding your employer is on a headlong flight into mediocrity and simply isn’t interested in doing anything well or better?
…”Their code has some duplication in it, and that burns them up inside.”…
Sometimes, code duplication is fine. I’ve been doing Windows desktop development as a hobby for almost 2 decades, and as part of a company for a decade. This includes very limited resource programming for embedded devices, to down-and-dirty internal engineering utilities w/little error handling just to get a job done, to database apps…
While I initially had the mentality that all code had be as efficient as possible, in all ways, execution speed and code size, I don’t think that way anymore for Windows Desktop apps….at all.
…things like….well an If/Then runs through 82,000 iterations a second, where as using an IIF only runs through 48,000 iterations a second…so I’ll use standard if/thens. (Not real numbers, but an example of how anal I used to be about it.)
Several months ago, I was writing new code for embedded device based on old code that an outside developer produced. This guy was great and making very small, tight, efficient code. However, this made it very hard to adapt the code, and use a large base of it for some other application, and resulted in re-writing the whole thing from scratch, which ended up being the faster route, with more re-usable code.
So I transferred that into my Windows desktop programming. Yes, I still want to use the least amount of programs steps to accomplish something, and still want to instantiate variables as little as possible, and use the least amount of mallocs/unmallocs, etc, but I’ve found it’s usually much faster (programming-wise) and easier to create 2 functions with 75% of the code the same, than to write one function, with a bunch of different parameters and types and program flow control. And this isn’t to say I don’t write “reusable” functions, but now know when it’s best to do so, and when it;s best to just write a 2nd similar function.
lopers in your organization to do their jobs at the level of quality they are capable of, or do you whip them into and out of each iteration?
Having moved to a new country and going through this process myself, I can’t help but think that this whole process of putting someone on the spot (White-board) is simply a bad idea. It is by no means an indication on whether the person is a good developer or not. Solving a complex data structure or algorithm problems on a white-board in front of others does not mean someone is a good employee either. Granted this should be job based, but my criteria to hire good developers breaks down to the following:
1- Skills
2- Experience
3- Behavior
For skills, there are various methods that can be used to determine whether the candidate actually knows their stuff or not. First step should always be a technical call where the interviewee asks general technical questions related to the position, for example if the position is a .NET C# Developer, I’m assuming the candidate can explain or have an understanding in most things related to such. Keep in mind that this is a phone call, so asking the candidate complex questions over the phone is not productive. You just want to have an idea of their background and their skill level. This also gives you a chance to have an idea on how the candidate can communicate.
If the technical call goes well and you’re happy with the candidate, the next step should be a take-home assignment. Give them a challenge of some sort and give them the proper time to complete and submit. This gives you an idea on whether they can get stuff done or not, and more importantly, how they solve problems. If you’re happy with their solution, then invite them on-site for a face-to-face interview. This should be the only interview conducted to save everyone time.
During the face-to-face interview, you can wrap everything up. Discuss the assignment they did and ask them why they chose that approach or if there are ways they can think of to improve (Discussing the assignment is important to determine that they indeed did it themselves). Also discuss their work experience in detail and ask them to describe the things they worked on in the past (If this is a person with no experience, ask them about school projects and hobbies). The point is you get to know more about their technical background and skill level through these discussions. Lastly, and this is extremely important, throughout the entire interview you should also watch their behavior and determine if they would be a good culture fit for the company or not. After all, these are people that you will work with, so a good positive attitude is just as important as the skill-set, the last thing you need is to hire someone that will make your environment toxic.
I think that should be it, but if you absolutely must test them further, don’t do it on a white-board, instead just give them a piece of paper with all the questions you want them to solve in the face-to-face interview, and give them a private space somewhere to solve them. Don’t stand over them and don’t put them on the spot, that way you’ll get a much more accurate answer on their skill level.
The next time someone asks me to solve programming questions with a paper and pencil, I am also going to request a pair of scissors and some tape. What I turn back into them will be a taped up bunch of scraps of paper with the final solution. If they can’t provide both, then that interview is over.
Asking me to do my work with paper and pencil is as ridiculous as expecting a bird to fly after removing its wings. You can’t cut, copy and paste with paper – unless you have scissors and tape!
I generally don’t produce perfect code in perfect order directly out of my head all the time (stunning, I know) – I tend to refactor as I go, letting the code inform me as to the correct solution. You can’t do that at all with paper and pencil. So, whatever you are getting from a paper and pencil test, you aren’t getting anything that resembles the persons skill level or their approach to problem solving.
I’ll mention briefly before I leave that “You know, it is much easier and faster to develop on a computer – do you require all of your developers to work with paper and pencils as well?”
Great post. As somebody who hires freelancers on regular bases, I constantly have to find creative ways to tests candidates before hiring.
One of the key things I keep in mind, which was mentioned, is multiple solutions to problems. I encounter so many people, who seem incapable, or afraid – school does that to people – to offer creative ways of solving problems. So many people will come to me with a problem, then expect me to give them the solution even though I hired them to come up with the solution. It’s crazy! As if people are worried that they are going to get sacked because they come up with ideas.
Anyway, I’m not a technical person, and find it hard myself to hire programmers, but one area I think is vital is if they fit in with culture at your business. I’m a bit of an introvert myself, and I like giving space to all the people who work for me. As most are like me, quiet, and work better on tasks alone. If somebody comes to the interview who is loud and imposing themselves, I know they may not fit in, even if they are immensely talented.
I wrote a an article recently on my experience on hireing programmers. And it covers this aspect of hiring. It might implement your article nicely.
You can find it here:
Anyway, thanks again for the great post.
I would add that if your test requires a large commitment of time that you had better provide the applicant with feedback whether or not you hire them. I have twice had interview tests that took approximately ~8 hours to complete (reported time was using the honor system), and one of them I completely blew the doors off of – I highlighted errors and issues in the test along with corrections, I provided dual solutions for each answer, etc., etc. After submitting I did not receive any feedback whatsoever on the test. I had to contact them via my head hunter and press for feedback several times before I got any information back.
I love it when a potential employer gives me technical tests – it allows me to seriously impress them. The only kind of technical test I can’t stand, and have actually terminated interviews when asked to do them, are the ones where they hand you a piece of paper and some pencils and ask you to code solutions to various questions.
I refactor code as I create it, and that requires a computer. I usually will create comment lines that describe what I need to do, then fill in the details. How are you supposed to do that on a piece of paper? Asking someone to write code with a pencil and paper is like asking a bird to fly after you tear off its wings.
But as I write this I have come up with a solution for the next time it happens. I will ask them to please provide me some tape and a pair of scissors along with the paper and pencil. I will then proceed to write portions of the solution as they occur to me, cut them out in lines and blocks, and tape them all into the correct order. As I turn over the taped up solution, I will probably mention that coding is actually a lot easier and less time consuming on a computer.
You’ve got some great tips for hiring the right programmer. I like how you said that it’s good to find someone who intuitively writes full documentation. To me, that just shows initiative and a good work ethic, so of course I’d want to hire someone like that if I owned a business.
Nice post and 5 signs to hire a programmer on the spot. Thanks for sharing.