Some engineers are genuinely better at using ChatGPT and friends to solve problems, and that's a skill worth including in evaluation of potential software engineers. I would advocate interviewers go with the flow, allow use of LLMs, but have candidates do it on the "shared screen" to see how they think - just like using a web browser in interviews.
I won't cry if "allow LLMs but watch how they use it / think" doesn't scale to "no skin in the game" offline tests where interviewers try to get candidates to spend an hour without the company matching that hour with an hour of employee/interviewer time. In fact, if (and it probably will) this kills offline testing, I think that's a win for labour.
What ChatGPT "cheating" reveals, imo, is how weak and artificial "whiteboarding" interview questions are: they claim to be problem solving tests but they're actually algorithm memorization tests. Interviewing without allowing use of LLMs is like requiring "no web browsing during interview": it artificially biases toward test-specific skills and away from real world strengths. If an LLM can straight up solve the problem, its not a very good match for skills required in 2024: its now a problem that SHOULD be solved via LLM.
It turns out that LLMs are better at rote memorization than we are. Good! The "whiteboarding algos" interview technique was already a bad match to real world SWE skills except in relatively rare job roles, and its a good thing that its artificiality caused it now be irreparably broken by ChatGPT. Bye bye, won't miss you ;-)
Most bog-standard software engineering work is boring things like good variable naming, effective use (and non-overuse) of encapsulation, designing a schema that won't make you hate life in 5 years, wise selection of dependencies, etc etc. Testing live with a candidate, watching them use ChatGPT, seeing how they think, this is what gets you a job-relevant-skill signal.
People were complaining about whiteboard coding interviews in the 1990s and they are complaining about it today. Meanwhile the tech industry has managed to hire millions of incredibly smart people and done pretty well for itself in that period. The interview process isn't going to change. There is no reason for it to change. Your options are to either suck it up and brush up on basic data structures and algorithms or interview at places that don't test for it. Just don't hold your breath waiting for the world to change for you.
The author says "whiteboard tests" are broken, but it seems like they're arguing that online coding assessments are broken, not in person interviews using an actual whiteboard.
Doing an in person interview on a whiteboard sidesteps the AI issue. As someone who's done a large number of remote interviews, there are some clear signs that some candidates try to cheat online tech interviews. I wonder if the trend will fuel more of returns to the office, or at least a return to in-person interviewing for more companies.
That's actually a valid question. It looks like it was an unpopular one.
Personally, I despise these types of tests. In 25 years as a tech manager, I never gave one, and never made technical mistakes (but did make a number of personality ones -great technical acumen is worthless, if they collapse under pressure).
But AI is going to be a ubiquitous tool, available to pretty much everyone, so testing for people that can use it, is quite valid. Results matter.
But don't expect to have people on board that can operate without AI. That may be perfectly acceptable. The tech scene is so complex, these days, that not one of us can actually hold it all in our head. I freely admit to having powerful "google-fu," when it comes to looking up solutions to even very basic technical challenges, and I get excellent results.
Amazon uses Hackerrank and explicitly says not to use LLMs. In that case it would be cheating. However, given that everyone is apparently using it, I now feel dumb for not doing so.
I wonder if OpenAI/Google/Microsoft, et al would hire a developer that leaned heavily on ChatGPT, etc to answer interview questions? Not that I expect them to have ethical consistency when there are much more important factors (profit) on the table, but after several years of their marketing pushing the idea that these are ‘just tools’ and the output was tantamount to anything manually created by the prompter, that looks pretty blatantly hypocritical if they didn’t.
At some point I assume that it’ll be so normal that you’ll almost have to say when you’re not using it.
I don’t need to say that I’m using a text editor, instead of hole punched cards. It’s also quite common to use an IDE instead of a text editor these days in coding interviews. When I was a student I remember teachers saying that they considered an IDE as cheating since they wanted to test our ability to remember syntax and to keep a mental picture of our code in our heads.
If your coding assessment can be done with AI and the code that the candidate is expected to write can’t be, doesn’t that by definition mean you are testing for the wrong thing during your coding interview?
The process is anything but simple at large companies. Even if the new hire is a complete fraud and can barely write code it'll still take an average manager 6-12 months to be able to show them the door. And it'll involve countless meetings and a mountain of paperwork, all taking away time from regular work. And then it'll take another 6 months to get a replacement and onboard them. That means your team has lost over a year of productivity over a single bad hire.
That comes after the decision that you can't fix the situation, which comes after you discovered that the hire was bad, which comes after a number of visible failures. That's a lot of wasted time/effort, even if the firing itself is simple.
Depends on the country I think - in Australia at least it seems like you can sue for unfair dismissal if you're angry about being kicked out, so HR departments only seem to get rid of someone as a last resort.
AI is likely to shape the future, gradually embedding itself across industries, fields, processes, tools, and systems. Similar to the introduction of electricity, where identifying practical applications was inevitable, trial and error will be an essential part of its evolution. While concerns about its impact are understandable, it's important to recognize that each generation often worries about the capabilities or standards of the next. However, such concerns are typically rooted in current frameworks of thought, which may not remain relevant in the future.
Right now, generative AI looks to be more like microwave ovens than electricity -- usefully novel but not obviously revolutionary or all-pervasive.
Even with the remarkable convenience and new opportunities that microwave ovens brought to kitchens, it still made a lot of sense for commercial kitchens and culinary schools to keep focused on more traditional fundamentals and their mastery when hiring and training candidates.
Same applies to our field for the foreseeable future.
>Since everyone started using AI, more candidates started clearing the first round with flying colors. The platforms had to recalibrate to let in their target percentage.
Is there any proof or data on this? Not saying it's wrong but I'm curious how big the effect actually is.
I'm not convinced it would be caused by AI. More people are laid off, so more unemployed people can grind leetcode all day, and more people are competing for fewer jobs.
The dirty secret of hiring processes is that the main goal of the earliest stages is to get from a lot of applications to something less than a lot, not to screen for fit. This has long been true, and it is now super true.
Every job posting gets flooded with hundreds of applicants, if not into the thousands. Most of those people are coming through the front door with no one to vouch for them and probably nothing on their resume that makes them a must-see candidate.
Most managers and HR teams probably don't even explicitly think about it this way, but by pure pragmatism have evolved processes that act as flow control.
The unacknowledged result is that the company will reject 90+% of applicants, regardless of fit, under the assumption that the filtering process will allow enough good people into the actual interview rounds that the team will be able to find someone they want. From this perspective, Hackerrank is not broken, it's doing exactly what is required of it by companies.
I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it. It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit.
People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process.
One might ask is there a better way to do this? Probably so. But if it were easy, it would already exist.
There would need to be an easier and cheaper way to filter applications.
One option for candidates is networking, this gets you in through the "vouched" side door.
This potentially means that the company should be encouraging employees with more than financial incentives to find candidates and recommend them. This means networking workshops, time swaps for attending networking/recruiting events, understanding the need to make synchronous contact with people who might be good candidates.
If you are interviewing based on what you do, and what the job application (which has been mutated to get through HR's posting requirements, but don't understand why and how you should be interviewing you are more likely to bring your past experience and biases to the interview. This is bad for your company and for candidates.
> I say all this because people who are in job search processes should frame the process accurately in their mind. It hopefully will help with not taking the process so personally or not getting so infuriated with it.
For people who consider honesty a part of their identity, it's infuriating that the system has a selection bias for dishonest people. It's not selecting a random 10%.
> It may also help you strategize how to find your way into the roles you want, if crushing these tests isn't your strong suit. People who are vouched for get to bypass all of this. The more confidently you are vouchced for by a trusted party, the more benefit of the doubt you get in the hiring process.
Networking is still selection bias, but one that favors white men. Racial minorities have fewer social connections to hiring managers, and women in tech have fewer friends working in the same industry.
I recall having to implement A* to search a nxn character grid in my AI course a few years ago. It took me close to a full day to wrap my head around the concepts, get used to python (we usually worked in C++), and actually implement the algorithm. Nowadays, an LLM can spit out a working implementation in seconds.
I think that's a big part of the issue with tests like Hackerrank - LLMs have been trained on a lot of the DSAs that those questions try to hit. Whereas, if you ask an LLM for a truly novel solution, it's much more likely to spit out a garbled mess. For example, earlier today, Google's search AI gave me this nonsense example of how to fix a dangling participle:
> To correct a dangling participle, you can revise the sentence to give the dangling modifier a noun to modify. For example, you can change the sentence "Walking through the kitchen, the smoke alarm was going off" to "Speeding down the hallway, he saw the door come into view".
LLMs have effectively made it impossible to test candidates for crystalline intelligence (e.g. remembering how to write specific DSAs quickly) remotely. Maybe the best solution would be to measure fluid intelligence instead, or front-load on personality/culture assessments and only rigorously assess coding ability in-person towards the end of the interview cycle.
i screen candidates all the time (not involved in technical rounds) and notice chatgpt cheating all the time but its easy to spot because the candidates read off answers in mechanical ways right after a flurry of typing
I think its time for the federal government to get involved, and it already has leverage
If your company benefits or plans to benefit from QSBS tax treatment, then evaluate engineers on things they’ll be doing in the company:
the application development process, sprint ceremonies, asynchronous communication and tooling
they wont be implementing data structures often, and if they are reinventing the wheel they are wasting company time, if your company needs a more efficient data structures for its super scalable problem - it should trigger a process audit to be honest.
I don't understand how QSBS has anything to do with interview practices. Are you saying the IRS should somehow audit any QSBS company to make sure they're not implementing any data structures or algorithms? Like, new companies must be audited if they try to use computer science?
Control behavior by regulating the intermediary, constitutional under any governance system, my favorite approach to anything is to look at what the thing relies on, and controlling access to that thing.
We can even add whistleblower bounties to current and prospective employees. The IRS already has a bounty program, probably doesn't even need Congress for this one. Its under the Treasury Department right? Can probably still Scott Bessent on linkedin about implementing it on Day 1
So, yes. You got it. Play devil’s advocate all you want, I’m ready. More likely you’ll have to put your reservations in during the comment period.
2025 lets go, new administration is up for anything and has no clue what theyre doing
The center is unguarded I bet I could get the 119th congress to pass anything since all the moneyed interests went to far extremes of their parties and will never get their causes passed the filibuster
I won't cry if "allow LLMs but watch how they use it / think" doesn't scale to "no skin in the game" offline tests where interviewers try to get candidates to spend an hour without the company matching that hour with an hour of employee/interviewer time. In fact, if (and it probably will) this kills offline testing, I think that's a win for labour.
What ChatGPT "cheating" reveals, imo, is how weak and artificial "whiteboarding" interview questions are: they claim to be problem solving tests but they're actually algorithm memorization tests. Interviewing without allowing use of LLMs is like requiring "no web browsing during interview": it artificially biases toward test-specific skills and away from real world strengths. If an LLM can straight up solve the problem, its not a very good match for skills required in 2024: its now a problem that SHOULD be solved via LLM.
It turns out that LLMs are better at rote memorization than we are. Good! The "whiteboarding algos" interview technique was already a bad match to real world SWE skills except in relatively rare job roles, and its a good thing that its artificiality caused it now be irreparably broken by ChatGPT. Bye bye, won't miss you ;-)
Most bog-standard software engineering work is boring things like good variable naming, effective use (and non-overuse) of encapsulation, designing a schema that won't make you hate life in 5 years, wise selection of dependencies, etc etc. Testing live with a candidate, watching them use ChatGPT, seeing how they think, this is what gets you a job-relevant-skill signal.
reply