- Recently: Given some basic intermediate output from LLM system, build out evaluation/quality tools, especially for dealing with hallucination
- ~8 years ago: Given a database table of restaurants, identify near-duplicates to merge
On the hiring manager side of things, my candidates generally enjoyed doing a take-home to build a ML model for the Li & Roth question classification dataset. That involved a bit more creativity and it was very role-related.
After being on both sides of the interview process, my current feeling is that take-homes are a decent fit for junior roles but take-homes are not a good use of time (on either side) for senior roles.
Pretty much every company I have ever worked at has done the take home assignment (EU based). There is almost never any issue, at least on the surface, from the candidates side as to doing them. But online, these discussions immediately turn into the majority deriding the process.
I have probably done about 6 take home assignments in my 14 year career and 5 resulted in a job offer. In a recent company I worked for I was the hiring manager and we had about a 40% assignment completed to offer ratio. Which seems like a fairly decent conversion ratio, from the average candidates side that would be close to 1 offer for every 2 assignments.
Is there a correlation with people hating on them because the ratio is skewed much less favorably? Do I have too subjective a view? I know there are companies that give an assignment as practically the first step in the process; but I would never engage with that since the risk/reward is weighted entirely in the company's favor (which under the surface, is an even bigger red flag).
My favorite take homes, and almost always what I have come across, are the "build a mini api" ones. I will typically just draw from the structure I have in place in other applications, implement the custom functionality, and do a readme that delves into the philosophy (in a light touch way) of the choices I make. Then the conversation of going through the implementation in a technical interview is generally a relatively relaxed process.
I never had a good one, although that's probably because I started refusing to do them years ago.
The worst one was about 15 years ago. The task was to design and implement an ML system to solve a complex network optimization problem. It was so complex that I was given three months to do it. My estimation was that I could do it in three months if I worked on it full time, which was clearly a ridiculous ask.
> The task was to design and implement an ML system to solve a complex network optimization problem. It was so complex that I was given three months to do it.
That's what I thought. I refused to do it and dropped out of the application process because it really sounded like they were trying to get free labor.
While I know there are some exceptionally few objective ethical companies out there I have completely lost faith in take home assignments. It’s one of those: that’s why we can’t have nice things.
If a potential employer wants to evaluate my ability and style for writing code I have a GitHub profile with numerous projects populated over numerous years. If that is too much trouble then why the fuck would I do a take home assignment for them?
I had one that I enjoyed many years ago, to write a service that calculated the Nth digit of Pi.
It was up to you what you wanted to do / how / if you wanted to deploy. They only asked that you write how you would've liked to receive such service from another org.
Then in the day of the in person interview we reviewed my code / README / tests ; was very nice.
Take-homes are highly flawed. If you provide a small/short challenge that could be solved with AI, it'll not help you measure the candidate. On the other hand, if you ask for a month worth of work, no one is going to take that up. Automattic does pay for it, and that's the only way I can see this attracting eligible candidates.
At your stage (startup with 2 people), you are much better focusing with designing your product than designing an interview process. Hire someone you know or freelancers.
The trick is to set the bar higher so that you can only solve it with AI. Don't ask them to do hackerrank level questions. Ask them to solve problems where part of the solution is a hackkerrank level component.
Architecture level tests are a sweet spot for nos. AI is not good at this, but senior candidates are often very good at them. You can give them template architecture to start with or work on, something that AI is known to trip over. The only way to know what AI is tripping on is to use it.
If the candidate is better at using AI to write code than the team, bring them in!
This was also my thinking — if you ask ChatGPT to design a DB schema design it often looks good at first glance but is actually pretty bad.
If a candidate uses AI as a help to solve their task, yet the output is still good, great! It's their job to solve the problem, with whatever tools they have.
[A random opinion from the internet and based on not knowing the actual details of your context.]
If it isn't worth four or eight or N hours of your staff's time to observe the candidate working on the problem, then it is probably not worth four or eight or N hours of the candidate's time.
Because what are you trying to measure?
How well the candidate lies about how long the project took?
How willing the candidate is to work many hours?
How much free time the candidate has to dedicate to their job search?
How well they GGS (Google and GPT and StackOverflow)?
How resourceful they are at hiring a freelancer?
Etc.
Sure those types of behaviors often resemble people's actual jobs, so maybe a take home coding task measures what your company is really looking for.
But realistically, inexperience and desperation are the most likely reasons someone will be excited about your take home challenge. The best skilled and most experienced candidates -- the kind people who know them want to work with -- probably won't be excited. Good luck.
The best ones I have done have been actual paid tasks for the team. Where they found something they needed from their backlog that was small and self-contained enough that I could do on my own system without needing any complex onboarding, and they paid me market rate for the time worked. We then came back together and reviewed the work as the next interview.
The ones that happened in the interview. I'm not working for free, I expect a modicum of investment from the hiring business - we're doing it live or not at all.
For context, the take home we're looking to do would happen after we've already interviewed them. We're not thinking of anything huge — something like an hour of work.
We're also doing a paid trial period, though that would be after the interview + take home.
Apologies if I came in too hot - none of this is a judgement against anything. I just see trends in the industry at large and expect a lot of newbies to be lurking. For every however many honest businesses, I get the sense more are unscrupulous
If nothing else, take this as a scar some of those who are interested may be carrying
It is exclusionary. Both ways. When I'm interviewing, the business is also. It's insurance, at a price. Filters should filter.
If I can't make a case live, but must go off with some pet project while imagining their behavior... I don't want that interview/work. Folks will certainly disagree, just here to remind of agency.
The best firms gave options. Am adult, will work under agreeable terms. One of those terms is "together" (time, not space), despite my own social anxiety. The same way the business hopes to select with this process, I will too.
None of this is to judge others - do what you prefer, but please do realize this 'take home' approach can be taken advantage of [again, for either side].
Yup, remote is often the opposite. I don't want to work with someone hovering over my shoulder. The hire should also be okay with nobody watching them for a while.
I get that some people want to be respected for their time, and that has to be built into the process before and after any take home tests are given - compensation, trial buddy, flight & hotel room, detailed feedback after the test.
Honest question - how are you planning to source candidates in the first place? stack-auth is a two person YC startup and you were previously at another YC startup. I would expect you to tap the YC network for candidate referrals that probably will need little to zero tech evaluation?
I would probably skip a take-home for candidates at all for now. You probably have some specific pain points already. Try to find experienced TypeScript developers with some experience similar to your pain points and recommended to you by people in your network.
If you made me create a take-home assignment, I might make it rather open-ended and say do a compare/contrast of the stack-auth REST API versus one of the other options out there using any programming language and be prepared to describe code in both examples and strengths/weaknesses? This will help you maybe identify developers who self-select on "I'm interested in the stack-auth problem space . . . " Plus, that feels like a reasonably-sized effort for a candidate to me and could maybe be fun.
If you had a technical component during actual interviews, I think it is totally ok to ask someone to code up a small, simple Typescript example function of limited scope. Frankly, that's probably enough for you to get "yes/no" hiring signal.
I'm, err, older than you and will gently ask that you don't focus entirely on people who graduated since you did.
recently i was given 72 hours for a take home. they required me to keep the “challenge” confidential and assign the rights to the code to them. i’m wondering what you all think of that.
In the sense of would this be a problem that might reasonably come up in the course of working elsewhere ... and how comprised would you be had it done so given you'd seen it before and signed right to your solution elsewhere.
How much are they paying for you to do the work? How does that compare to your hourly-rate, or salary? If for free they're clearly taking advantage, so I'd assess based on payment.
My first full-time web dev job was back in the 2010s, for a mom-and-pop solar business. I applied (on Craigslist, heh) and they asked me to bring a portfolio of past work (which was still common back then). I didn't really have one, but I learned jQuery and Bootstrap over the next few days and redid their website front page as a demo and brought it in with me to the interview. I got the job the next day. That was back when "whoa, you can make an ecommerce site" was still a valuable skill, lol.
Fast forward a decade and several jobs later, my favorite take-home so far is simply a no-nonsense assignment that reflected the actual work I would be doing. As a frontend person, that meant making a dashboard in React or whatever JS flavor I wanted, with whatever tools I wanted, for a co-review a few days later.
They asked me to make something like a movie browser, kinda like Netflix, using a provided list of 100 or so movie titles. It was a pretty simple project that only took a day or so, but showed enough basic React listing/filtering/searching/state/context usage, and some simple API lookups (used it to get cover/poster art and some metadata from IMDB). It wasn't a particularly challenging assignment, but it wasn't meant to be – it was just a normal frontend job at a traditional (non-tech) Fortune 1000 company.
After I submitted the assignment, the interviewer spent an hour or so walking through it with me, asking questions about certain parts and offering constructive feedback on others. It was a really good use of both of our time: them to see how I think through user-facing problems, and me to see how they like to manage work and communication.
I got the job, and did very well in it, and helped solve a few interesting challenges at that company (the dashboard was complex enough that it had interesting UX, IA, error handling, user flows, visualizations, etc.)
---------------
One take-home I did NOT really like was from an online interactive mapping company I really respect. They asked me to make an animated analog stopwatch, similar to the kind you'd hold in your hands at a race. I'd never really done any animation before, and their product didn't seem to have much of it, but that was the assignment.
I made one and it worked, with some caveats. The assignment didn't have much detail to begin with, so I submitted what I had and told them I'd be happy to modify it or rewrite the whole thing to spec, in whatever stack they wanted. A few hours later they told me sorry, but no thanks, and that was that. No further feedback.
It annoyed me because the take-home didn't really seem to reflect any sort of work I'd actually be doing anyway (web dev + mapping), which I actually had a moderate amount of prior experience with. The GIS (geo information systems) world has plenty of its own nuances and interesting challenges, but none of that was tested or discussed at all. To this day I still don't know why they wanted an animation challenge for a web mapping app.
Presumably they cared more about the graphics side of things, or it was just a chance for the right candidate to write something really cool from the ground up to impress. The assignment itself was flexible enough that it could've been anything from "here's a npm lib that does this with a SVG" to "I wrote an entire 3D renderer with ray-tracing to better simulate the watch face". I did not impress :)
----------------
Oh, and one more bad/ridiculous experience: One company's first "take-home" wasn't even a coding challenge, but some sort of convoluted personality test. They made you sign up for a third party platform to take a specific personality test that analyzed your personal and professional habits, etc.: https://www.assessmentday.com/ccat-criteria.htm
I laughed in their face and walked away. Guess that shows my personality, eh? Bullet dodged.
Some of the best take-home coding tasks typically resemble real-world work scenarios, allowing candidates to use any tools available, including debuggers and AI. Tasks that simulate actual job responsibilities are generally preferred, as they provide a clearer picture of the candidate's abilities. It's common for these challenges to take several hours or even days, making them more comprehensive than standard interview questions.
If they "resemble" work, shouldn't you pay applicants?
If not, are you going to miss a lot of people who have a no-take-home-problem policy, or are skeptical of take-homes that "resemble" work, really being someone else's work?
I think the account you're replying to is a spambot, FWIW.
But you do bring up an interesting point.
> If they "resemble" work, shouldn't you pay applicants? If not, are you going to miss a lot of people who have a no-take-home-problem policy, or are skeptical of take-homes that "resemble" work, really being someone else's work?
I think it's kind of a two-way street and self-selecting group, isn't it? People who don't like that sort of thing filter themselves out, and companies who want mid-tier candidates have an easy way to filter out both people with zero skill and people who are experienced enough they don't have to play games.
If you're the kind who can get work with a no take-homes policy, you probably don't need to be applying for jobs out of the blue like this anyway, rather than just working your connections.
Otherwise, as a candidate, I vastly prefer a reasonable take-home over the usual inane interview hoops (talk to four people in HR, a few managers from different departments, the dept head who you'll never work with, etc.). And arguably it SHOULD be like the real work, both for the company's and the candidate's sake, to see if the day-to-day would be a good match.
And if the company happens to use some of my code without hiring me? Okay, well, so what? I spent a few hours of my time, made some random corner of the internet a bit better. I got to practice a little, they get a bit of free work and I get a little cheated, ok, but life goes on? More productive than playing video games, at least, lol.
Those are good points, I admire your attitude. The flip side here is how jealously companies protect their "Intellectual Property". Why should we give away the sweat of our brows for free, of some corporations are going to claim it as their own, and keep anyone else from using it. Seems unfair. Anyway, thanks!
- Recently: Given some basic intermediate output from LLM system, build out evaluation/quality tools, especially for dealing with hallucination - ~8 years ago: Given a database table of restaurants, identify near-duplicates to merge
On the hiring manager side of things, my candidates generally enjoyed doing a take-home to build a ML model for the Li & Roth question classification dataset. That involved a bit more creativity and it was very role-related.
After being on both sides of the interview process, my current feeling is that take-homes are a decent fit for junior roles but take-homes are not a good use of time (on either side) for senior roles.
reply