I mean transparency in the sense of how the answers are assessed/evaluated. This basically gives candidates a little bit more guidance and structure.

An analogy that I like to use is rather silly, but it works: I might ask a candidate to describe to me how physically fit he are, and he tells me about how many weights he can lift and how fast you can run. But it turns out that I’m actually interested in flexibility and endurance rather than power and speed, and I’ll reject this candidate since he didn’t demonstrate flexibility or endurance. So it is true that he described physical fitness and that I’m assessing based on your physical fitness, but it’s also true that the information offered and what I wanted to assess were very different.

I'm looking forward to reading a post with reflections on lessons learned. :)

We had a few initial questions that we estimated would take ~20-60 minutes, and in retrospect I now imagine many candidates would have spent much longer than this (I know I would have).

Michael, I'm wondering if more transparency would have helped here? As a simplistic example, there is a big difference between these two questions:

Tell us about a time when you took initiative in a work context.

and

Tell us about a time when you took initiative in a work context. We are specifically looking for candidates that have done this in relation to people management, can describe the process and the results/impact, and can demonstrate taking initiative by doing something fairly innovative.

As I have spent more time interacting with job application processes,^[1] I lean more and more toward the opinion that broad/vague questions (such as ‘why are you interested in this job?’ and ‘why would you be a good fit?’) shouldn't be used. I'll ramble a bit about reasons, but I think the TLDR would be "poor applicant experience, and not very predictive of job performance."

On the organizational side, my observations are that there often isn't clear criteria for assessing / evaluating these questions^[2], which means that the unofficial criteria often ends up being "do I like this answer." I'd prefer something ever-so-slightly more rigorous, such as reject unless both A) there aren't grammar/spelling mistakes, and B) the answer demonstrates that this person has at least a basic understanding of what our organization does.^[3]

On the applicant side, there is a lot of uncertainty regarding what a good answer looks like, which makes the application feel like very arbitrary guess what the right answer is game. We might label is as low procedural justice. For a question such as "How did you hear about ORGANIZATION, and what makes you interested in working here?" an honest answer will probably be penalized, and thus I suspect that most applicants who care about getting the job will spend a good deal of effort on impression management, shying away from saying describing how the appeal is a combination of prestige, good salary, company culture, the professional network, and the feeling of making a positive impact.

These broad/vague questions are probably useful for eliminating particularly bad fit applications.^[4] But I do not have confidence in the ability of these question to do any more than to eliminate the bottom ~15% of applications.

1. ^{^}

   Both from the company side of filtering/selecting applications, and from the applicant side of submitting applications.

2. ^{^}

   But I have seen a minority of organizations that actually use a rubric and have clear and job-relevant criteria. Good for you guys!

3. ^{^}

   While also informing the applicant up front that "we don't expect you to write an essay about how you've have a lifelong desire to work in an entry-level research positions. We are just looking to make sure you have at least a surface level understanding of our industry and our mission. We'd like for you to demonstrate that you have some knowledge or experience related to our field/industry."

4. ^{^}

   "Bad fit" is a pretty fuzzy concept, but I'm thinking roughly about people who give answers  that don't demonstrate a modicum of knowledge or experience in the relevant field. If I am applying to Open Philanthropy, these would probably be answers such as "Overall I want to give pursue goodness for people, present and future," or "I can succeed in this role because of my experience as a JOB_TITLE. My organization and attention to detail enabled me to exceed expectations in that role." If I am the hiring manager, I want to see that the applicant has read the job description and is able to demonstrate some familiarity with the area of work.

Yes, there are ways to time submissions, and (from my perspective) they aren't particularly difficult to find or to use. I suspect that any organization not using them doesn't have can't find a timing tool as a reason, and more likely has chose not to devote the resources to improving this process, or  hasn't thought of it or hasn't bothered with it as a reason.

I would be very happy to read this piece, and I encourage you to write it (not just for my own desire to read it, but because I think that the world will be ever-so-slightly a better place if people in the EA community read and grapple with these ideas).

I don't have any advice, but I wanted to offer my sympathy.

I've been constrained by visa regulations previously, and it felt like such a burden. Not being able to leave a job until I had a new job already lined up, only being allowed to work in certain fields/areas, watching local citizen friends take a X-month sabbatical while I would have to leave the country within 2 weeks of leaving a job... I was pretty worn down by the general feeling of precarity combined with all all the options that were unavailable to me.

I'm so happy to see this guide! Thank you for posting it.

I like the idea of visiting local EA groups when I visit a new city, but how to get in touch is often ambiguous (fill in a form on the website? Reach out to random strangers on LinkedIn? Search for a Facebook group?) or I feel awkward asking a total stranger "I'm visiting, can I hang out with you?" Having this kind of a guide to serve as a 'home base' for visitors is a really nice service, and I think it makes the community feel much more welcoming/friendly.

I'd be interested to read about what you've learnt so far from writing these critiques.