Thanks, Pablo. Excellent questions!

how is taking a postdoctoral position at FHI seen comparatively with other "standard academia" paths? How could it affect future research career options?

My guess is that for folks who are planning on working on FHI-esque topics in the long term, FHI is a great option. Even if you treat the role as a postdoc, staying for say 2 years, I think you could be well set up to go to continue doing important research at other institutions. Examples of this model include Owain Evans, Jan Leike, and Miles Brundage. Though all of them went on to do research in non-academic orgs, I think their time at FHI would have set them up decently for that.

The other option is planning on staying at FHI for the longer term. For people who end up being a good fit, I think that's often an excellent option. Even though these roles are fixed term for 2 years, we're able to extend contracts beyond that if the person ends up being a good fit, provided we have the funding (which I expect us to do for the next couple of years).

What EU AI Policy we should push for?

My overall answer is that I'm not quite sure, but that it seems important to figure out! More reasoning below.

And what mechanisms do you see the EU AI policy having a positive impact, compared with the US for example?

Earlier this year, the EU Commission published its White Paper on AI, laying out a legislative agenda that's likely to be put into practice over the next couple of years. There's reason to think that this legislation will be very influential globally, due to it plausibly being subject to the Brussels Effect. The Brussels Effect (Wikipedia, recent excellent book) is the EU's legislation having a large effect globally, as it's adopted by companies and/or governments globally. We've seen this effect at work with regards to GDPR, food safety, etc. In brief, the reason this happens is that the EU has a large domestic market that is immovable, which means companies are strongly incentivised to operate on the market. Further, the EU tends to put in place legislation that is much harsher than other jurisdictions. As such, need to comply with the legislation at least in the EU market. In some cases, it won't make sense for companies to provide different products to different markets, and so they adhere to the EU standard globally. Once this occurs, there's pressure for the companies to push for EU-level standards in other markets and legislators in other jurisdictions may be tempted to put in place the EU-level standard. However, there's a number of ways one could imagine that the Brussels Effect won't be at work in the case of AI, which I'd be very keen to see someone start researching.

As such, I believe that the EU is much more likely than the US to determine what legislation tech/AI companies need to adhere to. The US could do the same thing, but it probably won't because the US legislator has a much smaller appetite for legislation than the EU.

What ways do you see for technical people to influence AI governance?

There are lots! I think technical folks can help solve technical problems that will help AI governance, for example following the Cooperative AI agenda, working on verification of AI systems, or otherwise solving bits of the technical AI safety problem (which would make the governance problem easier). They can also be helpful by working more directly on AI governance/policy issues. A technical background will be very useful as a credential (policy makers will take you more seriously) and it seems likely to improve your ability to think through the problems. I also think technical folks are uniquely positioned to help steer the AI researcher community in positive directions, shaping norms in favour of safe AI development and deployment.

Could you say more about which fields / career paths you have in mind?

No particular fields or career paths in particular. But there are some strong reasons for reaching out to people who already have or are on a good track to having impact in a field/career path we care about. These people will need a lot less training to be able to contribute and they will already have been selected for being able to contribute to the field.

The issue that people point to is that it seems hard to change people's career plans or research agendas once they are already established in a field. E.g. students are much more moveable. I think this is true, but I also think we haven't worked hard enough to find ways of doing this. For example, we currently have a problem where someone with over a decade of experience might not be comfortable with EA groups, because they tend to skew very young.

Things we could try more:

• Find senior researchers working on topics adjacent to those that seem important and who seem interested in the questions we're asking. Then get them involved by connecting them with an interesting research agenda, excellent collaborators, and potentially funding. This is something that a lot of the academic institutions in the longtermist space (which I know better than other EA causes) are trying out. I'm excited to see how it goes.
• Introduce more specialist community building. Perhaps this can be done by e.g. hosting more conferences on specific EA-related topics, where researchers already involved in a field can see how their research fits into the topic (this is something e.g. GPI seems to be doing a good job of).

Thanks, Jia!

Could you say more about the different skills and traits relevant to research project management?

Understanding the research: Probably the most important factor is that you're able to understand the research. This entails knowing how it connects to adjacent questions / fields, having well thought-out models about the importance of the research. Ideally, the research manager is someone who could contribute, at least to some extent, to the research they're helping manage. This often requires a decent amount of context on the research, often having spent a significant amount of time reading the relevant research and talking to the relevant people.

Common sense & wide expertise: One way in which you can help as a research manager is often to suggest how the research relates to work by others, and so having decently wide intellectual interests is useful. You also want to have a decent amount of common sense to help make decisions about things like where something should be published and what ways a research project could go wrong.

Relevant epistemic virtues: Just like a researcher, it seems important to have incorporated epistemic virtues like calibration, humility, and other truth-seeking behaviours. As a research manager, you might be the main person that needs to communicate these virtues to new potential researchers.

People skills: Seems very important. Being able to do things like helping people become better researchers by getting to know what motivates them, what tends to block them, etc. Also being able to deal with potential conflicts and sensitive situations that can arise in research collaborations.

Inclination: I think there's a certain kind of inclination that's helpful to do research management. You're excited about dabbling in a lot of different questions, more so than really digging your head down and figuring out one question in depth. You're perhaps better at providing ideas, structure, conceptual framing, feedback, than doing the nitty-gritty of producing all the research yourself. You also probably need to be fine with being more of a background figure, and let the researchers shine.

Thanks Misha!

Not sure I've developed any deep insights yet, but here are some things I find myself telling researchers (and myself) fairly often:

• Consider a wide range of research ideas. It's easy to get stuck in a local optimum. We often have people write out at least 5 research ideas and rate them on criteria like "fit, importance, excitement, tractability", e.g. when they join as a GovAI Fellow. You should also have a list of research ideas that you periodically look through.
• Think about what output you're aiming at from the start. It will determine the style, what literature you need to read, length, etc. Reworking a piece from one style to another can often take up to 20 hours.
• Make outlines often and early. This will help you be clear about what your argument is. Also, start writing earlier than you feel comfortable with. Often, things feel much clearer in your head than they actually are.
• If you have a high barrier to start writing, find ways to lower it. You can write outlines, dictate, say it out loud to a friend, set a timer during which you're just allowed to write, no editing.
• Discuss your ideas with others often. I find that a lot of good thinking, in particular on how to frame something, comes from discussion. Just the act of putting your thoughts into words helps.
• Clear writing is hugely valuable. A lot of research gets less attention than it deserves because it takes too much effort to parse. Trying to write clearly will also often highlight complexities or problems with your argument that can be hidden in unclear prose.
• Don't reinvent the wheel. There's been a lot of smart people trying to figure out things about the world before you. Make a proper effort to try to find their writings and learn from them.
• Quality over quantity. The impact of research is probably heavy-tailed, and we have okay means to tell the quality of a piece ex ante.
• If you're just starting out, find ways to work closely with people you can learn from. To do this, it's much easier to support some senior person's current agenda, rather than convincing them to help you on your research ideas. That way, they will very clearly get value out of working with you, and you'll get an inside view into their research intuitions. For example, I think working as a research assistant to a great researcher is one of the best ways you can learn the ropes.
• As a facilitator of research, there's also a bunch of common sense things you can do: Set deadlines for people, organise talks, connect people with the right people.
• There's a lot I could say about feedback, but I'll just say some things on how to ensure you actually get the feedback you need. Often this is difficult as the people whose feedback is most valuable to you have a lot of things pulling at their time.
• Make it easy. Give people instructions about what parts they should focus on, what questions you want them to think about etc. You can also pull out specific questions where you think they have expertise and just send those along.
• Be accommodating. Some people find it much less costly to give feedback after having listened to you give a talk or prefer giving feedback over a call. Others prefer doing everything via text.

Thanks for the question, Lukas.

I think you're right. My view is probably stronger than this. I'll focus on some reasons in favour of specialisation.

I think your ability to carry out a role keeps increasing for several years, but the rate on improvement presumably goes tapers off with time. However, the relationship between skill in a role and your impact is less clear. It seems plausible that there could be threshold effects and the like, such that even though your skill doesn't keep increasing at the same rate, the impact you have in the role could keep increasing at the same or an even higher rate. This seems for example to be the case with research. It's much better to produce the very best piece on one topic than to produce 5 mediocre pieces on different topics. You could imagine that the same thing happens with organisations.

One important consideration - especially early in your career - is how staying in one role for a long time affects your career capital. The fewer competitive organisations there are in the space where you're aiming to build career capital and the narrower the career capital you want to build (e.g. because you are aiming to work on a particular cause or in a particular type of role), the less frequently changing roles makes sense.

There's also the consideration of what happens when we coordinate. In the ideal scenario, more coordination in terms of careers should mean people try to build more narrow career capital, which means that they'd hop around less between different roles. I liked this post by Denise Melchin from a while back on this topic.

It's also plausible that you get a lot of the gains from specialisation not from staying in the same role, but primarily in staying in the same field or in the same organisation. And so, you can have your growth and still get the gains from specialisation by staying in the same org or field but growing your responsibilities (this can also be within one and the same role).