Summary

Hi, we are the Confido Institute and we believe in a world where decision makers (even outside the EA-rationalist bubble) can make important decisions with less overconfidence and more awareness of the uncertainties involved. We believe that almost all strategic decision-makers (or their advisors) can understand and use forecasting, quantified uncertainty and public forecasting platforms as valuable resources for making better and more informed decisions.

We design tools, workshops and materials to support this mission. This is the first in a series of multiple EA Forum posts. We will tell you more about our mission and our other projects in future articles.

In this post, we are pleased to announce that we have just released the Confido app, a web-based tool for tracking and sharing probabilistic predictions and estimates. You can use it in strategic decision making when you want a probabilistic estimate on a topic from different stakeholders, in meetings to avoid anchoring, to organize forecasting tournaments, or in calibration workshops and lectures. We offer very high data privacy, so it is used also in government setting.  See our demo or request your Confido workspace for free.

The current version of Confido is already used by several organizations, including the Dutch government, several policy think tanks and EA organizations.

Confido is under active development and there is a lot more to come. We’d love to hear your feedback and feature requests. To see news, follow us on Twitter, Facebook or LinkedIn or collaborate with us on Discord. We are also looking for funding.

Why are we building Confido?

We think there is a lot of attention in the EA space toward making better forecasts – investing in big public forecasting platforms, researching better scoring and aggregation methods, skilling up superforecasters and other quantitatively and technically minded people in the EA community, building advanced tools for more complex probabilistic models, etc.

This is clearly important and well done and we do not expect to add much to this effort.

However, we believe some other aspects of forecasting / quantified uncertainty are also valuable and currently neglected.

For example, little effort has gone into making these concepts and tools accessible to people without a math or technical background. This includes, for example, many people from:

• organizations working on animal welfare
• organizations working on non-technical AI safety, strategy and governance
• organizations working on biological risks and pandemic preparedness
• organizations working on improving policymaking and governance in general, think tanks, even government bodies

We think all of these would benefit from clearer ways of communicating uncertain beliefs and estimates, yet existing tools may have a high barrier of entry.

What makes Confido unique?

Confido is a tool for working collaboratively with probabilistic forecasts, estimates, beliefs and quantified uncertainty together with your team. Several features distinguish Confido from existing tools in this space:

• a strong emphasis on being easy to understand and convenient to use
• the ability to use it internally and privately, including self-hosting
• the ability to use it for more than just forecasting questions (more below)
• Confido is free and open source

Ease of use & user experience

Confido’s two main goals thus are to be maximally:

• easy to understand (easy to get started with, requiring minimal background knowledge, guiding the user where needed)
• convenient to use (requiring minimum hassle and extra steps to use it and perform tasks, pleasant to work with)

The second part is also very important because when a tool is cumbersome to use, people won’t use it very much. We have heard many a time from individuals and organizations: “Yes, we want to do more forecasting, but…” they never got around to doing it (for which we use the tongue-in-cheek term fore-crastination).

We are working with people experienced in UX design and education to accomplish these goals. We don’t claim to be quite there yet but we think we are making progress in this direction.

Internal vs public forecasting platforms

Confido is an internal forecasting platform. Each organization or group has its own workspace that is private by default. However, you can easily invite guest collaborators or even share parts of your workspace publicly.

There has been some pushback against using internal forecasting tools and in favour of just asking questions on e.g. Metaculus, where you have access to an existing pool of very good forecasters. However, there are several situations where one might prefer an internal solution:

• very specific and narrow questions or ones requiring arcane knowledge, concepts or terminology
• questions specific to a given organization, its projects and operations
• questions that cannot be made public, e.g. because of infohazards, PR risks or trade secrets
• you want answers from a specific group of people, such as a panel of experts you selected
• you want to ask something that is not a proper forecasting question (e.g. not easily resolvable); see below
• sometimes there is just a larger psychological barrier to asking a question publicly so you want to first try it within the safe space of your team

If you care a lot about data privacy and security, you can even self-host Confido on your own infrastructure.

Beyond forecasting: quantified uncertainty as a tool for better epistemics and communication

Forecasting requires a specific kind of questions — very precise and easy to evaluate after the fact.

However, we believe that the general principle of quantified uncertainty can be also useful for different kinds of questions, outside of formal forecasting.

People form beliefs all the time — while working on a project, researching a question or just as they go through life — even when that is not their main goal or focus or not as deliberate or systematic as when “doing forecasting”.

We think that even for such beliefs, there is value in trying to quantify them, to translate your “inner sense of certainty” into numbers. This both helps you clarify your own thinking and improves communication and cooperation within the team.

For this reason, Confido allows and encourages asking people on your team questions about their beliefs, even if these questions are less precise or it is not possible to evaluate the correct answer (either in practice or even in principle). We call these implicit belief questions.

Our ultimate goal: reaching far and wide

We are now offering Confido to the wider EA and EA-adjacent community with the belief it will be a useful tool and the hope that we will get valuable feedback that allows us to continue to iterate and improve upon it.

However, our main mission is to reach audiences outside the EA/rationalist bubble, who are not yet on the forecasting bandwagon – especially important institutions and decision-makers. We are already making first steps towards this – for example, Confido has been used to organize forecasting tournaments at the Dutch Ministry of Infrastructure and Water Management.

What does Confido look like?

When you receive your workspace, you can create rooms to put questions in. A room is a group of related questions that is shared with a given selection of users. It is a bit like a Slack channel. Here you can see an example of such a room, the “Example room.”

Once your create a room, you can start adding different types of questions:

Then you formulate your question and decide if it is a yes-no or numeric or date question. You can also decide who can see the group prediction:

Then you can add people to the room who can add predictions or estimates:

You and people who you have invited can then start to provide their estimates and comments. They can also upvote each other´s comments:

Once you collect enough responses you can click on the light blue icon and show the result on a projector screen:

How can you use Confido?

For strategic decision-making

Imagine you are tackling a very specific strategic question within your team, such as “Which of these three areas of neglected research should you assign to a newly recruited scientist in order to maximize the positive impact of their work?” or “If our organization sets this amount of time and resources and people, will we be able to meet a specific desired goal?” You create such questions in your Confido workspace and share them with relevant people within or outside your organization to gain the necessary group wisdom to and determine how confident they are about various decisions.

As Confido is private by default, you yourself determine which sets of questions (called rooms in Confido) to share with which people and what permissions to give them (whether they can add predictions, see results or even create their own questions).

You can ask both the true forecasting questions like “If we start a new EA coworking space in Prague, how many EA people will use it in the whole year 2023?”, but also belief questions that cannot be evaluated in principle, but still the answer can be valuable: “Was the name of the Effective Altruism movement a good idea?”

For more productive and inclusive meetings

If you happen to have meetings of seven or more people (whether live or online), it can often feel frustrating and as an inefficient waste of time. A larger group is an environment where it is more time consuming to find out what everyone thinks about a given issue; this risks having the course of the discussion mainly determined by the most extroverted participants rather than by whoever’s points are most relevant at a given time. Generally you need to have a discussion moderator who closely monitors the digressions and tries to get the discussion back on track. Another risk that group discussions face is for participants to get anchored in their opinions/answers by the previous speaker, thus making the group suffer from groupthink.

Again, Confido can help in these situations. Just create well-defined forecasting or belief questions before the meeting, ask at the beginning of the meeting if everyone understands them in the same way, and then just share the link or QR code and wait for the answers.

The advantage is that everyone thinks about the question at the same time and the more assertive/extroverted part of the group doesn’t anchor the rest. Only after the moderator can see the wisdom of the crowd of the whole group, and the level of opinion polarization, they can start a discussion and watch the group update their opinions as the discussion progresses. All updates can be seen in real time by the moderator and it is also possible to see an update history.

Confido also includes a presentation mode that allows the moderator to easily show current group opinion using a projector if that helps the discussion.

For talks and workshops

If you are organizing a lecture, talk or workshop, especially about a highly uncertain and controversial topic such as AI, it may be useful to see where your audience is at. You can simply create a Confido room with a few question before your talk, then share it using a QR code and let participants enter their guesses, opinions and estimates.

You can see them in real time, you can show a graph of the results on a projector with one click if you want. You can also encourage the audience to update their answers during the talk and see them evolve in real time. Or you can just ask one at the beginning and again at the end and compare the results.

Aside from you gaining useful information about your audience, this also allows the participants to feel more engaged with the topic and encourages them to think for themselves.

Confido is also useful when trying to explain forecasting itself.

Rather than (or on top of!) explaining the basic principles and empirical evidence behind forecasting, you can consider letting your audience experience, e.g., the wisdom of crowds effect firsthand.

Just create one or two questions, like how tall you are or when the first man was on the moon, generate a QR code and share it with your audience. Confido will then allow you to show estimates of the entire room in real time and for example see that they are surprisingly close to the true answer.

You can also use Confido to include a small calibration exercise as a part of your workshop.

(You never know who will be affected by such a workshop. In the Czech Republic, for example, the interactivity of Confido has made it popular with one ruling political party, which now wants to train and improve its calibration.)

For forecasting tournaments

The Confido app is good for running forecasting tournaments that can help collectives to find the best calibrated individuals who can be turned to in the time of high uncertainty and crisis or to make forecasting and its principles more widely known in the organization.

If you work for government or think about this kind of teambuilding or educational activity within your organization, you can use the advantage of Confido and ask targeted questions that are closely related to the field of activity in your industry.

Unlike other solutions on the market, Confido is free and you can run tournaments in several different organizations at the same time on one workspace.

The only current drawback is that we have not yet implemented scoring in Confido proper. However, you can export all predictions into a CSV file and score them yourself using e.g. LibreOffice Calc or Google Sheets.

Despite this inconvenience the demo version of Confido 1.0 has been already used for a internal tournament at Dutch Ministry of Infrastructure and Water Management or for a student forecasting tournament at a network of high-schools in Czechia.

What’s on the roadmap?

Confido is under active development, and currently less that a year old. There are many important features still missing.

These are some of the things we are planning (depending on available resources).

General features:

• Non-symmetrical probability distributions

• Displaying calibration curves based on you track record
• Scoring and weighted aggregations.
• More work on anchoring prevention. For example prevent anchoring by answer range by first having to select a range and only then add an estimate within that range:

Features to improve convenience of use and ease of adoption:

• Single sign-on (SSO) – this would rid organizations of the need to create and manage another set of user accounts.
• Integrations with everything. Ask a forecasting question on Slack, paste results into a Google Doc, etc.
• Internationalization and localization (translations, working with various unit systems etc.)

Features to improve ease of use and learning:

• Built-in hints, tips and links to resources (e.g. how to write a forecasting question, how to read a probability distribution plot, etc.)
• Built-in gamified calibration exercises and other training exercises
• Ability to switch between different UI modes (beginner / advanced / expert) to accommodate organizations where people with different backgrounds need to efficiently cooperate.

Who stands behind Confido?

Confido is being developed by Filip Štědronský, Václav Končický, Václav Peca, Jiří Sejkora and Blanka Havlíčková from the Confido Institute. We want to make expressing and understanding uncertainty the new standard. We will write more about our mission, research and project ideas in future posts. To see news and the most recent activities, follow us on Twitter, Facebook, LinkedIn or collaborate with us on Discord. You can also reach us at hello@confido.tools

We will be at EA Global London and EAGx Warsaw and will be happy to meet anyone interested in what we do. You can also meet us online May 31st 8:30 pm CET here.

Funding and connected challenges

The development of Confido was originally funded by the FTX Future Fund. After its crash we had to hit the brakes a little, with our team members juggling day jobs while fueling their passion for Confido's development. Therefore we are developing Confido slower than originally planned. Despite this fact, we still plan to add our part to solving the problem of human cooperation and the epistemic crisis. We welcome any recommendations and pointers towards funding sources that would allow the development of the Confido app to continue full steam ahead again.

Confido is open source and we also welcome any code contributions. If you’d like to help improve Confido, just send us a pull request on GitHub (our drop us an email to see what you can help with).

This article was written by Filip Štedronský and Blanka Havlíčková

We want to express thank you to Peter Hrosso, Nora Amman, Benjamin Tereick and Filip Murar, who helped us improve this post.