There is a famous investment firm where a partner was known for having candidates walk into the room without a greeting, sit across from him at a huge desk, and start right away by asking:
"We accept 20% of the people who apply. I'm 90% good at my job [of vetting]. What percentage of people who work here shouldn't have been accepted?"
I believe you can answer this on multiple levels. How do you go about tackling this brain teaser?
Most people here have zeroed in on the underdefined statement of "90% good". I would try to be a good Bayesian here and start with a few of my personal observations about the real world as priors.
1a) people who have formal training in assessing the quality of prediction models actively avoid using unspecific terms like "90% good".
1b) most managers, especially those bearing a title like "partner", have no formal training in metrics for prediction quality.
From these two, we can conclude that
he must be a layperson who doesn't know the basics of prediction quality metrics.
Which we can combine with another set of priors:
2a) most laypeople faced with the results of a prediction algorithm intuitively come up with the metric of precision and think it is a perfect metric for prediction quality.
2b) a hiring manager has no data available for calculating true and false negatives, since he typically loses connection to rejected job candidates and never learns whether they would have been good hires. On the other hand, the information of which hire turned out to be a bad call (false positive) is highly salient for him, for business reasons. Therefore, if he is able to calculate any metric for his prediction quality, it must be based on true and false positives only.
2c) A question whose intended answer is "you didn't provide enough data for me to give you the answer" is a very poor choice for one-upmanship. He must believe that his problem description is sufficient for an exact answer (which he knows, but hapless candidates might not find).
The above observations point in one direction: the metric he uses must be
precision, defined as: true positive cases divided by predicted-positive cases
This assumption about the metric leads to the simplest possible solution:
The question is, what percentage of people who got hired shouldn't been. This is the false positive rate of the hirer, which is 10%.
The reason why this is likely considered a brain teaser is
The problem is stated in a way inviting the answerer to use both numbers provided. But the question asks for an answer as a quotient of the people hired, so the first number (how many applied) is not relevant.
A few further thoughts on the underspecified metric
Under a realistic scenario, there are people who reject a job offer - making not just for missing data, but for data which is certainly not missing at random, because it is the best hires who can afford it. Also, it is very rare that absolutely all of the hired people were approved by a single hiring manager. So, if he has observed that 90% of people at the firm are good hires, his calculation that he is "90% good" (has a 90% precision) at hiring is also wrong. He not only picked a bad metric, but miscalculated it.
Morale of the story: Metrology is not for hobbyists.
Also, this guy is a self-important jerk, and I wouldn't want to work for him anyway.
The answer would simply be:
10%
because
if the partner has a 90% success rate at vetting, then 10% of the of candidates shouldn't have been accepted. It's irrelevant whether or not the 20% acceptance rate is applied before or after the vetting process. This also makes the assumption that the partner started before anyone else at the company
example
If 100 candidates were in the pool to begin with:
Accepting 20% leaves 20 candidates, 90% vetting rate means 18 should be hired, 2 shouldn't, 2 / 20 = 10%
I'm going to assume that when the partner says that they are 90% good at vetting they mean they accept an unqualified candidate 10% of the time, and reject a qualified one 10% of the time.
Then we need to find what percentage of candidates are actually qualified. Since we hire 20% of candidates that means that $0.2 = 0.9 \times x + 0.1 \times (1 - x)$. Solving for $x$ we see that 12.5% of candidates are actually qualified.
Now to find the portion of hires who are unqualified we take the portion of unqualified hires divided by total hires: $\frac{0.1 \times 0.875}{(0.9 \times 0.125 )+ (0.1 \times 0.875)} = 0.4375$ or 43.75% of hires are unqualified.
There are a lot of assumption that go into this, but this is probably the intended answer:
From the 20% intended accept rate, we get that
for every good applicant, there are 4 bad ones.
From the 90% success rate (assuming each kind on failed classification is as likely), we get that
90% of the good ones are accepted, and 10% of the bad ones are.
Continuing from these,
for every hired 0.9 good ones there are 0.4 hired bad ones. This means 4 out of every 13 new hires, or just over 30%, are bad hires.
The problem with this result is that
If you hire 90% of the intended 20%, and 10% of the unwanted 80%, you end up actually hiring 18%+8%=26% of the applicants.
This means we should probably also count the interviewer as being one of the people who shouldn't have been accepted. :-)
As DmiHawk said, the correct answer is 10% (making some reasonable assumptions: the ratio of good vs bad applicants in the general population is 50/50, he is equally able to judge good and bad applicants, he is the first to join the company, the number of employees is close to infinity, he only wants to hire good applicants etc. If any of the above assumptions are false, then answer would be: we can't know with the information provided).
The simple way to think about this is to focus on the fundamental question: "What percentage of people who work here shouldn't have been accepted?"
If the manager didn't hire an applicant, then his judgement was correct 90% of the time, but this is irrelevant since the applicant won't be working there.
If the manager did hire an applicant, then his judgement was correct 90% of the time. Making the reasonable assumption that he only wanted to hire good applicants and not bad ones, then 90% of the hires were good and 10% were bad.
One way of reading it is that the statement about 20% represents a hard limit. This means that his task is to hire the best 20% of people that apply.
This is the first way I looked at it, and the only criteria that would be meaningful with this hard limit is that 90% of the people he hired were the right choice, i.e. if there were 100 applicants, he correctly identified 18 of the 20 best candidates resulting in a 90% success rate.
Another way to look at it is that there is a large pool of applicants, some are qualified and some are not, and that they just happen to hire 20% of the people that apply. He is 90% correct for each individual in determining if they are a good hire or not.
I think this means that for each individual person, the right call is made 90% of the time. If the applicant should be hired, there is a 90% chance that they will be. If the applicant should not be hired, there is still a 10% chance that they will be hired.
This means he hires 90% of qualified people (Q) and 10% of unqualified people (U).
Using maths:
0.9Q + 0.1U = 0.2(Q+U)
10(0.9Q + 0.1U) = 10(0.2(Q+U))
9Q + U = 2Q + 2U
7Q = U
So for every qualified candidate, there are 7 unqualified candidates. In other words, out of 8 people, 1 is qualified and 7 are not.
Looking at 80 people, they would be divided into 4 categories:
* Qualified and Hired: 9/10 (90% correct)
* Qualified and Not Hired: 1/10 (10% mistake)
* Not Qualified and Hired: 7/70 (10% mistake)
* Not Qualified and Not Hired: 63/70 (90% correct)
So when the question asks "What percentage of people who work here shouldn't
have been accepted?"
If 16/80 people were hired and 7/16 of them were not qualified, the answer would be 43.75%
25%
The question posits that there are two groups of people:
First, consider the hypothetical case where the interviewer were 100% accurate. This would imply that all P candidates are accepted and all N candidates are not. Since we know the firm accepts 20% of candidates (presumed independent of the interviewer's skill), in a pool of 100 applicants P = 20 and N = 80.
However, we are told the interviewer is only "90% good." If we assume again this refers to accuracy, in the same pool of 100 candidates, we see FP + FN = 10 where FP are the wrongly accepted applicants and FN are the wrongly rejected applicants.
Where TP are the correctly accepted applicants, we have
TP + FP = 20 (all accepted)
P = TP + FN = 20 (should be accepted)
FP + FN = 10 (errors)
==> FP = 20 - TP = 20 - (20 - FN) = 20 - (20 - (10 - FP)) = 10 - FP
==> FP = 5 (25% of the 20 accepted).
The answer is:
There is not enough information to know the answer, but it is between 0-50% of hired candidates that should not have been accepted. To answer we would need to know more about the number of good candidates in the candidate pool (or other similar information such as the vetter's false negative rate).
For example in a group of 100 hundred employees we might have:
An equal performance rate for hiring good employees as rejecting bad which would give 10% bad candidates hired: 18 good/hired, 2 bad/hired, 8 good/not hired, 72 bad/not hired.
Alternatively we might find:
The vetter is highly specific but lets some good ones go, in which case we could have 0% bad candidate hired: 20 good/hired, 0 bad/hired, 10 good/not hired, 70 bad/not hired.
Or:
The vetter could be working with a poor candidate pool and had positions they needed to fill in which case they could have 50% bad candidates hired: 10 good/hired, 10 bad/hired, 0 good/not hired, 80 bad/not hired.
What percentage of people who work here shouldn't have been accepted?
To this, I say,
None, not anymore. 0%.
because
To know he's 90% good at vetting, he has to know how many good/bad decisions he's made, so he probably knows exactly who was a bad hire. If he knows that, then based on the strict 20% hiring rate and his rude lack of introduction, he's probably already fired them and wants me to know it.
