7
$\begingroup$

There is a famous investment firm where a partner was known for having candidates walk into the room without a greeting, sit across from him at a huge desk, and start right away by asking:

"We accept 20% of the people who apply. I'm 90% good at my job [of vetting]. What percentage of people who work here shouldn't have been accepted?"

I believe you can answer this on multiple levels. How do you go about tackling this brain teaser?

$\endgroup$
3
  • $\begingroup$ Is this really a puzzle, or just a vague statement that can be interpreted multiple ways? The phrase "90% good at my job" doesn't really have an established meaning. $\endgroup$ – Helena Dec 11 '20 at 18:54
  • 1
    $\begingroup$ As much as I enjoyed working on this, It's a pure math problem with nothing puzzling about it (although it is surprising) $\endgroup$ – Chris Cudmore Dec 11 '20 at 21:19
  • 1
    $\begingroup$ @ZenBalance [enigmatic-puzzle] is only supposed to be used if the problem statement/puzzle isn't immediately obvious Here it seems to be obvious what to do (find the percentage of wrongly accepted people), and answerers simply disagree on how to do it. $\endgroup$ – bobble Dec 11 '20 at 21:24
15
$\begingroup$

Most people here have zeroed in on the underdefined statement of "90% good". I would try to be a good Bayesian here and start with a few of my personal observations about the real world as priors.

1a) people who have formal training in assessing the quality of prediction models actively avoid using unspecific terms like "90% good".

1b) most managers, especially those bearing a title like "partner", have no formal training in metrics for prediction quality.

From these two, we can conclude that

he must be a layperson who doesn't know the basics of prediction quality metrics.

Which we can combine with another set of priors:

2a) most laypeople faced with the results of a prediction algorithm intuitively come up with the metric of precision and think it is a perfect metric for prediction quality.

2b) a hiring manager has no data available for calculating true and false negatives, since he typically loses connection to rejected job candidates and never learns whether they would have been good hires. On the other hand, the information of which hire turned out to be a bad call (false positive) is highly salient for him, for business reasons. Therefore, if he is able to calculate any metric for his prediction quality, it must be based on true and false positives only.

2c) A question whose intended answer is "you didn't provide enough data for me to give you the answer" is a very poor choice for one-upmanship. He must believe that his problem description is sufficient for an exact answer (which he knows, but hapless candidates might not find).

The above observations point in one direction: the metric he uses must be

precision, defined as: true positive cases divided by predicted-positive cases

This assumption about the metric leads to the simplest possible solution:

The question is, what percentage of people who got hired shouldn't been. This is the false positive rate of the hirer, which is 10%.

The reason why this is likely considered a brain teaser is

The problem is stated in a way inviting the answerer to use both numbers provided. But the question asks for an answer as a quotient of the people hired, so the first number (how many applied) is not relevant.

A few further thoughts on the underspecified metric

Under a realistic scenario, there are people who reject a job offer - making not just for missing data, but for data which is certainly not missing at random, because it is the best hires who can afford it. Also, it is very rare that absolutely all of the hired people were approved by a single hiring manager. So, if he has observed that 90% of people at the firm are good hires, his calculation that he is "90% good" (has a 90% precision) at hiring is also wrong. He not only picked a bad metric, but miscalculated it.

Morale of the story: Metrology is not for hobbyists.

Also, this guy is a self-important jerk, and I wouldn't want to work for him anyway.

$\endgroup$
6
  • 2
    $\begingroup$ I am aware that my solution has the same end result as another solution, posted before my answer. I hope people on a puzzling site will appreciate my way of arriving at it, so I posted it separately. $\endgroup$ – rumtscho Dec 10 '20 at 21:16
  • 2
    $\begingroup$ +1 for "Also, this guy is a self-important jerk, and I wouldn't want to work for him anyway." $\endgroup$ – Polygorial Dec 11 '20 at 9:00
  • $\begingroup$ I'd say it is 5%. One can assume that the 10% includes both error modes. i.e: Half the time he is wrong, he is wrongly dismissing qualified participants. $\endgroup$ – Stian Yttervik Dec 11 '20 at 10:33
  • 3
    $\begingroup$ @Quirk: The assumption in this answer is that 10% isn't a "theoretical" false positive rate; instead, it is the measured false positive rate amongst the accepted candidates. Meaning 10% of the accepted (=positive) candidates turned out to be bad, and 90% of the accepted candidates turned out to be good. If the hirer had been faced with only bad candidates, and they hired at least one of those candidates, then their measured false positive rate would be 100%, not 10%. $\endgroup$ – Stef Dec 11 '20 at 13:20
  • 1
    $\begingroup$ @Stef that describes 10% = FP / (FP + TP) = false discovery rate, not false positive rate = FP / (FP + TN) $\endgroup$ – Mass Dec 11 '20 at 16:33
11
$\begingroup$

The answer would simply be:

10%

because

if the partner has a 90% success rate at vetting, then 10% of the of candidates shouldn't have been accepted. It's irrelevant whether or not the 20% acceptance rate is applied before or after the vetting process. This also makes the assumption that the partner started before anyone else at the company

example

If 100 candidates were in the pool to begin with:
Accepting 20% leaves 20 candidates, 90% vetting rate means 18 should be hired, 2 shouldn't, 2 / 20 = 10%

$\endgroup$
1
  • 3
    $\begingroup$ This assumes the vetter is equally good at hiring good candidates as not hiring bad ones. $\endgroup$ – Barker Dec 10 '20 at 20:03
11
$\begingroup$

I'm going to assume that when the partner says that they are 90% good at vetting they mean they accept an unqualified candidate 10% of the time, and reject a qualified one 10% of the time.

Then we need to find what percentage of candidates are actually qualified. Since we hire 20% of candidates that means that $0.2 = 0.9 \times x + 0.1 \times (1 - x)$. Solving for $x$ we see that 12.5% of candidates are actually qualified.

Now to find the portion of hires who are unqualified we take the portion of unqualified hires divided by total hires: $\frac{0.1 \times 0.875}{(0.9 \times 0.125 )+ (0.1 \times 0.875)} = 0.4375$ or 43.75% of hires are unqualified.

$\endgroup$
1
  • $\begingroup$ Verified answer numerically. 43.75% is correct $\endgroup$ – Chris Cudmore Dec 11 '20 at 21:09
1
$\begingroup$

There are a lot of assumption that go into this, but this is probably the intended answer:

From the 20% intended accept rate, we get that

for every good applicant, there are 4 bad ones.

From the 90% success rate (assuming each kind on failed classification is as likely), we get that

90% of the good ones are accepted, and 10% of the bad ones are.

Continuing from these,

for every hired 0.9 good ones there are 0.4 hired bad ones. This means 4 out of every 13 new hires, or just over 30%, are bad hires.

The problem with this result is that

If you hire 90% of the intended 20%, and 10% of the unwanted 80%, you end up actually hiring 18%+8%=26% of the applicants.

This means we should probably also count the interviewer as being one of the people who shouldn't have been accepted. :-)

$\endgroup$
1
  • 4
    $\begingroup$ I think your first hidden block is mistaken. What the puzzle seems to assert is "for every accepted applicant, there are 4 rejected ones.", which is different from your assumption. $\endgroup$ – Evargalo Dec 11 '20 at 11:19
1
$\begingroup$

As DmiHawk said, the correct answer is 10% (making some reasonable assumptions: the ratio of good vs bad applicants in the general population is 50/50, he is equally able to judge good and bad applicants, he is the first to join the company, the number of employees is close to infinity, he only wants to hire good applicants etc. If any of the above assumptions are false, then answer would be: we can't know with the information provided).

The simple way to think about this is to focus on the fundamental question: "What percentage of people who work here shouldn't have been accepted?"

  1. If the manager didn't hire an applicant, then his judgement was correct 90% of the time, but this is irrelevant since the applicant won't be working there.

  2. If the manager did hire an applicant, then his judgement was correct 90% of the time. Making the reasonable assumption that he only wanted to hire good applicants and not bad ones, then 90% of the hires were good and 10% were bad.

$\endgroup$
2
  • 1
    $\begingroup$ In my opinion "good vs bad applicants in the general population is 50/50" is not a reasonable assumption for 2 reasons: 1) the question says absolutely nothing about there existing "good" and "bad" applicants, only those "should [and shouldn't] have been accepted" and 2) given this is a "famous investment firm" it is unreasonable to assume 50% of applicants are qualified. $\endgroup$ – Mass Dec 11 '20 at 17:24
  • $\begingroup$ If the ratio in the general population were 50/50 (which does not seem like a reasonable assumption at all to me), and the evaluator had 90% true positives and 90% true negatives as you suggest, the company would make offers to 90%*50% + 10%*50% = 50% of all applicants, not 20% of all applicants. So your assumptions contradict the known facts. $\endgroup$ – Nick Matteo Dec 11 '20 at 17:24
1
$\begingroup$

One way of reading it is that the statement about 20% represents a hard limit. This means that his task is to hire the best 20% of people that apply.

This is the first way I looked at it, and the only criteria that would be meaningful with this hard limit is that 90% of the people he hired were the right choice, i.e. if there were 100 applicants, he correctly identified 18 of the 20 best candidates resulting in a 90% success rate.

Another way to look at it is that there is a large pool of applicants, some are qualified and some are not, and that they just happen to hire 20% of the people that apply. He is 90% correct for each individual in determining if they are a good hire or not.

I think this means that for each individual person, the right call is made 90% of the time. If the applicant should be hired, there is a 90% chance that they will be. If the applicant should not be hired, there is still a 10% chance that they will be hired.

This means he hires 90% of qualified people (Q) and 10% of unqualified people (U).

Using maths:

0.9Q + 0.1U = 0.2(Q+U)
10(0.9Q + 0.1U) = 10(0.2(Q+U))
9Q + U = 2Q + 2U
7Q = U

So for every qualified candidate, there are 7 unqualified candidates. In other words, out of 8 people, 1 is qualified and 7 are not.

Looking at 80 people, they would be divided into 4 categories:

* Qualified and Hired: 9/10 (90% correct)
* Qualified and Not Hired: 1/10 (10% mistake)
* Not Qualified and Hired: 7/70 (10% mistake)
* Not Qualified and Not Hired: 63/70 (90% correct)

So when the question asks "What percentage of people who work here shouldn't
have been accepted?"

If 16/80 people were hired and 7/16 of them were not qualified, the answer would be 43.75%

$\endgroup$
1
$\begingroup$

25%

The question posits that there are two groups of people:

  • Those who should be accepted (P)
  • Those who should not be accepted (N)

First, consider the hypothetical case where the interviewer were 100% accurate. This would imply that all P candidates are accepted and all N candidates are not. Since we know the firm accepts 20% of candidates (presumed independent of the interviewer's skill), in a pool of 100 applicants P = 20 and N = 80.

However, we are told the interviewer is only "90% good." If we assume again this refers to accuracy, in the same pool of 100 candidates, we see FP + FN = 10 where FP are the wrongly accepted applicants and FN are the wrongly rejected applicants.

Where TP are the correctly accepted applicants, we have

TP + FP = 20 (all accepted)
P = TP + FN = 20 (should be accepted)
FP + FN = 10 (errors)
==> FP = 20 - TP = 20 - (20 - FN) = 20 - (20 - (10 - FP)) = 10 - FP
==> FP = 5 (25% of the 20 accepted).

$\endgroup$
2
  • 1
    $\begingroup$ Also the question asks of people that were hired, with the logic shouldn't it be 25% (FP = 5 out of 20)? $\endgroup$ – Jason Goemaat Dec 12 '20 at 0:00
  • $\begingroup$ I'm having some trouble following your numbers. I thought I understood that T means the vetting is 'T'rue, i.e. TP means it's a 'T'rue assessment of a 'P'ositive person (they were hired), FP means it's a 'F'alse assessment of someone that should have gotten hired (i.e. they were not hired), and likewise TN means a 'T'rue assessment of a negative person (not hired) and FN means 'F'alse assessment of an N which means they were hired. But then you start with 'TP + FP = 20', doesn't that mean 20 people should have been hired? And TP + FN = 20 means 20 were hired? $\endgroup$ – Jason Goemaat Dec 12 '20 at 0:05
0
$\begingroup$

The answer is:

There is not enough information to know the answer, but it is between 0-50% of hired candidates that should not have been accepted. To answer we would need to know more about the number of good candidates in the candidate pool (or other similar information such as the vetter's false negative rate).

For example in a group of 100 hundred employees we might have:

An equal performance rate for hiring good employees as rejecting bad which would give 10% bad candidates hired: 18 good/hired, 2 bad/hired, 8 good/not hired, 72 bad/not hired.

Alternatively we might find:

The vetter is highly specific but lets some good ones go, in which case we could have 0% bad candidate hired: 20 good/hired, 0 bad/hired, 10 good/not hired, 70 bad/not hired.

Or:

The vetter could be working with a poor candidate pool and had positions they needed to fill in which case they could have 50% bad candidates hired: 10 good/hired, 10 bad/hired, 0 good/not hired, 80 bad/not hired.

$\endgroup$
-1
$\begingroup$

What percentage of people who work here shouldn't have been accepted?

To this, I say,

None, not anymore. 0%.

because

To know he's 90% good at vetting, he has to know how many good/bad decisions he's made, so he probably knows exactly who was a bad hire. If he knows that, then based on the strict 20% hiring rate and his rude lack of introduction, he's probably already fired them and wants me to know it.

$\endgroup$
2
  • 1
    $\begingroup$ Welcome to Puzzling! This doesn't seem to be in the spirit of the problem, and note that this question isn't tagged [lateral-thinking] - so you're supposed to take information given at face value. $\endgroup$ – bobble Dec 11 '20 at 20:06
  • 1
    $\begingroup$ I've interpreted "I believe you can answer this on multiple levels" to mean it can be answered either literally or laterally. $\endgroup$ – Qaz Dec 11 '20 at 20:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.