A *Problem* for you

• How many possible ways?

The full answer is 11 letters long.

Hint1

You have to solve the top rebus first. After you've solved it, you'll hopefully realize, with (or without) the help of the knowledge-tag, what the problem is.

Hint2(Rebus)

What animal is it?

• I solved two-thirds of the top rebus, but unless this is meant to be the answer, I'm stuck on how they connect. Jan 20, 2022 at 16:07
• @Randal'Thor Ditto - not seeing how to pull together the separate parts yet. The red/purple part is this which altogether yields something like rot13(AQVATBBAB. Hfvat gur cneg bs gur snpr juvpu vg bpphcvrf, jr znl trg fbzrguvat yvxr ABFRAQVATBBAB, jurer gurer ner fbzr pbzcyrgr jbeqf ba qvfcynl, ohg abg frrvat nal fvtavsvpnapr whfg lrg...)
– Stiv
Jan 20, 2022 at 16:16
• @Randal'Thor Wait - rot13(AB.F RAQVAT BA 00) - now THAT looks promising...
– Stiv
Jan 20, 2022 at 16:18
• Guys, you're very close.. rot13(Gur vagragvba jnf abg ernyyl avgevgr ohg irel pybfr gb. Gurve purzvpny sbezhyn vf nyzbfg gur fnzr. Naq vs lbh jrer gb cynpr gur sbezhyn va n yvar, jung qb lbh trg jvgu n yvggyr ovg bs zbqvsvpngvba?) Jan 20, 2022 at 16:29
• @Stiv rot13(AQVATB ??? = ?????) Jan 20, 2022 at 16:41

Esther Klein, the mathematician who came up with the problem referenced in this puzzle.

Part 1, rebus

Inside the face we have the atomic number of Nitrogen, a Dingo and a picture of Nitrogen dioxide, i.e., NO2. These give
N + DINGO + NOO = N + DINGO + (No O) = EN + DINGO - O = ENDING

All of this is inside a happy face, so in total it gives
HAPPY ENDING

Part 2, the problem

The happy ending problem is a mathematical problem that says that for any positive integer N, there exists a large enough number M, s.t., if we place M points on a plane such that no three points are collinear, then there must be a subset of N points which form the vertices of a convex N-gon.

In particular, it was first proven that any five points must contain a convex quadrilateral. In that case, there are three different cases to consider pictured here

In the first case, all five points form a convex 5-gon, in which case any four points will form a convex quadrilateral so there are five ways to form the quadrilateral.

In the second case the convex hull is a quadrilateral with one point inside it. In this case either the corners of the convex hull can form a quadrilateral or we can draw a diagonal and leave out the corner point which ends up on the same side as the point in the middle. Thus, in this case, there are three ways to form the quadrilateral.

In the third case the convex hull is a triangle. In this case there is only one way to form the quadrilateral: with the two points inside and with the two points that would end up on the same side of the line extended from the two interior points.

The grids in the puzzle are all of the first or second type. The quadrilaterals can be formed in

5, 3, 5, 3, 5, 3

ways.

Part 3, extraction

The rebus here says
DIG + IT = DIGIT
indicating that we should take digits from the previous step and map them to the corresponding letter grids. OP commented that there is a small mistake in the puzzle so that the 5s of previous step are actually in position 4. Note also that the letter grids start at zero. Taking the correct letters gives

E KLEIN,

i.e., Esther Klein, who introduced this problem to Paul Erdős and George Szekeres.

• Well done! You got everything right and your idea of how to solve it is correct too. I made some small mistakes, so no wonder you couldn't solve it. rot13(rnpu qvtvg-tevq pbeerfcbaqvat gb n pbairk cragntba (jung lbh ersrerq gb nf pnfr 1) ner bss ol 1. Gur pbeerpg yrggre, ner va nyy guerr bs gurz, cynprq va cbfvgvba 4 vafgrnq bs cbfvgvba 5.) I can't make edits, since I've only been using my phone lately (no access to laptop) but hopefully it's clear what I mean. Aug 3, 2022 at 6:42
• So in grid 1 for example, the "E" should be in the position where the second "O" is... etc Aug 3, 2022 at 6:45
• @Prim3numbah Ah, makes sense! Edited the answer! Aug 3, 2022 at 7:11
• Looks good now. Nice job! Aug 3, 2022 at 8:33