In general, this is just a constraint satisfaction problem and there are usually several ways of solving such problems without using exponential amounts of time/space. Even backtracking works most of the time if you're careful to propagate constraints correctly.
For your tournament in particular:
I encoded your problem as a propositional boolean formula and solved it with a SAT solver. The code I wrote to encode the problem is here. I can solve a 5-round variant to get:
Round 1: (1, 2, 3) vs (4, 5, 6), (7, 8, 9) vs (10, 11, 12), (13, 14, 15) vs (16, 17, 18), (19, 20, 21) vs (22, 23, 24)
Round 2: (1, 4, 13) vs (8, 20, 22), (2, 5, 24) vs (7, 12, 18), (3, 17, 19) vs (9, 16, 21), (6, 10, 14) vs (11, 15, 23)
Round 3: (1, 5, 21) vs (11, 13, 16), (2, 18, 22) vs (9, 17, 23), (3, 12, 20) vs (10, 15, 19), (4, 8, 14) vs (6, 7, 24)
Round 4: (1, 6, 23) vs (12, 17, 24), (2, 7, 14) vs (3, 13, 21), (4, 15, 20) vs (5, 9, 11), (8, 10, 18) vs (16, 19, 22)
Round 5: (1, 10, 20) vs (14, 18, 21), (2, 15, 17) vs (8, 19, 24), (3, 4, 11) vs (12, 13, 23), (5, 7, 16) vs (6, 9, 22)
I could not find solutions for 7 rounds or even 6 within a few days. I played around with some symmetry-breaking constraints to speed up the search a little and the most effective was:
1. The first round is forced to be (1,2,3) vs. (4,5,6), (7,8,9) vs (10,11,12), etc.
2. The first teams are lexicographically ordered throughout rounds, e.g., team 1 is (1,2,3) in round 1, (1,4,22) in round 2, (1,6,8) in round 3, etc.
3. The two teams within each match are lexicographically ordered, e.g., the first match of round 1 is (1,2,3) vs (4,5,6) instead of the other way around
4. All matches within a round are lexicographically ordered by their constituent players, e.g., round 1 is (1,4,22) vs. (7,11,23), (2,5,13) vs. (12,14,21), ... because the tuples of all players within a match are ordered (1,4,7,11,22,23) < (2,5,12,13,14,21)
I could solve a variant of your original puzzle which might be interesting to you since even the 6-round case doesn't seem easily solvable. If you relax your second restriction and allow players to face off at most twice, you can get a 7-round tournament:
Round 1: (1, 2, 3) vs (4, 5, 6), (7, 8, 9) vs (10, 11, 12), (13, 14, 15) vs (16, 17, 18), (19, 20, 21) vs (22, 23, 24)
Round 2: (1, 4, 22) vs (7, 11, 23), (2, 5, 13) vs (12, 14, 21), (3, 16, 19) vs (9, 15, 17), (6, 20, 24) vs (8, 10, 18)
Round 3: (1, 6, 8) vs (13, 18, 23), (2, 15, 20) vs (4, 14, 24), (3, 7, 10) vs (5, 12, 19), (9, 16, 21) vs (11, 17, 22)
Round 4: (1, 7, 15) vs (2, 21, 24), (3, 8, 23) vs (10, 13, 16), (4, 11, 20) vs (6, 17, 19), (5, 14, 18) vs (9, 12, 22)
Round 5: (1, 11, 13) vs (6, 12, 15), (2, 9, 19) vs (10, 21, 22), (3, 4, 17) vs (7, 18, 24), (5, 8, 20) vs (14, 16, 23)
Round 6: (1, 16, 24) vs (18, 19, 22), (2, 4, 7) vs (9, 10, 20), (3, 5, 15) vs (8, 13, 21), (6, 11, 14) vs (12, 17, 23)
Round 7: (1, 21, 23) vs (10, 19, 24), (2, 16, 22) vs (12, 13, 20), (3, 6, 18) vs (7, 14, 17), (4, 8, 15) vs (5, 9, 11)
In fact, even an 8-round tournament is possible with the relaxed constraint.