Let's say $E$ is the maximum number of Pokemons we can evolve and $S$ is the number of Pokemons we sell, then we have:
(A): $E = \left\lfloor\frac{M+YS}{X}\right\rfloor \le \frac{M+YS}{X}$
(B): $S+E \le N$
(B) gives
$S \le N-E$
$\frac{M+YS}{X} \le \frac{M+Y(N-E)}{X}$
So replacing the first part with $E$ thanks to (A) we obtain
$E \le \frac{M+Y(N-E)}{X}$
$E + \frac{Y(E-N)}{X} \le \frac{M}{X}$
$E\left(\frac{Y}{X}+1\right) - \frac{YN}{X} \le \frac{M}{X}$
$E \le \frac{\frac{M}{X} + \frac{YN}{X}}{\frac{Y}{X}+1}$
$E \le \frac{M+YN}{Y+X}$
Which means, for us to get the highest possible value for E, that:
$E = \left\lfloor\frac{M+YN}{Y+X}\right\rfloor$
But with the limitation, as @BlueHairedMeerkat pointed out in the comments, that E cannot exceed N, such as the actual, more exact answer would be
$E = \text{Min}\left(\left\lfloor\frac{M+YN}{Y+X}\right\rfloor, N\right)$
As an example:
Let's say we have 6 Pokemons (N) and 11 Candy bars (M). We can evolve a Pokemon for 4 candies (X) and sell one for 3 candies (Y). Trivially, we find that selling 2 Pokemons (for 6 candies) gives us a total of 17 candies in order to evolve the 4 Pokemons we have left (with one candy left). What says the formula?
$E = \left\lfloor\frac{11+3*6}{3+4}\right\rfloor = \left\lfloor\frac{29}{7}\right\rfloor = \lfloor4.142...\rfloor = 4$
So it seems quite right (at least with this example).