This is my second attempt at a solution. It is vastly different from my first attempt, so I posted this as a separate answer. It achieves the smallest maximum result for a hand possible (with a hand whose result is $0$):
$\binom{52}{5} - 1 = 2598959$
Many thanks to @2012rcampion for some improvements to this solution (see the comments).
The given operators ($+$, $-$, $\times$, $/$) can be used to build a minimal program operating on a bounded number of allowed values. Since the problem is bounded, let's try it out! I'll define a series of functions for clarity. Each function does not use functions defined later, and does not call itself recursively, therefore the functions can be inlined trivially to form a single, huge function.
We will number the cards from $0$ to $51$.
The puzzle specifies that only the four operators $+$, $-$, $\times$ and $/$ are allowed, but does not indicate whether numerical constants are allowed. Assuming that the five inputs are implicitly allowed, we use the following function to extract the constant $1$:
$$\operatorname{ExtractOne}(a,b,c,d,e) = (a + b) / (a + b)$$
In the formulas that follow, any positive constant $n$ can be replaced by (this is the simplest solution, other schemes will give fewer terms):
$$\underbrace{\operatorname{ExtractOne}(a,b,c,d,e)+\cdots+\operatorname{ExtractOne}(a,b,c,d,e)}_{n\text{ times}}$$
We will take care to never perform a division by zero:
Given the fact that at most one of the inputs is zero (there is only one card with the ID $0$), we know that the sum of any two of the five inputs will be non-zero, and therefore can safely be used as the divisor in $\operatorname{ExtractOne}$.
We then write an $\operatorname{IsZero}$ function, which operates on positive integers between $0$ and $52\times 52=2704$. It returns $1$ when its input is $0$, and it returns $0$ otherwise:
$$\operatorname{IsZero}(x) = \frac{(x-1) \times (x-2) \times (x-3) \times (x-4) \times \cdots \times (x-2703) \times (x-2704)}{2704!}$$
The $\operatorname{IsZero}$ operates following this principle:
When the input is $0$, the formula becomes $\frac{(0-1) \times (0-2) \times (0-3) \times (0-4) \times \cdots \times (0-2703) \times (0-2704)}{2704!}$, which is the same as $\frac{(-1) \times (-2) \times (-3) \times (-4) \times \cdots \times (-2703) \times (-2704)}{2704!}$. Since there is an even number of negative numbers, the result of their product is positive and is equal to: $\frac{1 \times 2 \times 3 \times 4 \times \cdots \times 2703 \times 2704}{2704!} = \frac{2704!}{2704!} = 1$. When the input is non-zero, i.e. $x ∈ \{1,\ldots,2704\}$, the formula instead becomes $\frac{(x-1) \times \cdots \times (x-(x-1)) \times (x-x) \times (x-(x+1)) \times \cdots \times (x-2704)}{2704!}$. The $x-x$ results in a $0$ which nullifies the whole formula: $\frac{(x-1) \times \cdots \times (x-(x-1)) \times 0 \times (x-(x+1)) \times \cdots \times (x-2704)}{2704!} = \frac{0}{2704!}=0$
Since we only call $IsZero$ on integers which are squares in the set $\{0, 1, 4, 9 …, 51²\}$, we skip some of the terms (thanks @2012rcampion):
$$\operatorname{IsZero}(x) = \frac{(x-1) \times (x-4) \times (x-9) \times (x-4) \times \cdots \times (x-2500) \times (x-2601)}{1*4*9*⋯*2500*2601}$$
We can now compare numbers by comparing their squares (remember that our $\operatorname{IsZero}$ function only supports integers in the set $\{0,\ldots,52\times 52\}$, the domain of $\operatorname{IntEqual}$ will therefore be $\{0,\ldots,52\} \times \{0,\ldots,52\}$). The $\operatorname{IntEqual}(x,y)$ function returns $1$ when $x=y$ and $0$ otherwise:
$$\operatorname{IntEqual}(x,y) = \operatorname{IsZero}((x - y) \times (x - y))$$
We then define an $\operatorname{IfThenElse}(c,x,y)$ function, which returns $x$ when $c=1$ and $y$ when $c=0$:
$$\operatorname{IfThenElse}(c,x,y) = (c \times x) + ((1 - c) \times y)$$
Since we always use $0$ as the third argument of $\operatorname{IfThenElse}$, we can define a simpler version $\operatorname{IfThen}$ (thanks @2012rcampion):
$$\operatorname{IfThen}(c,x) = c \times x$$
We finally use $\operatorname{IfThenElse}$ to build a decision tree, which assigns a unique number to each combination of input IDs, regardless of their order. The formula is a bit cumbersome to read and write, so what follows covers only hands of $2$ cards among $5$:
$$\begin{array}{c}\;\;\;\operatorname{IfThen}\left(\operatorname{IntEqual}(a, 0),\left(\begin{array}{l}\;\;\;\operatorname{IfThen}(\operatorname{IntEqual}(b,1),0)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,2),1)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,3),2)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,4),3)\end{array}\right)\right)\\+\operatorname{IfThen}\left(\operatorname{IntEqual}(a, 1),\left(\begin{array}{l}\;\;\;\operatorname{IfThen}(\operatorname{IntEqual}(b,0),0)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,2),4)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,3),5)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,4),6)\end{array}\right)\right)\\+\operatorname{IfThen}\left(\operatorname{IntEqual}(a, 2),\left(\begin{array}{l}\;\;\;\operatorname{IfThen}(\operatorname{IntEqual}(b,0),1)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,1),4)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,3),7)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,4),8)\end{array}\right)\right)\\+\operatorname{IfThen}\left(\operatorname{IntEqual}(a, 3),\left(\begin{array}{l}\;\;\;\operatorname{IfThen}(\operatorname{IntEqual}(b,0),2)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,1),5)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,2),7)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,4),9)\end{array}\right)\right)\\+\operatorname{IfThen}\left(\operatorname{IntEqual}(a, 4),\left(\begin{array}{l}\;\;\;\operatorname{IfThen}(\operatorname{IntEqual}(b,0),3)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,1),6)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,2),8)\\+\operatorname{IfThen}(\operatorname{IntEqual}(b,3),9)\end{array}\right)\right)\\\end{array}$$
We use the form
$$\operatorname{IfThen}(c_1,v_1)+\operatorname{IfThen}(c_2,v_2)+⋯+\operatorname{IfThen}(c_n,v_n)$$
instead of
$$\operatorname{IfThenElse}(c_1,v_1,\operatorname{IfThenElse}(c_2,v_2,\ldots))$$
to avoid the rightward drift caused by deeply nested function calls.
With a hand of five cards, the decision tree would be five levels deep instead of two, and with 52 cards to choose from, each level would contain many more cases.
The solution proposed by Carl Löndahl can also be applied using this technique to compute the modulus.