I am one higher then pent up,
and a curse on your enemies.
The last civilization used me well,
and, out loud, a core of mathematics.

What am I?

You are:

Hex

I am one higher than pent up

The next highest numerical prefix after "pent-" is "hex-".

and a curse on your enemies

"Hex" is a term for a curse.

The last civilization used me well

Civilization V used hexagonal tiles with some success according to critics. (Thanks to Dan Russell on that one.)

and, out loud, a core of mathematics

When spoken, some dialects pronounce "hex" and "x" very similar. $x$ is an incredibly common variable name in algebra.

• I'd have imagined "hex" to be "hexadecimal" for the mathematics one, not necessary "x". – Josh May 3 '16 at 19:03
• Hexadecimal is not particularly core to mathematics, though. If the puzzle were speaking more specifically about computer science, I would agree. Also, if it were talking about hexadecimal, it probably wouldn't include the "out loud" note. – Ian MacDonald May 3 '16 at 19:07
• Good point. Hadn't thought of it that way. – Josh May 3 '16 at 19:19
• Must resist urge to weigh the pros cons of civ5 – Carl May 3 '16 at 19:54

Are you:

Hex

I am one higher than pent up

Pent + 1 = Hex

and a curse on your enemies

A hex on all y'all!

The last civilization used me well

Civ V the game went to hexagonal tiles

and, out loud, a core of mathematics

• I'd give you points also, but @Ian Macdonald has the most complete answer. Good work, though. – Seraviel May 3 '16 at 13:56

hex

I am one higher then pent up,

pent a prefix for five, hex a prefix for six

and a curse on your enemies.

hex is synonymous with curse

The last civilization used me well,

Referring to Greece?

and, out loud, a core of mathematics.