Integer overflow

Integer overflow is what occurs when a numeric variable exceeds its maximum capacity. Generally what happens, due to two's complement arithmetic, is that the variable wraps to the opposite end of the spectrum. So for a signed variable, a negative value too large to fit would become a large positive value. An unsigned variable that exceeds its capacity would become a small positive value, or zero. Subtracting too much from a small unsigned variable would produce a large value.

A signed 8-bit variable (which is what AC uses) can store -128 to 127 inclusive. If your AC is -126 and you put on a piece of armor that would reduce your AC by five points, then your AC would become -131. -131 is too large for a signed 8-bit variable, so your new AC would be 124. 124 AC is of course extremely bad, considering a naked human has 10 AC. Thankfully, this doesn't happen in the case of AC.

Some integers in NetHack, like AC and alignment, have protection against overflow. One way you can detect whether an overflow would occur is if subtracting from a negative value produces a positive value, or adding to a positive value produces a negative value. In the case of AC, the code calculates your current AC in a larger variable (varies, but most likely 32 bits), and then checks if the smaller AC variable (8 bits) would overflow.

However, most variables do not have overflow protection. For example, wielding the +100 Grayswandir would overflow parts of the to-hit calculation and you'd almost always miss.