Simplify the integer test machinery

[natecook1000]

It turns out casting is casting - up and down need all the same checks.

Checklist

• I've added at least one test that validates that my change is working, if appropriate
• I've followed the code style of the rest of the project
• I've read the Contribution Guidelines
• I've updated the documentation if necessary