There are many books on math in the Cox Library collection. You can find them mostly in the 510s:
510.1 Philosophy and theory
511 General principles of mathematics
512 Algebra
513 Arithmetic
514 Topology
515 Analysis
516 Geometry
518 Numerical analysis
519 Probabilities and applied mathematics