We got metal wire shelves and kept each shelf fairly close together (about 6-8 inches). We had 10 selves per rack and 18 modems per 3ft x 6ft shelf. 10 racks and related additional hardware (switches, terminals, routers, CSU/DSUs, etc) fit in about 600sq ft of office space with space to spare for technician workbenches. Could do much higher density these days of course, but it was the mid-90s and we thought we were doing good at the time.
We didn't quite get to the size where we needed that before we sold off the dial up business (for a pittance; we contributed to accidentally set off a price war in our market with incumbents with deep pockets - it was no fun, but I learned a lot)
I got lucky in that regard. The larger company in our area decided it was better to buy us out than compete on quality of service, so we were in a position to better negotiate terms of sale.