HIGHER MATH by Jennifer Ball ( Faber and Faber: $19.95; 256 pp.) . When is a novel not a novel? When it's a strange hybrid--is there a literary category called existential stand-up?--supposedly written by the now comatose Marissa (Moose) Minnion and edited by a quintet of scholars, including one Jennifer M. V. Ball. In fact, "Higher Math" seems like nothing so much as Ball's extended attempt to slither out of her old skin, as symbolized by Moose, and get on with the business of fiction. The good news is that her brave moments produce passages of rather startling clarity--but to get there the reader has to slog through miles of one-liners, which Ball heaps, one atop another, as though she did not yet trust her new voice and wanted to make sure everyone was properly entertained while she was getting her act together. But Moose's search for the meaning of life is sufficiently endearing to make the wiseacre act a frustration rather than an enhancement. Nobody's saying Ball should cut the comedy, but she could certainly take it down a notch; take off the literary equivalent of platform shoes and a spangled vest and relax a little.