PAYBACKS by Christopher Britton (Donald I. Fine: $16.95). This first novel by an ex-Marine deals with one of the most fascinating aspects of military life, involving one of the most controversial issues of the recent past--the court-martial and the Vietnam War. Yet, what you get is soap opera, and two-dimensional characters. There is the much-decorated drill instructor who has given his all to the Marines. There is the handsome young legal officer who defends the stern, but caring disciplinarian from a charge of murdering a recruit. There is the military prosecutor, nicknamed "Mad Dog," who, to further his career and offer a sacrifice to the anti-war rabble, is intent on railroading the DI into Leavenworth. There is--can any book of this kind be written without him?--the tough, cigar-chewing editor with the heart of gold. And there is the beautiful TV reporter who comes to realize that the Marine Corps is not merely a killing machine and that DIs are human beings too. She finds that even handsome young legal officers are human--very human--and it doesn't take much insight to guess how she finds that out. Yet, despite the cliches and flat characterizations, the book has its moments. There are some good sections on the fear and horror of combat in the jungles of Vietnam and the bitterness of ghetto blacks and poor whites who joined the Marines and wound up there. And it clearly reveals the love-hate feelings that many Marines and ex-Marines have for the Corps. Britton had a good thing going here. Too bad he drowned it in soap suds.