THE INVISIBLE COOK, A BOOK OF RECIPES: CAREFREE/FAIL-SAFE/ELEGANT by Ruth Mellinkoff (Pangloss, P.O. Box 4071, Malibu 90265: $20; 261 pp). Cookbooks are not often the subject of litigation. This one has been. Ruth Mellinkoff, author of a string of successful cookbooks as well as several respected works in medieval art history, licensed "Family Circle" magazine to publish one article and about 12 recipes from this, her latest collection. On Dec. 11, 1984, the magazine published two articles and 21 recipes. Mellinkoff sued. "Family Circle" settled out of court on June 20, 1985, for $10,000. The Pangloss Press news release on the settlement quotes the author: "The recipes are delicious. They ate too much." It would appear that this is a cookbook worth fighting over. In the preface, the author writes: "As my scholarly work has become more engrossing and demanding, my desire to discover new ways, or uncover forgotten old ones, of producing superb cuisine on my own terms has increased. On my own terms has meant: cooking as far ahead as possible; cooking when the time suits me best; storing the treasures safely; and making the delicious creations appear effortless. Working toward those goals, I have gradually become an almost invisible cook." The appetite of today's well-traveled scholar for European cuisine tends far to outrun his or her time or ability to produce it. Eventually, Mellinkoff's latest may prove as great a service to learning as it already is to gustatory conviviality.