An Orange County judge dismissed charges against three men accused of transporting 5 1/2 tons of marijuana from South America to California, ruling that Newport Beach police discovered the drugs through an illegal search of the men's boat. Superior Court Judge Francisco P. Briseno ruled that the fact that the man tending the 45-foot motor-driven sailboat Sea Dolphin in 1985 could not show them proper registration was insufficient grounds for boarding and searching the craft, and evidence found as a result of the boarding could not be used in a trial. Prosecutors told the judge they would appeal. It was the second dismissal of the case--on the same grounds--in a little more than a year. Charged were Richard W. Nelson, 27; Victor P. Lucini, 37, and David P. Choy, 35, all of Santa Cruz.