Lavery knows he is considered an annoyance, and at times even crazy, by people who "don't understand who I am or don't understand my situation."
He is undaunted by the criticism, or his latest legal setback.
Lavery estimates he has pocketed a few thousand dollars on his best days at Heisler Park, relying partly on such sales to help him pay his bills.
"There's a lot of artists out there who want me to win this because they're struggling to make a living too," he said.
So on Friday he stood behind his easel at Heisler Park, an instant magnet to tourists and other passersby as he painted white clouds into his latest rendering of Laguna.
"Wow! You've done it justice," said one onlooker.
"Very nice. That's incredible," another fan said.
This painting was not for sale. He had already found a buyer. Even if it were, he explained to the crowd, he could be arrested for trying to sell it.
"I don't get it," said Judy Cummings, visiting from Santa Cruz. "This adds local color. This is culture. This is like going to a museum right along the coastline."