After thinking about it for a while, this may be more common than i thought.
Flying from CMA to L35 @9500' today just north of EMT encountered a balloon of my left maybe 20-40' away same altitude. At 157kts it came up so quick i only had a millisecond to react. It didnt help that it was silver...