I don't do a lot of downshifting from higher speeds. It definately doesn't like being put into 2nd and 1st in particular until you are almost stopped. Mine is pretty quiet, but it has a lot to do with how the clutch is adjusted. There is a sweet spot you have to find. I also found if I use the heel shifter it seem less noticable and smoother.
To keep it in perspective, I had a guy who owns a Victory take mine for a ride. He thought it shifted nicer than his Victory and he thought the front brakes were awesome. Sometimes I think that we think we are worse off than other bikes, when actually, we are not. I swapped bikes with a Kawasaki rider once for a short ride as I had offered to let him ride my bike. What a horrible ride (I guess I was spoiled). Shakey and gutless. When I went to stop at a turn, it felt like I didn't have any brakes at all...it was scary. The grass always looks greener until you ride another bike, then you realize how good you have in spite of a few shortcomings and tradeoffs.