I can't recall from the last time that I had the wheel off but the manual shows the flange in towards the wheel. I suspect that if placed the other way there would be no danger but the seal would be exposed and grit might contaminate the bearing?
I've taken my wheel out tonight, due to a nail finding its way into my tyre :x
I don't think the wheel's been out since leaving the factory (except when I took it out just to see how difficult it was, and I would 100% definitely have reassembled it the way it came out) and the flanged side of the spacer faced inwards, towards the wheel.