I leave 'em on. Gives them that much more protection.

If you do decide to take them off I wouldn't worry about flavors. That is unless you are storing flavored cigars with non-flavored cigars, but even then the cello won't give you any protection from the flavors crossing.