All bushings shrink with use - even steel bushings become smaller due to sanding. So all bushings gradually 'wear out', at least relative to their use as dimensional references.
Delrin, HDPE, corian and similar materials can easily be turned to make bushings for the purpose of mounting blanks for turning - just understand that you need to actually measure the diameter of mating parts and turn the blank to the proper measured diameter.
I've used aluminum for specialty pen-making purposes - mandrels for closed-end pens, etc. It can be easily machined on a wood lathe, especially using carbide tools. But as others have noted, it can leave marks on the finished piece.