Hey John - If you have some fine grit sand paper, 1000 or better, roll some into a cylinder shape and lightly sand the inside of the brass bushings. Use oil as a lubricant. You can also lightly sand the outside of the sleeve as well, but use nothing less than 1000 grit, and always use oil while doing it.
And no, you shouldn't replace the delrin with metal shims, you'll have metal to metal contact in a location that's not designed for that.