^^^ I like you're explanation. I myself rarely use the AC. I'm thinking, for myself, just remove it. Like the above said, take your car to a shop. That would be your best diagnosis. After you find out the problem or problems according to the place you took your car, ask again here if someone knows of a fix. That's the best I can tell you if it's not a serious problem, you can wait, and you don't want to put out the cash.
Maybe someone else with more experience in these matters here can help or someone that went through the same things. Good luck in the matters at hand that await you.