You must pay-off the card first
Never close a credit card (even a secured credit card) without first paying it off. Unless you pay the card off first, it will be considered as a defaulted debt and it will affect your credit score negatively.
The logic behind that is that they had to use the collateral to satisfy the debt because you failed to pay it.
Pay the credit card off, send them a letter asking to close the credit card and ask for a refund of the deposit. Ask for a letter confirming that the account is closed with a $0 balance, and keep it forever. See close-credit-card.html for more information.