Is it possible to allow a user to edit their address book but not be able to edit any other part of their account?

In order for a user to be able to enter billing and shipping addresses, they need to be able to edit their address book.  The ability to edit the address book is controlled by the site group permission, "Can edit their own personal account info."  If that permission is off, then the user will not be able to edit any of their account information - including their address book.  If it is on, then the user will be able to edit their address book, but they will also be able to edit other aspects of their account - such as their password.  There isn't a way to isolate the address book from the rest of the user's account information.