Also you can follow the below things:
Just make sure your Web.Config file is up with Ajax scripts. Also, make sure - Scriptmanagers partial rendering is enabled
Disable eventvalidation (bad idea, because you lose a little of security that come with very little cost).
Use ASP.NET Ajax UpdatePanel. (Put the listbox in the Updatepanel and trigger a update, if you add or remove listbox. This way viewstate and related fields get updates and eventvalidation will pass.)
Forget client-side and use the classic postback and add or remove the listitems server-side.