To return a book to a library, use the verb 還 (huán; this character is also pronounced hái, for which it has an entirely different meaning):
I'd like to return this book. 我要還這本書. (Wǒ yào huán zhèi běn shū.)
To return a purchase (including a book!) to a store, use the verb 退還 (tuìhuán):
I'd like to return this book. 我要退還這本書. (Wǒ yào tuìhuán zhèi běn shū.)