Transceiver optimization for non-linear serial interference cancellation and optimal multiuser detection