It was taking the angle of the rotating sector object instead of the angle from the listener's position to it.