The ability of clustering the points into four planes comes from the gradient information between pixels. So the input of PlaneClustering has to be a soft mask as shown in the paper. Did you train this model by binary mask instead of pyramid mask?