How one can remedy situation with Imaginative and prescient framework in Swift exhibiting an excessive amount of background

I’m engaged on a characteristic for a Picture Sales space app in Swift to do background substitute utilizing the Imaginative and prescient framework and AVFoundation. Nevertheless, I’m having a difficulty the place the unique background is exhibiting greater than it ought to or perhaps is just not correctly aligned.

Right here is a picture of the difficulty I’m seeing:

Right here is my code for processing the ultimate preview:

func processFrameForFinalImage(
    pixelBuffer: CVPixelBuffer,
    orientation: CGImagePropertyOrientation,
    high quality: VNGeneratePersonSegmentationRequest.QualityLevel,
    showInPreview: Bool,
    completion: @escaping (UIImage?) -> Void
) {
    
    let request = VNGeneratePersonSegmentationRequest()
    request.qualityLevel = high quality
    request.outputPixelFormat = kCVPixelFormatType_OneComponent8

    let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer,
                                        orientation: .up,
                                        choices: [:])

    DispatchQueue.international(qos: .userInitiated).async {
        do {
            attempt handler.carry out([request])
            guard let maskBuffer = request.outcomes?.first?.pixelBuffer else {
                DispatchQueue.fundamental.async { completion(nil) }
                return
            }

            // Convert buffers to CIImage and apply constant orientation
            let cameraImage = CIImage(cvPixelBuffer: pixelBuffer).oriented(orientation)
            var maskImage = CIImage(cvPixelBuffer: maskBuffer).oriented(orientation)

            maskImage = maskImage.applyingFilter("CIGaussianBlur", parameters: [
                kCIInputRadiusKey: 2.0
            ])
       
            if let erode = CIFilter(title: "CIMorphologyMinimum",
                                    parameters: [kCIInputImageKey: maskImage,
                                                 "inputRadius": 1.0]),
               let output = erode.outputImage {
                maskImage = output
            }

            // Enhance distinction to push masks values towards 0 or 1
            if let controls = CIFilter(title: "CIColorControls") {
                controls.setValue(maskImage, forKey: kCIInputImageKey)
                controls.setValue(1.2, forKey: kCIInputContrastKey)
                controls.setValue(0.0, forKey: kCIInputSaturationKey)
                if let output = controls.outputImage {
                    maskImage = output
                }
            }

            // Clamp to [0, 1]
            if let clamp = CIFilter(title: "CIColorClamp") {
                clamp.setValue(maskImage, forKey: kCIInputImageKey)
                clamp.setValue(CIVector(x: 0, y: 0, z: 0, w: 0), forKey: "inputMinComponents")
                clamp.setValue(CIVector(x: 1, y: 1, z: 1, w: 1), forKey: "inputMaxComponents")
                if let output = clamp.outputImage {
                    maskImage = output
                }
            }
      
            var composited = cameraImage

            if self.isBackgroundReplacementEnabled,
               let bgImage = self.backgroundImage {
                
                // Scale background to match digicam picture dimension
                var bg = bgImage.reworked(by: CGAffineTransform(
                    scaleX: cameraImage.extent.width / bgImage.extent.width,
                    y: cameraImage.extent.peak / bgImage.extent.peak
                ))

                // Align background origin
                let dx = cameraImage.extent.origin.x - bg.extent.origin.x
                let dy = cameraImage.extent.origin.y - bg.extent.origin.y
                bg = bg.reworked(by: CGAffineTransform(translationX: dx, y: dy))

                // Mix the individual (digicam) over background utilizing masks
                composited = cameraImage.applyingFilter("CIBlendWithMask", parameters: [
                    kCIInputBackgroundImageKey: bg,
                    kCIInputMaskImageKey: maskImage
                ])
            }

            // Put together for preview output
            let finalImage = self.aspectFillImage(composited, targetSize: self.previewLayer.bounds.dimension)
            guard let cgImage = self.ciContext.createCGImage(finalImage, from: finalImage.extent) else {
                DispatchQueue.fundamental.async { completion(nil) }
                return
            }

            let uiImage = UIImage(cgImage: cgImage)

            DispatchQueue.fundamental.async {
                if showInPreview {
                    self.previewLayer.contents = cgImage
                }
                completion(uiImage)
            }

        } catch {
            print("Segmentation error: (error)")
            DispatchQueue.fundamental.async { completion(nil) }
        }
    }
}

And that is the code to setup the digicam, I’m solely utilizing the entrance digicam:

func setupCamera() {
    session.beginConfiguration()
    session.sessionPreset = .medium
    guard let gadget = AVCaptureDevice.default(.builtInWideAngleCamera, for: .video, place: .entrance),
          let enter = attempt? AVCaptureDeviceInput(gadget: gadget) else {
        print("⚠️ Couldn't create digicam enter")
        return
    }
    
    if session.canAddInput(enter) {
        session.addInput(enter)
    }
    
    let output = AVCaptureVideoDataOutput()
    output.setSampleBufferDelegate(self, queue: DispatchQueue(label: "digicam.body.queue"))
    if session.canAddOutput(output) {
        session.addOutput(output)
    }
    
    session.commitConfiguration()
}

I’m not positive what I’m doing fallacious. I feel it could possibly be one thing associated to the orientation.

One factor I discover is that I can solely get good outcomes utilizing VNImageRequestHandler with the orientation as up. Initially I attempted passing the identical worth because the one which I name processFrameForFinalImage, which is leftMirrored, however that resulted to the fallacious place of the masks.