Log in or register to post comments

Video Background stretched, cropped and wrongly rotated

November 20, 2019 - 5:09am #1

I am using Vuforia for iOS (native) together with Metal. I added Auto Rotation to the Metal Sample and added configureVideoBackgroundWithCameraMode: from the default Sample to the Metal Sample app. However, the Video Background doesnt rotate properly. The video background is split, stretched and is mirrored when rotating from the initial orientation to a different orientation.

Here is my code, only snippets that are new and not in the original Metal Sample:

ImageTargetsMetalViewController.mm

-(void)viewWillTransitionToSize:(CGSize)size withTransitionCoordinator:(id<UIViewControllerTransitionCoordinator>)coordinator {

    [super viewWillTransitionToSize:size withTransitionCoordinator:coordinator];

    [self handleRotation:[[UIApplication sharedApplication] statusBarOrientation]];

}

- (void) handleRotation:(UIInterfaceOrientation)interfaceOrientation

{

    dispatch_async(dispatch_get_main_queue(), ^{

        // ensure overlay size and AR orientation is correct for screen orientation

        [self handleARViewRotation:[[UIApplication sharedApplication] statusBarOrientation]];

        [self->vapp changeOrientation:[[UIApplication sharedApplication] statusBarOrientation]];

        [self->metalView updateRenderingPrimitives];

    });

}

- (void) handleARViewRotation:(UIInterfaceOrientation)interfaceOrientation

{

    // Retrieve up-to-date view frame.

    // Note that, while on iOS 7 and below, the frame size does not change

    // with rotation events,

    // on the contray, on iOS 8 the frame size is orientation-dependent,

    // i.e. width and height get swapped when switching from

    // landscape to portrait and vice versa.

    // This requires that the latest (current) view frame is retrieved.

    CGRect viewBounds = [[UIScreen mainScreen] bounds];

    int smallerSize = MIN(viewBounds.size.width, viewBounds.size.height);

    int largerSize = MAX(viewBounds.size.width, viewBounds.size.height);

    if (interfaceOrientation == UIInterfaceOrientationPortrait ||

        interfaceOrientation == UIInterfaceOrientationPortraitUpsideDown)

    {

        NSLog(@"AR View: Rotating to Portrait");

        CGRect viewBounds;

        viewBounds.origin.x = 0;

        viewBounds.origin.y = 0;

        viewBounds.size.width = smallerSize;

        viewBounds.size.height = largerSize;

        [metalView setFrame:viewBounds];

    }

    else if (interfaceOrientation == UIInterfaceOrientationLandscapeLeft ||

            interfaceOrientation == UIInterfaceOrientationLandscapeRight)

    {

        NSLog(@"AR View: Rotating to Landscape");

        CGRect viewBounds;

        viewBounds.origin.x = 0;

        viewBounds.origin.y = 0;

        viewBounds.size.width = largerSize;

        viewBounds.size.height = smallerSize;

        [metalView setFrame:viewBounds];

    }

}

-(void)configureVideoBackgroundWithCameraMode:(Vuforia::CameraDevice::MODE)cameraMode viewWidth:(float)viewWidthConfig andHeight:(float)viewHeightConfig {

    [[self vapp]configureVideoBackgroundWithCameraMode:cameraMode viewWidth:viewWidthConfig viewHeight:viewHeightConfig];

}

 

SampleApplicationSession.mm

 

- (void) changeOrientation:(UIInterfaceOrientation) ARViewOrientation {

    self.mARViewOrientation = ARViewOrientation;

    CGSize arViewBoundsSize = [self getCurrentARViewBoundsSize];

    int smallerSize = MIN(arViewBoundsSize.width, arViewBoundsSize.height);

    int largerSize = MAX(arViewBoundsSize.width, arViewBoundsSize.height);

    if (self.mARViewOrientation == UIInterfaceOrientationPortrait)

    {

        Vuforia::onSurfaceChanged(smallerSize, largerSize);

        Vuforia::setRotation(Vuforia::ROTATE_IOS_90);

    }

    else if (self.mARViewOrientation == UIInterfaceOrientationPortraitUpsideDown)

    {

        Vuforia::onSurfaceChanged(smallerSize, largerSize);

        Vuforia::setRotation(Vuforia::ROTATE_IOS_270);

    }

    else if (self.mARViewOrientation == UIInterfaceOrientationLandscapeLeft)

    {

        Vuforia::onSurfaceChanged(largerSize, smallerSize);

        Vuforia::setRotation(Vuforia::ROTATE_IOS_180);

    }

    else if (self.mARViewOrientation == UIInterfaceOrientationLandscapeRight)

    {

        Vuforia::onSurfaceChanged(largerSize, smallerSize);

        Vuforia::setRotation(Vuforia::ROTATE_IOS_0);

    }

    [self.delegate configureVideoBackgroundWithCameraMode:self.cameraMode

                                                viewWidth:arViewBoundsSize.width

                                                andHeight:arViewBoundsSize.height];

}

- (void) configureVideoBackgroundWithCameraMode:(Vuforia::CameraDevice::MODE)cameraMode viewWidth:(float)viewWidthConfig viewHeight:(float)viewHeightConfig

{

    float viewWidth = viewWidthConfig;

    float viewHeight = viewHeightConfig;

    // Get the default video mode

    Vuforia::CameraDevice& cameraDevice = Vuforia::CameraDevice::getInstance();

    Vuforia::VideoMode videoMode = cameraDevice.getVideoMode(cameraMode);

    // Configure the video background

    Vuforia::VideoBackgroundConfig config;

    config.mPosition.data[0] = 0.0f;

    config.mPosition.data[1] = 0.0f;

    [self performSelectorOnMainThread:@selector(updateOrientation) withObject:self waitUntilDone:YES];

    if (self.mIsActivityInPortraitMode) {

        float aspectRatioVideo = (float)videoMode.mWidth / (float)videoMode.mHeight;

        float aspectRatioView = viewHeight / viewWidth;

        if (aspectRatioVideo < aspectRatioView) {

            config.mSize.data[0] = (int)videoMode.mHeight * (viewHeight / (float)videoMode.mWidth);

            config.mSize.data[1] = (int)viewHeight;

        }

        else {

            config.mSize.data[0] = (int)viewWidth;

            config.mSize.data[1] = (int)videoMode.mWidth * (viewWidth / (float)videoMode.mHeight);

        }

    }

    else {

        // --- View is landscape ---

        if (viewWidth < viewHeight) {

            // Swap width/height: this is neded on iOS7 and below

            // as the view width is always reported as if in portrait.

            // On IOS 8, the swap is not needed, because the size is

            // orientation-dependent; so, this swap code in practice

            // will only be executed on iOS 7 and below.

            float temp = viewWidth;

            viewWidth = viewHeight;

            viewHeight = temp;

        }

        // Compare aspect ratios of video and screen.  If they are different we

        // use the full screen size while maintaining the video's aspect ratio,

        // which naturally entails some cropping of the video

        float aspectRatioVideo = (float)videoMode.mWidth / (float)videoMode.mHeight;

        float aspectRatioView = viewWidth / viewHeight;

        if (aspectRatioVideo < aspectRatioView) {

            config.mSize.data[0] = (int)viewWidth;

            config.mSize.data[1] = (int)videoMode.mHeight * (viewWidth / (float)videoMode.mWidth);

        }

        else {

            config.mSize.data[0] = (int)videoMode.mWidth * (viewHeight / (float)videoMode.mHeight);

            config.mSize.data[1] = (int)viewHeight;

        }

    }

    // Calculate the viewport for the app to use when rendering

#ifdef DEBUG_SAMPLE_APP

    NSLog(@"VideoBackgroundConfig: size: %d,%d", config.mSize.data[0], config.mSize.data[1]);

    NSLog(@"VideoMode:w=%d h=%d", videoMode.mWidth, videoMode.mHeight);

    NSLog(@"width=%7.3f height=%7.3f", viewWidth, viewHeight);

    //NSLog(@"ViewPort: X,Y: %d,%d Size X,Y:%d,%d", viewport.posX,viewport.posY,viewport.sizeX,viewport.sizeY);

#endif

    // Set the config

    Vuforia::Renderer::getInstance().setVideoBackgroundConfig(config);

}

- (void)updateOrientation

{

    UIInterfaceOrientation orientation = [[UIApplication sharedApplication] statusBarOrientation];

    if (orientation == UIInterfaceOrientationPortrait || orientation == UIInterfaceOrientationPortraitUpsideDown)

    {

        self.mIsActivityInPortraitMode = YES;

    }

    else if (orientation == UIInterfaceOrientationLandscapeLeft || orientation == UIInterfaceOrientationLandscapeRight)

    {

        self.mIsActivityInPortraitMode = NO;

    }

}

 

 

Here's a demonstration: https://streamable.com/w0ock I am not sure what to do here since i call Vuforia::onSurfaceChanged and update the VideoBackground I tried the VideoBackgroundAccess test sample aswell. I can confirm that it doesnt work with multiple orientations aswell. The VideoBackgroundAccess Sample only needed a change in the info.plist.

Log in or register to post comments