Log in or register to post comments

Light Matching

June 28, 2012 - 11:56pm #1

Hi!

Is there any way to link the ambient light sensor of the device with a light in Unity so virtual objects do not appear lighter or brighter that the camera video.

So, the intensity of the directional light wll be linked to the ambinet light sensor, or the overall brightness of the camera video.

Basically, its the same principle of color grading in compositing. How does Vuforia for Unity handles this?

Kind Regards.

Light Matching

October 24, 2020 - 10:02am #24

I just started learning Unity and Vuforia, so can you please tell me how to do it on Unity with detailed steps? Thank you very much.

Light Matching

March 26, 2020 - 8:08pm #23
 Type your code in the box. To create a new line within the box use SHIFT + ENTER. 

Since Vuforia still doesn't offer an ambient light estimation on 2020 other than with ARkit, here i share what i'm using with Unity 2019.3:

If anyone could to add color temperature estimation, would be nice.

using UnityEngine;
using System.Collections;
using Vuforia;

public class VuforiaLightEstimator : MonoBehaviour {

    public Light ambientLight;
    [Space]
    [Range(0f, 1f)] public float ambientLightWeight = 1f; //max percentage of light change according to scene light
    [Range(0f, 1f)] public float damping = 1f; //rate of change in time
    public int framesToSkip = 5; // number of frames to skip between lights estimations (free some cycles)

    private PIXEL_FORMAT mPixelFormat = PIXEL_FORMAT.UNKNOWN_FORMAT;
    private bool mAccessCameraImage = true;
    private bool mFormatRegistered = false;
    private byte[] pixels;
    private double totalLuminance;
    private float accLuminance;
    private float iniLuminance;
    private int frameCount;
   

    void Start() {
        // Grab Light intensity
        iniLuminance = ambientLight.intensity;
        accLuminance = iniLuminance;

#if UNITY_EDITOR
        mPixelFormat = PIXEL_FORMAT.GRAYSCALE; // Need Grayscale for Editor
#else
        mPixelFormat = PIXEL_FORMAT.RGB888; // Use RGB888 for mobile
#endif
        // Register Vuforia life-cycle callbacks:
        VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted);
        VuforiaARController.Instance.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
        VuforiaARController.Instance.RegisterOnPauseCallback(OnPause);
    }

    private void OnVuforiaStarted() {  //Called after vuforia camera init
                                       // Try register camera image format
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true)) {
            mFormatRegistered = true;
        } else {
            mFormatRegistered = false;
        }
    }

    void OnTrackablesUpdated() { // Called each time the Vuforia state is updated
        if ((mFormatRegistered) && (mAccessCameraImage)) {           
            if (frameCount <= 0) {
                frameCount = framesToSkip;
                Vuforia.Image image = CameraDevice.Instance.GetCameraImage(mPixelFormat);
                if (image != null) {
                    pixels = image.Pixels;
                    totalLuminance = 0.0;
                    if (pixels != null && pixels.Length > 0) {

                        for (int p = 0; p < pixels.Length; p += 64) {
                            totalLuminance += pixels[p] * 0.299 + pixels[p + 1] * 0.587 + pixels[p + 2] * 0.114;
                        }
                        totalLuminance /= ((pixels.Length) / 16);
                        totalLuminance /= 255.0;
                        accLuminance = (float)((8 * totalLuminance) * ambientLightWeight) + iniLuminance * (1 - ambientLightWeight);
                        if (accLuminance <= 0f) accLuminance = 0f;
                    }
                }
            }
            accLuminance = ambientLight.intensity * (damping) + accLuminance * (1 - damping);
            ambientLight.intensity = accLuminance;
            frameCount -= 1;
        }
    }

    // Called when app is paused / resumed
    void OnPause(bool paused) {
        if (paused) {         
            UnregisterFormat();
        } else {    
            RegisterFormat();
        }
    }
   
    // Register the camera pixel format
    void RegisterFormat() {
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true)) {
            mFormatRegistered = true;
        } else {
            mFormatRegistered = false;
        }
    }

    // Unregister the camera pixel format (e.g. call this when app is paused)
    void UnregisterFormat() {
        CameraDevice.Instance.SetFrameFormat(mPixelFormat, false);
        mFormatRegistered = false;
    }
}

 

Light Matching

November 20, 2019 - 1:54pm #22

Thanks for the script :)

Light Matching

September 21, 2018 - 10:09am #21

Caballero, es ud mi Dios... gracias por el script

Light Matching

June 18, 2018 - 1:27pm #20

Hi Vuforia folks, I wrote a script for getting light estimation in Vuforia. I can confirm that it works on both android and ios. It's been tested on Samsung S6, Pixel2, and iPhone 7. I'm more of a copy and paste coder so I cobbled this together from different attempts that I found on various forums. If anybody feels like making improvements to it, please feel free. Just give me a shout if you do. : )

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Vuforia;
using UnityEngine.UI;
using UnityEngine.Rendering;
using Image = Vuforia.Image;

/*       *************************** HOW TO USE **********************************

    Just Attach to your AR camera or Create an empty GameObject in your scene and attach this script
    Most likely you'll have at least one light in your sceen. add your light to the public Light To Effect var in the editor.
    You'll need to modify the code if you have more than one light you'd like to affect. 

    To see get dev/debugging readouts create a screen overlay canvas with a couple of text objects. add them to the lightoutput vars in the editor
    
    Can varify that a this light estimation script works in iphone7, Samsung S6, and pixel
    Should also work in the Unity editor as well.

         ***************************   ENJOY!   **********************************

*/
public class LightMatching : MonoBehaviour
{   private bool mAccessCameraImage = true;
    // camera image pixel 
    private Image.PIXEL_FORMAT mPixelFormat = Image.PIXEL_FORMAT.UNKNOWN_FORMAT;// or RGBA8888, RGB888, RGB565, YUV, GRAYSCALE
    // Boolean flag telling whether the pixel format has been registered
    private bool mFormatRegistered = false;
    // during development you might want to set up a couple of text objects in a UI screen overlay for debugging
    public Text LightOutput1;
    public Text LightOutput2;
    // boolean used to start debugging 
    public bool debugging;
    //This is the directional light I was using in my scene. 
    public Light m_LightToEffect;
    // This color variable is being used to change the ambient light in the scene. goes from white to black depending on brightness of lights
    // You might want to thake that out depending on how your scene is looking
    private Color lightColor = new Color(1,1,1,1);
    private float ligtColorNum;
    //Use this to make adjustments if your light estimation is looking too bright or too dark.  I don't know what a double is.
    public double intesityModifier = 10.0;
    //Use this to make light temperature adjustments
    public int temperatureModifier= 3000;
    // To be honest I'm not sure what these are for. I cobbled this code from a bunch of different scripts. It's not messing anything up so I'm leaving in 
    public float? intensity { get; private set; }
    public float? colorTemperature { get; private set; }
   

    void Start()
    {
        // I beleive these are required setting to make adjustments to the ambient light in the scene. I could be wrong about that. 
        GraphicsSettings.lightsUseLinearIntensity = true;
        GraphicsSettings.lightsUseColorTemperature = true;
        // set up pixel format
        #if UNITY_EDITOR
        mPixelFormat = Image.PIXEL_FORMAT.GRAYSCALE; // Need Grayscale for Editor
        #else
        mPixelFormat = Image.PIXEL_FORMAT.RGB888; // Use RGB888 for mobile
        #endif

        // API for getting Vuforia Callbacks as of Unity 2018.1.0f2. 
        //The OnVuforiaStarted event is required for getting the camera pixel data for sure
        VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted);
        // not sure if this is really needed but leaving in just to be safe
        VuforiaARController.Instance.RegisterOnPauseCallback(OnVuforiaPaused);
        //This behaves much like Update() - Most of the magic happens in OnTrackablesUpdated
        VuforiaARController.Instance.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
        // not needed. just thought I'd leave it incase anybody needs to access these callbacks
        //  DeviceTrackerARController.Instance.RegisterTrackerStartedCallback(OnTrackerStarted);
        //  DeviceTrackerARController.Instance.RegisterDevicePoseStatusChangedCallback(OnDevicePoseStatusChanged);
         
        //text used for debugging
        LightOutput1.text = "";
        LightOutput2.text = "";

    }

    private void OnVuforiaStarted()
    {
        // Try register camera image format
       
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
        {
            Debug.Log("Successfully registered pixel format " + mPixelFormat.ToString());
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register pixel format " + mPixelFormat.ToString() +
                "\n the format may be unsupported by your device;" +
                "\n consider using a different pixel format.");
            mFormatRegistered = false;
        }
    }
    /// 
    /// Called when app is paused / resumed
    /// 
    private void OnVuforiaPaused(bool paused)
    {
        if (paused)
        {
            Debug.Log("App was paused");
            UnregisterFormat();
        }
        else
        {
            Debug.Log("App was resumed");
            RegisterFormat();
        }
    }


    /// 
    /// Called each time the Vuforia state is updated
    /// 
    private void OnTrackablesUpdated()
    { //Here's the getting camera pixel part of the code. not sure how it works but it works
        if (mFormatRegistered)
        {
            if (mAccessCameraImage)
            {
                Vuforia.Image image = CameraDevice.Instance.GetCameraImage(mPixelFormat);
                if (image != null)
                {
                    string imageInfo = mPixelFormat + " image: \n";
                    imageInfo += " size: " + image.Width + " x " + image.Height + "\n";
                    imageInfo += " bufferSize: " + image.BufferWidth + " x " + image.BufferHeight + "\n";
                    imageInfo += " stride: " + image.Stride;
                    Debug.Log(imageInfo);
                    byte[] pixels = image.Pixels;
                    if (pixels != null && pixels.Length > 0)
                    {
                        Debug.Log("Image pixels: " + pixels[0] + "," + pixels[1] + "," + pixels[2] + ",...");
                       // I have no idea what the double type is or does. seems to be similar to float
                        double totalLuminance = 0.0;
                        for (int p = 0; p < pixels.Length; p += 4)
                        {
                            totalLuminance += pixels[p] * 0.299 + pixels[p + 1] * 0.587 + pixels[p + 2] * 0.114;

                        }

                        totalLuminance /= (pixels.Length);
                        //this takes the totalLuminance in the line above and puts it in the decimal range. Needs to be cast as a float.  My math is arbitrary but works out
                        ligtColorNum = (float)totalLuminance * 0.0255f;
                        //ligtColorNum is put in for RGB. will change color along the gray scale
                        lightColor = new Color(ligtColorNum, ligtColorNum, ligtColorNum, 1.0f);
                        Debug.Log("color ++++++++++++++++++++++++++++++++++++++++++++++++++++" + ligtColorNum);
                        // I got this math from someone else's code. seems to convert totalLuminance to smaller number for adjusting light luminance.
                        totalLuminance /= 255.0;
                        totalLuminance *= intesityModifier;
                        Debug.Log("Total luminance ========================" + totalLuminance);
                        m_LightToEffect.intensity = (float)totalLuminance;
                        //originally tried to change color of light in scene but it looked flickery. Could be fun though for someone to try light color effects
                        //  m_LightToEffect.color = lightColor;

                        //This is the secret ingredient in the special sauce. This adjusts the Ambient light that's always present in a scene.
                        //If you were to only adjust the lights in the scene it would never go completely dark if you turned the lights off in a room.
                        //still won't go completely dark because your screen will have some illumingation, but it gets pretty close.
                        RenderSettings.ambientIntensity = m_LightToEffect.intensity;
                        //I'm changing the color of the ambient light on the grayscale. It's a little redundant but I think it helps. Experiment and see how you feel. 
                        RenderSettings.ambientLight = lightColor;
                        Debug.Log("light intensity = " + m_LightToEffect.intensity);
                        //I'm not exacly sure what the color temp does. It was a setting in someone else's code, but seems to be working out.
                        colorTemperature = (float?)(totalLuminance * temperatureModifier);
                        m_LightToEffect.colorTemperature = (float)colorTemperature;
                        Debug.Log("calculating color temperature =========================" + colorTemperature);

                        //Used for debugging so you can see if light changes;
                        if (debugging == true){
                        if (m_LightToEffect.intensity != null)
                        {
                            LightOutput1.text = "light intensity = " + m_LightToEffect.intensity;
                        }
                        else
                        {
                            LightOutput1.text = "ambient intensity = " + RenderSettings.ambientIntensity;
                        }
                       // LightOutput2.text = "ambient intensity = " + RenderSettings.ambientIntensity;
                            LightOutput2.text = "color temp = " + m_LightToEffect.colorTemperature;
                    }
                        //


                    }


                }
            }
        }
    }

    /// 
    /// Unregister the camera pixel format (e.g. call this when app is paused)
    /// 
    private void UnregisterFormat()
    {
        Debug.Log("Unregistering camera pixel format " + mPixelFormat.ToString());
        CameraDevice.Instance.SetFrameFormat(mPixelFormat, false);
        mFormatRegistered = false;
    }
    /// 
    /// Register the camera pixel format
    /// 
    private void RegisterFormat()
    {
        if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
        {
            Debug.Log("Successfully registered camera pixel format " + mPixelFormat.ToString());
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register camera pixel format " + mPixelFormat.ToString());
            mFormatRegistered = false;
        }
    }


  
}

Light Matching

January 2, 2017 - 10:17am #19

I'm bumping this thread because i want to do something similar and i haven't the knowledge to update Ellio's script for Vuforia 6.

 

Basically i'm just looking for a simple solution for dynamic lighting for Vuforia and Unity! So if you have an other idea it would be great!

Light Matching

June 29, 2016 - 2:22pm #18

i have tried to add this code in my AR camera. But it make my app lack down after I install to my android phone.

Light Matching

May 2, 2015 - 9:57pm #17

Not sure where that file is, someone else is probably more framilar with the whole unity setup.

Light Matching

May 2, 2015 - 5:15pm #16

Sorry it has been a long time since i wrote the script. Also since i posted it. i'll take a look at the project tonight and let you know how it works!

Light Matching

April 30, 2015 - 7:43am #15

I have developed a bit of a hacky solution for something similar. (Only a method for using the camera input as a sort of fake realtime reflection anyway)

1) Create a Sphere

2) Drag and drop the Background Plane Behavior Script on to object.

3) Assign the standard unity 5 Uber Shader / Material to the sphere.

 

Tweak and play.

 

I created some bubbles, a videofeed particle system  and a magnifying glass. I also found that I could create a few interesting material variations.

Not as advanced as the video below but quite interesting and hopefully a small step in the right direction...

Light Matching

April 30, 2015 - 7:15am #14

mrmdesign wrote:

I would love to know how you got this set up working maximrouf.

Any chance of a tutorial at some point or a 'how to' guide?

 

Thank you,

Marc

I'm interested in this, too!

Light Matching

April 30, 2015 - 3:56am #13

I would love to know how you got this set up working maximrouf.

Any chance of a tutorial at some point or a 'how to' guide?

 

Thank you,

Marc

Light Matching

April 14, 2015 - 12:53pm #12

Thanks re7ox, I will fix it.

Light Matching

April 14, 2015 - 12:36pm #11

Hi maximrouf 

Thanks for the reply, though the video link is a dead link. 

Cheers

 

Light Matching

April 14, 2015 - 12:07pm #10

Light Matching

April 14, 2015 - 12:45am #9

Hi There,

I'm attempting to get this working with the latest Vuforia and User Defined sample and I get an error of namespace 'ITrackerEventHAndler' could not be found. Were do I locate this file? 

Many thanks for posting this script :P

Light Matching

March 4, 2014 - 3:21pm #8

Thanks for posting that!

Light Matching

March 4, 2014 - 2:58pm #7

Not sure how helpful it is now, but here is my current light matching script. I've attached it to the AR camera and it is accessing a directional light child component. It borrows heavily from the GetCameraImage script. The luminance to intensity is not perfect but the current version seems to work well.
Let me know if there are any huge mistakes or suggestions.

using System.Collections;
using UnityEngine;
public class LightMatching : MonoBehaviour, ITrackerEventHandler
{
		private Image.PIXEL_FORMAT m_PixelFormat = Image.PIXEL_FORMAT.RGB888;
		private bool m_RegisteredFormat = false;
		private byte[] pixels;
		private Light lightChild = new Light ();
		void Start ()
		{
				QCARBehaviour qcarBehaviour = (QCARBehaviour)FindObjectOfType (typeof(QCARBehaviour));
				if (qcarBehaviour) {
						qcarBehaviour.RegisterTrackerEventHandler (this);
				}
		}

		public void OnInitialized ()
		{
		}
		public void OnTrackablesUpdated ()
		{
				if (!m_RegisteredFormat) {
						CameraDevice.Instance.SetFrameFormat (m_PixelFormat, true);
						m_RegisteredFormat = true;
				}
				CameraDevice cam = CameraDevice.Instance;
				Image image = cam.GetCameraImage (m_PixelFormat);
				if (image != null) {
						pixels = image.Pixels;
						double totalLuminance = 0.0;
						for (int p=0; p<pixels.Length; p+=4) {
								totalLuminance += pixels [p] * 0.299 + pixels [p + 1] * 0.587 + pixels [p + 2] * 0.114;
						}
						totalLuminance /= (pixels.Length);
						totalLuminance /= 255.0;
						GetComponentInChildren<Light> ().intensity =(float)(8 * totalLuminance);
				}

		}
}

 

Light Matching

May 2, 2013 - 5:39am #6

One method that would also work is matching the colors/brightness of the image target points to the marker points from the live video.
This way you wont have to analyse the whole frame, just several points.    and it would also make sure the 3D object matches the colors on the location of the marker, instead of the whole scene.         this would take care of shadows and such.

Light Matching

April 3, 2013 - 11:22am #5

You can adjust the scene lighting programmatically in Unity, but there is currently no way to determine brightness through the Vuforia API. This sounds like a good request for the wish list - https://developer.vuforia.com/forum/general-discussion/wish-list

 

In the interim, here is an algorithm for approximating luminance using pixel analyis on the greyscale image..

http://b2cloud.com.au/tutorial/obtaining-luminosity-from-an-ios-camera

* keep in mind that this approach will be affected by the camera's exposure settings.

Light Matching

April 3, 2013 - 7:52am #4

Did you ever discover a solution for this?

I'm having the same problem at the moment, in that I need the brightness in my scene to match the brightness in the real world (or as close as is possible). If I'm somewhere very bright the app is fine, but anywhere dark the models are lit up far too brightly. 

Light Matching

June 29, 2012 - 4:38am #3

Reporting again, when you have a solution for it. Sounds very interesting. :)

Light Matching

June 29, 2012 - 1:47am #2

Hi salvadorlimones

I guess it would depend on whether you can get access to this from Unity, so probably worth checking in the Unity spec/forums in the first instance.

If you can then you may be able to use this as a control for the light within the scripts, thus achieving your desired effect.

HTH

Log in or register to post comments