Hi!
Is there any way to link the ambient light sensor of the device with a light in Unity so virtual objects do not appear lighter or brighter that the camera video.
So, the intensity of the directional light wll be linked to the ambinet light sensor, or the overall brightness of the camera video.
Basically, its the same principle of color grading in compositing. How does Vuforia for Unity handles this?
Kind Regards.
Did you ever discover a solution for this?
I'm having the same problem at the moment, in that I need the brightness in my scene to match the brightness in the real world (or as close as is possible). If I'm somewhere very bright the app is fine, but anywhere dark the models are lit up far too brightly.
You can adjust the scene lighting programmatically in Unity, but there is currently no way to determine brightness through the Vuforia API. This sounds like a good request for the wish list - https://developer.vuforia.com/forum/general-discussion/wish-list
In the interim, here is an algorithm for approximating luminance using pixel analyis on the greyscale image..
http://b2cloud.com.au/tutorial/obtaining-luminosity-from-an-ios-camera
* keep in mind that this approach will be affected by the camera's exposure settings.
One method that would also work is matching the colors/brightness of the image target points to the marker points from the live video.
This way you wont have to analyse the whole frame, just several points. and it would also make sure the 3D object matches the colors on the location of the marker, instead of the whole scene. this would take care of shadows and such.
Not sure how helpful it is now, but here is my current light matching script. I've attached it to the AR camera and it is accessing a directional light child component. It borrows heavily from the GetCameraImage script. The luminance to intensity is not perfect but the current version seems to work well.
Let me know if there are any huge mistakes or suggestions.
using System.Collections;
using UnityEngine;
public class LightMatching : MonoBehaviour, ITrackerEventHandler
{
private Image.PIXEL_FORMAT m_PixelFormat = Image.PIXEL_FORMAT.RGB888;
private bool m_RegisteredFormat = false;
private byte[] pixels;
private Light lightChild = new Light ();
void Start ()
{
QCARBehaviour qcarBehaviour = (QCARBehaviour)FindObjectOfType (typeof(QCARBehaviour));
if (qcarBehaviour) {
qcarBehaviour.RegisterTrackerEventHandler (this);
}
}
public void OnInitialized ()
{
}
public void OnTrackablesUpdated ()
{
if (!m_RegisteredFormat) {
CameraDevice.Instance.SetFrameFormat (m_PixelFormat, true);
m_RegisteredFormat = true;
}
CameraDevice cam = CameraDevice.Instance;
Image image = cam.GetCameraImage (m_PixelFormat);
if (image != null) {
pixels = image.Pixels;
double totalLuminance = 0.0;
for (int p=0; p<pixels.Length; p+=4) {
totalLuminance += pixels [p] * 0.299 + pixels [p + 1] * 0.587 + pixels [p + 2] * 0.114;
}
totalLuminance /= (pixels.Length);
totalLuminance /= 255.0;
GetComponentInChildren<Light> ().intensity =(float)(8 * totalLuminance);
}
}
}
I have developed a bit of a hacky solution for something similar. (Only a method for using the camera input as a sort of fake realtime reflection anyway)
1) Create a Sphere
2) Drag and drop the Background Plane Behavior Script on to object.
3) Assign the standard unity 5 Uber Shader / Material to the sphere.
Tweak and play.
I created some bubbles, a videofeed particle system and a magnifying glass. I also found that I could create a few interesting material variations.
Not as advanced as the video below but quite interesting and hopefully a small step in the right direction...
I'm bumping this thread because i want to do something similar and i haven't the knowledge to update Ellio's script for Vuforia 6.
Basically i'm just looking for a simple solution for dynamic lighting for Vuforia and Unity! So if you have an other idea it would be great!
Hi Vuforia folks, I wrote a script for getting light estimation in Vuforia. I can confirm that it works on both android and ios. It's been tested on Samsung S6, Pixel2, and iPhone 7. I'm more of a copy and paste coder so I cobbled this together from different attempts that I found on various forums. If anybody feels like making improvements to it, please feel free. Just give me a shout if you do. : )
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Vuforia;
using UnityEngine.UI;
using UnityEngine.Rendering;
using Image = Vuforia.Image;
/* *************************** HOW TO USE **********************************
Just Attach to your AR camera or Create an empty GameObject in your scene and attach this script
Most likely you'll have at least one light in your sceen. add your light to the public Light To Effect var in the editor.
You'll need to modify the code if you have more than one light you'd like to affect.
To see get dev/debugging readouts create a screen overlay canvas with a couple of text objects. add them to the lightoutput vars in the editor
Can varify that a this light estimation script works in iphone7, Samsung S6, and pixel
Should also work in the Unity editor as well.
*************************** ENJOY! **********************************
*/
public class LightMatching : MonoBehaviour
{ private bool mAccessCameraImage = true;
// camera image pixel
private Image.PIXEL_FORMAT mPixelFormat = Image.PIXEL_FORMAT.UNKNOWN_FORMAT;// or RGBA8888, RGB888, RGB565, YUV, GRAYSCALE
// Boolean flag telling whether the pixel format has been registered
private bool mFormatRegistered = false;
// during development you might want to set up a couple of text objects in a UI screen overlay for debugging
public Text LightOutput1;
public Text LightOutput2;
// boolean used to start debugging
public bool debugging;
//This is the directional light I was using in my scene.
public Light m_LightToEffect;
// This color variable is being used to change the ambient light in the scene. goes from white to black depending on brightness of lights
// You might want to thake that out depending on how your scene is looking
private Color lightColor = new Color(1,1,1,1);
private float ligtColorNum;
//Use this to make adjustments if your light estimation is looking too bright or too dark. I don't know what a double is.
public double intesityModifier = 10.0;
//Use this to make light temperature adjustments
public int temperatureModifier= 3000;
// To be honest I'm not sure what these are for. I cobbled this code from a bunch of different scripts. It's not messing anything up so I'm leaving in
public float? intensity { get; private set; }
public float? colorTemperature { get; private set; }
void Start()
{
// I beleive these are required setting to make adjustments to the ambient light in the scene. I could be wrong about that.
GraphicsSettings.lightsUseLinearIntensity = true;
GraphicsSettings.lightsUseColorTemperature = true;
// set up pixel format
#if UNITY_EDITOR
mPixelFormat = Image.PIXEL_FORMAT.GRAYSCALE; // Need Grayscale for Editor
#else
mPixelFormat = Image.PIXEL_FORMAT.RGB888; // Use RGB888 for mobile
#endif
// API for getting Vuforia Callbacks as of Unity 2018.1.0f2.
//The OnVuforiaStarted event is required for getting the camera pixel data for sure
VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted);
// not sure if this is really needed but leaving in just to be safe
VuforiaARController.Instance.RegisterOnPauseCallback(OnVuforiaPaused);
//This behaves much like Update() - Most of the magic happens in OnTrackablesUpdated
VuforiaARController.Instance.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
// not needed. just thought I'd leave it incase anybody needs to access these callbacks
// DeviceTrackerARController.Instance.RegisterTrackerStartedCallback(OnTrackerStarted);
// DeviceTrackerARController.Instance.RegisterDevicePoseStatusChangedCallback(OnDevicePoseStatusChanged);
//text used for debugging
LightOutput1.text = "";
LightOutput2.text = "";
}
private void OnVuforiaStarted()
{
// Try register camera image format
if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
{
Debug.Log("Successfully registered pixel format " + mPixelFormat.ToString());
mFormatRegistered = true;
}
else
{
Debug.LogError("Failed to register pixel format " + mPixelFormat.ToString() +
"\n the format may be unsupported by your device;" +
"\n consider using a different pixel format.");
mFormatRegistered = false;
}
}
///
/// Called when app is paused / resumed
///
private void OnVuforiaPaused(bool paused)
{
if (paused)
{
Debug.Log("App was paused");
UnregisterFormat();
}
else
{
Debug.Log("App was resumed");
RegisterFormat();
}
}
///
/// Called each time the Vuforia state is updated
///
private void OnTrackablesUpdated()
{ //Here's the getting camera pixel part of the code. not sure how it works but it works
if (mFormatRegistered)
{
if (mAccessCameraImage)
{
Vuforia.Image image = CameraDevice.Instance.GetCameraImage(mPixelFormat);
if (image != null)
{
string imageInfo = mPixelFormat + " image: \n";
imageInfo += " size: " + image.Width + " x " + image.Height + "\n";
imageInfo += " bufferSize: " + image.BufferWidth + " x " + image.BufferHeight + "\n";
imageInfo += " stride: " + image.Stride;
Debug.Log(imageInfo);
byte[] pixels = image.Pixels;
if (pixels != null && pixels.Length > 0)
{
Debug.Log("Image pixels: " + pixels[0] + "," + pixels[1] + "," + pixels[2] + ",...");
// I have no idea what the double type is or does. seems to be similar to float
double totalLuminance = 0.0;
for (int p = 0; p < pixels.Length; p += 4)
{
totalLuminance += pixels[p] * 0.299 + pixels[p + 1] * 0.587 + pixels[p + 2] * 0.114;
}
totalLuminance /= (pixels.Length);
//this takes the totalLuminance in the line above and puts it in the decimal range. Needs to be cast as a float. My math is arbitrary but works out
ligtColorNum = (float)totalLuminance * 0.0255f;
//ligtColorNum is put in for RGB. will change color along the gray scale
lightColor = new Color(ligtColorNum, ligtColorNum, ligtColorNum, 1.0f);
Debug.Log("color ++++++++++++++++++++++++++++++++++++++++++++++++++++" + ligtColorNum);
// I got this math from someone else's code. seems to convert totalLuminance to smaller number for adjusting light luminance.
totalLuminance /= 255.0;
totalLuminance *= intesityModifier;
Debug.Log("Total luminance ========================" + totalLuminance);
m_LightToEffect.intensity = (float)totalLuminance;
//originally tried to change color of light in scene but it looked flickery. Could be fun though for someone to try light color effects
// m_LightToEffect.color = lightColor;
//This is the secret ingredient in the special sauce. This adjusts the Ambient light that's always present in a scene.
//If you were to only adjust the lights in the scene it would never go completely dark if you turned the lights off in a room.
//still won't go completely dark because your screen will have some illumingation, but it gets pretty close.
RenderSettings.ambientIntensity = m_LightToEffect.intensity;
//I'm changing the color of the ambient light on the grayscale. It's a little redundant but I think it helps. Experiment and see how you feel.
RenderSettings.ambientLight = lightColor;
Debug.Log("light intensity = " + m_LightToEffect.intensity);
//I'm not exacly sure what the color temp does. It was a setting in someone else's code, but seems to be working out.
colorTemperature = (float?)(totalLuminance * temperatureModifier);
m_LightToEffect.colorTemperature = (float)colorTemperature;
Debug.Log("calculating color temperature =========================" + colorTemperature);
//Used for debugging so you can see if light changes;
if (debugging == true){
if (m_LightToEffect.intensity != null)
{
LightOutput1.text = "light intensity = " + m_LightToEffect.intensity;
}
else
{
LightOutput1.text = "ambient intensity = " + RenderSettings.ambientIntensity;
}
// LightOutput2.text = "ambient intensity = " + RenderSettings.ambientIntensity;
LightOutput2.text = "color temp = " + m_LightToEffect.colorTemperature;
}
//
}
}
}
}
}
///
/// Unregister the camera pixel format (e.g. call this when app is paused)
///
private void UnregisterFormat()
{
Debug.Log("Unregistering camera pixel format " + mPixelFormat.ToString());
CameraDevice.Instance.SetFrameFormat(mPixelFormat, false);
mFormatRegistered = false;
}
///
/// Register the camera pixel format
///
private void RegisterFormat()
{
if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true))
{
Debug.Log("Successfully registered camera pixel format " + mPixelFormat.ToString());
mFormatRegistered = true;
}
else
{
Debug.LogError("Failed to register camera pixel format " + mPixelFormat.ToString());
mFormatRegistered = false;
}
}
}
Type your code in the box. To create a new line within the box use SHIFT + ENTER.
Since Vuforia still doesn't offer an ambient light estimation on 2020 other than with ARkit, here i share what i'm using with Unity 2019.3:
If anyone could to add color temperature estimation, would be nice.
using UnityEngine;
using System.Collections;
using Vuforia;
public class VuforiaLightEstimator : MonoBehaviour {
public Light ambientLight;
[Space]
[Range(0f, 1f)] public float ambientLightWeight = 1f; //max percentage of light change according to scene light
[Range(0f, 1f)] public float damping = 1f; //rate of change in time
public int framesToSkip = 5; // number of frames to skip between lights estimations (free some cycles)
private PIXEL_FORMAT mPixelFormat = PIXEL_FORMAT.UNKNOWN_FORMAT;
private bool mAccessCameraImage = true;
private bool mFormatRegistered = false;
private byte[] pixels;
private double totalLuminance;
private float accLuminance;
private float iniLuminance;
private int frameCount;
void Start() {
// Grab Light intensity
iniLuminance = ambientLight.intensity;
accLuminance = iniLuminance;
#if UNITY_EDITOR
mPixelFormat = PIXEL_FORMAT.GRAYSCALE; // Need Grayscale for Editor
#else
mPixelFormat = PIXEL_FORMAT.RGB888; // Use RGB888 for mobile
#endif
// Register Vuforia life-cycle callbacks:
VuforiaARController.Instance.RegisterVuforiaStartedCallback(OnVuforiaStarted);
VuforiaARController.Instance.RegisterTrackablesUpdatedCallback(OnTrackablesUpdated);
VuforiaARController.Instance.RegisterOnPauseCallback(OnPause);
}
private void OnVuforiaStarted() { //Called after vuforia camera init
// Try register camera image format
if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true)) {
mFormatRegistered = true;
} else {
mFormatRegistered = false;
}
}
void OnTrackablesUpdated() { // Called each time the Vuforia state is updated
if ((mFormatRegistered) && (mAccessCameraImage)) {
if (frameCount <= 0) {
frameCount = framesToSkip;
Vuforia.Image image = CameraDevice.Instance.GetCameraImage(mPixelFormat);
if (image != null) {
pixels = image.Pixels;
totalLuminance = 0.0;
if (pixels != null && pixels.Length > 0) {
for (int p = 0; p < pixels.Length; p += 64) {
totalLuminance += pixels[p] * 0.299 + pixels[p + 1] * 0.587 + pixels[p + 2] * 0.114;
}
totalLuminance /= ((pixels.Length) / 16);
totalLuminance /= 255.0;
accLuminance = (float)((8 * totalLuminance) * ambientLightWeight) + iniLuminance * (1 - ambientLightWeight);
if (accLuminance <= 0f) accLuminance = 0f;
}
}
}
accLuminance = ambientLight.intensity * (damping) + accLuminance * (1 - damping);
ambientLight.intensity = accLuminance;
frameCount -= 1;
}
}
// Called when app is paused / resumed
void OnPause(bool paused) {
if (paused) {
UnregisterFormat();
} else {
RegisterFormat();
}
}
// Register the camera pixel format
void RegisterFormat() {
if (CameraDevice.Instance.SetFrameFormat(mPixelFormat, true)) {
mFormatRegistered = true;
} else {
mFormatRegistered = false;
}
}
// Unregister the camera pixel format (e.g. call this when app is paused)
void UnregisterFormat() {
CameraDevice.Instance.SetFrameFormat(mPixelFormat, false);
mFormatRegistered = false;
}
}
Hi salvadorlimones
I guess it would depend on whether you can get access to this from Unity, so probably worth checking in the Unity spec/forums in the first instance.
If you can then you may be able to use this as a control for the light within the scripts, thus achieving your desired effect.
HTH
N