Unable to see the joint trackbar when deployed in Hololens

[arunavanag591]

Hello I am building it on Universal windows platform. I have deployed the arm in the Hololens, I can get the robot model, however, the joint slide bars do not appear.

[shllybkwrm]

I have gotten the model working on HoloLens, however I'm working with a fairly old version of this repo so I'm unaware of any changes since Oct 2016.

My canvas settings look like this. Are yours similar?

[arunavanag591]

@shllybkwrm I tried replicating your canvas, however, it doesn't help, I still can't get the joint slide bars after deploying it in the hololens. Do I need to add any more setting? Right now the slide bars are coming for the controller.cs script.

[shllybkwrm]

@arunavanag591 Have you tried changing the canvas scale?
Also, I assume your Main Camera has been set up for HoloLens, correct?

[shllybkwrm]

Maybe this will be more helpful, I looked through my old files and I believe what I did was manually add Slider UI elements to the Canvas, position them where I wanted etc., and then modified the UR5Controller.cs to grab them automatically.

// Author: Long Qian
// Email: [email protected]

using UnityEngine;
using UnityEngine.UI;

public class UR5Controller : MonoBehaviour {

    public GameObject RobotBase;
    public float[] jointValues = new float[6];
    private GameObject[] jointList = new GameObject[6];
    private float[] upperLimit = { 180f, 180f, 180f, 180f, 180f, 180f };
    private float[] lowerLimit = { -180f, -180f, -180f, -180f, -180f, -180f };

    public GameObject CanvasObj;
    private Slider[] sliderList = new Slider[6];

    public InputField TextControl;

    // Use this for initialization
    void Start()
    {
        initializeJoints(jointList);
        initializeSliders(sliderList);

        TextControl.text = "(0,0,0,0,0,0)";

    }

    // Update is called once per frame
    void Update() {
        TextControl.text = string.Format("({0:0.0}, {1:0.0}, {2:0.0}, {3:0.0}, {4:0.0}, {5:0.0})",
            jointValues[5], jointValues[4], jointValues[3],
            jointValues[2], jointValues[1], jointValues[0]);

    }

    // Right before camera renders
    void LateUpdate() {

        for (int i = 0; i < 6; i++)
        {
            Vector3 currentRotation = jointList[i].transform.localEulerAngles;
            //Debug.Log(currentRotation);
            currentRotation.z = jointValues[i];
            jointList[i].transform.localEulerAngles = currentRotation;
        }
    }

    void OnGUI() {

        /*#if UNITY_EDITOR
                int boundary = 20;
                int labelHeight = 20;
                GUI.skin.label.fontSize = GUI.skin.box.fontSize = GUI.skin.button.fontSize = 20;
                GUI.skin.label.alignment = TextAnchor.MiddleLeft;
                for (int i = 0; i < 6; i++) {
                    GUI.Label(new Rect(boundary, boundary + (i * 2 + 1) * labelHeight, labelHeight * 4, labelHeight), "Joint " + i + ": ");
                    jointValues[i] = GUI.HorizontalSlider(new Rect(boundary + labelHeight * 4, boundary + (i * 2 + 1) * labelHeight + labelHeight / 4, labelHeight * 5, labelHeight), jointValues[i], lowerLimit[i], upperLimit[i]);
                }
        #else        
                Debug.Log("Entered else");
        #endif*/


        for (int i = 0; i < 6; i++) {
            jointValues[i] = sliderList[i].value;
        }
    }


    // Create the list of GameObjects that represent each joint of the robot
    public void initializeJoints(GameObject[] jointList_) {
        var RobotChildren = RobotBase.GetComponentsInChildren<Transform>();
        for (int i = 0; i < RobotChildren.Length; i++)
        {
            if (RobotChildren[i].name == "control0")
            {
                jointList_[0] = RobotChildren[i].gameObject;
            }
            else if (RobotChildren[i].name == "control1")
            {
                jointList_[1] = RobotChildren[i].gameObject;
            }
            else if (RobotChildren[i].name == "control2")
            {
                jointList_[2] = RobotChildren[i].gameObject;
            }
            else if (RobotChildren[i].name == "control3")
            {
                jointList_[3] = RobotChildren[i].gameObject;
            }
            else if (RobotChildren[i].name == "control4")
            {
                jointList_[4] = RobotChildren[i].gameObject;
            }
            else if (RobotChildren[i].name == "control5")
            {
                jointList_[5] = RobotChildren[i].gameObject;
            }
        }
    }

    // Create the list of GameObjects that represent each slider in the canvas
    public void initializeSliders(Slider[] sliderList_) {
        var CanvasChildren = CanvasObj.GetComponentsInChildren<Slider>();

        for (int i = 0; i < CanvasChildren.Length; i++) {
            if (CanvasChildren[i].name == "Slider0")
            {
                sliderList_[0] = CanvasChildren[i];

                sliderList_[0].minValue = lowerLimit[0];
                sliderList_[0].maxValue = upperLimit[0];
                sliderList_[0].value = 0;
            }
            else if (CanvasChildren[i].name == "Slider1")
            {
                sliderList_[1] = CanvasChildren[i];

                sliderList_[1].minValue = lowerLimit[1];
                sliderList_[1].maxValue = upperLimit[1];
                sliderList_[1].value = 0;
            }
            else if (CanvasChildren[i].name == "Slider2")
            {
                sliderList_[2] = CanvasChildren[i];

                sliderList_[2].minValue = lowerLimit[2];
                sliderList_[2].maxValue = upperLimit[2];
                sliderList_[2].value = 0;
            }
            else if (CanvasChildren[i].name == "Slider3")
            {
                sliderList_[3] = CanvasChildren[i];

                sliderList_[3].minValue = lowerLimit[3];
                sliderList_[3].maxValue = upperLimit[3];
                sliderList_[3].value = 0;
            }
            else if (CanvasChildren[i].name == "Slider4")
            {
                sliderList_[4] = CanvasChildren[i];

                sliderList_[4].minValue = lowerLimit[4];
                sliderList_[4].maxValue = upperLimit[4];
                sliderList_[4].value = 0;
            }
            else if (CanvasChildren[i].name == "Slider5")
            {
                sliderList_[5] = CanvasChildren[i];

                sliderList_[5].minValue = lowerLimit[5];
                sliderList_[5].maxValue = upperLimit[5];
                sliderList_[5].value = 0;
            }
        }
    }
}

[arunavanag591]

@shllybkwrm this occurred to me as well and I tried adding the sliders myself, however, I couldn't get it to animate. I saw your script, it doesn't look much different than mine. But my sliders are now just there for slow, I cannot really use it to manipulate the robot. My main camera is set up to look at the robot without any rotation. Looks fine when I launch the app in the editor.

Additionally, I am thrown with these errors, I am sure this has something to do with the fact i cant move the sliders and move the robot hololens: (apologies kinda new to c# and all these game development, my ultimate goal is to just get the joint status from the UR from the mixed reality and get it into my linux interface and work with it ). This is when I try to use your version of the code.

i tried adding the canvas: which fixes the issue however now they move like crazy and often hits limits and i am unable to get em back to normal condition. Thats the case in editor. Coming to hololens, the sliders still dont work, they just appear.

Also, a step ahead, do you have a clue, when and if I get to move the joints using the slider can I retrieve the current joint status at any instant and send it over a WebSocket maybe?

[shllybkwrm]

@arunavanag591 Hmm, I'm not sure, you can double check with my slider settings if you want. Also you can go ahead and remove the text field that I was using, or just add one in if that's easier.

Also, your sliders look pretty different than mine, what version of Unity are you using? I'm on 2017.1.1f1.

[arunavanag591]

HI @shllybkwrm ,
Apologies, I was working on a different projects in between. Yes you are right i was using the wrong sliders. Now I changed em, I got similar ones. However, I still cant manipulate them after deploying in the hololens. So I tried using gaze, and Tapped Event. However havent had any successful result yet. Did you have to program the slidebars to be able to manipulate them inside the hololens and take taps.

Additionally, is it possible to just select the robot jointwise and manipulate them which a slider. Or just by pinching the wrist will it be possible to manipulate the robot inside the hololens.

Thanks for your help.

[shllybkwrm]

@arunavanag591 Hey, I'm really sorry I never replied! Did you manage to fix this? If not, let me know what you still have problems with and I can let you know how I fixed that.

[arunavanag591]

Hi, @shllybkwrm Yea I fixed all the above issues. Having said which I am still manipulating the robot using the sliders which does not look appealing to a user. It would be great if I could manipulate the robot using the TCP (tool center point) and update the joint values.

Also just wondering, have you ever tried using mujoco physics simulator with unity, Mujoco released a plugin for Unity3D (http://www.mujoco.org/book/unity.html)

[shllybkwrm]

@arunavanag591 I haven't used that physics simulator, however I'm in the process of implementing something similar to what you want using inverse kinematics. You'll need to write a solver or find one online for the UR5, though.

[arunavanag591]

I am using the ur5 joint information sent over websocket from hololens to my Linux terminal where I am running motion planners. Backend's all set, it's just the interface I wanna make it look nicer.

@shllybkwrm sounds good, will be working on something similar. Probably have to stop using this model, and quite interested to use the physics simulator. Would be working on it soon. Let's keep in touch.

[arunavanag591]

@shllybkwrm just thought of sharing. I was able to use the physics engine to import a better model. The format of the model is similar to a URDF, so I got all the joint links and other information. Now I have a 3D target that controls entire arm movement, it's no more the previous individual joint sliders. This is much better.

[shllybkwrm]

@arunavanag591 That sounds very useful! I'm not familiar with the process of doing that - were there any guides you followed that you could point me to? Or, do you happen to have your own repo with that model?

[arunavanag591]

Hi @shllybkwrm I used as mentioned earlier: http://www.mujoco.org/book/unity.html
The new unity mujoco plugin. I wrote my own MJCF(the equivalent of URDF), however, you can use some already defined models in the resources: http://www.mujoco.org/forum/index.php?resources/

this is UR5 with a gripper: http://www.mujoco.org/forum/index.php?resources/universal-robots-ur5-robotiq-s-model-3-finger-gripper.22/

Additionally, I used the unity plugin BioIK (https://assetstore.unity.com/packages/tools/animation/bio-ik-67819) for computing the IK and control with a 3D target. If you are doing it for hololens, you can ask the creator of the plugin for a hololens version, as the one in the asset store has a lot of threading into it (made for unity editor), and we all know threading is not supported in UWP. Following which its pretty easy to configure your robot to the IK needed, it has examples in it which are easy to follow. No coding needed just to configure the robot IK but a lot of setting.

I am sorry, as its a project for the organization i work in, we are still in the process of whether to open source it. Its not in my git repo at present.
Let me know if you need further help.

[shllybkwrm]

Thanks so much @arunavanag591, I'm sure I can figure it out from those links (especially as I already have the UR5 IK done). This will be very useful for my work as well.