Assist is a peer-to-peer app with cross-platform capabilities. AR-enabled remote assistance can increase efficiency and accuracy of key scenarios for enterprise including training, inspection, assembly, maintenance, diagnosis and repair.
For first time users there is a Quick Start Guide available within the app (see: In-App Guides).
Some key features of Assist include:
- Remote web participants can see exactly what a Magic Leap 2 device participant can see to provide live assistance
- Collaboration through live annotation tools such as Pins, Arrows, and Draw Tools
- Live meshing of the physical environment to enhance collaboration
- A digital AR Workspace that moves with the device wearer and adapts to their work environment
Magic Leap Assist is available through the Featured Apps button on the Magic Leap 2 headset’s Home Menu and is compatible with Magic Leap OS 1.1.0 and above. Note: Featured Apps opens a web page in the browser. You may need to enable the Install unknown apps permission for the browser to install Assist after downloading it.
Release Notes
Assist V1.6.0 Release Notes
Additions:
- Added an option to Clear and Delete All content. You can now clear only annotations or delete all added content to a session to clear up space quickly.
- Added a session disconnect notification that gives the device host and web participants more information about why a session ended.
- Multi-Web User: Assist now supports up to 4 web participants within the same Assist Session for greater collaboration.
Visual Changes:
- Multi Docking UI: Improved UI for having multiple panels open at one time and docking multiple panels. See: Multi Docking Submenus.
Assist on Magic Leap 2
Installing Assist
Home Menu
The Home Menu will display when the Magic Leap 2 powers on. Use the Controller to interact with the Home Menu. Your Controller will cast a Ray to show where it is pointed.
Select Featured Apps
Scroll through the Home Menu to locate Featured Apps. Point the Ray at Featured Apps and press the Trigger to select it. A web browser will appear displaying featured applications for the Magic Leap 2.
Install Assist
Locate and select Assist from the list of featured applications. Selecting Assist will download and install the application. When Assist has finished downloading, navigate to OS Files, find the downloaded Assist application, click on it, and select Install (if this does not work, you can try clicking More (...) and manually selecting Open with to start the installation).
Note: On a factory reset ML2 device you may receive a notification saying "Web Browser - For your security, your device is not allowed to install unknown apps from this source." Click on Settings, and on the Unknown Apps page, toggle Allow From This Source on. You will then be prompted to install the already downloaded application
Launch Assist
Locate Magic Leap Assist in the Home Menu. Select Assist to launch the application.
Overview
When you launch Magic Leap Assist for the first time an option to experience an onboarding overview will appear.
This overview will introduce new users to the key features of Assist..
To experience the Overview again once inside of Assist, select the About Assist option from the Assist Menu and navigate to Overview.
Launch Permissions
When first launching Magic Leap Assist a series of permission requests will be displayed. Required permission are:
- Spatial Mapping - Access to the ML2’s spatial mapping and scanning capabilities to facilitate the Mesh visualization and scanning features
- Microphone - To capture audio to communicate with other participants
- Cameras - To share the current view of the device participant with other participants
These must be accepted before you will be permitted to continue into the rest of the application. Permissions can be disabled or reenabled later on by navigating to OS Settings -> Apps & Notifications -> Magic Leap Assist -> Permissions.
Denied Permissions
If permissions have been denied or previously allowed permissions were later denied in the OS Settings, you will see a warning screen when loading into Assist. This screen will show which permissions have been denied that are required to continue. To gain access to Assist to join or start a session, click the Open Settings button to be taken to the OS Settings page and allow the required permissions. Then close and restart Assist to gain full access to the application.
Controls
Assist Controls
The Magic Leap 2 Controller is used to navigate Assist. The following actions are mapped to the Controller:
Trigger
- Select
- Grab Object (object targeted)
- Draw (Drawing Tool active)
- Place Pin (Pin Tool active)
- Place Arrow (Arrow Tool active)
- Erase (Eraser Tool active)
Bumper
- Take Photo (Photo Capture feature active)
- Drop Annotation Tool (Annotation Tool active)
- Open Object Context Menu (object targeted)
Menu
- Hide/Reveal Navigation Menu (Press)
- Rotate Following elements into view (Single Press with elements out of center view)
- Re-dock Assist Menu and sub-menus (Double Press)
- Summon all (Long Press, centers all unlocked elements into center view)
Home
- Pauses Assist and opens Magic Leap 2 Home Menu
Touchpad
- Up/Down
- Move Annotation Cursor (Annotation Tool active)
- Move Object (object selected)
- Left/Right
- Scale Object (object selected)
- Rotate
- Rotate Object (object selected)
Assist Menu
After Assist loads, you will see the Assist Menu appear.
The Assist Menu contains the following:
- About Assist
- Session
- Files
- Annotate
- Photo Capture
- Wi-Fi Signal
- Time
- Display Mesh On/Off
- Headset and Controller Battery Status
Call Control buttons are disabled when a session is not active. A session is considered active when you are connected with a web participant in a live chat.
When a session is active, the following Call Controls are accessible:
- Mute/Unmute
- Camera On/Off
- End Call
Headset and Controller Battery Status
The Assist Personal Menu will display the current status of both your ML2 Headset and your Controller. The two indicators will change based on the charge status (on battery or plugged in and charging) and the current battery charge level.
Activating Submenus
You can open and close submenus by selecting buttons on the Assist Menu. Point your Ray at a menu button and select it by pressing and releasing the Trigger.
Selecting About Assist, Files, Annotate, or Session reveals a corresponding panel above the menu. Pressing a button when it is active, closes the corresponding panel.
Only one menu button can be active at a single time (unless submenus are undocked, see: Undocking Submenus). For example, if you open the Session Panel and then select the Annotate Tools, the Session Panel will automatically close and the Annotate panel will open in its place.
Navigation
Configurable For Any Task
The Assist workspace is highly configurable and can be customized to compliment the task at hand. Your menu, submenu panels and opened files can be positioned to accommodate the physical constraints of your environment.
Moving Menus
To move the Assist Menu, hover over the menu with your Ray and target the Move Bar located below the center of the menu. With the Trigger pressed, move the Controller to reposition the menu.
Undocking Submenus
Submenus, such as Annotate, can be undocked from the menu. Hover over the submenu panel and select the Move Bar. With the Trigger pressed, move the Controller to undock the submenu from the menu. The undocked submenu panel can be positioned anywhere on the workspace.
To redock a submenu, target the Move Bar and move the submenu back over the Assist Menu. You can also double press the Menu Button on your Controller to re-dock all undocked submenus.
Multi-docking Submenus
Assist will dock multi windows either side by side or push active windows back to highlight temporary panels and submenus. There are two types of panels in Assist: Persistent panels and Temporary panels. Persistent panels are panels such as the session panel or file folder, while temporary panels are submenus and functions like annotations, settings menus, and image captures. When you open multiple persistent panels they will be docked side by side automatically. Closing one of the persistent panels will bring the remaining panel into center focus. If you have persistent panels open then any temporary panels that open will push the already open panels back and reduce their opacity to show the change in focus. Once the temporary panel is closed they will return to main focus and full opacity. Only one temporary panel can be open at a time. Trying to open another temporary panel will close the previously docked panel.
You can undock persistent and temporary panels to keep them open. Undocked persistent panels will see any remaining persistent panels shifted to center focus above the Assist Menu. Undocked temporary panels will have the persistent panels return to focus in the foreground and at full opacity.
Customizing Your Workspace
By undocking submenus, you can have multiple submenu panels open at a single time. This can be helpful, for example, if you would like to have both the Annotate panel and the Session Panel open at the same time to assist you in performing a task.
All menus and submenus can be organized around you and within the world to match the work you need to do. Elements (menus, submenus, objects, panels, and other displayed content) can be put into either Follow or Not Following mode to customize your working environment. Elements in Follow mode will become body-relative (moving with you) while elements in Not Following will be world-relative and stay in place. Objects, files, and panels can also be locked to the world or left unlocked to be easily moved around as needed..
Follow Mode
Assist Menus, submenus, and other elements have different modes when loaded into the world. These modes allow you to operate within the world and organize AR content as needed to accomplish your tasks.
The Follow mode makes elements body-relative, which means they will base their position and movement behavior based on your position and move with you.
Not Following Mode
The Not Following mode will leave elements in a world-relative state, which means their placement will be based on where they are placed in the world. Not Following elements will remain in place as you move around the world.
Objects summoned into the world can additionally be set to a locked state from within the Object Menu. Locked elements are world-relative but will not be moveable until unlocked. Locked elements will also not respond to Summon and Re-dock commands from the Controller.
Billboarding
When in Follow mode, 2D elements will orient towards you when in view as you move around. This behavior is known as Billboarding. An element set to Follow can be placed in a desired body-relative position and it will automatically rotate and angle itself towards you.
When set to Not Following, 2D elements will return to a default vertical orientation that is now world-relative and will not Billboard, though they can be rotated and adjusted as needed.
3D elements when set to Follow will not Billboard. They will retain either a body-relative or world-relative position and not orient automatically. They can still be adjusted as needed though.
Accessing Modes
The Follow function can also be accessed via the different menus in Assist. For the Assist Menu you can find the Follow Button on the Move Bar. For docked submenus the Follow Button is hidden until it is undocked. For objects, hover over an object to bring up the Object Menu, then select the expand menu button to show the full menu. Then the menu option to Follow or Not Follow will be shown. You can also access the Follow status in the Object Menu hover state like any other menu element.
Hovering over the Move Bar will show the Follow status and hovering the Follow button for 2 seconds will bring up a tooltip showing the current Follow status.
When a menu element is undocked from a parent menu it will inherit the Follow status from that menu element. Duplicated objects and files will also inherit the same Follow status from the original element.
Locking an object will deactivate the Follow function on that object. Setting a Locked object to Follow will unlock the object. Unlocking a locked object will leave that Object in a World-Relative state and Not Following.
Reorganizing and Resetting the Workspace
Sometimes elements can get lost after a long session or if there are a lot of menus and objects in a session. In order to reorganize, reset, or simply recenter your main workspace you can use the Menu Button on your Controller. The Menu Button allows you to Rotate, Hide, Re-Dock, and Summon UI and object elements within the scene so that you can easily find them or clean up the session.
Hide/Reveal
You can Hide or Reveal the Assist Menu as needed by pressing the Menu Button once.
Rotate
Elements set to Follow will maintain their body-relative position but won’t rotate with you. In order to quickly bring Following elements to the center of your view, press the Menu Button once. This will cause the Following elements to rotate while maintaining their body-relative position to you with the Assist Menu taking a central position while other elements maintain their relative position. Once centered pressing the Menu Button again will revert to the Hide/Reveal function.
Re-Dock
If you have many menus and submenus open you can quickly Re-dock all menus into the Assist Menu by double pressing the Menu Button. This will bring the Assist Menu to the center of your view and Re-dock all submenus. This will keep the last opened submenu visible or, if it is active, the video panel.
Summon All
To gather all elements up into a center view, perform a long press on the Menu button. This will bring all unlocked elements into a centered grid view, whether they are Following or Not Following. If an element is locked it will remain where it was placed. All menus, submenus, and objects summoned this way will be set to Follow. If a video window is open it will take priority over other elements and take the center most position above the Assist Menu, which will also always take the center position. Files and objects will be resized to fit within this grid view.
Hosting a Session - Device Participant
Collaborative Sessions
The heart of Assist is a live, collaborative session between a device participant and a web participant. The device participant initiates a session and generates a Session Code. The web participant joins via the Magic Leap Assist website by entering the Session Code.
Starting a Session
To begin hosting a session you must open the Session Panel (open by default when starting the app) and enter a name to begin the process. The Next button is disabled until the name field is populated.
Enter Device Participant Name
Clicking the name input field also opens the keyboard. You can manually enter a name by using the keyboard.
Hovering over the name input field may display a tooltip with the previous name used in the previous session. Clicking the name in the tooltip automatically populates the input field with that name.
Invite a Participant
Once a name is entered, the Next button is enabled. By selecting Next, a Session Code will be generated in the next step.
Multi Web User Session
Assist can support up to 4 web participants in a single session. In order to start a multi user session simply send the same invite code to more participants. After the first web participant joins you will need to accept each additional web participant’s join request before they will be added to the session.
Share Web URL and Session Code
You need to share the Assist web app URL: assist.magicleap.cloud (Link: Magic Leap Assist) and Session Code with the web participant. The web participant can then join the session by entering the Session Code in the Assist web app.
Mesh Your Space
The Session Panel prompts you to mesh their space while waiting for the invited web participant to request to join the session. They can do this by slowly looking around their environment, allowing the camera on the headset to start tracking and mapping their space. Building up a mesh of the space allows the web participant to see a 3D representation of your environment which they can use to navigate and annotate on.
The Assist web app URL and Session Code are displayed at the bottom of the panel in case they are further needed.
Join Request
When the web participant enters the session lobby, a join request notification appears in your Session Panel. If the Session Panel is closed, it automatically opens to display the join request. To reduce visual noise, during a session, the visual AR Mesh is automatically toggled off when you select Accept from the web participant’s join request. Additional join requests will appear as a minimal banner notification across the top of the screen grouping and will not interrupt the session in progress.
Session
Connect Live with Video Chat
When the session is connected, a live video and audio feed of the web participant appears in the Session Panel.
When a session is live, the web participant can hear an audio feed of you and see a camera feed broadcast from your Magic Leap 2. The live camera feed allows the web participant to see your environment from your perspective.
The live video chat continues even if you close the Session Panel.
Call Controls
During a session, the Call Control buttons on the Assist Menu are enabled. You can mute/unmute your audio, turn your camera feed on/off, and end the session call at any time.
Participant Controls
When you add participants to your Assist session you will have a set of controls for managing participant interactions. These controls are only available to the device host of the session.
- Hide Annotations - This allows you to hide any annotations added by that participant. You can turn this on and off as needed.
- Mute Participant - This will mute that participant. As device host you can only mute a participant, they must unmute themselves.
- Remove Participant - This will kick that participant from the current session. A confirmation message will pop up to ensure you meant to remove the selected participant before fully kicking them. Any annotations they had will then be reassigned to your control.
Participant Status
When there are participants in the Assist session you will see various status indicators. There are indicators for: Muted, Annotations Hidden, Camera off, and Screen Sharing (which disables the camera).
Audio Issues
If you can't hear audio when the web participant shares their screen then be sure that the web participant is:
- Using the Chrome browser to share the content (required for Assist)
- Sharing specifically a Chrome Tab (not a window or full screen share)
- Has clicked Share Tab Audio
Audio will only be shared if you're sharing a tab in Chrome (not window or screen) and have checked the box Share Tab Audio.
Wi-Fi Signal Indicator
When in a session the Wi-Fi signal strength will be displayed on the Assist Menu as shown above. This indicator will relay the current signal strength of your Wi-Fi connection. This feature requires enabling the Location permission in the Assist OS level Settings. The Wi-Fi indicator can be clicked, which will bring you to the Wi-Fi settings menu.
If your Wi-Fi signal becomes reduced to the point that video performance will suffer, a warning window will appear warning you to return to a location closer to your Wi-Fi signal source. To stop seeing these messages, should signal issues persist, click the Do not prompt this again option and click OK.
If you completely lose Wi-Fi connection then a notification will appear prompting you to reconnect to Wi-Fi. Click on the Open Settings button to be taken to the OS Settings page to reconnect to a Wi-Fi network.
End Session
When you wish to exit a session and end the call you can press the End Call button in the bottom right of the Assist Menu, the red phone icon. Selecting this button will bring up an Exit Session prompt.
Ending the session as a device participant will end the session completely for the Web participant. To enter a new session both participants must go through the whole Start Session flow again.
Session Disconnect Notification
If the session encounters an error or a participant leaves then a Session Disconnection notification will appear for the device participant still in the session.
If the web participant left the session by selecting the Exit Session button the notification will give the device participant the option to either exit the session themselves or remain in the session temporarily. If the device participant leaves the session then the notification will still show for the web participant, however there will be no option to remain in the session.
Selecting Exit Session will immediately return the participant to the session starting screen. Selecting Stay in Session will leave the participant in the session for 18 seconds while the message “Web participant has left the session” is displayed on the session page in case the web participant attempts to rejoin the session. After 18 seconds the participant will automatically be returned to the session starting screen.
If a participant was disconnected due to a server error the participant will be automatically returned to the session start page with a notification alerting them to a server error.
Select the OK button or the X to clear the notification and attempt to start or join a new session.
If the session is interrupted by network errors then a notification featuring a 18 second countdown will appear.
During this 18 second period the participant can attempt to reconnect to their Wi-Fi network by selecting Open Settings, exit the session by selecting Exit Session, or wait to see if the interruption clears up by itself. If the connection is re-established during this 18 second period the notification will clear itself automatically and alert the participant of the reconnection.
If the session does not reconnect before the countdown ends then a Session Disconnection notification will display alerting the participants.
Select OK to return to the Session Start page.
Port Forwarding
In some situations when on corporate or company networks you may need to have your network admin or IT department allow for the following ports to be forwarded and open:
- 443 TCP (HTTPS)
- 3478 UDP, 5060-5061 TCP, 5060 UDP (WebRTC)
- ICE server requires UDP ports above 50000
This is not required for S0H0 or home routers.
Photo Capture
Photo Capture
This feature allows you to take a photo of the session, including any 3D content in view, and to share that photo immediately with the web participant for review. The photo is also saved to the Files Folder, which is visible to you and web participants.
On entering Photo Capture, you will be presented with a camera reticle showing what content will be captured once a photo is taken.
On pressing the Bumper, a photo is taken and processed. Once processed a preview of the photo is displayed to you. The preview contains the photo, photo name prefix, and a notification that the photo has been shared with the web participant.
The photo preview will remain for a short moment and then fade away. In order to dismiss the preview immediately you can press the Bumper or Trigger when not targeting anything in view.
Once cleared, you will return immediately to the Photo Capture reticle view to be able to take another photo.
Files Folder
The Files Folder can be accessed by selecting the Files button from the Assist Menu. The Files Folder contains a library of sample files including 3D objects and 2D assets. The Files Folder also contains photos taken with the Photo Capture feature.
Opening Files
Files can be opened by dragging and dropping them into your immediate environment.
Hover over the file and press the Trigger to select it. While holding the Trigger, drag the file to your physical environment, and then release the Trigger. The file opens where it is dropped.
Deleting Files
To delete files uploaded to the Assist session or photo and video captures taken during the session, first navigate to Files from the Assist Menu. Here all files that are active in the session can be found.
Each file is represented in a grid view with a More button. Selecting More will present additional options for that file. From the More option you can Open the file into the session or Delete it from the session. Deleting a file will remove it from the session for both the Device and Web participant.
Multiple files can also be selected for deletion by using the Edit button in the upper right of the Files Folder menu.
To delete multiple files, select Edit and then select each file you wish to delete. After selecting multiple files the Edit button will change into a Delete button. Once all the files you want to delete have been selected you can select Delete to delete all of them at once. A warning prompt will appear letting you know that these files will be permanently removed from the session. Select Delete to continue or Cancel to go back and select different files.
Sample files cannot be deleted from the session and deleted files cannot be recovered once deleted and must be re-uploaded.
Objects & Files
Receiving Objects and Files
The web participant can send you objects and files such as schematics, instructions, or 3D models to help assist you with tasks in your environment. When the web participant sends you a file, a notification displays in AR view. Select accept and the file will appear as an object in your environment.
Transforming Objects
Assist has an in-depth object control and transform system which utilizes various object states and transform handles. With the object transforms you can move, scale, rotate, and for some objects perform a non-uniform scale.
Select States
Objects in Assist have several states when it comes to interacting with them. These states are: Default, Hover, Select, and Active.
Default State
When an object is in the Assist scene, not targeted nor selected, it is in the default state.
Hover State
When targeting an object it will enter the Hover State. When in the Hover State an object will display gray transform handles showing the bounding box for the object and the Object Menu button will appear.
Select State
Selecting an object with the Trigger will cause it to enter the Select State. In the Select State you will see the transform handles around the object allowing you to Scale, Rotate, and (for primitives) Non-Uniform Scale. The transform handles only show on the side of the object you are targeting. You will also see the Object Menu button. The object will remain in the Select state until you either: Select a different object, pull the Trigger with the Ray off of the object, Pull the trigger with the Ray on the object but without selecting any of the menu or transform handles, or if the selection times out after 4 seconds.
Active State
Transforming or moving an object will make it enter the Active State. When in the active state other participants cannot interact with the object. While the object is being moved or transformed the transformation handles will disappear except for the handle being used which will be highlighted in blue. The Active State persists until the object or handle is released, it will then return to the Select State.
Move Objects
To move an object, target it, press and hold the Trigger, and move the Controller. Release the Trigger to place the object.
Scale Objects
To scale an object, target the object and select it to enter the Select State. Once in the Select State use the corner handles of the bounding box to scale the object up or down. Target the corner handles, press and hold the Trigger and move the Controller to scale the object. Release the Trigger to set the scale. Objects can also be scaled by selecting them and sliding up or down on the touchpad.
Rotate Objects
To rotate an object, select the object to enter the Select State. Once in the Select State use the vertical side handles to rotate the object. Target the handle with the Ray, press and hold the Trigger, and move the Controller to rotate the object. Release the Trigger to set the rotation. Objects can also be rotated by selecting the object and sliding left or right on the touchpad.
Non-Uniform Scale
For primitive objects you can also perform a non-uniform scale where you can squash and stretch an object along one or more axes. To perform a non-uniform scale target and select a primitive to bring up the Select State. On the bounding box you will see a handle in the center of the bounding box. Select that handle and hold the Trigger then move the Controller to perform an non-uniform scale. Release the Trigger to set the scale.
Annotations
Annotations can also be transformed just like objects. Just like with objects, target the annotation to bring them into a Hover State to see if you can transform the annotation.
Object Cursor
Direct Object Transformation
The Object Cursor appears when the Controller is in close proximity to an object.
The Object Cursor gives you the ability to move and rotate digital objects as if they were physical objects.
Objects can be directly transformed by placing your Controller on, or in close proximity to, the digital object. When your Controller is close to the object, the Ray is replaced with the Object Cursor. Press and hold the Trigger to grab the object. While holding the Trigger you can move or rotate the object free-form, as if you were holding it in your hand. Release the Trigger to place the object.
Object Menu
Object Menu
All objects have an Object Menu. The Object Menu can be revealed by hovering over or selecting the Object. You can access the Object Menu that appears when targeting an object by selecting the Object Menu Button. The Object Menu can also be accessed by targeting the object and pressing the Bumper. The Object Menu contains the following options:
- Duplicate
- Lock
- Actual Scale
- Delete
Additional options may be available for specific object types.
Duplicate
Selecting Duplicate creates an identical copy of the object. The duplicate object appears next to the original object.
Lock
Objects can be locked or unlocked. By default objects are unlocked. When an object is locked, it cannot be moved. Locked objects have a lock symbol appear next to the Object Menu.
Actual Scale
Selecting Actual Scale transforms the object to its true size. The object’s true size is not always the same as its spawn size.
Delete
Selecting Delete permanently removes that instance of the object. Deleting an open object does not remove the object from Files or from the File Manager.
Object Controls
Object Controls
Some files such as videos and PDFs have Object Controls that appear next to the Object Menu Button. Object Controls are revealed when targeting an object.
Video
Object Controls for videos include a Play/Pause Button, Mute/Unmute Button, and a Progress Bar. The Progress Bar can be scrubbed to jump to a specific moment in the video.
Object Controls for PDFs include Previous and Next. The current page and total number of pages in the PDF are displayed.
Animation Controls
FBX models which have baked in animations can be played similarly to a video object. After spawning the 3D model with animations you can access the models animation controller by [Need to double check this flow]. Animation controls allow you to play and pause animations, loop the animation, select the next animation, and select a previous animation. Changing between animations is done by pressing Next or Previous. There is no menu for selecting a specific animation, just the Next and Previous buttons to scroll through all baked in animations present already in the model.
Object Shortcuts
Shortcuts
The Controller has incorporated various shortcuts to optimize your productivity.
Object Menu
Press the Bumper while targeting an object to open the Object Menu.
Quick Duplicate
Press and hold the Bumper while dragging an object to Quick Duplicate.
Quick Delete
Double tap the Bumper while targeting an object to Quick Delete.
Annotate
Mark Up Your Space
Annotations are visual communication tools. Both you and the web participant can annotate directly in your space using digital tools.
Annotation tools can be accessed by opening the Annotate submenu panel from the Assist Menu. The following annotation tools are available: Draw, Pin, Arrow, Eraser, and Color Palette.
You can place arrows to direct attention or guide someone through a large space. You can use pins to mark components or label detailed parts on a piece of machinery. You can use drawings to highlight important areas or communicate free-form.
Using Annotation Tools
When an annotation tool is active, the Ray is replaced with a custom Annotation Cursor. A label is mapped to the Touchpad of the Controller displaying the name of the active tool (ex: Draw, Pin, Arrow, Eraser).
You can drop an active tool at any time by pressing the Bumper on the Controller. An active tool can also be dropped or unselected by selecting the tool button a second time.
Swipe to Customize Cursor Position
The Annotation Cursor position in your environment can be adjusted. Swipe up on the Touchpad to move the Cursor farther away from the Controller. Swipe down on the Touchpad to move the Cursor closer to the Controller. Adjusting the position can help extend your reach.
Color Palette
The Color Palette allows you to select a color for your annotations. The selected color is used when creating drawings, placing pins, or placing arrows.
The current active color is displayed at the center of the Color Palette wheel and is reflected in the tool icons.
When the Color Palette is active, it replaces the Annotate panel. Select the X icon to close the Color Palette.
Draw Tool
Drawing tools can be used to create freeform lines in space. There are two drawing tools available: Draw Thick and Draw Thin.
When the Draw Tool is active, a crosshair appears as the Annotation Cursor floating in front of your Controller.
Place the Cursor point where you would like to start drawing. Then press and hold the Trigger while moving the Controller to draw. Releasing the Trigger completes the drawing.
Pin Tool
A pin is a digital marker with a label that can be used to identify objects in your space.
When the Pin Tool is active, a crosshair appears as the Annotation Cursor floating in front of your Controller. Place the Cursor where you would like the pin tip to point. Then press and release the Trigger to place the pin. Hover over the pin and select the form field to add a label.
The label is a helpful communication feature that is unique to pins. Imagine using pins to identify and label small parts on a piece of machinery.
Arrow Tool
Arrows are used to direct attention towards points of interest and as a wayfinding mechanism.
When the Arrow Tool is active, a crosshair appears as the Annotation Cursor floating in front of your Controller. Place the Cursor where you would like the arrow tip to point. When you press the Trigger, the arrow appears with the point originating at the crosshair center. While holding the Trigger, you can move the Controller to change the arrow’s length and direction. Releasing the Trigger places the arrow.
Eraser Tool
The Eraser Tool can be used to delete individual objects.
When the Eraser Tool is active, a crosshair appears as the Annotation Cursor floating in front of your Controller. Target individual pins, drawings, or arrows by hovering over the annotation. When the annotation is targeted, the crosshair turns red. Press and release the Trigger to erase the targeted annotation.
Delete All Annotations
Delete All (trash icon) removes all annotations from the session including annotations that were created by you and annotations created by the web participant.
Selecting the Delete All button displays a confirmation message dialogue box. Annotations will not be deleted until confirmed through the dialogue box.
Hide Annotations
Device participants can hide the annotations of the web participants…
About Assist
About Assist
About Assist displays application version information, copyright information and provides access to other features such as a short tutorial or in-app settings.
Overview
The tutorial can be accessed at any time through the About Assist panel. In the submenu, the tutorial tab is active by default when this panel opens.
To view the tutorial, select Start Overview.
Magic Leap Spectator
Assist is compatible with Magic Leap Spectator, a mobile application that allows participants to view streamed AR content on their mobile devices when in an Assist Session. To enable Magic Leap Spectator connectivity toggle the option Allow Magic Leap Spectator Connectivity.
When enabled this menu also allows the device participant to set a unique identifier for their headset when trying to connect with Magic Leap Spectator which will be displayed on the connection screen in the mobile app.
Developer Logs
Logs display technical and performance information. Logs are intended for developer use.
Magic Leap Spectator
Magic Leap Spectator is a mobile application that makes it easy for anyone to capture and share high quality mixed reality content streamed from a Magic Leap 2 device. ML Spectator supports a wide range of use cases including: Creating video documentation, broadcasting sessions sessions for an audience, acting as a seconds screen for demos, and more. For full details on how to use Magic Leap Spectator see: Link
To connect ML Spectator to your Assist session first follow the connection instructions in the Magic Leap Spectator mobile app. For detailed instruction on connecting ML Spectator to your Assist session see: Connect to a Magic Leap 2 Device.
Assist Web Application
Assist Web App Setup
Locate A Desktop or Laptop Computer
The Assist web application requires a desktop or laptop computer. Tablets and mobile devices are not supported. Verify that your computer is connected to the internet.
Open Web Browser
On your computer, open up a web browser. Chrome is the preferred browser for Magic Leap Assist.
Enter Assist Web Portal URL
Go to assist.magicleap.cloud to begin joining a session.
Joining a Session - Web Participant
Joining a Session
You will be prompted to enter your name. The Next button remains disabled if the name field is empty. After entering a name, click Next to continue.
Enter Session Code
You can enter the Session Code given by the device participant. Click the Next button to validate your Session Code.
Sessions with Multiple Users
If the session being joined has multiple web users, then you must wait for the session host to accept your join request or for an available slot to open up. Assist sessions can only accommodate up to 4 web users at a time. Once a slot opens up and/or the host accepts your request you will be brought into the session.
Camera Preview Page
Once the code is validated, you can proceed to the camera preview page. Here you are able to preview your Camera View and adjust their camera and audio settings. When they are ready, you can click Join Session to send a request to the device participant to join the session.
Loading Session
A loading animation is displayed in the Session Panel while the connection is being established once the device participant accepts the request.
Session
Live Collaboration
In a session you are connected directly with the device participant. The device participant can hear you and see a live video feed of you. You can hear the device participant and see a live camera feed streamed from the device participant’s Magic Leap 2.
In a live session you can:
- Video call with the device participant
- Interact with a 3D mesh of the device participant’s space
- See a live camera feed from the device participant’s Magic Leap 2
- Annotate the device participant’s space
- Share your screen
- Send files to the device participant
- Receive files from the device participant
Mesh Window
The Mesh Window displays a 3D representation of the device participant’s space. You can interact with the mesh to examine the device participant’s environment by zooming, panning, and orbiting. Within the Mesh Window you will see a pyramid shaped icon that represents where the device participant is looking and what their field of view is.
Headset Feed
The headset feed displays a live camera feed from the device participant’s Magic Leap 2, allowing you to see what the device participant is seeing. The headset feed can be paused by clicking the pause button in the headset feed Control Bar.
Participant Feed
In a multi web user session an additional panel will display the other web users camera feeds. This panel will automatically update and scale the panel for 2/3/4 users. Hovering over a user’s camera feed will bring up controls for that feed: Mute, Hide/Show Annotations, and Maximize.
Note: Once a user is muted you will not be able to unmute them, the user themselves must unmute their microphone. This is done for security reasons.
Controls
The Control Bar located at the bottom of the web app contains the following: About Assist, Your Video Window, Mute/Unmute Microphone, Hide/Show Video, Screen Share, Files, Session information, and Exit Session.
Additional controls can be revealed by hovering over individual windows, as well as in the top left corner of the screen (headset icon). This icon allows you to directly select a specific window to maximize. Annotation tools are located in the device participant’s camera window allowing you to draw, add pins, or clear annotations.
Toolbar
The Toolbar located on the right side of the web app contains the following: Select, Draw, Pin, Color Palette, Eraser, and Delete All.
Session Information
The Session menu brings up a pop-up window with the Session code, all participants, and their annotation color coding. Next to the name of each participant will also be an icon indicating if they are the device user (headset icon) or a web user (PC icon). The session code has a copy button on the right that will copy the Session code for easy sharing.
End Session
When you wish to exit a session and end the call you can press the End Call button in the bottom right of the Control Bar, the red phone icon. Selecting this button will bring up an Exit Session prompt.
To finish exiting the session select Exit from the prompt. To continue with the session select Cancel. After leaving or ending a session you will be prompted to join a new session if you want. If not, you can exit the browser or tab.
Annotate
Annotations
Annotations are visual communication tools. You can directly augment the device participant’s environment by drawing or placing pins in the Mesh Window or Camera Window. Annotations you create will immediately appear in the device participant’s environment.
Annotation Mode
To annotate in the Mesh Window, select an annotation tool, hover over the Mesh Window, and click to annotate. This will activate Annotation mode and invite any other web participants to join in adding annotations. While the annotation tools are active you can still navigate within the Mesh Window with the following controls:
- CTRL + Click - Look around
- ALT/OPTION + Click - Zoom
- SHIFT + Click - Pan
You can bring up this list of commands by clicking the information (small i) icon in the Mesh Window. The available navigation tools will also appear as a tool bar in the Mesh Window when it is in focus.
Annotating In The Camera Window
To annotate in the Camera Window the camera feed must be paused. Once you select an annotation tool, your cursor will turn into a pause icon. Click on the camera window or mesh window to pause it and begin annotating.Then select an annotation tool and click directly in the Camera Window to annotate. Annotations created in the Camera Window will also be visible in the Mesh Window. Once finished annotating, click the Done button at the top of the window to return to a live camera feed and exit Annotation mode. In multi web user sessions you will need to unhide your annotations if they are hidden to continue annotating. If a web participant is in Annotation mode, the option to hide their annotations is disabled until they exit annotation mode.
Draw Tool
The Draw Tool can be used to create free-form lines. When the Draw Tool is active, your Cursor appears as a crosshair when hovering over the Mesh Window or Camera Window. Place the crosshair where you would like to start drawing. Click and drag to draw. Releasing the click completes the drawing.
Pin Tool
Pins can be used to mark and label specific parts or objects. When the Pin Tool is active, your Cursor appears as a crosshair when hovering over the Mesh Window or Camera Window. Place the crosshair where you would like the pin tip to point. Then click and release to place the pin. Hover over the pin and select the form field to add a label. Adding or editing a label to a pin only works in the Mesh Window.
Annotation Color
The web participant’s annotation color is determined by their join order and cannot be changed. Your annotation color is indicated by a dot next to your name tag.
Eraser Tool
The Eraser Tool can be used to delete individual objects in the Mesh Window. Erasing only works in the Mesh Window and not in the Camera Window. When the Eraser Tool is active, your Cursor appears as a crosshair when hovering over the Mesh Window. The crosshair rotates and turns red when hovering over a targeted drawing or pin. With the annotation targeted, click to erase the annotation.
Delete Annotations
Selecting the Trash icon deletes all annotations from the session including annotations created by you, other web participants, and the device participant.
Screen Sharing
Share your Screen
You can share your screen with the device participant by selecting the screen share button from the Control Bar. This allows you to share your entire screen, a specific window, or an individual tab in a web browser. When clicking on the screen share button a pop-up will appear with available sharing sources listed. Once a screen is selected to be shared the window will maximize and begin to stream its content to the device participant with the shared content window displaying the shared content and highlighted in yellow.
While sharing your screen you can click away from the shared content window by clicking on the expand icon on a different window. When doing this the shared content window will minimize while the selected window will maximize. The shared content window will remain highlighted in yellow when minimized and the stop sharing button will still be available. When sharing your screen the participant view on the device side will be minimized down into an icon to save on bandwidth.
The window layout will not change when opening the File Drawer. The content windows will simply adjust in size to accommodate the File Drawer.
End Screen Share
To end the screen share, select the Stop Sharing button from the top of the screen share window. .
Audio issues with Screen Sharing
If the device participant can't hear audio when sharing an audio source or video with audio then be sure that you are:
- Using the Chrome browser to share the content (required for Assist)
- Sharing specifically a Chrome Tab (not a window or full screen share)
- Has clicked Share Tab Audio
Audio will only be shared if you're sharing a tab in Chrome (not window or screen) and have checked the box Share Tab Audio.
Uploading Files
File Manager
The File Manager is a quick and easy file management tool for uploading and sharing your files with the device participant. It contains an Upload button, sample files, Photo Captures and any already uploaded files you have available in the session. To open the File Drawer click the Files button on the Control Bar at the bottom of the screen.
File Order
Files are shown newest to oldest, top to bottom, showing uploaded files and then captured images and videos in the File Manager. The most recently uploaded file will always be on the top of the File Drawer. If a lot of files have been uploaded during a session, and the File Drawer is full, you can navigate up and down using the arrows on the sides of the File Drawer.
Photo Captures
Any images taken during the session by the device participant using Photo Capture will show up in the File Drawer. They will be ordered newest to oldest, top to bottom, just like any other file uploaded to the session.
New File Notification
When new files are uploaded, a yellow dot will appear by the Files button on the Control Bar, if it is not already open. With the File Drawer open new files that have been uploaded but not interacted with yet will also have a yellow dot next to their name. New files uploaded will need to be downloaded before they can be viewed and spawned. Files requiring downloading will have a download icon appearing over their preview image.
Sharing Files
Files uploaded into the File Drawer on the web app are not automatically shared with the device participant (i.e, they will not see the files in their local File Folder). In order to share a file from the File Drawer with the device participant, simply click and drag the file into the Camera or Mesh View Windows. The content of the File Drawer is then shared between you and device participants.
Uploading
You can upload multiple files simultaneously, though folders cannot be uploaded. Any incompatible files are grayed out.
The file(s) will be visible in the File Drawer once the uploading process begins. An uploading animation will appear to inform you of the upload status. Files that are still uploading cannot be opened or displayed.
Once files have been successfully uploaded, the thumbnail transforms into a preview of the image or object, and an upload confirmation icon will appear momentarily.
Deleting Files
To delete files uploaded to the Assist session or Photo and Video Captures taken during the session, first navigate to the Files Folder from the Assist Menu. Here all files that are active in the session can be found.
Each file is represented in a list view with a More button. Selecting More will present additional options for that file. From the More option you can Open the file into the session or Delete it from the session. Deleting a file will remove it from the session for both the Device and Web participant.
Sample files cannot be deleted from the session and deleted files cannot be recovered once deleted and must be reuploaded.
Deleted File Notification
When a file is deleted you will receive a notification alerting you which file was deleted and who deleted the file.
Upload Errors
If an error occurs during uploading, an alert will appear and display the reason for the failure. The file will be automatically deleted from File Drawer after a short pause.
File Limitations
If you select a file from the local directory that exceeds the maximum allowed file size or that is an unsupported file type, an alert will appear to notify you that it cannot be uploaded. The upload will not begin and the file will not appear in the File Drawer.
The error box will close when you dismiss it or when you attempt to upload a new file/group of files. It will also disappear on its own accord, after a short pause.
See: File Specifications for details about allowed file types and sizes.
File Specifications
Compatible File Types
- Portable Document Format (PDF)
- Images (JPG, JPEG, PNG)
- 3D Models (GLB, GLTF, FBX, 3DS, OBJ)
- Video (MP4, TS, MKV, 3GP, WEBM)
Maximum File Size
- PDFs: Up to 250MB
- Images: Up to 50MB
- 3D Models: Up to 75MB
- Video: Up to 1GB
File Storage
Files loaded into an Assist session will be temporarily stored in RAM and loaded as active elements are needed. At the end of an Assist session the uploaded files will be cleared from the ML2 headset’s local storage.
Preparing 3D Models
3D Models loaded into an Assist session require some preparation to ensure proper performance. Models should limit the total number of polygons that the object is composed from. There is an upper limit of 1 million polygons for 3D models of any file type. Loading high detail models will cause performance to slow and responsiveness to suffer as the upper limit is approached. If a high detail model must be loaded, reduce the number of other models being displayed on the canvas. If complex scenes need to be displayed ensure they are populated with low detail models.
Recommendations:
- Use as few materials as possible (5 or fewer)
- Reduce texture resolutions where possible (2K resolution or lower)
- Reduce total number of Tris where possible (150k maximum)
- Avoid 3D model scenes featuring too many elements. Load individual 3D models or reduce the total scene complexity to meet the above limits
Preparing Images
Images are automatically resized to fit within a 2048 x 2048 pixel display size within Assist. Files exceeding this size will be converted and compressed to fit within that size limit while preserving their aspect ratio. Images exceeding 6000 x 6000 pixels will not be accepted. If your images exceed this size limit you will need to manually resize them to be smaller than 6000 x 6000 pixels.
Recommendations:
- Keep image size below 6000 x 6000 pixels maximum
- Images are compressed upon upload, if compression may be an issue consider using a lossless image file type like PNG over JPG or JPEG
In-App Guides
Interaction Guide
In the Sample Assets located in the File Manager is a helpful PDF detailing how to use the Magic Leap Controller when working in Magic Leap Assist. The PDF can be referred to as needed, either during a session or solo.
Quick Start Guide
A quick start guide is included in the sample files which will help orient new users on how to use Magic Leap Assist.