Assist is a peer-to-peer app with cross-platform capabilities. AR-enabled remote assistance can increase efficiency and accuracy of key scenarios for enterprise including training, inspection, assembly, maintenance, diagnosis and repair.

Some key features of Assist:

• Live meshing enables spatial awareness and collaboration features
• Experts can see what you’re seeing and provide live assistance
• Collaborate with live annotation tools: label, point arrows and draw
• Workspace moves with you and adapts to your work environment

Assist is available through the "Featured Apps" menu option available starting with Magic Leap OS 1.1.0. Note: "Featured Apps" opens a webpage in the browser. You may need to enable the "Install unknown apps" permission for the browser to install Assist after downloading it.

Refer to this article for additional questions and troubleshooting tips for Assist.

Assist on Magic Leap 2

Installing Assist

Home Menu

A home menu will display when the Magic Leap 2 powers on. Use the controller to interact with the menu. Your controller will cast a ray to show where it is pointed.

Select Featured Apps

Scroll through the home menu to locate Featured Apps. Point your controller ray at Featured Apps and press the trigger button to select it. A web browser will appear displaying featured applications for the Magic Leap 2.

Install Assist

Locate and select Assist from the list of featured applications. Selecting Assist will download and install the application. When Assist is finished downloading, it will appear as an item in the home menu.

Launch Assist

Locate Assist in the home menu. Select Assist to launch the application.

Controls

The Magic Leap 2 controller is used to navigate Assist. The following actions are mapped to the controller:

Trigger Button

• Select
• Grab Object (Object Targeted)
• Draw (Drawing Tool Active)
• Pin (Pin Tool Active)
• Place Arrow (Arrow Tool Active)
• Erase (Eraser Tool Active)

Bumper Button

• Take Photo (Photo Capture Feature Active)
• Drop Annotation Tool (Annotation Tool Active)
• Open Object Context Menu (Object Targeted)

Menu Button

• Hide/Reveal Navigation Menu

Home Button

• Pauses Assist and opens Magic Leap 2 Home Menu

Touchpad

• Up/Down
  • Move Annotation Cursor (Annotation Tool Active)
  • Move Object (Object Selected)
• Left/Right
  • Scale Object (Object Selected)
  • Rotate
  • Rotate Object (Object Selected)

Menu

After Assist loads, you will see the menu appear. For first time users, this happens after a brief tutorial.

The menu contains the following:

• Settings
• Files
• Photo Capture
• Annotate
• Session
• Mesh On/Off
• Call Controls

Call control buttons are disabled when a session is not active. A session is considered active when you are connected with a web-user in a live chat.

When a session is active, the following call controls are accessible:

• Mute/Unmute
• Camera On/Off
• End Call

Activating Submenus

You can open and close submenus by selecting buttons on the menu. Point your controller ray at a menu button and select it by pressing and releasing the trigger button.

Selecting Session, Annotate, Files, or Assist reveals a corresponding panel above the menu. Pressing a button when it is active, closes the corresponding panel.

Only one menu button can be active at a single time (unless submenus are undocked, see Undocking Submenus). For example, if you open the Session panel and then select the Annotate button, the Session panel will automatically close and the Annotate panel will open in its place.

Navigation

The Workspace

The Workspace is a body-relative, invisible sphere that houses your menu, tools, and files.

The Workspace travels with you, acting as a mobile workstation, providing easy access to your menu, tools and files regardless of where you are. You are always at the center of the Workspace and when you move, it moves with you.

Configurable for Any Task

The Workspace is highly configurable and can be customized to compliment the task at hand. Your menu, submenu panels, and files can be positioned to accommodate the physical constraints of your environment (see 7.0 Customizing Navigation).

Customizing Navigation

Moving The Menu

To move the menu, hover over the menu with your controller ray and select the move button located below the center of the menu. With the trigger button pressed, move the controller to reposition the menu.

When moving the menu around the Workspace, a dot pattern is visible indicating the surface of the Workspace.

Undocking Submenus

Submenus, such as Annotate, can be undocked from the menu. Hover over the submenu panel and select the move button. With the trigger button pressed, move the controller to undock the submenu from the menu.

The undocked submenu panel can be positioned anywhere on the Workspace.

Customizing Your Workspace

By undocking submenus, you can have multiple submenu panels open at a single time. This can be helpful, for example, if you would like to have both the Annotate panel and the Session panel open at the same time to assist you in performing a task.

Body-Relative or World-Relative

By default, your menu and files are body-relative and attached to the Workspace. Your menu and files can be made world-relative by undocking them from the Workspace and placing them in a world-relative position. If an object is world-relative, then it remains in a fixed position in your environment rather than traveling with you on the Workspace.

To make your menu (or a submenu panel) world-relative, hover over the menu and select the move button. With the trigger button pressed, push or pull the menu off of the Workspace sphere. As the menu detaches from the Workspace, you will notice that the menu is no longer surrounded by dots. When the menu is undocked from the Workspace, it remains world-relative in a fixed position in the environment.

Resetting the Menu

You can hide or reset the menu by pressing the menu button on the controller. Pressing the menu button once hides the menu. Pressing the menu button a second time resets the menu. Resetting the menu automatically closes and docks any submenus that may have been active or undocked.

Start A Session

Invite A Participant

The heart of Assist is a live, collaborative session between you and a web-user. You are the session host and the web-user is the participant. As the host, you initiate the session by inviting a participant.

Enter the web-user’s email address by selecting the Participant Email field. A keyboard will open. Use the keyboard to fill in the email address. Select done to close the keyboard. Select the Your Name field, enter your name, and then close the keyboard.

When both fields are complete, select the Invite button. A session invitation will be emailed to the web-user.

Invitation Confirmation

A confirmation message is displayed letting you know the invitation has been sent. Select the Next button to proceed.

If you need to resend the invitation or send an invitation to a new participant, select the Send New Invitation button.

Scan Your Environment

Scan your environment by looking around in all directions. Scanning creates a digital mesh of your environment that is shared with the web-user in the Assist Web App.

The mesh visualization is automatically turned on when advancing to this panel. You can turn the mesh visualization off by selecting the Mesh button in the menu. Scanning continues in the background even if the mesh view is turned off.

Accept Join Request

A notification appears when the web-user requests to join the session. You must accept the web-user’s request for the session to begin.

Session

Connect Live with Video Chat

When the session is connected, a live video and audio feed of the web-user appears in the Session panel.

When a session is live, the web-user can hear an audio feed of you and see a camera feed broadcast from your Magic Leap 2. The live camera feed allows the web-user to see your environment from your perspective.

The live video chat continues even if you close the Session panel.

Call Controls

During a session, the call control buttons on the menu are enabled. You can mute/unmute your audio, turn your camera feed on/off, and end the session call at any time.

Objects & Files

Receiving Objects and Files

The web-user can send you objects and files such as schematics, instructions, or 3D models to help assist you with tasks in your environment.

When the web-user sends you a file, a notification displays on the Workspace. Select accept and the file will appear.

Object Menu

The Object Menu displays information about the object and provides access to actions such as delete and scale to actual size. To open the Object Menu, hover over the object and select the menu icon. You can also open the Object Menu by hovering over the object and pressing the bumper button.

Animating Special Objects

Special objects can be disassembled, allowing you to see all of the object's parts and better understand the object’s construction. To disassemble or assemble a special object, hover over the object and select the Assemble/Disassemble button or open the Object Menu and select Assemble/Disassemble.

Move, Scale and Rotate Objects

To move an object, hover over it, press the trigger button, and move the controller while holding the trigger button. Release the trigger button to place the object.

To scale an object, hover over the object, press and hold the trigger button, and press left on the touchpad to scale the object down or right on the touchpad to scale it up.

To rotate an object, select the object, and rotate clockwise or counterclockwise on the touchpad while holding the trigger button.

Direct Object Manipulation

Direct object manipulation gives you the ability to move and rotate digital objects as if they were physical objects.

Objects can be directly manipulated by placing your controller on, or in close proximity to, the digital object. When your controller is close to the object, the controller ray is replaced with a special cursor. Press and hold the trigger button to grab the object. While holding the trigger button you can move or rotate the object free-form, as if you were holding it in your hand. Release the trigger button to place the object.

Files Folder

The files folder can be accessed by selecting the Files button from the menu. The files folder contains a library of pre-loaded 3D objects and 2D assets. The files folder also contains photos taken with the capture feature.

Opening Files

Files can be opened by dragging and dropping them onto the Workspace.

Hover over the file and press the trigger button to select it. While holding the trigger button, drag the file onto the Workspace, and then release the trigger button. The file opens where it is dropped.

Photo Capture

Photo Capture Feature

The photo capture feature allows you to take photos during a session. To activate the photo capture feature, select the Capture button on the menu. When the photo capture feature is active, a head-locked viewfinder appears.

Taking Photos

Look at the area where you would like to capture a photo. Press the bumper button to take a photo. The photo captures both the real-world and digital content within the viewfinder. The photo is automatically saved in the files folder.

The photo capture feature is only available when a session is active and when your video feed is on.

To exit the capture feature, select the Capture button in the menu.

Photos Folder

Accessing Photos

Open the files folder by selecting the Files button from the menu. Select the Photos folder in the files panel. The Photos folder contains all photos created using the capture feature.

Photos

A preview image is displayed for each photo. Photos are named with the time and date they were taken. Photos can be opened by dragging and dropping them onto the Workspace.

Sharing Photos

Photos generated with the capture feature can be sent to the web-user. A share icon appears on each photo thumbnail.

Select the share icon to send the photo to the web-user. A dialogue box will display confirming you would like to send the photo.

Annotate

Mark Up Your Space

Annotations are visual communication tools. Both you and the web-user can annotate directly in your space using digital tools.

Annotation tools can be accessed by opening the Annotate submenu panel. The following annotation tools are available: Draw, Pin, Arrow, Eraser, and Color Palette.

You can place arrows to direct attention or guide someone through a large space. You can use pins to mark components or label detailed parts on a piece of machinery. You can use drawings to highlight important areas or communicate
free-form.

Using Annotation Tools

When an annotation tool is active, the controller ray is replaced with a custom cursor. A label is mapped to the touchpad of the controller displaying the name of the active tool (ex: Draw, Pin, Arrow, Eraser).

The user can “drop” an active tool at any time by pressing the bumper button on the controller. An active tool can also be “dropped” or unselected by selecting the tool button a second time.

Swipe to Customize Cursor Position

The custom cursor (ex: crosshair) position can be adjusted. Swipe up on the trackpad to move the cursor farther away from the controller. Swipe down on the trackpad to move the cursor closer to the controller. Adjusting the position can help extend your reach.

Color Palette

The color pallet allows you to select a color for your annotations. The selected color is used when creating drawings, placing pins, or placing arrows.

The current active color is displayed at the center of the color pallet wheel and is reflected in the tool icons.

When the color pallet is active, it replaces the Annotate panel. Select the X icon to close the color pallet.

Draw

Drawing tools can be used to create freeform lines in space. There are two drawing tools available: thick and thin.

When a drawing tool is active, a crosshair appears floating in front of your controller.

Place the crosshair point where you would like to start drawing. Then press and hold the trigger while moving the controller to draw. Releasing the trigger completes the drawing.

Pin

A pin is a physical marker with a label that can be used to identify objects in your space.

When the pin tool is active, a crosshair appears floating in front of your controller. Place the crosshair where you would like the pin tip to point. Then press and release the trigger to place the pin. Hover over the pin and select the form field to add a label.

The label is a helpful communication feature that is unique to pins. Imagine using pins to identify and label small parts on a piece of machinery.

Arrows

Arrows are used to direct attention towards points of interest and as a wayfinding mechanism.

When the arrow tool is active, a crosshair appears floating in front of your controller. Place the crosshair where you would like the arrow tip to point. When you press the trigger button, the arrow appears with the point originating at the crosshair center. While holding the trigger, the user can move the controller to change the arrow’s length and direction. Releasing the trigger places the arrow.

Eraser

The eraser can be used to delete individual objects.

When the eraser tool is active, a crosshair appears floating in front of your controller. Target individual pins, drawings, or arrows by hovering over the annotation. When the annotation is targeted, the crosshair turns red. Press and release the trigger button to erase the targeted annotation.

Delete All Annotations

Delete All (trash icon) removes all annotations from the session including annotations that were created by you and annotations created by the web-user.

Selecting the delete all button displays a confirmation message dialogue box. Annotations will not be deleted until confirmed through the dialogue box.

Tutorial

The tutorial can be accessed at any time through the Assist panel. To open the Assist panel, select the Assist icon from the menu. The tutorial tab is active by default when the Assist submenu panel opens.

To view the tutorial, select Start Tutorial.

About Assist

About Assist displays application version information and copyright information.

Developer Logs

Logs display technical and performance information. Logs are intended for developer use.

Assist Web Application

Assist Web App Setup

Locate A Desktop or Laptop Computer

The Assist Web Application requires a desktop or laptop computer. Tablets and mobile devices are not supported. Verify that your computer is connected to the internet.

Open Web Browser

On your computer, open up a web browser. Chrome is recommended for the best experience.

Access Email

In the web browser, open your email. Locate the Assist invitation that was sent to you by the session host (Magic Leap 2 user).

View Assist Invitation

Open the Assist invitation in your email.

Click Join Session

Select the Join Session button to launch the Assist Web App.

Join Assist Session

Enter Your Name

When the Assist Web App loads, you will see welcome messaging and a prompt to enter your name. Select the name field and type your name. Select the Next button to continue.

Preview Your Video and Audio

A preview of your video feed is displayed. Adjust the video input by selecting the dropdown arrow next to the camera icon. Adjust the audio input and output by selecting the dropdown arrow next to the microphone icon.

Mute or unmute your microphone by selecting the microphone icon. Disable or enable your video feed by selecting the camera icon. For the best experience, enable both your video and audio.

When you are satisfied with your video and audio preview, select the Join button.

Request to Enter Session

When you join the session, you will be placed into the session lobby. The session host (Magic Leap 2 user) receives a join request. When they approve your join request, you will be placed into the live session.

Session

Live Collaboration

In the session you are connected directly with the host (Magic Leap 2 user). The host can hear you and see a live video feed of you. You can hear the host and see a live camera feed streamed from the host’s Magic Leap 2.

In a live session you can:

• Video chat with the host
• Interact with a 3D mesh of the host’s space
• See a live camera feed from the host’s Magic Leap 2
• Annotate the host’s space
• Share your screen
• Send files to the host
• Receive files from the host

Mesh Window

The mesh window displays a 3D representation of the host’s space. You can interact with the mesh to examine the host’s environment by zooming, panning, and orbiting. In the mesh you will see a frustum that represents the host’s field of view.

Camera Window

The camera window displays a live camera feed from the host’s Magic Leap 2, allowing you to see what the host is seeing. The camera feed can be paused by clicking the pause button in the camera window control bar.

Control Bar

The control bar located at the bottom of the web app contains the following: About Assist, Your Video Window, Mute, Hide/Show Video, Screen Share, Files, and Exit Session.

Toolbar

The toolbar located on the right side of the web app contains the following: Select, Draw, Pin, Color Palette, Eraser, and Delete All.

Side by Side View

You can customize the layout of the web app to best serve your current task. By default the camera window and mesh window are positioned side by side.

Match the Magic Leap 2’s View

The mesh window can be oriented to match the host’s view by selecting the match view button in the mesh window control bar. The mesh will remain in this position until manipulated—it will not live follow the host’s view. This feature is a quick way to reorient your view and help you understand where the host is located.

Maximizing Mesh or Camera Window

Get a closer look at the mesh window or camera window by selecting the maximize button in the mesh window control bar or camera window control bar. Return to the side by side view by selecting the side by side icon in the maximized window control bar.

Minimize Your Video Window

You can minimize your video window to place it in the main control bar at the bottom of the web app by selecting the minimize icon in your video window control bar.

Annotate

Annotations

Annotations are visual communication tools. You can directly augment the host’s environment by drawing or placing pins in the mesh window or camera window. Annotations you create will immediately appear in the host’s environment.

Annotating In The Mesh Window

To annotate in the mesh window, select an annotation tool, hover over the mesh window, and click to annotate.

Annotating In The Camera Window

To annotate in the camera window, you must first pause the camera feed by pressing the pause button on the camera window control bar. Select an annotation tool and click directly in the camera window to annotate. Annotations created in the camera window will also be visible in the mesh window.

Draw

The draw tool can be used to create freeform lines. When the draw tool is active, your cursor appears as a crosshair when hovering over the mesh window or camera window. Place the crosshair where you would like to start drawing. Click and drag to draw. Releasing the click completes the drawing.

Pin

Pins can be used to mark and label specific parts or objects. When the pen tool is active, your cursor appears as a crosshair when hovering over the mesh window or camera window. Place the crosshair where you would like the pin tip to point. Then click and release to place the pin. Hover over the pin and select the form field to add a label.

Color Palette

The color palette allows you to select a color to annotate with. The selected color is reflected in the center of the color pallet icon, in the drawing tool icon, and the pin icon.

Eraser

The eraser can be used to delete individual objects in the mesh window. When the eraser tool is active, your cursor appears as a crosshair when hovering over the mesh window. The crosshair rotates and turns red when hovering over a targeted drawing or pin. With the annotation targeted, click to erase the annotation.

Delete All

Selecting the trash icon deletes all annotations from the session including annotations you created and annotations created by the host.

Share Files

Open Files Drawer

Selecting the files button from the control bar opens the files drawer. The files drawer contains pre-loaded 3D objects and images that can be sent to the host. The files drawer also contains any photos that the host has shared.

Drag and Drop to Share Files

To share a file, click the file and drag it over the mesh window or camera window, and then drop the file into the window. A confirmation message will appear when the file has been sent.

Download Shared Photos

The host can share photos with you during the session. When a new photo has been shared, a dot appears over the file folder icon. In the file drawer, the new photo is highlighted. Select the download icon on the photo to download and view the image.

Select Screen Share from Control Bar

Share your screen with the host by selecting the screen share button from the control bar.

Choose Screen, Window or Tab

You can choose to share your entire screen, a window, or a tab.

Screen Share Window Maximizes

The screen share window is maximized and begins live streaming your content to the host.

End Screen Share

To end the screen share, select the screen share button from the control bar.