Interactive markers for teleoperation in ROS - Deep Dive

Overview - Interactive markers for teleoperation

What is it?

Interactive markers are special visual tools in ROS that let users move, rotate, or control robots by dragging markers in a 3D space. They provide a way to teleoperate robots by directly interacting with these markers in a graphical interface. This makes controlling robots more intuitive and visual, especially when precise movements are needed. They combine visualization and control in one simple interface.

Why it matters

Without interactive markers, teleoperating robots would rely on typing commands or using separate controls, which can be slow and error-prone. Interactive markers let users see and manipulate the robot's position and orientation directly, reducing mistakes and speeding up tasks. This improves safety and efficiency in real-world robot control, especially in complex environments.

Where it fits

Before learning interactive markers, you should understand basic ROS concepts like nodes, topics, messages, and visualization tools like RViz. After mastering interactive markers, you can explore advanced teleoperation techniques, robot motion planning, and integrating sensors for autonomous control.

Mental Model

Core Idea

Interactive markers are like virtual handles in 3D space that you grab and move to control a robot's actions directly and visually.

Think of it like...

Imagine a puppet controlled by strings; interactive markers are the strings you pull or twist to move the puppet's limbs exactly how you want.

┌─────────────────────────────┐
│        RViz Window          │
│  ┌───────────────┐          │
│  │ Interactive   │          │
│  │ Marker (cube) │◄── Drag  │
│  └───────────────┘          │
│          ▲                  │
│          │                  │
│   Robot moves accordingly   │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding ROS visualization basics

Concept: Learn how ROS uses RViz to show robot data and simple markers.

RViz is a tool that shows robot data like sensor readings and robot models. Markers are simple shapes (like arrows or cubes) that RViz can display to represent points or objects. These markers are static visuals without interaction.

Result

You can see your robot and some shapes in RViz but cannot move or control them.

Understanding how RViz displays data is essential before adding interactivity to markers.

2

Foundation

Basics of interactive markers

Concept: Interactive markers add user control to markers, letting you drag or rotate them in RViz.

Interactive markers are special markers that respond to mouse actions in RViz. You can click and drag them to change their position or orientation. They send feedback messages to ROS nodes about these changes.

Result

You can move markers in RViz, and your program receives updates about their new positions.

Interactive markers bridge visualization and control by turning static visuals into manipulable controls.

3

Intermediate

Creating interactive markers in ROS

Concept: Learn how to program interactive markers using ROS's interactive_marker package.

You create an interactive marker server in your ROS node. Then you define markers with controls (like move or rotate) and insert them into the server. The server handles user input and sends feedback to your node.

Result

Your node can create markers in RViz that users can manipulate, and your code reacts to those changes.

Knowing how to set up the interactive marker server is key to building teleoperation interfaces.

4

Intermediate

Handling feedback for teleoperation

🤔Before reading on: do you think feedback from interactive markers is sent continuously or only once after interaction? Commit to your answer.

Concept: Interactive markers send feedback messages whenever the user moves or rotates them, allowing real-time control.

When a user drags or rotates a marker, the interactive marker server sends feedback messages with the new pose. Your node subscribes to these messages and can command the robot to move accordingly.

Result

Robot movements update live as the user manipulates the marker in RViz.

Understanding feedback timing is crucial for smooth and responsive teleoperation.

5

Intermediate

Combining multiple controls on one marker

🤔Before reading on: do you think a single interactive marker can have multiple control types (move, rotate) at once? Commit to your answer.

Concept: Interactive markers can have several controls, like moving along different axes and rotating, all on one marker.

You can add multiple control handles to a marker, such as arrows for moving along X, Y, Z axes and rings for rotation. This lets users manipulate the robot's pose in all directions from one marker.

Result

Users get a rich interface to control robot position and orientation intuitively.

Knowing how to combine controls makes teleoperation flexible and user-friendly.

6

Advanced

Integrating interactive markers with robot motion

🤔Before reading on: do you think interactive marker feedback directly moves the robot or requires additional processing? Commit to your answer.

Concept: Interactive marker feedback usually needs processing to convert marker poses into robot commands safely.

Your node receives marker pose updates and translates them into robot motion commands, considering constraints like joint limits or obstacles. This may involve motion planning or safety checks before sending commands to the robot.

Result

Robot moves smoothly and safely in response to user input via interactive markers.

Understanding the gap between marker input and robot motion prevents unsafe or impossible commands.

7

Expert

Optimizing interactive markers for complex teleoperation

🤔Before reading on: do you think interactive markers can handle multiple robots or complex environments easily? Commit to your answer.

Concept: Advanced use involves managing many markers, synchronizing states, and handling network delays for multi-robot teleoperation.

In complex systems, you may have multiple interactive markers controlling different robots or robot parts. You need to manage marker states carefully, handle feedback latency, and ensure consistent robot states. Techniques include state synchronization, feedback filtering, and UI design for clarity.

Result

Teleoperation remains responsive and reliable even in complex multi-robot or multi-user scenarios.

Knowing these challenges helps build robust teleoperation systems beyond simple demos.

Under the Hood

Interactive markers work by running a server node that manages marker objects and their controls. When a user interacts with a marker in RViz, RViz sends input events to the server, which updates the marker's pose and sends feedback messages back to the user's ROS node. This feedback loop allows real-time control. Internally, the server tracks marker states and publishes updates on ROS topics. RViz renders the markers and handles mouse events, translating them into pose changes.

Why designed this way?

Interactive markers were designed to unify visualization and control in ROS, avoiding separate interfaces for commanding robots. The server-client model allows multiple nodes to create and manage markers independently. Using ROS messages and topics fits ROS's distributed architecture, enabling flexible teleoperation setups. Alternatives like custom GUIs were less flexible and harder to integrate with ROS tools.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│   RViz GUI    │◄─────►│ Interactive   │◄─────►│ Robot Control │
│ (User input)  │       │ Marker Server │       │    Node       │
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                      ▲                      ▲
       │                      │                      │
   Mouse events          Marker state           Robot commands
   & rendering           & feedback             & motion

Myth Busters - 4 Common Misconceptions

Quick: Do interactive markers automatically move the robot without extra code? Commit to yes or no.

Common Belief: Interactive markers directly control the robot's movement as soon as you move them in RViz.

Reality: Interactive markers only send feedback about their new pose; you must write code to translate that into robot commands.

Why it matters: Assuming automatic control leads to confusion when the robot does not move, causing wasted time debugging.

Quick: Can one interactive marker control multiple robot parts independently? Commit to yes or no.

Common Belief: A single interactive marker can control multiple robot joints or parts separately at the same time.

Reality: Each interactive marker controls one pose; controlling multiple parts requires multiple markers or complex code to split controls.

Why it matters: Misunderstanding this can cause overly complex marker designs and user confusion.

Quick: Do interactive markers work perfectly with zero latency in all networks? Commit to yes or no.

Common Belief: Interactive markers provide instant, lag-free control regardless of network conditions.

Reality: Network delays can cause lag in feedback, making teleoperation less smooth and requiring handling in code.

Why it matters: Ignoring latency can cause unsafe or jerky robot movements in real deployments.

Quick: Are interactive markers only useful for teleoperation? Commit to yes or no.

Common Belief: Interactive markers are only for teleoperating robots and have no other uses.

Reality: They are also useful for debugging, visualization, and setting goals or waypoints interactively.

Why it matters: Limiting their use reduces creativity and misses opportunities for better robot interfaces.

Expert Zone

1

Interactive marker feedback can be throttled or filtered to reduce network load without losing control quality.

2

Combining interactive markers with action servers allows asynchronous, cancellable robot commands improving robustness.

3

Custom controls can be created beyond standard move/rotate, enabling complex user interactions like scaling or mode switching.

When NOT to use

Interactive markers are not ideal for high-speed or highly precise control where direct joystick or sensor input is better. For fully autonomous robots, direct teleoperation via markers is unnecessary. Alternatives include joystick teleoperation, voice commands, or autonomous planners.

Production Patterns

In production, interactive markers are often integrated with safety layers that check commands before execution. They are used in surgical robots, warehouse robots, and drones for intuitive control. Multi-user teleoperation setups use synchronized markers with role-based access. Feedback smoothing and state synchronization are common to handle network issues.

Connections

Human-Computer Interaction (HCI)

Interactive markers apply HCI principles of direct manipulation and visual feedback to robot control.

Understanding HCI helps design interactive markers that are intuitive and reduce user errors.

Distributed Systems

Interactive markers rely on ROS's distributed messaging system to communicate between GUI and robot nodes.

Knowing distributed system concepts clarifies how marker feedback travels and why latency matters.

Video Game Controllers

Interactive markers function like virtual joysticks or control sticks in games, mapping user input to character movement.

Recognizing this similarity helps grasp how interactive markers translate user actions into robot commands.

Common Pitfalls

#1 Ignoring feedback messages and expecting robot to move automatically.

Wrong approach:

Creating interactive markers but not subscribing to feedback or sending robot commands.

Correct approach:

Subscribe to interactive marker feedback and translate pose updates into robot motion commands.

Root cause:

Misunderstanding that interactive markers only provide input, not direct control.

#2 Using a single control type when multiple are needed for full pose control.

Wrong approach:

Adding only move controls on X axis and expecting full 3D control.

Correct approach:

Add move controls on X, Y, Z and rotate controls for full 6DOF manipulation.

Root cause:

Underestimating the complexity of robot pose control.

#3 Not handling network latency causing jerky robot movements.

Wrong approach:

Sending every feedback message directly as a robot command without filtering.

Correct approach:

Implement feedback throttling or smoothing before commanding the robot.

Root cause:

Ignoring real-world network and processing delays.

Key Takeaways

Interactive markers let users control robots by moving virtual handles in a 3D visualization tool.

They separate visualization from control logic, requiring your code to handle feedback and command the robot.

Combining multiple controls on one marker enables intuitive full pose manipulation.

Handling feedback timing and network delays is essential for smooth teleoperation.

Advanced use involves managing multiple markers and integrating safety and motion planning for real-world robots.

Practice

1. What is the main purpose of interactive markers in ROS teleoperation?

easy

A. To visually control robots by moving markers in 3D space

B. To write robot control code without any user input

C. To display static images of the robot status

D. To log robot sensor data for offline analysis

Solution

1. Step 1: Understand interactive markers role

   Interactive markers allow users to move and rotate markers in a 3D view to control robots easily.

2. Step 2: Compare options with purpose

   Only To visually control robots by moving markers in 3D space describes visual control via markers; others describe unrelated tasks.

3. Final Answer:

   To visually control robots by moving markers in 3D space -> Option A

4. Quick Check:

   Interactive markers = Visual robot control [OK]

Hint: Interactive markers let you move robot parts visually [OK]

Common Mistakes:

• Thinking markers are only for displaying data
• Confusing teleoperation with offline logging
• Assuming no user input is needed

2. Which ROS message type is commonly used to create an interactive marker for teleoperation?

easy

A. std_msgs/String

B. sensor_msgs/Image

C. geometry_msgs/Twist

D. visualization_msgs/InteractiveMarker

Solution

1. Step 1: Identify message type for interactive markers

   The message type visualization_msgs/InteractiveMarker is designed to define interactive markers in ROS.

2. Step 2: Eliminate unrelated message types

   sensor_msgs/Image is for images, geometry_msgs/Twist for velocity commands, std_msgs/String for text messages, so they don't create interactive markers.

3. Final Answer:

   visualization_msgs/InteractiveMarker -> Option D

4. Quick Check:

   Interactive marker message = visualization_msgs/InteractiveMarker [OK]

Hint: Interactive markers use visualization_msgs/InteractiveMarker type [OK]

Common Mistakes:

• Choosing geometry_msgs/Twist which is for velocity commands
• Confusing image or string messages with markers
• Not knowing ROS message types

3. Given this snippet handling interactive marker feedback in ROS:

void processFeedback(const visualization_msgs::InteractiveMarkerFeedbackConstPtr &feedback) {
  if (feedback->event_type == visualization_msgs::InteractiveMarkerFeedback::POSE_UPDATE) {
    geometry_msgs::Pose new_pose = feedback->pose;
    // Update robot command with new_pose
  }
}

What happens when the user moves the marker?

medium

A. The robot command updates with the marker's new pose

B. The marker resets to its original position

C. The feedback event_type is ignored

D. The robot stops moving immediately

Solution

1. Step 1: Analyze feedback event type

   The code checks if the event_type is POSE_UPDATE, which means the marker was moved or rotated.

2. Step 2: Understand the effect of POSE_UPDATE

   When POSE_UPDATE occurs, the new pose is extracted and used to update the robot command, so the robot moves accordingly.

3. Final Answer:

   The robot command updates with the marker's new pose -> Option A

4. Quick Check:

   POSE_UPDATE triggers robot command update [OK]

Hint: POSE_UPDATE means marker moved, update robot pose [OK]

Common Mistakes:

• Assuming marker resets automatically
• Ignoring the event_type check
• Thinking robot stops without command

4. You wrote this callback for interactive marker feedback but the robot does not move:

void feedbackCallback(const visualization_msgs::InteractiveMarkerFeedbackConstPtr &feedback) {
  if (feedback->event_type = visualization_msgs::InteractiveMarkerFeedback::POSE_UPDATE) {
    // update robot command
  }
}

What is the error?

medium

A. Missing return statement in the callback

B. Using assignment '=' instead of comparison '==' in the if condition

C. Incorrect message type for feedback parameter

D. Not publishing the updated robot command

Solution

1. Step 1: Check the if condition syntax

   The condition uses '=' which assigns a value instead of '==' which compares values.

2. Step 2: Understand effect of assignment in if

   Assignment always returns true, so the condition is always true but does not properly check event_type, causing logic errors.

3. Final Answer:

   Using assignment '=' instead of comparison '==' in the if condition -> Option B

4. Quick Check:

   Use '==' to compare event_type, not '=' [OK]

Hint: Use '==' to compare, '=' assigns value [OK]

Common Mistakes:

• Confusing '=' and '==' in conditions
• Assuming missing return causes no movement
• Ignoring need to publish commands

5. You want to create an interactive marker that allows the user to rotate a robot arm joint only around the Z axis. Which control type and orientation should you use in your marker setup?

hard

A. Use a BUTTON control with no orientation

B. Use a MOVE_PLANE control with orientation aligned to the X axis

C. Use a ROTATE_AXIS control with orientation set to align Z axis with the joint axis

D. Use a MOVE_AXIS control aligned to the Y axis

Solution

1. Step 1: Identify control type for rotation

   ROTATE_AXIS control allows rotation around a single axis, suitable for rotating a joint.

2. Step 2: Set orientation to align with joint axis

   To rotate around Z axis, set the control's orientation so its Z axis matches the joint's rotation axis.

3. Final Answer:

   Use a ROTATE_AXIS control with orientation set to align Z axis with the joint axis -> Option C

4. Quick Check:

   ROTATE_AXIS + Z orientation = rotate joint around Z [OK]

Hint: ROTATE_AXIS control with Z orientation rotates joint around Z [OK]

Common Mistakes:

• Using MOVE_PLANE which moves in a plane, not rotate
• Choosing BUTTON which is for clicks, not rotation
• Aligning control to wrong axis