Remote Observation & State Capture

What

Remote Observation & State Capture turns what the robot knows into observation packages. It reads the Vehicle / Robot Model's authoritative state and any configured scene or sensor source, then publishes each result on a named stream.

The same stage serves two consumers:

• Remote-side Assistance reads a local observation such as pose, heading, terrain, or other vehicle state before it calculates the next command.
• Downlink Communication receives feedback streams such as pose and camera view, then applies the configured communication conditions before the operator sees them.

These are separate packages and separate paths. A study can keep a state stream local while sending only video to the operator; the current TurtleBot reconstruction does not use robot-state.

Why

Observation is not the same as presentation. The robot may have an exact pose and sensor state locally while the operator receives an older camera frame. Keeping capture separate from downlink transport makes that difference explicit.

Stream-level configuration lets a study keep local state separate from delayed video, or compare pose and video streams that arrive at different ages.

How

The module uses the same vehicle snapshot twice in a tick: first to refresh local observations for robot-side decisions, then to package the updated state for operator feedback. For example, a local robot-pose package can let SlopeBoundaryGuard check the next command, while the post-update robot-pose and robot-view packages describe the result sent through the downlink.

In one control tick, the sequence is:

1. Read the current snapshot. Remote-side Assistance reads the local observation streams it declares through ReceiveObservation and TryGetLatestObservation. In the TurtleBot demo, SlopeBoundaryGuard reads robot-pose; robot-state has no current consumer.
2. Update the vehicle. The Vehicle Model applies that command, updates its kinematics and actuation records, and associates the latest sensor data with the new snapshot.
3. Publish the result. Observation & State Capture reads the updated snapshot. Local state is refreshed immediately; robot-view and any other configured downlink stream are sent to the operator through Downlink Communication.

So the controller acts on the previous snapshot, while the operator eventually receives feedback about the vehicle's latest action.

Boundary

Contract

An observation package is identified by its stream id and carries a study-defined payload. A source can project selected fields from the complete vehicle state, or package a sensor result such as a camera frame. Downstream modules do not need to know how the source captured it.

public abstract class RemoteObservationSource<TObservation>
    : RemoteObservationSource
    where TObservation : class
{
    internal sealed override bool TryCapturePackage(
        double timestampSeconds,
        out Package package)
    {
        if (!TryCapture(timestampSeconds, out TObservation observation))
        {
            package = null;
            return false;
        }

        package = new Package(
            observation,
            timestampSeconds,
            StreamId);
        return true;
    }
}

• Payload is the pose, camera frame, sensor state, or other observation.
• SourceTimestampSeconds records when the observation was captured.
• StreamId routes the package to the local consumer or downlink channel.
• PublishToDownlink controls whether the post-control package enters delayed feedback; local assistance can still read it when this is false.

Remote-side Assistance reads a local stream with TryGetLatestObservation<TObservation>. Downlink Communication routes feedback packages by the same stream id and applies its configured channel condition.

Examples

Each example projects the same vehicle snapshot into a different observation stream.

public sealed partial class RobotViewSource
    : VehicleStateObservationSource<TurtleBotState, ViewFramePackage>
{
    protected override bool TryCapture(
        TurtleBotState state,
        double timestamp,
        out ViewFramePackage observation)
    {
        TurtleBot3WafflePiEnhanced vehicle = (TurtleBot3WafflePiEnhanced)Vehicle;
        CameraFrameSnapshot camera = state.Snapshot.Sensors.Camera.CapturedFrame;
        if (!vehicle.TryCopyCameraFrame(
                camera,
                out RenderTexture frame,
                out TurtleBotState stateAtCapture))
            return false;

        observation = new ViewFramePackage(
            frame,
            camera.SampledAt,
            camera.Sequence,
            camera.WorldToCameraMatrix,
            camera.ProjectionMatrix,
            stateAtCapture,
            vehicle.RecycleCameraFrame);
        return true;
    }
}

public sealed partial class RobotStateSource
    : VehicleStateObservationSource<TurtleBotState, RobotStatePackage>
{
    protected override bool TryCapture(
        TurtleBotState state,
        double timestamp,
        out RobotStatePackage observation)
    {
        observation = new RobotStatePackage(state, timestamp);
        return true;
    }
}

Config

1. Configure the observation sources. Point pose and any study-defined state sources at the Vehicle / Robot Model's LatestState; assign the camera source to the robot view.
2. Give each source a stream id. Use the exact name expected by its consumer, such as robot-pose, robot-state, or robot-view.
3. Connect local assistance when the study uses it. Remote-side Assistance reads only the stream it declares through TryGetLatestObservation. A local-only state stream needs no downlink channel, but it is unnecessary unless a configured local module consumes it.
4. Configure feedback channels separately. Add a Downlink Communication channel only for streams the operator should receive, then configure delay, loss, jitter, or bandwidth per stream.