Operator-side State Reconstruction

What

Operator-side State Reconstruction runs after a feedback package has crossed the downlink. It receives the released package, queues it until the next module step, and applies its payload to a local representation of the remote world. A pose package places the remote robot at the pose that was sampled remotely; a camera or lidar package updates the corresponding delayed sensor view; map or landmark packages can update a local environment model.

The module reconstructs received evidence, not the authoritative robot. It must not read the Vehicle / Robot Model, the live robot transform, or the current remote camera. The local world therefore remains at the latest remote time for which the required packages have arrived. Interpolation, prediction, filtering, and visual overlays are separate choices made by later operator-side modules.

Why

Communication delay makes the operator's view a different state from the robot's current state. Reconstruction gives downstream modules a coherent local representation of that delayed world: the remote robot can be placed in a local scene, delayed sensor data can be attached to it, and a study can update a map or SLAM model as new observations arrive. The shipped TurtleBot implementation currently stores delayed pose and video; it does not yet build a map or move a separate local robot representation.

How

Operator-side State Reconstruction turns released feedback into the latest local representation of the remote world. It consumes only what has crossed the downlink; it does not read the live vehicle.

Downlink Communication can fan out the same released package to more than one consumer. Operator-side State Reconstruction keeps the operator-side remote-world model; it is not a forwarding stage. In the TurtleBot demo, the model contains delayed pose and video. A study that sends lidar, map, or landmark data can extend the same module to maintain a richer local environment.

• Receive. ReceivePackage only enqueues; it does not update operator state immediately.
• Process. On the next module step, every pending package is passed to ReconstructPackage in queue order.
• Apply. The implementation updates the remote-world model for the payload type. Older pose data is replaced, newer sensor data is attached to the corresponding remote time, and older camera frames are released when replaced.

Boundary

Contract

The base class implements IPackageConsumer. Your implementation provides ReconstructPackage; the base class owns the queue and calls your method during the module step:

using System.Collections.Generic;
using Unite.Core;

public abstract class OperatorSideStateReconstructionModule
    : TeleroboticsModule, IPackageConsumer
{
    private readonly Queue<Package> pendingPackages = new();

    public void ReceivePackage(Package package)
    {
        if (package != null)
            pendingPackages.Enqueue(package);
    }

    internal sealed override void Step(double timestampSeconds)
    {
        while (pendingPackages.Count > 0)
            ReconstructPackage(pendingPackages.Dequeue(), timestampSeconds);
    }

    protected abstract void ReconstructPackage(
        Package package,
        double timestampSeconds);
}

• ReceivePackage is called by Downlink Communication when a package is released.
• timestampSeconds is the current kernel time when reconstruction runs; it is not the package's sample time.
• The payload carries its own capture metadata, such as SampledAt. Preserve that metadata when creating a representation.
• The base module does not publish a replacement package. Consumers read the implementation's properties or other local state.

Examples

The TurtleBot implementation is in Assets/UNITE/Demo/every-move-you-make/Runtime/EveryMoveStateReconstruction.cs. The second tab shows an EnvironmentUpdatePackage published by Remote Observation & State Capture and consumed by the reconstruction component.

using Unite.Core;
using Unite.Kernel;
using UnityEngine;

public sealed class EveryMoveStateReconstruction
    : OperatorSideStateReconstructionModule
{
    public PosePackage LatestPose { get; private set; }
    public ViewFramePackage LatestViewFrame { get; private set; }
    public Texture CurrentViewFrame { get; private set; }
    public double CurrentViewSampledAt { get; private set; }
    public long CurrentViewSequence { get; private set; } = -1;

    protected override void ReconstructPackage(
        Package package,
        double timestampSeconds)
    {
        if (package.Payload is PosePackage pose)
        {
            LatestPose = pose;
            return;
        }

        if (!(package.Payload is ViewFramePackage view) || !view.Frame)
            return;

        ViewFramePackage previous = LatestViewFrame;
        LatestViewFrame = view;
        CurrentViewFrame = view.Frame;
        CurrentViewSampledAt = view.SampledAt;
        CurrentViewSequence = view.Sequence;

        if (previous != null && previous != view)
            previous.Release();
    }
}

using System.Collections.Generic;
using Unite.Core;
using Unite.Demo.EveryMoveYouMake;
using Unite.Kernel;
using UnityEngine;

public sealed class EnvironmentUpdatePackage
{
    public double SampledAt { get; }
    public IReadOnlyList<Vector3> OccupiedPoints { get; }

    public EnvironmentUpdatePackage(
        double sampledAt,
        IReadOnlyList<Vector3> occupiedPoints)
    {
        SampledAt = sampledAt;
        OccupiedPoints = occupiedPoints;
    }
}

public sealed class RemoteWorldModel
{
    private readonly List<Vector3> occupiedPoints = new();

    public Vector3 RobotPosition { get; private set; }
    public float RobotHeading { get; private set; }
    public double LastSampledAt { get; private set; }
    public IReadOnlyList<Vector3> OccupiedPoints => occupiedPoints;

    public void ApplyPose(PosePackage pose)
    {
        RobotPosition = pose.Position;
        RobotHeading = pose.Heading;
        LastSampledAt = pose.SampledAt;
    }

    public void ApplyEnvironment(EnvironmentUpdatePackage update)
    {
        occupiedPoints.Clear();
        occupiedPoints.AddRange(update.OccupiedPoints);
        LastSampledAt = update.SampledAt;
    }
}

public sealed class StudyStateReconstruction
    : OperatorSideStateReconstructionModule
{
    public RemoteWorldModel RemoteWorld { get; } = new();

    protected override void ReconstructPackage(
        Package package,
        double timestampSeconds)
    {
        if (package.Payload is PosePackage pose)
        {
            RemoteWorld.ApplyPose(pose);
            return;
        }

        if (package.Payload is EnvironmentUpdatePackage update)
            RemoteWorld.ApplyEnvironment(update);
    }
}

Config

1. Implement OperatorSideStateReconstructionModule and handle the pose, sensor, and environment payloads the study sends through Downlink Communication.
2. Assign one implementation to the agent's Operator-side State Reconstruction field. The Kernel connects Downlink Communication to it.
3. Keep the reconstructed world separate from the Vehicle / Robot Model. Do not store a live Transform, camera, or authoritative state reference in the reconstruction module.
4. Release owned resources when a representation is replaced or the module is destroyed. The TurtleBot implementation releases replaced ViewFramePackage textures.