MOSAICModular System for Adaptive
Human–AI Collaboration

A Configurable Testbed for Human–AI TeamingIEEE SMC 2026 · Bellevue, WAgithub.com/iHuman-Lab/mosaic

Bennett Kwaku Dogbey · Elahe Oveisi · Hemanth Manjunatha

October 4, 2026

Meet the iHuman Lab

We study how people and intelligent systems work together.

Hemanth Manjunatha Principal Investigator

Elahe Oveisi PhD Candidate

Bennett Kwaku Dogbey PhD Student

Our work spans brain–computer interfaces, human–robot interaction, human-AI interaction, and cognitive augmentation.

iHuman Lab · Oklahoma State University

Studying Human–AI Teaming

When an AI teammate advises people in dynamic, high-stakes tasks, researchers ask questions like these:

Trust and reliance

When do people follow AI advice, and when is that reliance appropriate?

Decisions under pressure

How do uncertainty and time constraints affect reliance?

Situation awareness

What happens when the person and AI have access to different information?

Cognitive state

How does AI advice affect attention and workload?

AI teammate design

Which teammate behaviors and communication strategies improve team performance?

Adaptive assistance

When should assistance change with task demands or human performance?

Answering these questions requires observing the interaction as it unfolds over time.

These are the field’s core questions. Each one plays out during the interaction, as decisions, advice, and consequences build on each other.

Why Do We Need a Configurable Testbed?

That takes four capabilities in one experiment. Existing tools rarely offer all four together.

01

Interactive decision making

People make a sequence of decisions, and each outcome changes what happens next.

03

Configurable task conditions

Researchers need to vary information access, uncertainty, time pressure, and the consequences of mistakes.

02

A customizable AI teammate

Researchers need to vary what the AI knows, how reliable it is, and when and how it communicates.

04

Synchronized data

Task events, human actions, AI advice, physiological signals, and sensor measurements must share a common timeline.

To study the interaction over time, researchers need an interactive task, a customizable teammate, conditions they can vary, and synchronized data. The next slide shows how MOSAIC brings them together.

What Is MOSAIC?

MOSAIC is a modular testbed for human–AI teaming research. Configure its components for each study.

01

Interactive task

Search and rescue, one decision at a time

03

Configurable conditions

Map, view, timing, and scoring per study

MOSAIC

modular · configurable

02

Customizable AI teammate

Scripted or LLM, with behavior you set

04

Synchronized data

Actions, advice, and signals on one clock

MOSAIC lets researchers vary the task and AI teammate, then record the interaction.

MOSAIC stands for Modular System for Adaptive Human–AI Collaboration: an open-source Python platform for studying human–AI teaming through configurable tasks, built on MiniGrid. Like a mosaic built from tiles, it assembles separate parts (task, teammate, interface, sensing) into one study. Each box answers the need in the same position on the last slide: the task, a teammate you can customize, conditions you set per study, and one synchronized record. Modular: the environment, AI teammate, interface, sensing, and analytics are separate components, so you replace one without touching the others. Configurable: map, hazards, view, scoring, and the teammate are set per study, not hard-coded. Search and rescue is the first testbed; the same pieces can support other collaborative tasks (ihuman-lab.github.io/mosaic).

How MOSAIC Works

Each part is a separate module: the participant acts on the task, the AI teammate advises from task context, and LSL aligns every stream into one session record.

Human participant

Observes, decides, and acts.

actions

observations

MOSAIC task

Runs the task and the human-facing interface.

task context

advice

AI teammate

Interprets task context and provides advice.

physiological signals

task and interaction stream

Study sensors

Physiological data: EEG, eye tracking, pupil diameter

Time-stamped stream

Lab Streaming Layer (LSL)

Aligns streams on a shared clock

Synchronized record

Session data

Ready for analysis and replay

Here is MOSAIC. You’re the participant on the left. You act, the task updates, and when you ask, the teammate reads the task context and answers. Everything the researcher varies sits in configuration, and everything that happens, including gaze and physiology, lands on one clock. Underneath, the reusable parts live in the mosaic package and each study’s choices live in experiment/; the code map at the end shows that split.

The Task: Search and Rescue

Mission

Search a multiroom building and rescue the real victims.

Real victims and decoys look alike, so an AI teammate’s advice can help or mislead.

Part Three plays the mission.

Search and rescue provides a controlled but consequential setting for studying human–AI decisions.

The Human–AI Teaming Loop

At every decision, the participant can act alone or ask for advice.

next decision

1 Observe

Read the room and the mission state.

2 Decide

Act alone, or ask the AI teammate.

act alone

3 Act

The participant always chooses the final action.

4 Outcome

The environment updates and the score changes.

Optional advice path

ask

use or ignore

Request advice

MOSAIC sends task context to the AI teammate.

advice

Evaluate advice

Compare it with what the participant can see.

Recorded throughout Task state · advice request · AI response · participant action · outcome

The central question is when and how AI advice changes human decisions.

Next, we will install MOSAIC, launch a mission, and change the task ourselves.

Part Two

Install MOSAIC

Check the system, install MOSAIC, and run the mission

Why Install Experience Configure

Companion notebook On the tutorial server: every setup command to copy, and the mission with nothing to install Sign in with your password, then open 02_mosaic_human_ai.ipynb jupyter.ihuman-lab.work

Before You Start

Check four things on your laptop before installing.

Python 3.10 or 3.11 check: python --version

Git Any recent version check: git --version

Terminal PowerShell, macOS Terminal, or a Linux shell

Display A local desktop: the game opens its own window

Installed for you by pip install -e .

Task

minigrid gymnasium numpy

Interface

pygame-ce pygame_gui

Study tools

pandas pyyaml python-dotenv

Nothing to install by hand: the next two slides set everything up.

No install, or prefer to copy and paste? The companion notebook is on the tutorial server: jupyter.ihuman-lab.work

Clone MOSAIC and Create the Environment

Everything in the tutorial runs from one repository and one environment.

• macOS / Linux
• Windows PowerShell
• Conda · Any Platform

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
python3 -m venv mosaic_env
source mosaic_env/bin/activate

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
python -m venv mosaic_env
.\mosaic_env\Scripts\Activate.ps1

git clone https://github.com/iHuman-Lab/mosaic.git
cd mosaic
conda create -n mosaic_env python=3.11 -y
conda activate mosaic_env

1 Download the code

Clones the MOSAIC repository

2 Enter the folder

Stay in mosaic for the whole tutorial

3 Create the environment

A private Python named mosaic_env

4 Activate it

Every later command runs inside it

Your prompt begins with (mosaic_env), and python --version reports 3.10 or 3.11.

Choose either venv or Conda, not both. Ubuntu attendees who see “ensurepip is not available” should install python3-venv, delete the incomplete mosaic_env, and repeat the step. PowerShell needs the leading dot-backslash on the activation script. Everyone stays in the mosaic directory for the rest of the tutorial.

Install MOSAIC

Five commands, run from the mosaic folder with mosaic_env active.

python -m pip install --upgrade pip
python -m pip install -e .
python -m pip uninstall -y pygame
python -m pip install --force-reinstall "pygame-ce>=2.5.2"
python -c "import mosaic.sar.env, mosaic.gui.main; print('MOSAIC ready')"

The check prints a pygame-ce banner, then MOSAIC ready. If it says cannot import name 'DIRECTION_LTR', rerun lines 3 and 4.

1 Update pip

Avoids old-installer errors

2 Install MOSAIC

-e means your Part Four edits apply without reinstalling

3–4 Switch to pygame-ce

The interface needs it; plain pygame hides it

5 Check it

Imports the task and the interface

Lines 3 and 4 work around a packaging gap: MOSAIC’s pyproject.toml declares pygame, but pygame_gui needs pygame-ce, and both install into the same pygame folder. Skipping them ends in ImportError: cannot import name 'DIRECTION_LTR' from 'pygame' (reproduced on a clean clone of iHuman-Lab/mosaic main, 1018587, Python 3.11). --force-reinstall is required: uninstalling pygame deletes files pygame-ce shares. pip check then reports “mosaic 0.1.0 requires pygame”; that is harmless. Once upstream depends on pygame-ce the two lines do nothing harmful and can come off the slide.

Run the Included Mission

• macOS / Linux
• Windows PowerShell

PYTHONPATH=src python -m experiment.main

$env:PYTHONPATH = "src"
python -m experiment.main

Window It opens fullscreen. F11 toggles a window (macOS: fn+F11, or set fullscreen: false in configs/experiment.yaml).

Move Arrow keys turn and walk the agent

Terminal Font warnings and connect_all failed lines are normal

You should see this: the room on the left, the panels on the right.

This is MOSAIC’s own study runner from the cloned repository — no extra file is needed. Hold here until most attendees can move the agent. Ask everyone to keep the game open: Part Three walks through it live. The terminal fills with pygame_gui font warnings and “Timeout during mission generation: connect_all failed” lines on every launch; both are harmless (the level generator retries).

First Checkpoint

Confirm

The window opened

Arrow keys move the agent

The information panel updates

It stays open for Part Three

Symptom	Recovery
cannot import name 'DIRECTION_LTR'	Rerun install lines 3 and 4 (pygame-ce)
No module named 'mosaic'	Activate mosaic_env, rerun pip install -e .
No module named 'experiment'	Run from the repo root with PYTHONPATH=src
python --version shows 3.9 or older, or 3.12 and newer	Recreate mosaic_env with Python 3.10 or 3.11
No window, or No available video device	Run locally, not over SSH

Everyone with a window open is ready for Part Three. Leave it open.

Part Three

Experience the SAR Mission

Play one mission and learn to read the screen

Why Install Experience Configure

The Search-and-Rescue Mission

Explore, tell real victims from decoys, and rescue them before time runs out.

Explore

Move room to room. A locked door needs the key of the same color.

Identify

Both are red T shapes. A real victim’s stem is centered; a decoy’s is off-center.

Decide

A real victim earns a point, a decoy costs one, and the clock keeps running.

Every choice has a consequence you can see and score.

No code in this part: the goal is to know the mission the way a participant does. The clip is the real interface: a real victim (green flash, +1), a decoy (red flash, -1), then another real victim. In today’s run the clock only counts down: at 0:00 the timer stops and the mission keeps going (appendix slide “Time Limit”); the lab’s study runner is what ends a trial.

Game Interface

What each tile means, and where to read the mission.

Game viewport — the green flash is rescue feedback

Information panel — rescued, remaining, steps, inventory

Mission status and score — reward and the clock

Controls, always on screen

Chat panel — where AI teammate advice arrives

You the red triangle

Victim rescue with Tab

Decoy costs a point

Lava ends the mission

Door Space opens it

Locked door needs the key of its color

Key Tab picks it up

Victims and Decoys

Real victims and decoys are both red T shapes. Only real victims have health.

REAL VICTIM DECOY

100% 60% 25% 0%

In the study runner, the white bar shows a victim’s health.

Symbols A real victim’s stem sits in the middle of the bar. A decoy’s is off-center.

Health In the lab’s study runner, health drains while a victim is in view, faster near lava, and the victim dies at zero. In today’s mission it stays full.

Rescue Stand next to a victim, face it, and press Tab. The score updates at once.

+1.0 Real victim rescued

−1.0 Decoy rescued

−2.0 Victim died · study runner

Real victims earn points and decoys cost them. Telling them apart is where advice matters.

Health lives on every victim (health, deplete_rate in mosaic/sar/objects.py). The study’s LavaRiskVictimPlacer sets starting health by facing and the drain rate by distance to lava; the study env TunedPickupVictimEnv drains victims in view each step and emits victim_died at zero. The bar is drawn only while advice is on screen (the study reveals it on each reply). In the tutorial’s experiment.main run, health does not drain, so the health bar and the −2.0 outcome do not appear; say so if asked. The reusable VictimPlacer places real victims only; decoys come from the study layer.

Controls and Gameplay

Four actions. Each clip shows what its keys do.

Navigate

↑ Move forward one tile

←→ Turn to face a new direction

Open

Space Open the door you are facing

Locked doors open only while you carry the matching key.

Pick up or rescue

Tab Take the key, or rescue the victim, you are facing

Request advice

Alt Ask the AI teammate

The reply appears in the chat panel. Today’s placeholder teammate answers “Currently, no commands are available.” Part Four gives it a real one.

The advice clip uses the grounded teammate from the Part Four slide “An AI Teammate Without a Key”, so the reply is true for that building. In the attendees’ own run the runner attaches the keyless dummy teammate, so Alt replies “Currently, no commands are available.”

Break · 15 minutes

Take a Break

Next · Part Four: Configure MOSAIC

Fifteen minutes. Leave this slide up for the whole break. Helpers use the time to finish any install that is still stuck. Before starting Part Four, check that everyone can move the agent; anyone who closed the game relaunches it with the command on “Run the Included Mission”. Part Four edits the file that window was launched from, so the game has to be working first.

Part Four

Configure MOSAIC

experiment/main.py in four parts: SAR, GUI, LLM, Sensing

Why Install Experience Configure

Companion notebook On the tutorial server: every change in this part, as the edit and as a cell you can run Sign in with your password, then open 02_mosaic_human_ai.ipynb jupyter.ihuman-lab.work

Runtime Architecture

Four parts you can change, built on Gymnasium and MiniGrid.

↑ ← → Space Tab Alt

Human participant

Watches the screen and presses keys.

keys

frames

4.2 GUI mosaic.gui

Game view, panels, chat, and feedback.

action

observation

4.1 SAR mosaic.sar

Rooms, victims, rules, and rewards.

physiological signals

context · advice

built on

4.4 Sensing experiment/

Gaze and EEG on the game’s clock, through LSL.

4.3 LLM mosaic.llm

Reads the task, replies in the chat.

obs, reward, terminated, truncated, info = env.step(action)

Gymnasium + MiniGrid

The step API and the grid world.

Point at the attendee’s own window: the keys they press, the GUI that draws it, the environment that decides what happens. Each numbered block is one section of Part Four. The chat image is a real reply from the keyless teammate on “An AI Teammate Without a Key”. Sensing is study-layer only — the tutorial run has no sensors attached. Sensors never talk to the game directly: each one streams into Lab Streaming Layer, which puts game and sensor samples on one clock. The env.step line is src/mosaic/gui/user.py:53.

One File, Four Parts

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

src/experiment/main.py · lines 31–51

    env = build_sar_env(
        screen_size=800,
        num_rows=3,
        num_cols=3,
        room_size=10,
        victim_placer=LavaRiskVictimPlacer(
            num_real_victims=game_config.get("num_real_victims", 6),
            num_fake_victims=game_config.get("num_fake_victims", 12),
        ),
        lava_placer=SectorSpreadLavaPlacer(
            lava_per_room=game_config.get("lava_per_room", 8),
        ),
        locked_room_placer=LockedRoomPlacer(locked_room_prob=0.5),
        camera_strategy=AgentFOVCamera(),
    )
    env.reset()
    llm_client = build_llm_client(
        provider=config.get("provider", "dummy"), model=config.get("model")
    )
    gui = SAREnvGUI(env, config=config, llm_client=llm_client)
    gui.run()

4.1 SAR

The building, what fills it, the camera

4.2 GUI

The participant’s window

4.3 LLM

Who answers Alt

4.4 Sensing

What every step records

This is the file attendees launched. Step through the highlights; each one is a section of Part Four, in this order: SAR (the building, what fills it, the camera), GUI (the window), LLM (who answers Alt), and Sensing (what every step records). Rewards are a keyword argument you add to the SAR call. This is main.py as the tutorial needs it: imports enabled and the provider defaulting to “dummy”. Upstream MOSAIC must carry that default before the session (iHuman-Lab/mosaic main, 1018587, has the imports but still defaults the provider to “openai”, so Alt replies “Import tabulate failed”). If it has not landed, have everyone add provider: dummy as a top-level line in configs/experiment.yaml before Part Three.

4.1 · SAR

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Lines 31–44 build the task. Notebook: notebooks/02_mosaic_human_ai.ipynb

src/experiment/main.py lines 31–44

    env = build_sar_env(
        screen_size=800,
        num_rows=3,
        num_cols=3,
        room_size=10,
        victim_placer=LavaRiskVictimPlacer(
            num_real_victims=game_config.get("num_real_victims", 6),
            num_fake_victims=game_config.get("num_fake_victims", 12),
        ),
        lava_placer=SectorSpreadLavaPlacer(
            lava_per_room=game_config.get("lava_per_room", 8),
        ),
        locked_room_placer=LockedRoomPlacer(locked_room_prob=0.5),
        camera_strategy=AgentFOVCamera(),

Three things to change

1 The building Notebook

YAML counts · num_rows · locked_room_prob

How big, crowded, and locked

2 Camera Next

camera_strategy=

What the participant sees

3 Rewards Try it

action=RescueAction(...)

What a mistake costs

Lines 31–44 are the whole SAR task: building size, the placers that fill it, and the camera. Counts are per room (12 real victims across a 3×3 building is 108). The building settings are Step 6 of the notebook, which is in the repository attendees cloned (notebooks/02_mosaic_human_ai.ipynb; it also needs matplotlib, scipy, pylsl, and pyxdf). Its own small building has no decoys; its make_mission() mirrors these lines and has them, and the notebook also shows the three cameras (Step 2) and the reward change (Step 6). The camera and rewards get the next two slides. Code lives in src/mosaic/sar/, src/mosaic/core/, and src/experiment/placers.py.

What the Participant Sees

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Line 44 sets the camera. One walk through a door, three cameras:

Edge-following

EdgeFollowCamera · scrolls with the agent

Whole room

AgentFOVCamera · the default

Forward cone

AgentConeCamera · only what is ahead

Agent same tile in all three   ·   Room the agent’s room

Same task. The camera sets what the participant can see.

To swap it: import the class from mosaic.core.camera and change line 44. Cameras are a strategy interface: CameraStrategy in src/mosaic/core/camera.py returns an image from get_crop(), and env.switch_camera() swaps it mid-session. Avoid FullviewCamera and AgentCenteredCamera: both raise AttributeError on env reset because they have no reset() method.

Rescue Rewards

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it Make rescuing a decoy cost 5 points instead of 1.

src/experiment/main.py an import, then inside build_sar_env

+from mosaic.sar.actions import RescueAction, RescueRewards
@@ inside build_sar_env(...) @@
         camera_strategy=AgentFOVCamera(),
+        action=RescueAction(
+            rewards=RescueRewards(fake_victim=-5.0)),

Run Relaunch and rescue a decoy.

What will the reward read?

Before · the default

After · fake_victim=-5.0

action= goes inside the build_sar_env(...) call, after camera_strategy=. The defaults in RescueRewards (src/mosaic/sar/actions.py) are +1 / -1 / -2 (real_victim_alive, fake_victim, real_victim_dead); the lab’s study passes 10 / -10 / -20 (src/experiment/game.py line 86). Every rescue also reports an event in info["events"]. A NameError on RescueAction means the import is missing.

4.2 · GUI

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Lines 50–51 open the participant’s window.

src/experiment/main.py lines 50–51

    gui = SAREnvGUI(env, config=config, llm_client=llm_client)
    gui.run()

Three things to change

1 Feedback flash Demo · next

vignette=EdgeVignette(...)

How a mistake feels

2 Info panel Appendix

info_panel=

What progress is shown

3 Time limit Appendix

max_time: (YAML)

How much time pressure

The GUI lives in src/mosaic/gui/. SAREnvGUI takes the env, the whole YAML config, the teammate, and optional replacements for the info panel, chat panel, vignette, and prompt builder.

Feedback Flashes

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Presenter demo Make the red flash after a decoy stronger and longer.

src/experiment/main.py an import, then the gui = line

+from mosaic.gui.feedback import EdgeVignette, VignetteStyle
@@ further down @@
-    gui = SAREnvGUI(env, config=config, llm_client=llm_client)
+    gui = SAREnvGUI(env, config=config, llm_client=llm_client,
+        vignette=EdgeVignette(800, styles={"wrong_victim":
+            VignetteStyle((200, 0, 0), 200, 1.5, "fade")}))

Run Relaunch and rescue a decoy.

How will the mistake feel different?

Before · default, 0.6 s

After · stronger, 1.5 s

One style per event in DEFAULT_VIGNETTE_STYLES (src/mosaic/gui/feedback.py): color, strength, length, and fade or pulse. Your styles override the defaults. SAREnvGUI(info_panel=..., chat_panel=...) replaces the side panels, and gui.set_advice_callbacks(...) runs code when advice arrives.

4.3 · LLM

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Lines 47–49 pick the AI teammate. Today it is the dummy placeholder.

src/experiment/main.py lines 47–49

    llm_client = build_llm_client(
        provider=config.get("provider", "dummy"), model=config.get("model")
    )

Today, after Alt

Three things to change

1 Keyless teammate Try it · next

llm_client=ReliableTeammate(0.7)

How reliable the advice is

2 Real LLM On your own

provider: in the YAML, key in .env

Who answers Alt

3 Prompt Appendix

prompt_builder=

What the teammate is told

The teammate contract lives in src/mosaic/llm/; hosted providers are built in src/experiment/llm.py. Any teammate implements one method of LLMClient, query(prompt) -> str. prompt_type: sparse or detailed (top of the YAML) changes how a real LLM talks; llm_nudge_interval: N makes it speak every N steps without Alt.

An AI Teammate Without a Key

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Try it Swap in a teammate that reads the map and is right 7 times in 10.

Get advisor.py into src/experiment/, from
github.com/Bkdogbey/mosaic-smc-tutorial · presentation/labs/

src/experiment/main.py imports, then the gui = line

+import json
+from .advisor import ReliableTeammate
@@ further down @@
-    gui = SAREnvGUI(env, config=config, llm_client=llm_client)
+    gui = SAREnvGUI(env, config=config,
+                    llm_client=ReliableTeammate(0.7),
+                    prompt_builder=json.dumps)

Run Relaunch and press Alt.

Is the advice always right?

Before · the dummy placeholder

After · ReliableTeammate, grounded in the map

labs/advisor.py is in the tutorial repository (github.com/Bkdogbey/mosaic-smc-tutorial, presentation/labs/). Any teammate implements one method of LLMClient, query(prompt) -> str (src/mosaic/llm/client.py). At 0.7, about three replies in ten point at a decoy: ask the room whether anyone noticed. That is the reliance question from the start of the tutorial. The “after” image was captured at reliability 1.0. No API key is used in this session, so this is the one teammate attendees see give grounded advice.

Connect a Real LLM

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

On your own With your own key: two packages, one key, two lines of YAML.

terminal once, with mosaic_env active

python -m pip install llama-index-llms-openai tabulate

.env a new file in the mosaic folder

OPENAI_API_KEY=<your key>

configs/experiment.yaml new top-level lines

+provider: openai
+model: gpt-4o-mini

Run Relaunch and press Alt.

How it answers

1 Prompt The game state, as rules and a table

2 Model One reply, a few seconds later

3 Chat The participant decides whether to follow it

Not run live: no API key is used in this session. Walk through the three cards so attendees can do it afterwards with their own key, then reveal how the reply is produced. A key belongs only in a local .env, never in a repository. main.py already reads provider and model from the top of the YAML and loads .env (load_dotenv()); build_llm_client connects on the first Alt, not at launch, so no key or a typo shows as an error in the chat and the game keeps running. The model name must be one the package recognizes. tabulate builds the prompt’s table (DataFrame.to_markdown()); MOSAIC does not declare it yet, and without it Alt replies “Import tabulate failed”. What the model is told is on the appendix slide “What the Teammate Is Told”.

4.4 · Sensing

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Each step produces an observation. The study runner streams it with sensors on one clock.

src/experiment/main.py line 46

    env.reset()

obs → agent_x, agent_y, direction, grid,
      saved_victims, remaining_victims,
      victim_health, step_count, …
    + action, reward, llm_response
    → study runner: one JSON line per frame on LSL

Three things to change

1 Recorded fields Notebook

print(sorted(obs))

What each step holds

2 Custom fields

observation= in build_sar_env

Add or drop fields

3 Sensors Demo · next

register_sensor(...)

Gaze or EEG on the game’s clock

Observations are built in src/mosaic/sar/observations.py; the study runner is in src/experiment/. read_data() is in src/experiment/game.py (lines 108–124, SARGameTrial). The notebook (notebooks/02_mosaic_human_ai.ipynb) prints the fields in Step 4 and records them with gaze in Steps 8–9. Change what each step records by subclassing ObservationProcessor and passing it as observation=. replay.py plays a recording back. The full study runner needs the lab’s ixp package and LSL, which this tutorial does not install.

Live Demo: Eye Tracking

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

Presenter demo A Tobii eye tracker on the lab’s study runner, while a volunteer plays.

src/experiment/eye_demo.py the study runner

     experiment = Experiment(config)
+    experiment.register_sensor(
+        name="TobiiEyeTracker", sensor_cls=TobiiEyeTracker,
+        sensor_config={})
+    experiment.calibrate_sensor(
+        "TobiiEyeTracker", screen=config["display"],
+        fullscreen=config["fullscreen"])
     experiment.add_task(task_cls=SARGameDemo, ...)

What lands next to the game data?

Watch for

1 Calibration Five dots

2 Play A short mission

3 Two streams, one clock TobiiEyeTracker (gaze, 60 Hz) and SARGame (game state, every frame), side by side in LabRecorder

Presenter only: the lab’s ixp package (not on PyPI), tobii_research, PsychoPy, pylsl, ray, and LabRecorder. Machine setup and the rehearsed steps are in EYETRACKER_SETUP.md. Run PYTHONPATH=src python -m experiment.eye_demo from the mosaic folder. The two calls on the slide are the same ones the lab’s full study uses (experiment.py); eye_demo.py adds them to a single two-minute mission with the keyless teammate, so no API key is used and the study files are not edited. Seat the volunteer 60 to 65 cm from the screen. Calibration is five points (center and four at 0.2 from the edges): look at each dot until it disappears; SPACE accepts, R redoes. Esc ends the mission. Reveal the third item once LabRecorder is on screen: TobiiEyeTracker is gaze at 60 Hz, 9 channels; SARGame is one JSON state per frame, about 30 a second. Gaze and game land on one clock, so you can see where the volunteer looked before and after each rescue; notebook Step 9 reads the saved recording. If the device fails, go to the next slide, which shows the rehearsal recording.

The Synchronized Record

Map 4.1 SAR 4.2 GUI 4.3 LLM 4.4 Sensing

One recorded mission gaze and game on the same clock

What to look for

Where Which part of the screen holds the gaze?

When What do the eyes do in the second after a rescue?

Advice Where does gaze go when the teammate replies?

Gaze and game share one clock, so every look lines up with what happened.

The figure is real: tools/make_gaze_figure.py drew it from the rehearsal recording, the LabRecorder file of the slide 33 demo, with the same code as notebook Step 9. Top: each circle is a fixation, bigger for longer, on the screen’s three areas. Bottom: one clock, the victims saved and under it the area the gaze was in; dark gaps are moments the tracker lost the eyes. In this recording the gaze stayed on the game view (99%) and did not go to the info panel after any of the three rescues. The three rows are the questions a researcher asks of a record like this. Say that the third needs a real LLM teammate: this mission ran with the keyless one, so it has no advice in it. This slide is also the fallback: if the tracker fails, skip the live demo and show this. Rehearsal calibration was rough, so the gaze sits low on the screen; after the venue rehearsal redraw it with python tools/make_gaze_figure.py <recording.xdf> and render again.

Where to Go Next

The parts you changed today, and where they live.

src/experiment/ One study built on the package. Copy this folder to start yours.

main.py

placers.py

llm.py

pacing.py

game.py

replay.py

sensors/

API build_llm_client · LavaRiskVictimPlacer · SARGameTrial

composes

src/mosaic/core/ 4.1

Level generation and cameras

level.py

placers.py

camera.py

API Placer · CameraStrategy

src/mosaic/sar/ 4.1 4.4

The search-and-rescue task

env.py

placers.py

actions.py

objects.py

observations.py

API build_sar_env · RescueAction

src/mosaic/gui/ 4.2

What the participant sees

main.py

user.py

chat.py

info.py

feedback.py

API SAREnvGUI · EdgeVignette

src/mosaic/llm/ 4.3

The teammate contract

client.py

process_prompts.py

prompts.yaml

pathfinding.py

parser.py

API LLMClient · ask

Have the repository open in an editor beside this slide. mosaic/ is the reusable package; experiment/ is one study built on it, and it is the folder attendees copy and change for their own work (its sensors/ folder is the rest of 4.4). Every file named here exists in the MOSAIC repository. The badges match the four parts of Part Four.

MOSAIC in a Study

What you configured today, put to work in a study of people and an LLM teammate.

LLM-Mediated Human-AI Interaction in Search and Rescue: Impact of Expertise on Attentional Allocation Elahe Oveisi · Hemanth Manjunatha

Task The search-and-rescue mission, with victims whose health runs down.

Teammate Two LLM teammates and a no-LLM baseline, three trials per participant.

Sensing A Tobii eye tracker and the game on one clock, through LSL.

Question Does guidance change where people look, and does expertise matter?

Monday, October 5 · 14:00–14:15 Grand C · Session MoA10 Scan for the talk’s slides

Come and hear what they found: Monday at 14:00, Grand C.

An invitation, and an application of what attendees just configured: say what the study did, and leave the results for the talk. Paper MoA10.3 in the regular session Decision Support System I, Monday October 5, 14:00 to 14:15, room Grand C. Thirteen participants each played three trials in random order: two with an LLM teammate and one no-LLM baseline, up to 15 minutes or 1,000 moves. Victim health deteriorates, and Alt asks for advice and reveals victim health for five seconds. Gaze and pupil data came from a Tobii eye tracker, on one clock with the game through LSL: the same pieces as Part Four. Participants were grouped into experts and novices. The code opens the talk’s slides (ihuman-lab.github.io/presentations/ieee-smc-eo/); the program entry is conf.papercept.net, SMC26 program, Monday, MoA10.3. Check the time and room against the final program on the day.

Thanks!

Questions?

MOSAIC github.com/iHuman-Lab/mosaic

Appendix

More changes and reference

Each change starts from the stock files

Locked Rooms

Set the chance that a room is locked: 0.5 locks about half, 0.9 most.

src/experiment/main.py line 43

-locked_room_placer=LockedRoomPlacer(locked_room_prob=0.5),
+locked_room_placer=LockedRoomPlacer(locked_room_prob=0.9),

Run Relaunch and walk to the nearest door.

How much of the building can you still reach?

Before · 0.5: 4 locked doors

After · 0.9: 8 locked doors

Keep locked_room_prob below 1.0: at 1.0 generation never finishes. The count is max(1, int(rooms × prob)), so a 3×3 building always locks 4 rooms at 0.5 and 8 at 0.9; which rooms changes run to run. LockedRoomPlacer lives in src/mosaic/sar/placers.py; its place_all() picks which rooms lock and hides each key in an unlocked room. Placement is a pluggable interface: Placer in src/mosaic/core/placers.py has one method, place_all.

Time Limit

Give the mission 2 minutes instead of 5.

configs/experiment.yaml a new top-level line, not under game:

 display: &display 0
 fullscreen: &fullscreen true
+max_time: 2

Run Relaunch and watch the timer.

How will you play with only 2 minutes?

Before · the default

After · max_time: 2

At 0:00 the timer stops; the mission keeps going.

main.py passes the whole YAML to SAREnvGUI (src/mosaic/gui/main.py), which reads max_time (minutes) from the top level. The max_time under game: configures the lab’s full study runner, not this run. On its own the timer only counts down: at 0:00 the mission keeps going. The lab’s study runner (SARGameTrial.execute) is what ends a trial.

The Info Panel

Hide how many victims remain.

Get panel.py into src/experiment/, from
github.com/Bkdogbey/mosaic-smc-tutorial · presentation/labs/

src/experiment/main.py an import, then the gui = line

+from .panel import NoProgressPanel
@@ further down @@
-    gui = SAREnvGUI(env, config=config, llm_client=llm_client)
+    gui = SAREnvGUI(env, config=config, llm_client=llm_client,
+                    info_panel=NoProgressPanel(800, 400, 400))

Run Relaunch and look at the mission box.

What does the participant no longer know?

Before · rescued and remaining

After · no remaining count

NoProgressPanel (labs/panel.py, tutorial repo) subclasses InfoPanel (src/mosaic/gui/info.py) and overrides one method, _mission_html. The three numbers are the panel’s x offset, width, and height in the 800-px window. chat_panel= accepts a ChatPanel subclass the same way.

What the Teammate Is Told

Before trusting its advice, look at what it is told.

Every Alt sends

1 Rules Rescue real survivors, nearest first, never through lava

2 Every real victim Even ones out of your view

3 No decoys Left out, so it never points at one

Change it SAREnvGUI(..., prompt_builder=...)

The start of one prompt

Agent: room (0,1) facing West | …

Object       Direction  Visible  PathLength
Victim4      East       Yes      9
Victim2      South      Yes      0
RedKey1      East       Yes      5
PurpleKey1   East       Yes      6
PurpleDoor1  South      Yes      34
RedDoor1     East       No       10

build_prompt and _build_table are in src/mosaic/llm/process_prompts.py; _build_table skips FAKE_VICTIM cells (line 60), so decoys never reach the model. Every real victim, key, and door in the building is listed with direction, reachability, and path length; Visible marks what is near the participant. The rules are the preamble in src/mosaic/llm/prompts.yaml. This asymmetry matters for any study of reliance: the AI knows where the real victims are, and the participant has to tell them from decoys.