Physical AI Team Intern

The team

I’m one of four engineers on Janicki’s founding Physical AI team, reporting directly to the CEO and CTO. Our team is tasked with applying modern robot learning and language models to the company’s manufacturing floor.

Within this brand new team, my job as an intern is to learn each new area of robotics fast, prove it on real hardware, and teach it back to my mentors. My first task was to design and assemble a simple automated guided vehicle (AGV) with scrap metal and simple features: suspended mecanum wheel drivebase, a powerful brain capable of processing any model, lidar and camera sensor capabilities, and a 6-axis robotic arm. Once done, I was tasked with integrating LLMs and VLMs onto the AGV, getting control systems to respond to simple English commands. Then I learned sensor fusion, combining any number of camera feeds to teach the robot what itself and the world around it looks like. Then I fused depth-camera output with a 6-axis arm to find a specific object and pick it up with 100% accuracy, knowing at all times exactly where both the arm and the target object were located.

I finished my assigned platform R&D (all of the above) in three weeks, then pitched and delivered the rest of the summer myself: making a legacy industrial arm autonomous.

Making a legacy arm autonomous

Most arms used in industry are incredibly precise, but blind. They repeat taught paths exactly, with no idea what’s in front of them: set a fixture offset by an inch and the program breaks. Blind also means unsafe. Keeping people clear of a traditional arm takes cages, light curtains, and other heavy equipment that makes operation significantly harder than necessary.

My work aimed to solve … all of those issues. Not only did I integrate this decade-old operating system with 2026 ROS, I also built my own safety layer. The cameras I added in the environment around the arm successfully provided a guidance feed, but were also layered into a separate, analog control system using onboard CV models to detect humans and stop the arm if someone got too close.

This expands on what I’d already seen on the AGV. Its onboard arm was blind too, until I taught myself sensor fusion and let the robot see both its own arm and the target.

×−+

industrial-arm.glb (coming soon)

awaiting model

6-axis industrial arm

[ model loading soon… ]

The work broke into three pieces:

• Expose the arm as a ROS2 node. I brought the arm’s controller onto ROS2 through its real-time control interface (RSI), running on an onboard embedded compute module. Once the arm is a node in my system, it becomes something the rest of a modern robotics stack can talk to.
• Give it sight. Stereo depth cameras mapped the workspace in 3D. I fused their outputs in real-time to create a 3D model of the robot and its environment, extending a custom vision model so the arm could find and align to its tooling anywhere.
• Put the trained motion models on real hardware. My team and I trained targeted motion models, using reinforcement learning on thousands of simultaneous simulations on Isaac Sim.

Talking to a robot in English

One of the features I’m most proud of was how I incorporated a locally hosted LLM that turned a plain-English operator instruction into a deterministic, real-time motion procedure.

The key constraint was that we could never let the language model drive the arm. No current model can be trusted to do so because we can’t weigh a human life against even the smallest chance of an LLM hallucination. Instead, we can instruct an LLM to interpret human commands, then select and parameterize procedures that are deterministic and verified. A model that occasionally hallucinates works as an effective interface. It does not work as a motion controller. That sharp boundary is what makes natural-language control safe on potentially deadly hardware.

Language, vision, and speech all ran entirely on-robot. No round-trip to the cloud meant no API latency, no waiting on slow Wi-Fi, no data leaving the building, and importantly no cost. Leaning on small deterministic models made everything significantly faster, since each model is trained to do exactly one job. Each model we could turn deterministic saved maybe 0.1s per command. Each of these minuscule time-saves is what enabled the whole system to run in real time.

Autonomous mobile robots

Alongside the arm work, I built and programmed an autonomous mobile robot using the AGV frame I created previously:

• Sensor-fused navigation on ROS2 (Humble and Jazzy)
• LIDAR + depth-camera for 3D point-cloud mapping
• An onboard 6-axis arm to make the platform mobile and manipulative
• A natural-language agent layer, the same idea as above applied to a moving base
• Multiple smaller agentic layers to control each aspect of the AGV

Those agent layers ran as a stack of small LLM agents, each handling one aspect of the AGV. Since failures of this robot don’t run anywhere near the cost of a human life, I was able to push language models to their limits and see what level of real-world interaction was truly possible.

Operator dashboard

Autonomy is only useful if a human can supervise it. I designed and programmed an operator dashboard with live sensor feeds, manual arm control, task macros, and the plain-English console. With it, an operator can see exactly what the robot sees, take over instantly, and issue new work without touching code.

What I took from it

The gap between simulation and reality is where the engineering lives. A model that’s 95% accurate in simulation is a robot that crashes one time in twenty, and these robots run each action far more than twenty times. Most of our work was dedicated to defining when a model could be trusted as truly accurate, testing motion not only in simulation but in real-world environments as well.

LLMs are not the answer to everything. Large language and vision-language models are astonishingly capable. Harnessed properly, they can take on almost any task. But once I’d loaded them onto a robot I’d built myself, their limits got concrete. Even locally hosted, running several of them in parallel is slow enough to blow past real-time latencies, sometimes taking minutes just to process a set of images. And they take significant effort to properly use for accurate models: you can train the hallucinations down, but you can never be fully certain an output is correct.

Small deterministic models are the counterweight. Each model is trained to do a single thing, like detecting one specific part. At production scale, deterministic models are just as accurate at that one job as an LLM, often more so, because it doesn’t need to know how to do anything else. It runs orders of magnitude faster, and you can run five, ten, a hundred of them side by side. You’ll never understand everything in an environment the way an LLM can, but for 99.9% of what a factory robot needs, you don’t have to. Teach the robot exactly what it needs to recognize with deterministic models, put a light speech-to-text layer and an LLM for interpretation on top, and you get something that listens to English commands without losing real-time functionality.