Using a Nintendo Switch as ROS2 controller

Inspiration and resources

Control your ROS robot from your phone! - is the Articulated Robotics video and blog post that inspired me originally. It introduces a basic browser-based tool for teleop and hints at the potential of Foxglove for building an even more capable robot controller. Later Josh wrote a dedicated Foxglove extension for joysticks that looks great for controlling robots in development, testing, production and to review recorded data - check the video, discussion and repo. Not clear to me if one actually needs ROS installed on the console to go for this solution. Then there is also this video and discussion of a Foxglove webcam panel, and this one in which Josh describes the advantages of the Steamdeck .
Steam Deck as a Robot Controller is a blog post by A. Kamath, which discussses various ways to install ROS2 on the Steamdeck including ubuntu, robostack and distrobox, as well as other useful tools such as Foxglove, Plotjuggler, etc.
Robodeck is a Youtube playlist by Polyhobbyist / Lou Amadio about running ROS2 on Steamdeck using Robostack

At this point I could barely resist the urge to buy a Steamdeck. Then I came across this twitter thread about a cool project that used Nintendo Switch Joy-Cons to train a 2-arm robot, more details on the blog post and repo. Shortly after that, I learnt about @r_frojd running ROS on a Nintendo Switch.

This was clearly the Universe speaking to me, because I happen to own a Nintendo Switch, so I realized I was destined to enter history as the pioneer who found the way to use the Nintendo Switch in lieu of the Steamdeck as a robot controller - or at least the first who published it.

A Google search informed me that installing Linux on the console involves hacking it first, and not all consoles are hackable. As a matter of fact, the one I owned was not suitable.

No surprise most of the hacking tutorials I found online (e.g. [this one](How to Mod Your Nintendo Switch! FULL GUIDE (Latest firmware not supported) )) were directed at gamers, focused on working from a Windows PC, and involved backing up existing data to allow dual boot with the original firmware.

This guide focuses instead on installing Linux on a fresh SD card, without dual boot or mixing with the original OS or homebrew alternatives, and it assumes working from a Linux PC.

Bill of Materials

I rushed to buy an old unpatched Nintendo Switch console from ebay for ~$125. Not such a bargain considering I had to pay additional ~70€ in taxes(!) when a few weeks later it arrived from Japan without Joy-Cons, dock, or charger. A set of new Joy-cons I ordered from Amazon added 67.90€ to the bill. That's ~270€ altogether, so maybe hunting for old S/N in the local CeX would have been a better idea - probably faster and cheaper.
A fast UHD-I microSD card: the switchroot guide recommends the 128Gb U3 class Samsung Pro Plus, 28.58€ from Amazon
An RCM jig, 6.29€ from Amazon

Check that your console can be hacked

Newer consoles have been patched against the vulnerability exploited by this method, so you'll need to:

Find out the Serial Number of your console (Where? TLDR; read the physical sticker on the side of the console, or go to HOME Menu > System Settings > Scroll down the left-hand menu on the side and select System > Serial Information > Console Serial Number)
Check if you're in luck using one of the online SN checkers, e.g. SSNC or ismyswitchpatched.com

Prepare the SD card

Format the SD card to FAT32. The fastest method in Linux is using the terminal.

Download Hekate 6.0.6 or newer: I downloaded hekate_ctcaer_6.2.2_Nyx_1.6.4.zip from here

Unzip and drag all contents onto the SD

Prepare the Hekate payload

We need a payload launcher for Linux.

Install dependencies and clone the fusee-launcher repo.

$ sudo apt update && sudo apt upgrade
$ sudo apt install python3 python3-pip python3-usb git 
$ python3 -m pip install --upgrade pip
$ git clone git clone https://github.com/Yankas/fusee-launcher.git

Copy the file hekate_ctcaer_6.2.2.bin from the zip to the fusee-launcher folder

Booting the console into RCM Mode

Source: YouTube tutorial

Remove the right Joycon and slide the RCM clip fully into the rail
Power off the console fully Power button long press > Power Options > Turn off
Plug the console to one of the blue USB 3.0 ports of the computer
While holding Volume Up button press Power button once. Screen should remain black. We're in!

Inject the Hekate payload

$ cd fusee-launcher
$ sudo python3 fusee-launcher.py hekate_ctcaer_6.2.2.bin
Important note: on desktop Linux systems, we currently require an XHCI host controller.
A good way to ensure you're likely using an XHCI backend is to plug your
device into a blue 'USB 3' port.

Identified a Linux system; setting up the appropriate backend.
Found a Tegra with Device ID: b'@\x03\x00\x18\x00\x00\x00\x08\x87x-d\x01\x10\x10b'

Setting ourselves up to smash the stack...
Uploading payload...
Smashing the stack...
The USB device stopped responding-- sure smells like we've smashed its stack. :)
Launch complete!

Power up the console. The first time we need to set the time, then we are in Hekate Home. You may unplug the console and remove the RCM clip.

Now we can follow the switchroot tutorial to install Ubuntu 22.04. This video is a good help. theofficialgman-ubuntu-jammy-5.1.2-2024-10-18.7z

Make a partition of the SD card

Tools > Partition SD Card

Nyx Settings > Dump Joy-Con BT

When dumping the Joy-con BT I got a message that it failed to get IMU calibration, see screenshot. Joy-Cons are brand new, I checked in the original firmware they work fine and even recalibrated along these instructions: https://www.youtube.com/watch?v=BAMB-kfmqYw

Ta-da!

Configure Foxglove

Sign in at https://foxglove.dev

Open connection > rosbridge

Chromium Menu > Cast, save and share > Create shortcut...

rename and click Open as window

Right click on the icon > Allow launching

Foxglove resources

https://docs.foxglove.dev/docs/connecting-to-data/frameworks/ros2

Foxglove gives your team observability-driven features to bring robotics to market faster: record device data, tag notable events, collaborate, and more.

Here are some useful resources to get started:

email [email protected] , join our Slack community, or explore our developer docs.

Joy-Cons in Linux

A quick'n'dirty GUI to display Joy-Con orientation

I mocked up a quick'n'dirty GUI to display the orientation of the Joy-Cons. As a first step joy_con_gui.py draws two blocks resembling the Joy-Cons and allows to rotate them with the keyboard.

Keyboard Controls:

Left Joy-Con (Neon Red)

W → Rotate Down (tilt backward, decreasing X-axis angle)
S → Rotate Up (tilt forward, increasing X-axis angle)
A → Rotate Left (tilt counterclockwise around Y-axis)
D → Rotate Right (tilt clockwise around Y-axis)

Right Joy-Con (Neon Blue)

I → Rotate Down (tilt backward, decreasing X-axis angle)
K → Rotate Up (tilt forward, increasing X-axis angle)
J → Rotate Left (tilt counterclockwise around Y-axis)
L → Rotate Right (tilt clockwise around Y-axis)

Note: the Yaw rotation axis is missing !

Getting orientation out of the Joy-Cons

Next I want to get the orientation from the motion control data streamed by the Joy-Cons. Like this:

Inspiration came from this project https://github.com/AlmondGod/Nintendo-Aloha (see the twitter thread and blog post) which used this promising python library: https://github.com/tocoteron/joycon-python (there may be some recent forks with added functionality)

$ pip install joycon-python hidapi pyglm

Unfortunately I could not make it work: Joy-Cons are connected and send data via evtest or evdev (only buttons though, not motion control) but still hid.enumerate()finds only mouse and keyboard but not the Joy-Cons, therefore joycon-python does not work.

I did extensive troubleshooting and tried many things:

tried installing cython-hidapi from here: https://github.com/trezor/cython-hidapi and hid
added udev rules to permanently give permissions $ sudo nano /etc/udev/rules.d/50-nintendo-switch.rules:

# Switch Joy-con (L) (Bluetooth only)
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", KERNELS=="0005:057E:2006.*", MODE="0666"

# Switch Joy-con (R) (Bluetooth only)
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", KERNELS=="0005:057E:2007.*", MODE="0666"

# Switch Pro controller (USB and Bluetooth)
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="057e", ATTRS{idProduct}=="2009", MODE="0666"
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", KERNELS=="0005:057E:2009.*", MODE="0666"

# Switch Joy-con charging grip (USB only)
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="057e", ATTRS{idProduct}=="200e", MODE="0666"

After editing the file it is necessary to reload the rules with: $ udevadm control --reload-rules

Somehow I came to the conclusion that this is the problem:

$ sudo modprobe hid_nintendo
modprobe: FATAL: Module hid_nintendo not found in directory /lib/modules/5.15.0-131-generic

I am on ubuntu 20.04 and my kernel version 5.15.0-131 does not include the hid-nintendo driver (it was added in kernel 5.16 which corresponds to ubuntu 21.10)

Tried upgrading the kernel with the following steps and broke ubuntu (no wifi) so I went back.

Add the latest mainline kernel repository:

$ sudo add-apt-repository ppa:cappelikan/ppa
$ sudo apt update

Install the latest kernel (e.g., 6.5 or 6.6):

$ sudo apt install mainline
$ mainline --install-latest

Reboot:
```
$ sudo reboot
```
Verify the upgrade:
```
$ uname -r
```

After upgrading, try:

$ sudo modprobe hid_nintendo

If successful, dmesg | grep -i nintendo should show logs.

Anyway, I had to revert. Instead I tried to install it manually:

Download the hid-nintendo Source Code:

$ cd /usr/src
$ sudo git clone https://github.com/nicman23/dkms-hid-nintendo.git
$ cd dkms-hid-nintendo

$ sudo dkms add .
$ sudo dkms build nintendo -v 3.2
$ sudo dkms install nintendo -v 3.2

After this it seemed to work...

$ sudo modprobe hid_nintendo
$ lsmod | grep hid_nintendo
hid_nintendo           49152  0
ff_memless             24576  1 hid_nintendo
hid                   147456  6 hid_nintendo,i2c_hid,hidp,usbhid,hid_multitouch,hid_generic
$ dmesg | grep -i nintendo
[...]
[ 3340.136843] nintendo 0005:057E:2007.0005: delta=136 avg_delta=15
[ 3344.471943] nintendo 0005:057E:2007.0005: compensating for 7 dropped IMU reports
[ 3344.471951] nintendo 0005:057E:2007.0005: delta=136 avg_delta=15
[ 3348.041780] nintendo 0005:057E:2007.0005: compensating for 7 dropped IMU reports
[ 3348.041791] nintendo 0005:057E:2007.0005: delta=136 avg_delta=15
[...]

But still hid.enumerate() did not detect the joycons.

I'll try on a more modern kernel...

Other options to explore

JoyconLib is a library for Unity featuring button/stick polling, HD rumble, and accelerometer data processing. By Evan Kahn / @wormyrocks. Discussion: https://gbatemp.net/threads/joy-con-unity-library.486629/
JoyCon-Driver is a Windows Driver for the Nintendo Switch JoyCons and Pro Controller with support for analog stick and motion controls
Nintendo Switch Reverse Engineering all you ever wanted to know about the Nintendo. See this issue: Gyroscope / Accelerometer Status #18
A Joy Con Web HID - Web HID driver for Nintendo Joy-Cons with support for all buttons, analog sticks, and the device’s gyroscope and accelerometer sensors.
Using Nintendo Joy-Cons to Control Rock ‘Em Sock ‘Em Robots
Using Nintendo Switch controllers on Linux discussion claims installing dkms-hid_nintendo as a module and the joycond daemon should do the trick, but someone in the comments complains this doesn't work in 20.04...
joycon is a Joy-Con input driver for Linux, but motion control is still in their To Do list

The one that worked

This is a great summary review of the different options available by 2020... Claims hid-nintendo is present since kernel 5.10, see also this

install dkms_hid_nintendo if the kernel does not include hid_nintendo

$ git clone https://github.com/nicman23/dkms-hid-nintendo
$ cd dkms-hid-nintendo
$ sudo dkms add .
$ sudo dkms build nintendo -v 3.2
$ sudo dkms install nintendo -v 3.2

install joycond driver:

$ git clone https://github.com/DanielOgorchock/joycond
$ sudo apt-get install libevdev-dev
$ cd joycond
$ cmake .
$ sudo make install
$ sudo systemctl enable --now joycond

install the cemu hook

$ pip3 install git+https://github.com/joaorb64/joycond-cemuhook

Run the hook.
Connect the Joy-Cons to the PC. First wake them up pressing the black centre button on the side until the LED starts scanning. Otherwise the Connect slide switch in the Bluetooth Settings window will not respond.
Pair them: pressing SL+SR buttons to pair each Joy-Con individually or ZL+ZR to pair them combined

$ joycond-cemuhook 
Looking for Nintendo Switch controllers...
================= ('127.0.0.1', 26760) ================
Device         LED status   Battery Lv   MAC Addr    
1 🕹️ L          ■ □ □ □      55.0 ▄       AC:FA:E4:A4:A2:5B
2 🕹️ R          ■ ■ □ □      70.0 ▅       AC:FA:E4:A4:36:BC
3 ❎ 
4 ❎ 
=======================================================


================= ('127.0.0.1', 26760) ================
Device         LED status   Battery Lv   MAC Addr    
1 🎮 L+R       ■ ■ ■ □      55.0 ▄       AC:FA:E4:A4:36:BC
2 🎮 L+R       ■ ■ ■ □      55.0 ▄       AC:FA:E4:A4:A2:5B
3 ❎ 
4 ❎ 
=======================================================

But.. how is motion control data exposed? Turns out.. new dedicated devices have appeared:

$ evtest
No device specified, trying to scan all of /dev/input/event*
Not running as root, no devices may be available.
Available devices:
/dev/input/event20:	Nintendo Switch Left Joy-Con IMU
/dev/input/event22:	Nintendo Switch Right Joy-Con IMU
/dev/input/event23:	Nintendo Switch Combined Joy-Cons
Select the device event number [0-23]: 20
Input driver version is 1.0.1
Input device ID: bus 0x5 vendor 0x57e product 0x2006 version 0x8001
Input device name: "Nintendo Switch Left Joy-Con IMU"
Supported events:
  Event type 0 (EV_SYN)
  Event type 3 (EV_ABS)
    Event code 0 (ABS_X)
      Value    287
      Min   -32767
      Max    32767
      Fuzz      10
      Resolution    4096
    Event code 1 (ABS_Y)
      Value    -18
      Min   -32767
      Max    32767
      Fuzz      10
      Resolution    4096
    Event code 2 (ABS_Z)
      Value   4216
      Min   -32767
      Max    32767
      Fuzz      10
      Resolution    4096
    Event code 3 (ABS_RX)
      Value      0
      Min   -32767000
      Max   32767000
      Fuzz      10
      Resolution   14247
    Event code 4 (ABS_RY)
      Value   1000
      Min   -32767000
      Max   32767000
      Fuzz      10
      Resolution   14247
    Event code 5 (ABS_RZ)
      Value    999
      Min   -32767000
      Max   32767000
      Fuzz      10
      Resolution   14247
  Event type 4 (EV_MSC)
    Event code 5 (MSC_TIMESTAMP)
Properties:
  Property type 6 (INPUT_PROP_ACCELEROMETER)
Testing ... (interrupt to exit)
Event: time 1738461230.860033, type 4 (EV_MSC), code 5 (MSC_TIMESTAMP), value -711924796
Event: time 1738461230.860033, type 3 (EV_ABS), code 3 (ABS_RX), value -1999
Event: time 1738461230.860033, type 3 (EV_ABS), code 4 (ABS_RY), value 3000
Event: time 1738461230.860033, type 3 (EV_ABS), code 5 (ABS_RZ), value -1999
Event: time 1738461230.860033, type 3 (EV_ABS), code 0 (ABS_X), value 289
Event: time 1738461230.860033, type 3 (EV_ABS), code 2 (ABS_Z), value 4218
Event: time 1738461230.860033, -------------- SYN_REPORT ------------
Event: time 1738461230.860035, type 4 (EV_MSC), code 5 (MSC_TIMESTAMP), value -711919796
Event: time 1738461230.860035, type 3 (EV_ABS), code 3 (ABS_RX), value -4998
Event: time 1738461230.860035, type 3 (EV_ABS), code 4 (ABS_RY), value 1000
Event: time 1738461230.860035, type 3 (EV_ABS), code 5 (ABS_RZ), value -999
Event: time 1738461230.860035, type 3 (EV_ABS), code 0 (ABS_X), value 290
Event: time 1738461230.860035, type 3 (EV_ABS), code 2 (ABS_Z), value 4220
Event: time 1738461230.860035, -------------- SYN_REPORT ------------
Event: time 1738461230.860036, type 4 (EV_MSC), code 5 (MSC_TIMESTAMP), value -711914796
Event: time 1738461230.860036, type 3 (EV_ABS), code 3 (ABS_RX), value -5997
Event: time 1738461230.860036, type 3 (EV_ABS), code 4 (ABS_RY), value 3000
Event: time 1738461230.860036, -------------- SYN_REPORT ------------
Event: time 1738461230.875011, type 4 (EV_MSC), code 5 (MSC_TIMESTAMP), value -711894796
Event: time 1738461230.875011, type 3 (EV_ABS), code 3 (ABS_RX), value 5997
Event: time 1738461230.875011, type 3 (EV_ABS), code 4 (ABS_RY), value 2000
Event: time 1738461230.875011, type 3 (EV_ABS), code 5 (ABS_RZ), value -1999
Event: time 1738461230.875011, type 3 (EV_ABS), code 2 (ABS_Z), value 4221
[...]

Net step is writing a python script that connects to the device using evdev (see this python-evdev tutorial) and monitors the stream of events to updates an orientation state variable. Note the sign conventions.

one_joy_con.py- first attempt at displaying Roll Pitch Yaw for a JoyCon.

two_joy_cons.py - improved and encapsulated in a class to display RPY for both JoyCons. Kalman filter is implemented but deactivated because I could not get it to work. Roll is ok, Pitch is clipped +/-90 degrees, Yaw is still a drama...

Check out the video in this February 3, 2025 at 4:09 AM post by @aergenium.bsky.social:

I’m looking at using a Nintendo Switch to control my robots. Joy-Cons promise to be fun, and I had a first breakthrough today. A short video, and details in the repo: github.com/mhered/ninte...

Next step is to link this with the GUI - done in joy_con_gui.py

To Do: a nicer GUI using CAD model instead of brick

Found this post on Rendering STL files with matplotlib using numpy-stl.

numpy-stl is python library to manipulate meshes and 3D objects.

Downloaded a CAD model of the Joy-Cons in STL from GrabCAD, split it in parts using FreeCAD and simplified to a low-poly mesh (~2000 vertices) each using https://3dless.com/

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
assets		assets
src		src
.gitignore		.gitignore
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Using a Nintendo Switch as ROS2 controller

Inspiration and resources

Bill of Materials

Check that your console can be hacked

Prepare the SD card

Prepare the Hekate payload

Booting the console into RCM Mode

Inject the Hekate payload

Configure Foxglove

Foxglove resources

Joy-Cons in Linux

A quick'n'dirty GUI to display Joy-Con orientation

Keyboard Controls:

Left Joy-Con (Neon Red)

Right Joy-Con (Neon Blue)

Getting orientation out of the Joy-Cons

Other options to explore

The one that worked

To Do: a nicer GUI using CAD model instead of brick

About

Releases

Packages

Languages

mhered/nintendo_switch_robot_controller

Folders and files

Latest commit

History

Repository files navigation

Using a Nintendo Switch as ROS2 controller

Inspiration and resources

Bill of Materials

Check that your console can be hacked

Prepare the SD card

Prepare the Hekate payload

Booting the console into RCM Mode

Inject the Hekate payload

Configure Foxglove

Foxglove resources

Joy-Cons in Linux

A quick'n'dirty GUI to display Joy-Con orientation

Keyboard Controls:

Left Joy-Con (Neon Red)

Right Joy-Con (Neon Blue)

Getting orientation out of the Joy-Cons

Other options to explore

The one that worked

To Do: a nicer GUI using CAD model instead of brick

About

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages