Neuro:Talk
Inventor | Masha Kosanovic and Nadja |
micro:bit IDE | MakeCode Editor |
Best Used | Makerspace |
Difficulty
Hardware | Medium |
Software | Hard |
Special Requirement
SG90 Micro Servo | Required |
3D printer | Required |
Glue Gun | Recommended |
Project Overview
Welcome to Neuro:Talk, Your Gateway to Expressive Communication Through ASL!
These instructions will guide you through the process of building and programming your Neuro:Talk device, which takes Electromyography (EMG) signals from American Sign Language (ASL) gestures, converts them into Morse code, and displays the output via tones, an LED display, and a rotating pointer on a servo and cup. Let's dive in and explore this unique method of communication!
Build Instructions
Materials
- 2x spiker:bit kit (spiker:bit, 3x electrodes, and 1x orange cable)
- 2x micro:bit
- 1x servo motor
- 1x cup (for the rotating pointer)
- 1x cardboard, tape, and scissors for housing
- (Stick and glue gun not shown)
1. Start by grabbing the cup and tape
2. Tape the cup for improved grip, this will matter when we attach the servo
3. Tape the cup onto the piece of paper
4. Now grab the stick, in this case it's a 3D printed part. Apply glue from a glue gun to where the servo hat will connect
You can download the stl file for 3D printing part from here.
5. Attach the servo to the stick. The servo should then be connected to the micro:bit
6. Apply glue from the glue gun onto the tape cup and then place the servo onto the tape cup
7. Add a 0, letters, and a space matching your letter array. This is best done by running the code and having the servo move through the array
Code
Now that your device is assembled, it's time to program it to decode ASL gestures into Morse code and display it. This code is split into two parts, send and receive. This first chunk is sending ( it looks long, but most of that is due to the arrays for alphabet and morse code ).
The program calibrates hand movies for dots and dashes, converts these into letters, and then controls a servo to select the letters chosen. This information is communicated from micro:bit to micro:bit via the radio command.
Sender (Morse code) code
- Block Code
- Python
- Js
def on_received_string(receivedString):
if receivedString == "yes":
basic.show_string(receivedString)
servos.P0.set_angle((alphabet_list.index_of("y") + 1) * 6)
basic.pause(500)
servos.P0.set_angle((alphabet_list.index_of("e") + 1) * 6)
basic.pause(500)
servos.P0.set_angle((alphabet_list.index_of("s") + 1) * 6)
elif receivedString == "no":
basic.show_string(receivedString)
servos.P0.set_angle((alphabet_list.index_of("n") + 1) * 6)
basic.pause(500)
servos.P0.set_angle((alphabet_list.index_of("o") + 1) * 6)
basic.pause(500)
servos.P0.set_angle(0)
radio.on_received_string(on_received_string)
sentence = ""
samples: List[number] = []
spikerbit.start_muscle_recording()
radio.set_group(90)
servos.P0.set_angle(0)
morse_code_list = ["01",
"1000",
"1010",
"100",
"0",
"0010",
"110",
"0000",
"00",
"0111",
"101",
"0100",
"11",
"10",
"111",
"0110",
"1101",
"010",
"000",
"1",
"001",
"0001",
"011",
"1001",
"1011",
"1100",
"1111",
"1110",
"0011"]
alphabet_list = ["a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z"]
sending = True
total_trials = 4
recording_time = 1000
threshold = 5
while sending:
morse_code = ""
for index in range(total_trials):
basic.show_icon(IconNames.HEART)
basic.clear_screen()
basic.pause(recording_time)
samples = spikerbit.signal_block(recording_time)
non_zeros = 0
zeros = 0
i = 0
while i <= len(samples) - 1:
if samples[i] == 0:
zeros += 1
else:
if samples[i] < threshold:
non_zeros += 1
i += 1
if zeros == len(samples):
basic.show_leds("""
. . # . .
. # . # .
. # # # .
. # . # .
. . # . .
""")
else:
if non_zeros == 0:
morse_code = "" + morse_code + "1"
basic.show_leds("""
. . . . .
. . . . .
. # # # .
. . . . .
. . . . .
""")
else:
morse_code = "" + morse_code + "0"
basic.show_leds("""
. . . . .
. . . . .
. . # . .
. . . . .
. . . . .
""")
position = 0
while position <= len(morse_code_list) - 1:
if morse_code_list[position] == morse_code:
if morse_code == "1111":
basic.show_icon(IconNames.SMALL_DIAMOND)
sentence = "" + sentence + " "
elif morse_code == "1110":
basic.show_leds("""
. . # . .
. # . . #
# # # # #
. # . . #
. . # . .
""")
sentence = sentence.substr(0, len(morse_code_list))
elif morse_code == "0011":
basic.show_icon(IconNames.YES)
basic.show_string(sentence)
radio.send_string(sentence)
sending = False
basic.clear_screen()
else:
basic.show_string("" + (alphabet_list[position]))
sentence = "" + sentence + alphabet_list[position]
servos.P0.set_angle((position + 1) * 6)
position += 1
radio.onReceivedString(function (receivedString) {
if (receivedString == "yes") {
basic.showString(receivedString)
servos.P0.setAngle((alphabet_list.indexOf("y") + 1) * 6)
basic.pause(500)
servos.P0.setAngle((alphabet_list.indexOf("e") + 1) * 6)
basic.pause(500)
servos.P0.setAngle((alphabet_list.indexOf("s") + 1) * 6)
} else if (receivedString == "no") {
basic.showString(receivedString)
servos.P0.setAngle((alphabet_list.indexOf("n") + 1) * 6)
basic.pause(500)
servos.P0.setAngle((alphabet_list.indexOf("o") + 1) * 6)
}
basic.pause(500)
servos.P0.setAngle(0)
})
let sentence = ""
let samples: number[] = []
spikerbit.startMuscleRecording()
radio.setGroup(90)
servos.P0.setAngle(0)
let morse_code_list = [
"01",
"1000",
"1010",
"100",
"0",
"0010",
"110",
"0000",
"00",
"0111",
"101",
"0100",
"11",
"10",
"111",
"0110",
"1101",
"010",
"000",
"1",
"001",
"0001",
"011",
"1001",
"1011",
"1100",
"1111",
"1110",
"0011"
]
let alphabet_list = [
"a",
"b",
"c",
"d",
"e",
"f",
"g",
"h",
"i",
"j",
"k",
"l",
"m",
"n",
"o",
"p",
"q",
"r",
"s",
"t",
"u",
"v",
"w",
"x",
"y",
"z"
]
let sending = true
let total_trials = 4
let recording_time = 1000
let threshold = 5
while (sending) {
let morse_code = ""
for (let index = 0; index < total_trials; index++) {
basic.showIcon(IconNames.Heart)
basic.clearScreen()
basic.pause(recording_time)
samples = spikerbit.signalBlock(recording_time)
let non_zeros = 0
let zeros = 0
for (let i = 0; i <= samples.length - 1; i++) {
if (samples[i] == 0) {
zeros += 1
}
else {
if (samples[i] < threshold) {
non_zeros += 1
}
}
}
if (zeros == samples.length) {
basic.showLeds(`
. . # . .
. # . # .
. # # # .
. # . # .
. . # . .
`)
} else {
if (non_zeros == 0) {
morse_code = "" + morse_code + "1"
basic.showLeds(`
. . . . .
. . . . .
. # # # .
. . . . .
. . . . .
`)
} else {
morse_code = "" + morse_code + "0"
basic.showLeds(`
. . . . .
. . . . .
. . # . .
. . . . .
. . . . .
`)
}
}
}
for (let position = 0; position <= morse_code_list.length - 1; position++) {
if (morse_code_list[position] == morse_code) {
if (morse_code == "1111") {
basic.showIcon(IconNames.SmallDiamond)
sentence = "" + sentence + " "
} else if (morse_code == "1110") {
basic.showLeds(`
. . # . .
. # . . #
# # # # #
. # . . #
. . # . .
`)
sentence = sentence.substr(0, morse_code_list.length)
} else if (morse_code == "0011") {
basic.showIcon(IconNames.Yes)
basic.showString(sentence)
radio.sendString(sentence)
sending = false
basic.clearScreen()
} else {
basic.showString("" + (alphabet_list[position]))
sentence = "" + sentence + alphabet_list[position]
}
servos.P0.setAngle((position + 1) * 6)
}
}
}
Receiver (Hand sign) code
- Block Code
- Python
- Js
def on_received_string(receivedString):
global counter
basic.show_string(receivedString)
basic.pause(2000)
basic.show_icon(IconNames.HAPPY)
basic.pause(record_time)
counter = spikerbit.num_peaks_in_last(record_time)
if counter >= 2:
basic.show_icon(IconNames.YES)
radio.send_string("yes")
elif counter == 1:
basic.show_icon(IconNames.NO)
radio.send_string("no")
radio.on_received_string(on_received_string)
counter = 0
record_time = 0
spikerbit.start_muscle_recording()
radio.set_group(90)
record_time = 2000
counter = 0
radio.onReceivedString(function (receivedString) {
basic.showString(receivedString)
basic.pause(2000)
basic.showIcon(IconNames.Happy)
basic.pause(record_time)
counter = spikerbit.numPeaksInLast(record_time)
if (counter >= 2) {
basic.showIcon(IconNames.Yes)
radio.sendString("yes")
} else if (counter == 1) {
basic.showIcon(IconNames.No)
radio.sendString("no")
}
})
let counter = 0
let record_time = 0
spikerbit.startMuscleRecording()
radio.setGroup(90)
record_time = 2000
counter = 0
Operating Instructions
Now that your device is ready, it's time to use it!
- Power On: Turn on both spiker:bit devices.
- Input Morse Code: Every second after you see the "Heart" displayed on the screen "type” the Morse code you want to send using the proper arm movement. If the code length is shorter than four characters, ensure you “type” an “empty code” at the end to complete the sequence.
- Send Command: After entering your message, use the “send command” to transmit it to the receiver.
- Receive the Code: The receiver (second micro:bit) should receive the sender’s message if everything is set up correctly.
- Respond: The receiver should respond by performing a “yes” or “no” action as demonstrated in the accompanying video.
- Verify Response: The sender should confirm that the correct response was received from the receiver.
Understanding correct arm movement - Morse Code
In this project, there are three different morse code you can perform
- Dot: Flexing the arm once.
- Dash: Tightening the arm.
- Empty: Do nothing
The system uses International Standard Morse Code for the alphabet only. Use this as a reference to choose each code
Special Commands -Morse Code
For this project, some specific codes allow special actions:
---- (dash, dash, dash, dash): Add a space.
---• (dash, dash, dash, dot): Delete the last character.
••-- (dot, dot, dash, dash): Send the combined message.
Practice Makes Perfect
As you practice using the Neuro:Talk, you'll become more adept at controlling the output, ensuring that your ASL gestures are accurately represented.
Customizing Your Neuro:Talk Device (Future Work)
You can further enhance your device with additional features:
Multiple Output Modes
- Visual Mode: Use the LED display for quiet environments.
- Auditory Mode: Use the buzzer for audible communication.
- Kinetic Mode: Use the rotating pointer for a more tactile feedback option.
Advanced EMG Mapping
- Explore more complex ASL gestures and map them to longer Morse code sequences.
- Integrate additional sensors for more nuanced control and feedback.