Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MASTRA-2376] Add documentation for speech to speech #2942

Merged
merged 45 commits into from
Mar 14, 2025
Merged
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
136e048
[MASTRA-2371] add methods to support realtime voice providers
YujohnNattrass Mar 10, 2025
7371074
lint
YujohnNattrass Mar 10, 2025
74b9fae
types for core event listeners
YujohnNattrass Mar 10, 2025
696ef79
changeset
YujohnNattrass Mar 10, 2025
0ba8aa4
use same tool types as agent
YujohnNattrass Mar 10, 2025
727550a
init commit
YujohnNattrass Mar 11, 2025
ea06fe4
feat: openai realtime voice provider
YujohnNattrass Mar 11, 2025
026a4cd
update voice and agents
YujohnNattrass Mar 11, 2025
13d7d8e
add realtime voice methods to agents
YujohnNattrass Mar 11, 2025
293edb4
set up realtime speech providers with instructions and tools
YujohnNattrass Mar 11, 2025
ecaadd2
update name
YujohnNattrass Mar 11, 2025
83fbd25
update tests and fix
YujohnNattrass Mar 11, 2025
d05cb9a
changeset
YujohnNattrass Mar 11, 2025
346adc1
remove log
YujohnNattrass Mar 11, 2025
608b7f4
fix conditional
YujohnNattrass Mar 11, 2025
fe615ce
remove logs
YujohnNattrass Mar 11, 2025
a3a2476
cleanup
YujohnNattrass Mar 11, 2025
905c16e
Merge branch 'main' into MASTRA-2371-mastra-voice-s2s-methods
YujohnNattrass Mar 11, 2025
a2a2978
openai realtime docs
YujohnNattrass Mar 11, 2025
6504367
update voice event map
YujohnNattrass Mar 11, 2025
b3ea4db
update readme
YujohnNattrass Mar 11, 2025
11f5853
update ref
YujohnNattrass Mar 11, 2025
2ce3f91
add deprecation notice
YujohnNattrass Mar 11, 2025
831123f
update names
YujohnNattrass Mar 11, 2025
5dce279
update tests
YujohnNattrass Mar 11, 2025
09dd7e3
update README
YujohnNattrass Mar 11, 2025
4da6db1
update realtime methods
YujohnNattrass Mar 11, 2025
7fc7cce
update adding tools
YujohnNattrass Mar 11, 2025
565a6e8
fix types
YujohnNattrass Mar 11, 2025
28e9201
use audio delta update instead of full stream
YujohnNattrass Mar 11, 2025
bc22c84
update method names
YujohnNattrass Mar 11, 2025
bef9271
log deprecation warning
YujohnNattrass Mar 11, 2025
d142c10
update relay to send for realtime
YujohnNattrass Mar 11, 2025
d2bc1f0
openai realtime api docs
YujohnNattrass Mar 12, 2025
557d94e
update docs
YujohnNattrass Mar 13, 2025
95d990e
update docs
YujohnNattrass Mar 13, 2025
087faea
fix voice implementation for composite-voice
YujohnNattrass Mar 13, 2025
9a67c93
merge
YujohnNattrass Mar 13, 2025
32331a8
reset with parent branch
YujohnNattrass Mar 13, 2025
22b7188
support int16array
YujohnNattrass Mar 13, 2025
ad46783
Update vast-corners-deny.md
YujohnNattrass Mar 13, 2025
808a179
Merge branch 'voice-openai-realtime-api' of https://github.com/mastra…
YujohnNattrass Mar 13, 2025
c218477
remove dup
YujohnNattrass Mar 13, 2025
dd262ca
Merge branch 'main' of https://github.com/mastra-ai/mastra into opena…
YujohnNattrass Mar 14, 2025
4c2c504
update llms-full.txt and llms.txt
YujohnNattrass Mar 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
types for core event listeners
YujohnNattrass committed Mar 10, 2025
commit 74b9fae8dd1bf9c539f9aa3d393eee874c56ea3c
31 changes: 25 additions & 6 deletions packages/core/src/voice/voice.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
import { MastraBase } from '../base';
import { InstrumentClass } from '../telemetry';

// Define standard voice events
export type VoiceEventType = 'speaking' | 'writing' | 'error' | string;

// Define event data structure for each event type
export interface VoiceEventMap {
speaking: { text: string; audioStream?: NodeJS.ReadableStream };
writing: { text: string };
error: { message: string; code?: string; details?: unknown };
thinking: { prompt?: string };
listening: { audioStream?: NodeJS.ReadableStream };
[key: string]: unknown; // Allow for custom events
}

interface BuiltInModelConfig {
provider: string;
name: string;
@@ -30,7 +43,7 @@ export abstract class MastraVoice<
TTuneConfig = unknown,
THuddleConfig = unknown,
TTools = unknown,
TEventArgs = unknown,
TEventArgs extends VoiceEventMap = VoiceEventMap,
TSpeakerMetadata = unknown,
> extends MastraBase {
protected listeningModel?: BuiltInModelConfig;
@@ -154,20 +167,26 @@ export abstract class MastraVoice<

/**
* Register an event listener
* @param event Event name
* @param callback Callback function
* @param event Event name (e.g., 'speaking', 'writing', 'error')
* @param callback Callback function that receives event data
*/
on(_event: string, _callback: (...args: TEventArgs[]) => void): void {
on<E extends VoiceEventType>(
_event: E,
_callback: (data: E extends keyof TEventArgs ? TEventArgs[E] : unknown) => void,
): void {
// Default implementation - voice providers can override if they support this feature
this.logger.warn('on not implemented by this voice provider');
}

/**
* Remove an event listener
* @param event Event name
* @param event Event name (e.g., 'speaking', 'writing', 'error')
* @param callback Callback function to remove
*/
off(_event: string, _callback: (...args: TEventArgs[]) => void): void {
off<E extends VoiceEventType>(
_event: E,
_callback: (data: E extends keyof TEventArgs ? TEventArgs[E] : unknown) => void,
): void {
// Default implementation - voice providers can override if they support this feature
this.logger.warn('off not implemented by this voice provider');
}