Speech Recognition Copilot for the A2A B-17

[lequinne]

Kilstorm,

When you get version 1.2.0 (should be up tomorrow - just want to do a few more test flights) try deleting or renaming your copilot.cfg, and let me know if you get the same issue. Hopefully I fixed it.

Farley

[Kilstorm]

Kilstorm,

When you get version 1.2.0 (should be up tomorrow - just want to do a few more test flights) try deleting or renaming your copilot.cfg, and let me know if you get the same issue. Hopefully I fixed it.

Farley

Will do. Got an error on the Prop levers after take off when moved them to take over power. Up came an error prompt window, I "x" it out and all was good. I think what that is about is that I have two prop levers set for 1&2 and 3&4 and they might of been out of sync when I moved them. No biggy as in P3Dv3 there is also a CHProducts issue where it doesnt recognize them in a way that it gives them generic labels. Its been fixed by LM and will be available on the next update which will be the first for V3.

[lequinne]

Actually Kilstorm,

That's a nasty and embarrassing bug in my code, which I've actually already found and fixed for the new version. I was using those boxes for debugging, and forgot to fully remove one of them, sort of like a surgeon leaving a hemostat inside a patient. whoops...

I'll just go ahead and upload the new version right now, since it's ready and will be a big improvement I think. Keep letting me know if you find bugs, hopefully there aren't any left, but if there are, I'll try to fix them quicker.

Farley

[lequinne]

hello all,
VERSION 1.2.0 IS UP!

New Features
-Added "Hard Mute" key
-Added Radiocompass set command
-Added option to lock or hold prop levers after RPM set
-Copilot will return gear switch to neutral if left down or up by pilot
-Added PDI off/on command
-Added optional interphone system modeling -EXPERIMENTAL
-Added "...but don't prime" (etc) option to start engine command for starting flooded engines
-Added Prime engine X Y strokes command for extreme cold weather.

Plus fixes for some unpleasant and less unpleasant bugs, tweaks and performance optimization.

NOTE:
The new features haven't been added to the manual yet. I uploaded the new version anyway because it fixes some annoying bugs, and I'll add the new features to the manual tomorrow.

-Everything in the manual is still valid, only new features have been added.
-Try setting the "radio compass"! If I did it right, it should be flexible enough that you don't really need an explicit outline of the accepted syntax - Let me know if it works!
-If the "interphone simulation" box is checked:
When headphones and/or oxygen masks are on, the crew will not hear you unless your jackbox is in "inter" or "call" and the PTT switch is depressed.
If in doubt, just uncheck the box in the settings window.

PLEASE:
Let me know if there are any problems. Let me know if it works. Let me know what you like and don't like, and let me know what future features you would like to see.

Happy flying to all!
Farley

[Kilstorm]

Did a ground test to try out some of the new stuff. Downloaded the new zip file, copy and pasted the dll file into the panel folder overwriting the one that was in there. Started the sim and got a test positive on the following:

Start engine dont prime
Prime engine 3 two times
Intercom would only work if I had the headphones or oxygen mask on, set to intercom or call and held down the PTT key command as you had designed. With none set as my PTT key command could not be heard by crew or seen in text.

Will have to do a test flight later this weekend.

I really like what you did with the intercom except I dont like having to push to talk. This is an area I always thought it would of been great if A2A would of used a additional sound set for where all the voices sounded as if they were talking thru a headset instead of the cabin sound set. If there was one feature I would request it would be to take the "yes sir", "Right Away', "you got'em Capt", "they're all mine" sounds and make a duplicate of those but as if they are being spoke and heard in a headset and use those sounds for when headphones are on.

Also if there was a way to map a joystick button as a replacement to a key command for the PTT, that would be cool.

Im totally happy with what this is, as it is but you asked for feedback of what things we might think of and those are the only things that come to mind as this is a well thought out addon you have created!

[lequinne]

Kilstorm,

These are both things I've thought about.

As for direct joystick support, I will likely include it eventually. I haven't done it so far because it seems a little tricky (though obviously possible), getting keypress messages from the sim is extremely easy, and I thought my time would be better used expanding the program in other directions. Of course, a PTT key on the keyboard is highly inconvenient, but there's an easy workaround that I use: assign a key in the Copilot, then use FSUIPC to program a joystick button to virtually press that key (and repeat while held). According to the P3D forum, FSUIPC is very recently up and running for P3Dv3.

As for alternate audio over the interphone:

Of course, I'd love to have it too, it's just the sort of immersion that I seek to create, but:
Barring expanded access suddenly being granted by A2A, there doesn't seem to be anything I can do to change any of the installed voice sounds short of extensively hacking A2A's code, which I'm sure they would frown upon, to say the very least.

The alternative, then, is to have a voice-over recording session, make my own voice set, and implement my own voice engine. The problem is that it would be much more expensive and time consuming than what I've done so far... If I do it, I would want to do it thoroughly and well, (e.g. recorded on a genuine throat mic, etc...) and would not be able to release it for free. The A2A B-17, though brilliant, is aging, people tend to gravitate to the newer addons, a very small number of people have responded to what I've made so far, and frankly I am not sure I would be able to recoup the financial expenditure and time that producing such a product would require.

What I would like, is for A2A to produce a new and improved heavy, another 17 or perhaps a B-29 or - my preferred choice - a Liberator, and in such a case, I will likely be able to re-use a lot of my current code and program structure, and be all over recording and voice sets from the get-go. But that's up to them, obviously.

Sincerely,
Farley

[N602AC]

I thought I'd post some feedback on the update.

I see you can set a hard mute key now, which I did so. I also enabled the interphone system thing as well. This is a definite improvement, I like it. I don't have as much of a problem getting it to hear me correctly which I will go into detail on below.

I believe this problem lies with Microsoft's speech recognition itself, even after doing 5 training sessions in Microsoft's speech recognition, it almost never hears me correctly. Here's an example: I say "Fuel pumps on" and on the status bar above I see it says "off fuel pumps off". Every time I say that, it thinks I say "off fuel pumps off".

Furthermore almost anything I say seems to be automatically interpreted to be preceded with the word "off" which at this point is making this annoying. It's usable, but somewhat annoying.

Here are some examples:
I say, "Air cleaners on" it thinks I say "off air cleaners off"
I say, "Waste gates eight" it thinks I say "off waste gates eight" (Which is equivalently interpreted as "turbos off")
I say, "Turn on the APU" or "Start up the putt putt" it thinks I say "off turn on the a p u" or "off putt putt off" (Which is equivalently interpreted as "a p u off")
And so on.
I'd say it hears me correctly about 25%-40% of the time at this point.

The weirdest thing is, after I trained it 5 times, I did a couple flights and it worked 'ALRIGHT'. But then after that it went to hell. Let me be clear I am blaming Microsoft's speech recognition for this, not your DLL, as I had similar issues as this with various FS2Crew products which also use Microsoft speech recognition software.

Here is a HUGE HUGE HUGE question I have for you:
Microsoft's speech recognition sucks compared to Google's speech API, which I believe is free for developers to use. (think "OK Google" if you have a recent Android smartphone then you must know what I'm talking about)
Can you incorporate Google's speech recognition into this instead of Microsoft, or at least make it an option?

I have a challenge for you to try, and anyone else reading this right now.
Find a page in a book somewhere near you, or a newspaper article, or whatever.
Open Notepad and get Microsoft speech recognition ready.
Read the article completely.
Did Microsoft speech recognition screw up what you said? If so, how badly?

Now, go here: https://www.google.com/intl/en/chrome/demos/speech.html (Google Web Speech API demonstrator tool)
Now read that same article into this webpage that is using Google speech recognition.
Did Google speech recognition screw up what you said? If so, how badly?

I challenge everyone to try this exercise. I almost completely guarantee that Google is 1,000% better.

Let me know your thoughts on all of this,
Dan

[lequinne]

Thanks, Dan

This is extremely valuable feedback, and exactly the sort that I want. I've obviously been using the program extensively from the ground up, learning the commands cumulatively as I wrote them rather than all at once, and the recognizer and I have figured each other out pretty much perfectly. So I am at a disadvantage when it comes to optimizing the speech recognition for other users.

I do have some thoughts. First off:

Since you said that recognition is better while on interphone and are also having issues with FS2Crew, I am first inclined to suspect your microphone. But I'm sure you've tried experimenting with different mic placement, distance from mouth and angle. I personally find that recognition suffers significantly if my boom mic is at angled above or below the line of my mouth.

Secondly, there may be some issue with the recognizer in your system specifically - I have an issue where perhaps one out of every hundred or so times I load the airplane recognition seems horrible, but when I reload it it works fine again. May have to do with processor or memory load, I'm not sure. It could be you're getting this all the time... I'm still researching trying to figure this one out.

Thirdly, your feedback is especially valuable because there are some changes I can make to my implementation which may help to mitigate your especially annoying spurious "off" problem, and improve recognition a bit in general. I will try to have an improved version up tomorrow night.

Google speech API, while brilliant (it uses a more advanced recognition algorithm, I understand), doesn't seem yet to have an interface which I can conveniently use in this type of application. I am hopeful that this will change soon, however if it does, it's probable that converting will still be very involved and take some time. I chose Microsoft speech primarily because it's what everyone else seemed to be using (FS2Crew etc), and it comes with windows, so people would already have it installed and it would be simple to set up. That said, I will look into other speech recognition engines. If there's one which is better and I could convert to without completely rewriting everything, it's definitely something I will seriously consider.

Cheers,
Farley

[Kilstorm]

Thanks, Dan

Since you said that recognition is better while on interphone and are also having issues with FS2Crew, I am first inclined to suspect your microphone.
Cheers,
Farley

I have FS2Crew and using my aviation headset, it could never hear the word "off" I forget what word it would think I said but flying the Dash 8 I always had to say skip. I recently put my computer headset in my flight sim and the mic on that allows the program to hear the word off now. I wouldnt say its 100% the problem Dan might be having but do agree with lequinne that its certainly worth a look at trying a different mic if only to test it out.

[Tomas Linnet]

i like the idea of not using MS speech as I can't use it having a Danish W7. if it's possible to use this addon any way, please let me know. Some other guy made a voice control addon for the mighty B-17, but I never got to try it for the same reason. this must be an issue for all not english W7??

[lequinne]

Hahahaha I didn't realize there already was one, I completely missed that, wouldn't have spent the time making this at all... But I had fun I guess.

Thomas, you should be able to use it if you download and install the English MUI from Windows, and switch your display language to English and your recognition language to English(US) while you use it - let me know how it goes.

sincerely,
Farley

[Tomas Linnet]

I'm on W7 Home Premium, which doesn't support multi language

[lequinne]

VERSION 1.2.5 IS UP

Implements adjustments to the grammar file which should improve recognition, especially spurious opening "offs".

This version also allows realistic engine starts with the mixture in idle-cutoff, which is a feature i'm pretty excited about. As pilot you move the mixture control to Auto Rich as soon as the engine catches, and not prior to engine start, as you would flood the induction system. This is the correct starting procedure for practically all american warplanes of this period. This feature works on all engine starts, regardless of whether they are commanded or done manually by you. See manual for further details.

Work has begun on a Google-based version of this copilot. At this point it seems possible, but expect it to take several weeks at the very least.

Cheers,
Farley

[Tomas Linnet]

Work has begun on a Google-based version of this copilot. At this point it seems possible, but expect it to take several weeks at the very least.

Cheers,
Farley

[N602AC]

VERSION 1.2.5 IS UP

Implements adjustments to the grammar file which should improve recognition, especially spurious opening "offs".

This version also allows realistic engine starts with the mixture in idle-cutoff, which is a feature i'm pretty excited about. As pilot you move the mixture control to Auto Rich as soon as the engine catches, and not prior to engine start, as you would flood the induction system. This is the correct starting procedure for practically all american warplanes of this period. This feature works on all engine starts, regardless of whether they are commanded or done manually by you. See manual for further details.

Work has begun on a Google-based version of this copilot. At this point it seems possible, but expect it to take several weeks at the very least.

Cheers,
Farley

YESSSSSSSSSSSSSS

Awesome. So glad to hear it.

Looking forward to it!

Dan