LUKE AIR FORCE BASE, Ariz. -- Luke Airmen working with an Arizona State University student have developed a 3D augmented reality system that will revolutionize the way pilots conduct mission debriefs and will represent the Air Education and Training Command at Spark Tank 2021.
Maj. Kevin Hawkins, 56th Operations Support Squadron wing intelligence chief, and 1st Lt. Adam Treece, 56th OSS intelligence readiness chief, Lt. Col. Alexander Turner, 56th OSS director of operations, and Dylan Kerr, ASU digital culture Master of Arts student, teamed to adapt Microsoft HoloLens2 smartglasses into a tool aircrews can use to conduct detailed mission debriefs from an interactive, three-dimensional perspective.
The debrief process “is literally fundamental to how we learn every time we go out to train and fly,” said Hawkins.
The 56th Fighter Wing’s mission is to train the world’s greatest fighter pilots and combat ready Airmen. Instructors train students to fly the F-16 Fighting Falcon and the F-35A Lightning II aircraft in preparation for combat operations. While flying, aircraft sensors capture flight characteristics including critical data such as position, airspeed, altitude and actions in a 3D environment, explained Hawkins.
After each flight, instructors review the flight details with students to evaluate their performance while seeking ways to help them develop in a post-flight meeting called a “debrief.”
According to Hawkins, aircrews currently perform debriefs using whiteboards and maps, which are conventionally 2D. While pilots can us computers to view portions of the flight, such as the position of the aircraft on the map, software limitations make it challenging to visually recreate the full perspective of each aircraft involved in the mission, he said.
He said forcing 3D data into a 2D environment is not natural, and during that translation of data “there's opportunity to make mistakes or it's just harder to learn and complete,” said Hawkins.
Hawkins and Turner envisioned a concept that would take full advantage of data captured by the aircraft’s sensors, and Kerr programmed the software to enable aircrews wearing the smartglasses to see the mission in 3D.
“Augmented reality enables individuals who wear the smartglasses to interact with others and their surroundings,” said Hawkins. “We think it’s going to be really effective because we are able to see if the student pilots understand the information or not.”
An additional advantage about using smartglasses is that it moves debriefs into the virtual domain. As a result, pilots who are geographically separated can discuss the mission together, Treece said.
“Right now, if I want to debrief with someone who flew a mission from a different base we'd have to do that over the telephone,” said Treece. “But imagine if you put these headsets on and could debrief virtually; and, both could see the same thing at the same time. That would make it a much more efficient process.”
Initially Air Force units submitted 305 Spark Tank 2021 projects to participating major commands for evaluation. The MAJCOMs selected 15 projects, including Next-Gen Debrief, to advance forward for Air Force-level review. In December, Air Force leadership will select a final six from the submissions, and project leaders for those submissions will present their innovative idea to a board consisting of the Secretary of the Air Force, Chief of Staff of the Air Force, and Chief Master Sergeant of the Air Force before the winner is selected. The winning entry will be announced during the Air Force Association Aerospace Warfare Symposium, February 2021.
Hawkins said he is confident Next-Gen Debrief will succeed.
“If we go to the finals and win Spark Tank and they ask us what our way forward is, we think we can get units to use the system within six months or so after finishing Spark Tank,” said Hawkins.
Hawkins said due to the classification level of some aerial operations, not all units would initially be able to use the first version of the smartglasses to debrief, but they are navigating the approval processes for classified use. However, input from aircrews who use the technology following unclassified missions will be invaluable for future updates, he added.
The team’s vision is for aircrews to maximize the use of augmented reality while planning, executing and debriefing the mission. To enable the vision, the team hopes to gain visibility during the competition and show people this is what the process should look like moving forward, said Treece.
“I think that there are many opportunities to use this as a jump off point to introduce this type of technology into the way we do business,” said Treece. “In my mind, there are many more opportunities to make what we do that much more efficient.”
Hawkins said the goal in the six to 18 months following Spark Tank 2021 is to increase efficiencies in the way the Air Force operates. With this technology, they think they can do just that.