No, your alarm won’t go off with your AirPods in. I have experienced it myself numerous times, and it’s better because I have multiple alarms buzzing throughout the day- from reminders to taking my multivitamins to calling someone back. Having my AirPods in is basically a personality trait at this point.

How iPhone Alarms React When You Have AirPods In
Since I have with my AirPods on almost everyday with multiple alarms buzzing from am to pm, I have observed that the alarm sounds can be heard through the iPhone only.
Apple has designed it in such a way that one does not miss the alarm even if the sounds are not heard through AirPods because the alarm sounds are so loud. Not only this, but even when my phone is on DND mode, I can still hear the alarm going off.
Why Your Alarm Is Not Going Off When You Have AirPods In?
Technically, the Alarm should always go off regardless of whether you are wearing your AirPods or not. However, there are certain cases where your alarm would not be ringing. That is not totally related to your AirPods.
One time, my alarm didn’t go off and I missed a very important morning meeting.
This was because my phone ran out of battery while I was sleeping. So, make sure it your phone is charged enough so that it has battery to ring your alarm in the morning.
Final Thoughts
In short, the answer to your question is ‘no’, your alarm won’t go off even with your AirPods in. It always buzzes through your iPhone and is loud enough to wake you up from the sweetest of dreams, sadly.