Do It Yourself

Obligatory I have never insulated a garage door, but I do like insulating things.

Replacing with newer glass could work if the company that makes your door makes replacement windows and you can get ones with more panes or something? Otherwise might be kinda tricky, unless they're a standard size.

Is the rest of the door insulated and are the edges properly weather sealed? Even though the windows probably let a lot of heat through per area and they're harder to work with, the whole door is a lot more area and easier to insulate so could try starting there. And of course weather sealing will keep you from exchanging as much air.

If keeping the windows working as windows isn't too important, you can probably cover them with the same stuff you would cover the rest of the door with, seems like people usually use foam and some kind of radiant barrier, often integrated in one product. If you do want the windows to work, then maybe the cling film type products that work for windows would work? I could also see bubble wrap or something working pretty well for a cheap improvised alternative, especially the kind with the big interconnected bubbles that's no fun to pop.

Anyway, I hope one of those ideas is helpful and you can insulate your door!

Garages have motors with liquids that produce fumes (aka voc in the air). If you are insulating the garage door well, what are you doing to make the air remain breathable if a motor starts to leak? An explosion or poison gas in your house (if you have an attached garage) may be worth putting off air sealing.

My garage doors just have plexiglas windows. I’ve insulated the doors and cut out the windows — they don’t appear to have too big an impact on heat gain/loss.

Unheated well ventilated garages are better for your car. So I would insulate the wall between house and garage, and stop heating the garage.

The premise behind most to all insulation is to prevent air movement as much as possible with as little material as possible. For windows, this usually means more panes, which means more trapped layers of isolated air (or other gases). The best bang for your buck for garage door windows is window insulation film. You can buy it in a number of stores. If you have a single pane window in your garage door, you have almost no insulation at all for the window. Double pane would be better, but will still let a fair amount of heat transfer. Given the small window size, you may want to try putting the film on the outside as well as the typical inside layer, giving an additional layer of insulating air.

The other thing about windows is they are poorly insulated objects put in a hole punched through better-insulated objects (the garage door). Sometimes the hole the window is put in isn't properly sealed, and that again allows air flow. Check for any air flow around the windows and caulk them before installing the film. This may require removing the trim and removing any old or decayed caulking before applying the new caulking.

If you in fact have single-pane windows in your garage door, you may want to see about getting double-pane windows instead. This won't be cheap, and may require modification or replacement of the trim for the windows. The windows are likely double-pane, though, and even if the seal is worn out the improvement from a new set of double-pane windows won't be much.