Programming

I'm a fan of early returns.
So

if(isalpha(letter) == false) return letter
if(letter whatever) {
   Do thing
   Return letter
}
if(letter something else) {
   Do something else
   Return letter
}
throw error("unprocessable letter")

I find it lets me mentally walk through all the paths more easily.
And if something gets too complex for this, then I need to break it down into further functions

It's not at all necessary, but I find it makes much easier to read code if you instead only use if statements and just return early when you're in a function. For example, you could check isalpha(letter) == true is true then check letter + key <= 90 do the letter += key; return letter; then since letter + key must be > 90 if it didn't already return a value, then you can do the while statement and return letter without needing an if statement at all. Then the isalpha(letter) == false is also unncessary, just return letter at the end.

Like this:

char rotate(char letter, int key)
{
  if (isalpha(letter)
  {
    if (letter  + key <= 90)
    {
        letter += key;
        return letter;
    }

    do the while loop here
  }

  return letter;
}

you are checking the result of a method that returns a boolean (true/false) there is no reason to check for true, else, FILE_NOT_FOUND. you can also forgo equating its return value to something to get a boolean value required for the if statement since its already a boolean value.

e.g.

if (isalpha(letter)) 
{ 
   // ...
} 
else 
{ 
   // ...
}

I second the early return suggestions, but the other thing you should be aware of is that isalpha() is being evaluated twice unnecessarily. If its cheap and not call frequently, not a real big problem, but it is a waste of cycles. If you want to document the else, you could try:

...
}
else  // isAlpha() == false
{
...

Also, if isAlpha was something that could change between evaluation, such as isTuesday, you are at risk of the first call returning false, and then the second call returning true, which would skip both cases.

https://en.wikipedia.org/wiki/Defensive_programming might be an interesting read.

Right, the compiler isn't smart enough to recognize that isalpha() is pure and deterministic.

Expect isalpha is part of the standard library not an arbitrary function, a compile should be able to optimize standard calls.