Ordinarily, I would say that you should pursue your dreams and should just quit and dedicate yourself full time toward looking for the permanent position in a library. However, for most jobs, it is easier to get one if you already have one, and the US is heading into a recession now and will likely be there for at least a year or three, and jobs are soon going to be even harder to get.

So instead of quitting, I would recommend that you keep your present jobs, try to make yourself more valuable at the place you like the most, and send out resumes in your spare time.

Start by telling your most sympathetic supervisor that you want to do more work and be more useful and helpful, and ask how you can do that. Then work like crazy to do it. It may work out, it may not (library management, I have discovered, can be both political in the bad sense of the word and capricious), but in either case, it will be good training that you didn't have to pay for.