I have been fired from jobs and it's came as a surprise to me. It took me a while to learn that when most bosses say, I want you to be honest, that they may not really want you to be honest. Also I have had to deal with difficult colleagues, it took me a while to realize that a difficult colleague is dealt with in a different way to a difficult person you could encounter outside of work. For instance, in the second, I might say let's talk about it, or I may be offended, at work there is a game to be played, or just different rules, and they apply to dealing with customers too. I'm still learning things all the time too, we don't stop learning I hope.
I don't think it has to be Aspergers, it's just something that's happened and you've learned to position yourself differently with difficult customers. As much as a customer may be being difficult, how long can it last for? You are not there all day and the library must close eventually, so you are getting paid to listen to someone complain to you! How many people go home and don't get anything to listen to siblings or someone else have a rant
Also, it could be other things, poor management, maybe someone wanted to have a friend work there, millions of different possibilities, so fwiw I would say don't blame yourself too much for it, and don't worry too much about picking up signs imo, we can all work on these things going forward, think of setbacks as just something that's happened, or an opportunity to improve or re-evaluate something, and move forward. I would say don't just think it's because it's all your fault, you are bad and such things (if you do) because you'll just hurt yourself and not make any progress (at least this is what i've found and practice, so it might work for you too.)
It's happened, and no ones life goes perfect from what I can see, I hope you feel better soon and my heart went out to you.
Edit: I've also had jobs i've been really great at, so sometimes there might not be any reason for it.
I remember hearing before that we are far more likely to remember the bad things in life than the good things that have happened to us, I did a quick google search and found a lot of articles, but like this one
http://science.howstuffworks.com/lif...-than-good.htm
Anyway, I just wanted to show that don't just think you suck and think about all the sucky things that happened, a lot of good things have happened too in regards to picking up on things and good things you have done for distressed customers.