Looking back on my work history, I think that the weight placed on references varies greatly from one company and one job to another. Maybe do some thinking and research about what type of company may just want to know that you are honest and not a troublemaker and will focus more on your skills. Where I live applying for permanent jobs in big companies is more competitive in terms of good references. Smaller companies are more likely to need someone who can walk in and do the job at short notice.

I got back into work myself through an agency who knew that someone had walked out and left a boss in the lurch. At that moment in time he didn't much care who I was so long as I could get in and take the problem off his hands. It wasn't an easy situation but it gave me a much-needed leg up.