There are many students in my classes who were 24-25ish all the way up to in their 60s. Since there's not a massive age gap, you'll probably fit right in as students may not even be able to tell whether you're 24, 25, 21 or 20 years old. You don't have to explain why you're at college at 24 years old, unless the administration asks you. To be friendly to younger students, you can give a summary of the explanation because if it's a long, complicated story, telling all of it may turn some people away as it could evoke a cloud of misery and sadness around you.

Using myself as an example, I graduated high-school at a younger age than most. I started pre-school 1 year early (most start at around 5-6 years old, I started just shy of 4 years old), skipped a grade in elementary school (or rather 2 grades since they had split-grade classes), etc... . Fellow students and random people even to this day mistake me for being a high-school kid as I look pretty young, despite being in my final year. It sucked at first because I felt out of place with the majority of other students who were 1-2 years older. I tried to dress in various styles of clothing hoping that would help, although it made some things worse as my friends became confused why I was showing up to classes with such sudden shifts in clothing style.

However, I realized that being younger or older doesn't matter for one simple reason: the majority of students are there to learn and study. I found this out when part-time students in their 40s-50s were in study groups with other 20-something students. I thought to myself, "if that guy has over 30 years on them yet they're fine with it, then surely being a few years younger wouldn't matter". I was right. I'm a member of 4 clubs on campus and go to some of their events. While the clubs are more social than academic, it's still an academic setting so the other students don't really care how old you are or your life story of what brought you to the campus. I don't mean to sound cynical if I am, it's a set of observations that time after time have shown to be consistent.

Good luck on your studies.