It's not easy to find a reliable $2,500 car, because if people expected it to be reliable it would be worth more.
Your best bet is to find a car with complete records that suggest it has been thoroughly maintained and not too troublesome. And a seller that is just looking to sell an unneeded car without much hassle, for just a little more than a dealer would give them.
Newer is definitely better. Among older cars Hondas and Toyotas are easily the most reliable, but for this reason they're especially hard to find under $2,500.
You might be able to find the best deals on cars from brands that have either been discontinued or with weak images, like Pontiac, Mitsubishi, and Suzuki (but not the Forenza, Reno, or Verona). The Chevrolet Impala and Ford Taurus are two larger cars that tend to sell for low prices.