Realistically, you've got an '11 Sienna that is (I'm assuming) rated to tow 3,500 lbs (imho, that means you're fine to tow 5,000 lbs; just be careful) and it has a relatively reasonable cost of ownership. You don't really NEED anything else; the Sienna should do just fine. But, granted, there are smaller vehicles that will do what the Sienna will do. Namely the V6 Venza, which is basically the same powertrain as your Sienna (also the same powertrain as the v6 Camry) in a smaller package. So that would be the logical "trade down in size" option for your situation. Also, the v6 Rav4 with the towing package has the same towing capacity.
Some small cars actually have some reasonable towing capacities. The Toyota Matrix is rated for 1500 lbs; if you can make that work for all but a couple days a year, just borrow/rent something bigger for when you're towing more than 2500 lbs. You can even get them with awd.
Also, you could probably email a bunch of (Toyota?) dealers that you'd like to make a roughly even-value trade to get into a smaller, X towing capacity vehicle. You'll probably find one that has a Venza or something like that that's been sitting a while that they'd be willing to swap for your Sienna with less markup than if you walked in asking for a price and trade in value. Just compare your blue book values for whatever is offered to your vehicle.