Unfortunately there are going to be very few unique, fun to drive premium cars that are offer a manual transmission these days, especially if you eliminate VW and the wonderful Golf R/GTI from the equation as well as the A4. You might consider cars with modern dual clutch automated manuals, as they really do offer the best of both worlds and this would open your options up a bit. Here are a couple of thoughts:
Mini Clubman: The new versions are larger and have decent space and they certainly have a unique look. Best thing is that they still offer a 6 speed with all models. Get the upgraded engine for good performance. They can get expensive with options, but Mini's are essentially BMW's and offer BMW level luxury. This is the only Manual car that I could find that fits your requirements.
BMW offers a manual in the 3 series Sedan, but not wagon. There are manual cars like the Nissan Juke or Hyundai Veloster, which are unique, but I'm assuming this isn't your taste, especially if you value material quality and workmanship like I do.
Volvo V60 R Design Wagon (or Polestar): Great looking car and unique in the USA given that it is wagon shaped. No manual, but manual shifting mode is said to be good. Turbo charged engine puts out good power in T5 form and great power in T6 form. FWD or AWD offered.
Mercedes GLA250: Mercedes calls this an SUV, but it is basically a fun hot hatchback with a slight lift. Being a Mercedes hatchback, it is slightly unique in the USA. The 2 liter turbo 4 provides decent power. Again, no manual, but it does have a seven speed dual clutch with paddle shifters. Interior layout is top notch and clean.
Ford Flex: Probably a strange suggestion, but it is very unique, and is essentially a large wagon, built on the Taurus chassis. With the optional 365 HP V6 it can be a lot of fun, although won't be agile like a smaller car. Kids will love the space. Traditional Automatic is a big downside but at least it won't blend in with the sea of regular people and their crossover SUV's.
Here is a 2016 list of cars offered in the USA with a manual transmission:
As you can see, it is mostly trucks, high performance cars, or base model sedans/hatchbacks. As I said before, if you have never tried a good dual clutch transmission, you should. I have had several VW's with DSG's and my GTI transmission is great. Manual mode with the paddle shifters or gear lever is truly fun and shifts are lightning quick. Completely different than the pseudo manual mode with Automatic transmission cars with torque converters.