No, not at all. That would look really good. However, unless it's a blazer or a sportscoat I wouldn't wear just a plain suit jacket. suit jackets are really meant to be worn with a suit, and peopole can tell that they are part of a suit. with some suit jackets you can get away with it though. You'll probly look good anyway.