You bet you should go! The crowds for the TT are larger than the Manx Grand Prix, (at least it used to be that way) and the MGP draws large crowds meaning island resources can become pretty saturated, but it's still worth the trip.
A single day, however, is going to be pretty tough unless you fly into Ronaldsway airport and rent transport. And you'll miss most of the color and feel of the event. Not to mention being able to see the island (one of the most beautiful places on earth, IMHO) and appreciate the complexity of the Mountain Course - 37 3/4 miles of public roads.
You can take a ferry from either Liverpool or Heysham but you will have to book quickly, especially if you want to travel with a car or motorcycle. I've done both and prefer Heysham a lot more than Liverpool. Much easier to get to. As I recall, the ferry trip takes around 3 to 3 1/2 hours.
Finding a place to stay also might be difficult though not impossible. Douglas will be very crowded. Some of the smaller coastal villages should be better bets for accommodations. I've always preferred Port Erin on the southwest coast. It's quaint, and has a good selection of hotels. The Ocean Castle Hotel is nice, or was the last time I was on the island.
Take a look at the Isle of Man Tourist Board website, and make your plans soon. Good luck and enjoy. It's spectacular!
A closing thought: People aspire to different levels of greatness. I've always thought that being Admiral of the Manx Herring Fleet would be a great title to be awarded.