Holidays

I'm in a bit of a quandary over the up-coming holidays.
In the past, I have always liked getting together with my family. Mom would always cook up way to much food. The house would be so crowed that getting to the bathroom was a problem, never mind whether or not somebody would be in there when you did make it. The noise and confusion was just a regular part of the holidays.
These days, I find myself wanting to avoid the crush. If I get into a crowd of more than five or six, all of the energy just gets drained right out of me. I tire very easily, and the sound of voices gives me a headache that lasts for hours.
My brother wants to have a big family do for Thanksgiving. (That's on October 8th here in Canada.) I just can't do it. For one thing, he wants to serve the meal rather late. My wife has a condition that does not allow her to eat after a certain time. Besides that, he is dating a woman with two young children. Put the three of them together with my brother, sister, three nieces and my parents, and you are talking about major chaos.
I do like to spend time with my family. It has just outgrown me. I need them to come in small, calm dosages. The real problem is, how do you say that to your family without hurting some body's feelings?
The other side of the coin is my in-laws. I love them dearly, and want to spend some quality time with them. They are the second best part of being married. Is it wrong for me to choose a peaceful afternoon with them on one or two of the holidays of the year?
Okay, maybe I'm trying to rationalize not taking part in my brother's big plans for Thanksgiving, but all of the things you have just read are true. When does the truth become the wrong thing? I would hope that somebody reading this could give me a little insight.