That's a hard one.  Which would you be happier doing?  The advise I've always gotten is not to overload myself, but then again, you are paying a pretty penny for your education.  Think about the level of difficulty you will be facing, as well as other activities you will be involved in.  Will you have time to sleep, eat, and study, or will you only be able to choose two?

As a graduate with a physics degree from another school, I would recomend going the electrical engineering route, because you you will be taking the same prerequisites that will allow you to study quantum computing, modern and quantum physics, without the headache of classical physics and other required courses unique to each program. We had a chemistry student in our quantum physics and quantum computing class, and he was in no way at a disadvantage. A computer science major would have probably had an advantage in the computing class. I am not trying to deter you from majoring in physics, but you must understand that if computer science is your passion, than the physics workload will definately get in the way if you choose a double major.