A small carb is ALWAYS easier to set up than a big one. The fuel metering signal is much stronger on a small carb.
At 5,500rpm and 100% volumetric efficiency a 480cfm carb would be perfect. You don't have 100% ve. After all of those turns and bumps and valves closing a surging the air back and forth you would be lucky to see 80% on a good set of heads at the rpms we really run. A 390cfm would be great on that. A 500cfm AFB would be perfect.
So why is your carb not working well? It's too damn big. Try to blow through a 2' section of 3" pipe and see if you can feel the air on the other end of it. The carb needs to feel ther air. It draws off of air moving through. A carb doesn't force anything--the engine does the work. Does a little 302 move enough air to create a reliable signal in a 670cfm carb?
No.
I'm sure it could be crutched to run okay, but it will never have crisp response and you will not need the secondaries. You need a carb with will have the throttle plates standing wide open at full power. Forget all the homerisms about carbs. You have to drive this on the street. I would find something around 400cfms.
I could get that running, what you have, but it won't run as well as the right carb would.