Most of the fuel pumps have a check valve in them to keep the fuel from draining back to the tank. If it is an older fuel pump. It may have some leak back due to the check valve not seating all the way. Another culprit could be, and easier to check is you plunger valve in your carburetor. If the leather is dried up (ethanol gas can do that). When you press the pedal to get a shot of gas when starting cold, and it’s dried up you will not get much help starting.
Engine cool, but within an hour or so of running, Engine off, air cleaner off, look down the carburetor and give your throttle mechanism a full stroke. You should see two good streams of gas shooting out. If not pull your accelerator plunger out of the top of the carburetor and check to see if the leather has shrunk or hard.
If you have full streams, check it again after setting a week or so. (Your normal times between drives). If no streams or week streams, your fuel pump is likely draining back.
Original rubber fuel lines that are hard and brittle will also usually have small leaks that could allow drain down.