Pot holes are caused by water collecting in the road instead of running off the road.
Get the dirt away from road so water can run off. I use a combination of dirt, rock or gravel to fill holes. Let it get rained on. Pack in by driving on it. Get the holes fixed before you top coat it with an inch or so with rock or gravel. Where I come rock is white rock. Gravel is cheaper, more rounded. Make the water run off the road, not down the middle. Crown the road. Keep top coating until you get a good road base. I like to use #2 for the base and top with #8. It may take years to work the #2 in, depending on how heavy the traffic is. I have one drive for the trash man. He has a very heavy truck.
You want dirt, I like to use clay, around the rock to displace the water and hold the gravel in place. George