I'd be kinda concerned with backfilling too early without weight to help hold it down. Might also check into a poured concrete walls ?
You used to see many places that did the foundation then put the floor & joists on then made a roof to cover it and lived in the "basement house" as they then finished the upper section over time. I even heard that some made the temporary roof part so it came apart or folded up ? and then made the walls out of it. Got to be plans out there for this.