Unfortunately this is TYPICALLY the symptoms of compression loss. After too much wear (IE if you have defects, overheated the car, neglected oil changes, bad combustion over time etc etc etc) the piston rings and cylinders will not mate and seal properly causing low compression and blow-by. While cold, the ring-cylinder mate is tight, keeping the combustion gasses from blowing by and allowing the engine to run. As the engine heats up the ring-cylinder mate becomes looser so that the engine will run smoother and easier...but if there is too much wear the combustion gases will begin to blow by the ring-cylinder surfaces and will cause each stroke to lose power. Enough blow-by will eventually mean that each stroke is not powerful enough to maintain engine rotation.
To determine if it is bad rings/cylinders, go to a mechanic or go buy a compression tester (maybe Auto-Zone or other auto stores rent them?). Test the compression of each cylinder per the tool instructions and note the compression of each cylinder. From what I could find the BARE minimum compression for engines in general is 90 psi...but it may be more or less for your engine. You also typically want cylinders to be within about 10-15psi of each other.
IF it is indeed bad compression, you will have to buy new piston rings and have your cylinders re-honed...or just do a straight up engine swap (which would likely be MUCH easier)...but let us know how this test goes first!
PS- internet searches seem to turn up compression problems for later model Cavaliers