Jesse Spencer was born in England in 1786. He was married to Polly Southwick in 1813, and they had five children: Hiram (1811-1840), Elizabeth (1814-1868), Calvin (1822-1907), David (1822-1880), and Daina (1824-?). He was an early Pennsylvania settler who left England for Smethport, Pennsylvania on Potato Creek. He built a cabin there, and they made money by felling the pine timber. The family may have later moved to Keating Township in McKean County, Pennsylvania.