Stony Point began being built in 1818 by William Smith and was completed by his son, Joel Smith, in 1829. In 1808, William Smith established the second post office in Greenwood County and Joel Smith was one of the largest stockholder in the Graniteville Manufacturing Company. The Smith’s also ran a store near Stony Point. Joel would serve serve as a postmaster and then in the SC House of Representatives.