Joel is the son of John Douglas and Nancy Denman Walden. He and his wife had 9 children and lived their entire lives in Lawrence County, Mississippi.