Whitney Conway Ellsworth (Jim Beaver) is an experienced prospector who has pursued the color all over the country, even having once worked as a miner at wage and an overseer on sites owned by the Hearst mining company. Having left his position with the company, disgusted at the nonchalant attitude to the well being of the miners, he is introduced in the first season as one of the many individuals who has traveled to the town with the promise of wealth in the gold rich hills, revealing himself to have a 'dead-eye' for the color having successfully managed to eke out a comfortable living in this profession.
He was a regular of the Gem Saloon like many of the town's prospectors and was liked by the Gem's employees including Trixie and Dan Dority. Also understanding the dangerous nature of their employer, he keeps his witnessing of Brom Garret's murder to himself lest he meet a similarly unfortunate accident. In return for his silence, Dan Dority does not make known to Swearengen of Ellsworth's status as a witness.
After being hired by Seth Bullock to manage Alma's claim to keep her title active, he begins growing an attachment and a strong sense of loyalty towards her and the young orphan Sofia in her care with whom he begins to form a father-daughter bond. By the second season, Ellsworth has gone from a mere prospector to completely overseeing Alma's claim and the digging operations as well as a trusted friend and confidant. He defends her claim by chasing off Hearst's Geologist, Francis Wolcott, when he comes to spy out the territory - the two familiar with one another from Ellsworth's days working for the company - and advises Alma to stand her ground when rumor is spread about the camp about the future stability of gold claims once the town is annexed. He also seems to grow distanced from the Gem as well, refusing to genuflect in the presence of the likes of Dority as he once did - a sign of his moving away from his older life and his growing self-assurance and confidence. He has formed friendships with Joanie Stubbs and Sol Star.
Later he is advised by Trixie to wed Alma once it is clear she has become pregnant by Sheriff Bullock. Trusting it is the right thing to do and with a genuine wish to help her save face and help raise both Sofia and the future child, he proposes. She demurs but soon accepts his proposal and marries him the following week in front of their gathered friends and townspeople.
When George Hearst attempts to force Alma to sell him her gold claim, Ellsworth is one of the few in town who will stand up to him. But his relationship with Alma falls apart when she miscarries and begins using dope again. The final straw comes when she gets high and attempts to consummate their relationship. Despite this, they remain friends. Near the end of Season 3, he is shot to death by one of Hearst's agents while supervising Alma's claim. His body is brought to camp in the back of a wagon, and many people, including Trixie, are shocked and upset by his death.