Woollen manufacturing emerged as a major Canadian industry between 1840 and 1870. Mills were built in areas like the Mississippi River Valley where water power, labour and supplies were abundant. James Rosamund built mills at Carleton Place and Almonte in the 1840s and 1850s. His sons, Bennett and William, in partnership with George Stephen of Montreal, began the Rosamund Mill in Almonte in 1866. For the next 40 years, it was one of the largest, most progressive mills in Canada.

The entire mill complex is one of a few surviving Ottawa Valley textile manufacturing complexes from the late 19th century. In 1988, the Mississippi Valley Textile Museum acquired the Annex of the mill complex for use as a museum.

The warehouse portion of the Annex was built around 1872. It is a rubble stone structure with a blank elevation on the south and 12 bays on the north elevation with 16-over-16 double-hung sash windows. The office was added about 1904. It is a two-storey structure and shares a party wall and connecting passageways with the warehouse. The exterior of the first floor has segmental arch windows and door openings with radiating voussoirs and keystones done in contrasting dressed and polished red sandstone, accented with red sandstone quoins. The first-floor window sashes are double-hung one-over-one panes with a decorative stained glass transom depicting sheep and floral designs in an Art Nouveau fashion. The door has a similar decorative transom with the written words, Rosamund Woolen Co.

The Textile Museum was designated a National Historic Site in 1986. The Ontario Heritage Trust secured a heritage easement to conserve the building in 1989, and in 1994 the Town of Almonte designated it under the Ontario Heritage Act.