In October 2, 1962, a provincial historical plaque commemorating James Baby was unveiled on the grounds of his former home, now Duff-Baby House. In 1986, the Ontario Heritage Trust replaced the original plaque with this bilingual marker:
JAMES BABY 1763-1833
The first member of Upper Canada's French community to gain prominence in government circles, Baby was born in British-controlled Detroit, the son of a well-established trader. He was educated in Quebec and, after returning to this area, entered the mercantile business. In 1792, through his family's influence, he received lifetime appointments to the Executive and Legislative Councils, Upper Canada's pre-eminent political bodies. Three years later Baby moved from Detroit to the south shore and in 1807 took up residence in this commodious 18th-century house. For his long and loyal government service he was granted the post of Inspector General of Public Accounts in 1815. To attend to the duties of this position, Baby moved to York (Toronto), where he lived until his death.