From the Milton College Preservation Society web site:
"Joseph Goodrich came to Wisconsin from Allegany Co., New York, in the summer of 1838, and selected the site of Milton for a settlement. It was his idea that a school be opened to provide a more advanced education than that offered by the few common schools in the area which were usually held only three months in the year in private homes.
Instruction in the school was designed almost exclusively to aid young men in preparing for business pursuits and to prepare young men and women for teaching in the public schools. Every advantage offered to young men was offered also to young ladies, and they recited in the same classes. It was thought that a school of this kind would attract families to settle in the vicinity.
The academy building was built in 1844 near the north- west corner of the public square when there were only four dwellings in the village. Construction cost about $300, paid by Joseph Goodrich, who managed the school and bore its expense in its first years.
It opened as a select school at the beginning of December 1844, with the Rev. Bethuel C. Church as its first teacher. He paid no rent for the building, and he retained all tuition income. Samuel S. Child succeeded him for a short time, and then the Rev. S. S. Bicknell was engaged as principal. It was he who in the spring of 1847 issued the first catalog. Tuition was $3 per term of 11 weeks, of which there were three a year; board with private families was $1 to $1.50 per week; and there were 67 students, 40 gentlemen and 27 ladies.
The success of the Academy had proved its worth to the community, and in the winter of 1847-48 a group of citizens combined to secure a charter for the school, placing it under the control of a board of trustees. It was named "The Du Lac Academy, a name that was never popular, nor used beyond the charter and the correspondence of the officers of the school."
...
The old grout academy building became untenable, and classes met in a private house for a part of the last two years. "For want of suitable accommodations" the school was suspended for two-thirds of the year 1853.
A new interest in the school led to a larger association of persons and a new charter was obtained from the legislature in March 1854, naming the institution "The Milton Academy". The school was reorganized in the spring of that year under the Spicers and classes were held in private residences until the fall of 1855 when the "beautiful and commodius building of Milwaukee pressed brick" was ready for use. It was 40' x 44', three stories high, and cost "something over $4000", paid largely by the subscriptions of the stockholders of the Academy.
"In the summer and fall 1857, the need of another building for dormitory purposes being greatly felt, the trustees authorized Hon. Joseph Goodrich and Hon. Jeremiah Davis . . . to erect one" for fifty students. Its cost was nearly $5000, and was finished by the beginning of the winter term 1857.
...
President Whitford, realizing the need for a science building, obtained permission from the Board of Trustees at their annual meeting in July 1901 to seek funds for construction of such a building. His efforts were cut short by his sudden death in May,1902.
During the commencement exercises of that year, it was proposed that a science hall on the spot where the commencement tent stood would be a fitting memorial to President Whitford. The proposition was approved by the alumni, money was raised and the cornerstone was laid in June, 1904. The building was completed in the fall of 1906 and dedicated at commencement exercises in 1907.
...
The home purchased for President Daland and his family in 1902. It was later known as "the Studio" where the School of Music was located, "fortunately, far enough removed from the other departments to cause them no pain." (Prof. L. H. Stringer, the Bulletin, Dec. 1948)
When the Department of Music moved into the Daland Fine Arts Center, the president's home/Studio became the Administration Building.