George Peabody kept in close touch with Baltimore while he was in London. He knew that the city, as an important center of trade, was enjoying unprecedented commercial and economic success, but he was equally aware that its cultural and educational development lagged far behind. Baltimore, prosperous and with a burgeoning population, had no public library, art gallery, school of music, or learned journals. His American colleagues, however dedicated, were at a disadvantage before the more cultured and educated Europeans.

Peabody met with his Baltimore friends Charles James Madison Eaton, then President of the struggling Library Company of Baltimore, William Edward Mayhew, and John Pendleton Kennedy to find a solution. It was Kennedy -- author, statesman, and former Secretary to the Navy -- who had known Peabody since their days as volunteers in 1812, who did the most to shape the Peabody Institute. He worked with Eaton and Mayhew to outline a formal plan to achieve Peabody's aims. George Peabody's goal was a practical one: to stimulate individual initiative by providing citizens with an opportunity to receive an education in the arts and humanities. It was hoped that this would stimulate an institutional revolution within the city, and in fact, it was to do so. Adhering to Peabody's concept of a multi-faceted institution, the committee developed a structure that would expose the citizens of Baltimore to the finest in literature, music, the fine arts, and contribute to the formation of literary and scientific taste in the city.

During Peabody's visit to Baltimore in 1857, Kennedy drafted the founding letter which Peabody signed and published on 12 February. Within months, a financial panic temporarily halted plans for the Institute. Despite the ensuing turmoil, a charter of incorporation was obtained by the trustees in March 1858.

After locating a suitable building site on Mt. Vernon Place, a high point on the edge of town overlooking the city, the Trustees planned a building to house a library of 100,000 books, a lecture hall to accommodate 3000, an art gallery 200 feet long, offices and a music salon that was to be plain but elegant and capable of "harmonious adjustment." Architects from New York, Virginia and Baltimore submitted drawings, but after concluding that none of the nine final choices were entirely satisfactory, the trustees divided the $1200 in prize money among the finalists with the understanding that their drawings would be used in the formulation of a final design. Architects Lind and Murdock of Baltimore were appointed to prepare the plans. The reality of cost estimates forced cutting in half the size of the halls, reducing the library, office and gallery space, and eliminating altogether quarters for the academy of music. While the cornerstone was laid in 1859, the Civil War intervened, disrupting all normal activities and dividing the loyalties of the Trustees themselves. The building was not to be dedicated until 1866.

The dedication of the Institute was to mark the dawn of a new era of cultural opportunity, and George Peabody returned from London for the occasion. The dedication took place in the Lecture Hall of the building on 25 October 1866.