In 1953, the NFL awarded Baltimore a franchise that took the name of the old AFC Colts. The Baltimore Colts played their inaugural first game at Memorial Stadium on September 27, 1953. Over 46,000 fans filled the two tier structure to watch the Colts play the Chicago Bears. Memorial Stadium underwent several expansions during its existence. Box seats were added in 1961 increasing the seating capacity to 49,000. By 1964, the upper deck was lengthened, with two sections added at both ends of the horseshoe. The seating capacity was increased to 54,000 for baseball and 65,000 for football. In 1970, a digital scoreboard replaced the original "Gunther" scoreboard, that sat beyond the endzone. Memorial Stadium was a great place to watch a baseball game, but an terrible place to watch football. During football season, the field was laid out so that the upper deck extended sixty yards from the end zone of the horseshoe, past the 50-yard line to the 40-yard line where it ended on both sides. Not only were quality seats missing, but along with the obstructed seating, the "horseshoe" shape of the stadium was terrible for football, because the 40-50 yard line seats actually took fans away from the field. Consequently by the early 1980s, the Colts became dissatisfied with Memorial Stadium. The team wanted their own stadium to play at, so they decided to move to Indianapolis after the 1983 season. The Colts played their last game at Memorial Stadium on December 18, 1983.

After the Colts left, the Baltimore Orioles (MLB) were the only tenant at Memorial Stadium. The team constructed a state of the art ballpark in downtown Baltimore and moved from Memorial Stadium after the 1991 season. For five years, Memorial Stadium was home to various minor league teams. Meanwhile, in the early 1990, in Cleveland, owner of the Browns, Art Modell, sought a new facility for his team. After failing to reach an agreement with the City of Cleveland to have a stadium built, Modell decided to move his team to Baltimore, where the city offered to build a new stadium. The Cleveland Browns became the Baltimore Ravens. While a new stadium was constructed, the Ravens temporarily played at Memorial Stadium. An excited 64,124 fans watched the Ravens play their first game at Memorial Stadium on September 1, 1996. The Ravens played only two seasons at Memorial Stadium before moving into M&T Bank Stadium in downtown Baltimore. The last professional football game at Memorial Stadium was on December 14, 1997. Memorial Stadium remained vacant for the rest of its existence. The city of Baltimore finally decided to demolish Memorial Stadium in February 2001.