England
Based on his success in the U.S. both professionally and with the national team, several teams from both England and Scotland expressed an interest in signing Brown. In August 1932, as the Caledonia Cruise liner neared the dock, representatives from these teams awaited him. However, Scott Duncan, manager of Manchester United, took a tugboat out to the liner and signed Brown on board. Brown played from 1932 to 1934 with United, scoring 17 goals in 40 games, the second highest on the team. Notably, he scored directly from a corner kick in his first game against Grimsby Town within 90 seconds. While Brown scored regularly with United, he alienated the team management with his outspoken support for a players' union. On May 6, 1934, before United transferred Brown to Second Division Brentford for £300, he scored the only goal in the Manchester (Senior) Cup final match against cross-town rival, Manchester City, at Old Trafford, 1–0. Unfortunately, as with Manchester, his union sentiments quickly soured his relationship with the Brentford's team ownership. He made only one appearance for the first team, but scored 53 goals in 74 games for the reserves and won the 1934–35 London Challenge Cup.
In September 1936, newly promoted, First Division Brentford sent Brown to Tottenham Hotspur for a transfer fee of £1,200. In his one season there, he played only four first team games, but scored twenty-one goals in thirty games for the reserves. In July 1937, Brown moved to semi-professional Guildford City of the Southern Football League for a fee of £750. Over his two seasons with Guildford City, Brown scored 148 goals in 150 games and helped the club win the Southern League title during the 1937–38 season. During the 1938/39 season, Brown recorded five Hat-Tricks and scored seven goals (six of them headers) in one game versus Exeter City. Guildford finished in second place, losing out on a second consecutive title by just one point. James had been sick and out of most of the play during the last month of the season. He was loaned out to Brentford's First Division team, to play center-half in the FA Cup match against Fulham, where he scored a goal. He left Guildford City at the end of the 1939/40 season, because of increasing cartilage problems. Guildford City suspended all League play in September 1940. He then moved north to finish his career with Scottish First Division club Clyde, signing in November 1940 and played two matches with his brother John against Queen's Park in February and March 1941. In the February match with Clyde, just like his debut with Manchester United, Brown scored directly from a corner kick. He retired due to injuries soon after. Clyde finished in second place, only three points behind eventual champions Rangers during the 1940/41 season.
