A number of large companies, particularly banks, are likely to pack up at least some of their workers and move them out of London now that the U.K. has voted to exit the EU. In all, the city that has long been considered the financial capital of Europe could lose as many as 40,000 workers in the wake of Brexit.

Much of the exodus could come from the big U.S. banks. Foreign financial firms were some of the remain camp’s biggest backers, saying the consequences of voting to leave the EU would make London a less advantageous place to do business. Goldman Sachs spent at least $500,000 helping to fund the remain campaign. Goldman has 6,000 employees in London. It hadn’t said how many it may move out if the vote went for “leave.”

Other banks have been more upfront in saying that they would have at least some of their staff exit London. Last week, Morgan Stanley said it would relocate as many as 1,000 workers if the U.K. was to leave the EU. Morgan Stanley now says it has no definite plans to relocate workers. J.P. Morgan Chase, prior to the vote, said it was likely to move 4,000 employees out of Europe. But on Friday morning after the vote, Dimon said JPMorgan was committed to keep a large staff in London. Still, he said J.P. Morgan is likely to move at least 1,000 people out of London.

“For the moment, we will continue to serve our clients as usual, and our operating model in the U.K. remains the same, Dimon said in a memo to staff obtained by Business Insider. “In the months ahead, however, we may need to make changes to our European legal entity structure and the location of some roles.”

In the past few years, banks have threatened a number of times to move workers out of London because of special bonus taxes and caps. Few firms have actually followed up on those threats.

Many U.S. banks have set up shop in London because it gives them access to trade European markets without the restrictions that you face outside of the EU. John Cryan, Deutsche Bank’s CEO, has said it would be odd for his bank to trade the sovereign debt of EU nations from an office outside of the EU. Deutsche Bank has 12,000 employees in London, and it said it was looking to relocate a good portion. In the wake of the vote, Cryan, a Briton by birth, said it was a bad day for Europe.

Even British banks are looking to leave Britain. HSBC, the most global of the large U.K. banks, said it was likely to cut as many as 1,000.

Bank analysts at Keefe, Bruyette & Woods have estimated that the big U.S. banks allow could move just over 7,200 workers outside of London. What’s more, Lloyds of London, the insurance firm, has said that a material number of the 34,000 employees in the insurance industry could be moved out of the U.K. PriceWaterhouseCoopers estimates that Brexit could cost between 70,000-100,000 financial services jobs by 2020.

In all, there are around 360,000 workers in the City of London, which is the EU’s financial center. Nearly 11% of those people, or roughly 40,000, come from other places in the EU. While no-one even in the Leave campaign has raised the threat of forced repatriation, whatever the U.K. and EU agree among themselves in how to govern their relations in future is unlikely to make them feel more secure about their positions.