It is remarkable just how unremarkable America’s massive financial surveillance system has become to most people. Americans were rightly outraged when Edward Snowden revealed the government’s widespread spying campaigns on online communications. Yet every day, our financial transactions are subject to similar scrutiny. The programs aren’t even secret: you can read up about them on official government websites. But for some reason, we accept this surveillance as a fact of life. We shouldn’t.

If you give an agent a surveillance program, he will try to expand it. This is the case with the many legally questionable financial reporting requirements sprung forth from the Bank Secrecy Act of 1970 (BSA), which is kind of like the PATRIOT Act for money.

Most recently, the Federal Reserve and Treasury Department have proposed expanding what is called the “travel rule” to capture international funds transfers above $250. Currently, financial institutions are required to make certain reports on customers when they send international transactions in excess of $3,000. This has been the threshold since the travel rule was first adopted in the U.S. in 1996, despite inflation since then.

Here’s how it works: Let’s say someone wants to send $5,000 to someone else in the U.S. or abroad. That person goes to their bank and tells them where they’d like to send the money. The bank, by law, must collect, store, and send certain identifying data to the receiving financial institution, including the name, address, and account information for the sender and receiver. This data must be passed along intermediary financial institutions and stored for at least five years. It isn’t immediately shared with the government unless it is determined to be “suspicious” enough to trigger Suspicious Activity Report (SAR) requirements under the BSA. In other words: banks must keep this data on hand in case the government needs it.

These surveilled people are suspected of no crime, nor are they given any opportunity to opt out of this data collection. Still, the government preemptively requires that their transactions be tagged and tracked as if they had done something wrong.

The threat of government involvement is apparent. It has effectively deputized banks to keep treasure troves of transaction data on hand in case it should become useful.

But there are many other good reasons that innocent people should oppose these programs that don’t have to do with the government at all. Forcing third parties to maintain financial records on transactions gives them an intimate window into your life. As Supreme Court Justice William Douglas wrote of the BSA in 1971:

“The records of checks—now available to the investigators—are highly useful. In a sense, a person is defined by the checks he writes. By examining them, the agents get to know his doctors, lawyers, creditors, political allies, social connections, religious affiliation, educational interests, the papers and magazines he reads, and so on ad infinitum.”

Maybe you just don’t want the data quality manager at Bank of America to have access to the knowledge that you’ve been sending money to your preferred political or religious causes. It’s not their business and you haven’t done anything wrong. Plus, you need to trust that they will protect this data and not expose it to hacks or leaks. Yet this is the current state of play for American funds transfers, and it may soon be considerably expanded.

The proposed rule change would apply to traditional currency transfers as well as cryptocurrency transactions. The travel rule also applies to domestic funds transfers, but the limit for those would be kept at $3,000.

There is some good news: The Treasury Department generally understands the distinction between custodial cryptocurrency transactions (those that are facilitated by third parties like exchanges) and non-custodial or peer-to-peer cryptocurrency transactions that involve no third party. It also understands that software developers and miners have no direct control over fund transfers. Non-custodial transactions, developers, and miners are exempt from surveillance requirements. So there is at least a little bit of privacy breathing room when it comes to non-custodial cryptocurrency uses.

(Of course, not every federal regulator is this astute when it comes to the networking properties of cryptocurrencies: the Department of Justice recently described the use of privacy-preserving cryptocurrencies to be “a high-risk activity” that is inherently “indicative of possible criminal conduct.”)

Still, it’s worrying that government agencies don’t even consider personal privacy when proposing new regulations. My colleagues at Coin Center have filed a comment on the proposed travel rule change pointing to the lack of privacy considerations.

By law, federal agencies must issue cost-benefit analyses that weigh the trade-offs of a proposed new rule to industry and society. The travel rule analysis only considers the costs that would be imposed on banks on regulators. The extreme cost to privacy for millions of Americans is not even an afterthought: it’s not a thought at all. That’s a big problem.

If the Federal Reserve and Treasury Department had considered the proposed $250 travel rule’s privacy costs on individuals, perhaps it would not pass a cost-benefit test. Actually, maybe it would prompt the agencies to rethink the architecture of our financial surveillance altogether.

Justice Douglas foresaw the grave dangers to privacy posed by intermediated financial surveillance all the way back in 1971. Today, when so much more of our financial lives are channeled through third parties, the danger is that much greater.

The many problems with America’s financial surveillance system are apparent, setting aside these grave threats to our personal privacy. It creates compliance and hacking risks for institutions that must store this data. And it doesn’t even work very well. Criminals are routinely able to get the finance they need despite this web of data tracking. Meanwhile, innocent people may have trouble making transactions or get caught in the hassle of some overzealous agent. It’s a big mess.

Let’s hope that financial regulators listen to the many public comments encouraging an explicit consideration of how privacy is affected by financial surveillance programs. But these questions should not only be considered by regulatory agencies: it is perhaps time for the Supreme Court to once again examine the legality of these surveillance programs that hoover so much of our financial lives into exploitable central datasets.

ANDREA O’SULLIVAN is the Director of the Center for Technology and Innovation at the James Madison Institute in Tallahassee, Fla. Her work focuses on emerging technologies, cryptocurrency, surveillance, and the open internet.