The tentative agreement means the government-owned bank would need to take a $1.44bn charge in the second quarter to pay the penalty. The rest will come from money it has already set aside.
It moves the bank a step closer to shaking off government ownership. The government, which owns 71% of the RBS, paid $45.5bn to bail out the bank after the financial crisis.
RBS chief executive Ross McEwan said this was a “milestone moment for the bank”.
He said: “Reaching this settlement in principle with the US Department of Justice will, when finalised, allow us to deal with this significant remaining legacy issue and is the price we have to pay for the global ambitions pursued by this bank before the crisis.”
The bank warned that the agreement was subject to the Department of Justice (DOJ) and RBS entering into a legally binding agreement.
RBS is the latest bank to settle claims of mis-selling in the run-up to the financial crisis. Barclays agreed to a $2bn (£1.4bn) settlement with the DOJ in March.
“Removing the uncertainty over the scale of this settlement means that the investment case for this bank is much clearer,” Mr McEwan said.
With the penalty paid, RBS can now open talks with UK regulators about the resumption of dividend payments.
It may also allow the UK Chancellor Philip Hammond to sell a stake in the bank.
“I welcome the agreement in principle to resolve this long-standing issue which will, when finalised, remove a major uncertainty for the UK taxpayer,” Mr Hammond said in a statement.
“It marks another significant milestone in RBS’s work to resolve its legacy issues, and will help pave the way to a sale of taxpayer-owned shares.”
The company’s stock closed up nearly 4%.