The probe is part of a wider investigation into banks' involvements in Yakuza-related financial transactions; and comes after Mizuho Financial, the nation's second largest bank, admitted to lending 200 million yen ($2 million) to criminal gangs.
The money had gone mostly to about 230 small auto loans; and were made by Orient Corp a consumer financing company affiliated with and financed by Mizuho.
Mitsubishi UFJ Financial Group and Sumitomo Mitsui Financial Group, the two other top Japanese banks, were given notice on Tuesday of the upcoming probe - with FSA inspectors expected at their branches next week.
This time, FSA investigators will examine transactions elsewhere in the company, including the parent bank, officials said.
"Megabanks clearly need to take responsibility for a wider array of issues ranging from management to compliance as they have many companies under their umbrellas," said Takehito Yamanaka, an analyst at Credit Suisse Group AG in Tokyo, to Bloomberg.
"The FSA's inspection is likely to check if the banks' top managers are handling things in a responsible manner."
Mizuho has already sanctioned 54 of its executives, including its president, for ignoring the loans made to the Yakuza.
On Monday, lawyers hired by Mizuho to look into the transactions said "many officials and board members were aware of, or were in a position to be aware of, the issue".
But the lawyers' report also said that Mizuho failed to recognise it as a problem, believing that the compliance division "was taking care of it".
Mizuho president Yasuhiro Sato will give up six months' pay for failing to stop the loans. Takashi Tsukamoto will resign as chairman of Mizuho Bank, while keeping the post at the parent company, and take a similar pay cut.