Senior U.S. and Iranian officials met separately for 80 minutes on the sidelines of the negotiations in Vienna.
Details were not given, but such bilateral talks were inconceivable before the 2013 election of Hassan Rouhani, a relative moderate, as president of Iran. U.S.-Iranian dialogue is seen as crucial to any breakthrough nuclear agreement.
"The conversation was productive and focused mainly on how the comprehensive talks will proceed from here," a senior U.S. State Department official said on condition of anonymity after Under Secretary of State for Political Affairs Wendy Sherman's meeting with Iranian Deputy Foreign Minister Abbas Araqchi.
Sherman headed the U.S. delegation, while Iranian Foreign Minister Mohammad Javad Zarif and Araqchi led Tehran's negotiating team at the table with Britain, China, France, Germany, Russia and the United States.
After Sherman's meeting with Araqchi, the Iranians met on Tuesday evening with all six powers to continue talks on how to approach future negotiations, diplomats said. The discussions will resume on Wednesday roughly at 10:30 a.m. (0930 GMT) and could run into Thursday.
"Much of the first day was focused on discussions about process for how the comprehensive talks will proceed," a senior U.S. official said. "We made clear that every issue is on the table as part of the comprehensive negotiations, and now it's time to dig into the details and get to work."
In the evening session between Iran and the six "substantive issues began to be discussed", the U.S. official added.
A European diplomat said no decisions had been taken yet on how the talks will proceed in the future - the six powers led by the EU foreign policy chief Catherine Ashton want to get a deal done within six months.
"We had quite detailed discussions, productive and in a positive atmosphere," the diplomat said. "But this is day one and we have at least another day."
COMPLEX PROCESS AHEAD
The Americans and Iranians have played down expectations.
Iranian Supreme Leader Ayatollah Ali Khamenei, the man with the final say on all matters of state in the Islamic Republic, declared again on Monday that the talks "will not lead anywhere" - while also reiterating that he did not oppose the delicate diplomacy with the six world powers.
Hours later a senior U.S. administration official also tamped down expectations, telling reporters on Monday that it will be a "complicated, difficult and lengthy process" and "probably as likely that we won't get an agreement as it is that we will".
It is the first round of high-level negotiations since a November 24 interim deal that, halting a decade-long slide towards outright conflict, has seen Tehran curb some nuclear activities for six months in return for limited relief from sanctions to allow time for a long-term agreement to be hammered out.
The stakes are huge. If successful, the negotiations could help defuse many years of hostility between energy-exporting Iran and the West, ease the danger of a new war in the Middle East, transform power relationships in the region and open up vast new possibilities for Western businesses.
Araqchi sounded upbeat about the initial 40-minute discussions with the six nations but appeared to draw a line against Tehran's ballistic missile programme being addressed in any future talks.
"We had good discussions ... and we are trying to set an agenda. If we can agree on an agenda in the next two to three days, it means we have taken the first step. And we will move forward based on that agenda," he said. "This agenda ... will be about Iran's nuclear programme and nothing else, nothing except Iran's nuclear activities can be discussed."
He was answering a question about Iran's ballistic missile work after U.S. officials said they want Tehran to accept limitations on any nuclear-capable missile technology as part of any long-term deal reached by Iran and the powers.
There may be other sticking points in the talks. Iran says it will not cede its "right" to install advanced centrifuges to enrich uranium, signalling defiance in a manner that may irk the United States and its European allies.
Despite his public scepticism about chances for a lasting accord, Khamenei made clear Tehran was committed to continuing the negotiations between Iran and the six powers.
"What our officials started will continue. We will not renege. I have no opposition," he told a crowd in the northern city of Tabriz on Monday to chants of "Death to America" - a standard refrain since the 1979 Islamic Revolution.
Western diplomats said it was difficult to predict the chances of getting a final agreement over the next six months that would be acceptable to all sides. "The one thing we know is they want the sanctions to go away, which will work in our favour," a Western diplomat told Reuters.
During a decade of fitful dialogue with world powers, Iran has rejected allegations by Western countries that it is seeking a nuclear weapons capability. It says it is enriching uranium only for electricity generation and medical purposes.
Tehran has defied U.N. Security Council demands that it halt enrichment and other proliferation-sensitive activities, leading to a crippling web of U.S., EU and U.N. sanctions that has severely damaged the OPEC country's economy.
Khamenei's approval of serious negotiations with the six powers despite the scepticism he shares with hardline conservative supporters is driven by Iran's worsening economic conditions, analysts say.
Another major factor was the overwhelming election last year of Rouhani, who is determined to relieve Tehran's international isolation based on "constructive interaction" with the West.
The goal of the talks for the United States and its European allies is to extend the time that Iran would need to produce enough fissile material for a nuclear weapon.
For that goal to be achieved, experts and diplomats say, Iran would have to limit enrichment to a low concentration of fissile purity, deactivate most of its centrifuges now devoted to such work, curb nuclear research to ensure it has solely civilian applications and submit to more intrusive monitoring by U.N. anti-proliferation inspectors.
Khamenei and other Iranian officials have often made clear that they could not accept any such cuts in nuclear capacities. The trick will be devising compromises that powerful constituencies on both sides can live with.
Western governments appear to have given up on the demand, made in a series of Security Council resolutions since 2006, that Iran should totally halt the most disputed aspects of its programme - all activities related to uranium enrichment at the underground Natanz and Fordow plants and production of plutonium at the planned Arak heavy water reactor.
Diplomats privately acknowledge that the nuclear programme is now too far advanced, and too much a cornerstone of Iran's national pride, for Tehran to agree to scrap it entirely.
But while Iran may keep a limited enrichment capacity, the West will insist on guarantees that mean any attempt to build a nuclear bomb would take long enough for it to be detected and stopped, possibly with military action.
Israel, which called the November deal a "historic mistake" because it did not dismantle its arch-enemy's enrichment programme, made its position clear ahead of the Vienna talks.
"We are giving a chance for (a) diplomatic solution on condition that it provides a comprehensive and satisfactory solution that doesn't leave Iran with a nuclear breakout capability," Strategic Affairs Minister Yuval Steinitz said.
"In other words, that it doesn't leave (Iran) with a system ... that would permit it to remain close to a bomb," Steinitz he told Israeli radio.