Deputy Prime Minister Uhuru Kenyatta, who faces international charges of crimes against humanity linked to the last election, was provisionally ahead of Prime Minister Raila Odinga by 53 to 42 percent with about half the votes counted.
But Kenyatta, son of the country's independence leader and one of Africa's richest men, could still fall short of victory as the count goes on. Odinga's camp says results are not yet in from many of his strongholds.
If no candidate secures a 50 percent majority, a second round would have to be held.
Kenyans hope the vote will restore their nation's image as one of Africa's most stable democracies after bloodshed at the last election, when more than 1,200 people were killed in ethnic violence after a disputed result. Kenyatta denies charges at the Hague-based International Criminal Court (ICC) that he unleashed death squads of his followers after the 2007 vote.
This time around, at least 15 people were killed in pockets of violence, but that is being seen as a broadly peaceful outcome and a big turnout is a positive sign. The real test will be whether the candidates and their backers accept the result.
"Nobody should celebrate, nobody should complain," election commission chairman Isaac Hassan told journalists, saying work was going on to resolve glitches and speed up the count. "We therefore continue to appeal for patience from the public."
The commission says provisional results may not be tallied until Wednesday, meaning an official declaration will not come until then or later.
Kenyatta's lead has held up so far, but with results from central areas arriving before those from remoter regions, 60 percent of polling stations have yet to report.
Odinga's camp said counting in their strongholds had not been completed yet and a debate over the fate of a sizeable number of rejected votes could help shift the balance.
Voting has often been on ethnic lines in Kenya, with Kenyatta hailing from the largest tribe, the Kikuyu, and Odinga from the smaller Luo tribe. Both men are also receiving support from the tribes of their vice presidential running mates.
The United States and Western donors have watched the vote closely, concerned about the stability of a nation seen as a regional ally in the fight against militant Islam.
They also worry about what to do if Kenyatta wins, because of the charges he faces in the Hague.
With memories of that violence still fresh, many Kenyans are determined to prevent a repeat and have vowed to accept the outcome, as have the candidates.
In a reminder of the regional importance of a country that has been allied with the West against Islamist militants, an explosion struck a predominantly Somali neighbourhood in the capital Nairobi late on Tuesday, injuring one person.
"People should be patient; in 2007 Mr Odinga was leading against Mwai Kibaki in preliminary results, the following day when we woke up, things turned upside down and Kibaki won the elections. I believe the same thing would happen," said Christopher Otieno, 31, a seller of household wares.
Investors breathed a sigh of relief after voting passed off calmly, initially strengthening the Kenyan shilling against the U.S. dollar. The shilling later lost some of its gains after the slow count cast doubt on whether a first-round victor would emerge. Analysts said an outright win would be preferred to a run-off, which would prolong uncertainty.
The inspector general of the Kenyan police, David Kimaiyo, told a news conference he would not allow demonstrations anywhere in the country over delays in releasing the election results because of concerns protests could turn violent.
With turnout estimated by the election commission at more than 70 percent, a total of about 10 million votes must be tallied in the nation of 14.3 million eligible voters.
"Initially the sentiment was in favour of a first round win for Uhuru, but it's kind of difficult to gauge that now," said Ignatius Chicha, head of markets at Citibank.
William Ruto, Kenyatta's running mate who also faces charges of crimes against humanity at the ICC, called the vote "free, fair and credible".
Should Kenyatta and Ruto take office, Kenya would join neighbouring Sudan as a country with a leader wanted by the court in the Hague. However, the two leaders say that unlike Sudan's Omar al-Bashir, they will not become fugitives.
During voting Ruto said: "We shall cooperate with the court with a final intention of clearing our names."
Odinga's CORD party said it was still confident it would get back into the race as the counting continued, but also pointed to irregularities in the process including late voting by voters in some areas, hinting at legal challenges ahead. Odinga had questioned preparations before the poll.
Odinga's running mate, outgoing Vice President Kalonzo Musyoka, discouraged his rivals from "premature celebrations".
"We appeal for calm and call on our supporters to relax because we are confident that after all votes are in, CORD will carry the day," Musyoka said told a news conference.
The election commission acknowledged a polling clerk had been caught issuing extra ballots and said manual voter lists were used where the electronic registration system failed. But it has said there were no significant problems in voting.
Raising the stakes in the race, Odinga could be facing his last crack at the presidency after narrowly missing out in the disputed 2007 result to now-outgoing President Mwai Kibaki, who has served the maximum two five-year terms.
Kenya's African neighbours, whose economies felt the shockwaves from the violence last time, have watched intently. Some landlocked states which rely on Kenya for access to the coast stockpiled fuel and other material, worried that the vital trade route could again be cut if violence flared.
But violence on election day was limited to pockets in the north and east of the country.
At least 15 people were killed in two attacks by machete-wielding gangs in the Mombasa region hours before the vote on Monday. Police officers blamed a separatist movement, the Mombasa Republican Council, suggesting different motives to the ethnic killings after the 2007 vote. The MRC denied any role.
Alongside the presidential race, there were elections for senators, county governors, members of parliament, women representatives in county assemblies and civic leaders.
(Additional reporting by Joseph Akwiri in Mombasa, Richard Lough, Beatrice Gachenge, Duncan Miriri, Kevin Mwanza in Nairobi; Writing by Edmund Blair and James Macharia; Editing by Peter Graff)