Residents broke into celebrations on the stroke of midnight, with a fireworks display lighting up the skies over the city.
India's new Prime Minister Narendra Modi was among the first to congratulate Rao, promising his "complete support to the people & Government of Telangana" in a message on Twitter.
"India gets a new state! We welcome Telangana as our 29th state. Telangana will add strength to our development journey in the coming years," Modi added.
"Telangana's birth comes after years of struggle and sacrifices by several people. We pay our respects to them today."
Telangana has been created by splitting the state of Andhra Pradesh in two. Hyderabad, an IT hub, will serve as the capital of both states for the next decade.
The campaign to create a separate state in one of India's most economically deprived regions began in the late 1950s, with its champions arguing it has been neglected by successive state governments.
However wealthier coastal regions of Andhra Pradesh fiercely opposed the split, fearing that it would trigger economic upheaval.Andhra Pradesh's chief minister resigned earlier this year on the eve of a vote by parliament to approve Telangana's creation.
The build-up to the vote was marred by chaotic scenes in parliament, with lawmakers coming to blows while one representative even squirted pepper spray at colleagues.
Analysts had seen the decision by the then Congress government to back the state's creation as a strategic move designed to increase its support in the south in a general election year.
However Congress won only two of the 42 seats in the undivided Andhra Pradesh when the election results were announced last month.
India last redrew its internal boundaries in 2000, with the creation of three new states in economically deprived areas in the north.
Critics say the bill could open a "Pandora's box" of demands for statehood by other regional groups in the ethnically diverse nation, which also has a host of separatist movements.