Serena was playing in her first tournament since giving birth last September and faced her sister for the 29th time, with Venus winning all but four of their meetings.
Serena saved three break points to hold her first service game but lost her third to love as Venus took a 4-2 lead and served out for the set in 37 minutes.
Venus raced to a 3-0 lead in the second set after breaking her sister twice and had a match point at 5-3 which Serena saved, before Venus saved a break point of her own at 5-4 and went on to take the match.
The No 8 seed will go on to face Latvian Anastasija Sevastova, who beat Julia Goerges in straight sets 6-3 6-3.
Second seed and Australian Open champion Caroline Wozniacki is through to the fourth round after seeing off Aliaksandra Sasnovich 6-4 2-6 6-3, while US Open champion Sloane Stephens lost to Daria Kasatkina 6-4 6-3.
Caroline Garcia beat Daria Gavrilova 7-5 6-4 to set up a match with Angelique Kerber, who beat Elena Vesnina 7-5 6-2.
Wildcard Danielle Collins beat Sofya Zhuk 6-4 6-4 and will face Carla Suarez Navarro, who beat No 4 seed Elina Svitolina 7-5 6-3.