The Supreme Court on Monday rejected an appeal seeking to give people born in American Samoa U.S. citizenship.

In leaving in place an appeals court decision, the court also passed up an invitation to overturn a series of decisions dating back to 1901 known as the Insular Cases, replete with racist and anti-foreign rhetoric. Justice Neil Gorsuch had called for the cases to be overturned in April.

But the justices refused to take up an appeal from people born in American Samoa, and living in Utah, who argued that a federal law declaring that they are “nationals, but not citizens, of the United States at birth” is unconstitutional.

A trial judge in Utah ruled in their favor, but the federal appeals court in Denver said Congress, not courts, should decide the citizenship issue. The appeals court also noted that American Samoa’s elected leaders opposed the lawsuit for fear that it might disrupt their cultural traditions.

American Samoa is the only unincorporated territory of the United States where the inhabitants are not American citizens at birth.

Instead, those born in the cluster of islands some 2,600 miles (4,184 kilometers) southwest of Hawaii are granted “U.S. national” status, meaning they can’t vote for U.S. president, run for office outside American Samoa or apply for certain jobs. The only federal election they can cast a vote in is the race for American Samoa’s nonvoting U.S. House seat.

The Insular Cases, which arose following the Spanish-American War, dealt with the administration of overseas territories.

In their conclusion that residents of territories had some, but not all, rights under the Constitution, justices wrote in stark racial and xenophobic terms. Citizenship could not be automatically given to “those absolutely unfit to receive it,” one justice wrote.

That history prompted Gorsuch to comment in a case involving benefits denied to people who live in Puerto Rico, decided in April. He wrote that the Insular Cases were wrongly decided because they deprived residents of U.S. territories of some constitutional rights.