The Swedes thrashed Tunisia by the same scoreline to launch their campaign, but came crashing back down to Earth in Houston.
Potter's men, who would have reached the last 32 with a win, were 4-0 down after two goals each from Brian Brobbey and Cody Gakpo.
Anthony Elanga pulled one back before Crysencio Summerville put the seal on a five-star display for Ronald Koeman's side.
"We have to congratulate them, and as I said, I think we'll learn a lot from the game," said the Englishman Potter.
"The scoreline affects you; we became a bit passive then. We didn't quite have the belief in what we wanted to do.
"So that was a disappointment. It was a learning point for us, of course, we'll be better for it."
Potter, whose team are still in with a decent shout of the next stage but face a dangerous Japan next, felt the final scoreline was "harsh."
Sweden had their chances in both halves, racking up 20 shots in total, nine of them on target.
But they were shaky at the back throughout, with Sunderland striker Brobbey needing just five minutes to open the scoring.
Sweden were 2-0 down at half-time, but Potter saw positive signs in the second as they had numerous goal-scoring chances come and go.
'They deserved to win'
"I thought the reaction after the half-time break was really, really good. I thought we were the better team, actually," he said.
"But unfortunately, we conceded again and, obviously, if you concede goals, it's a problem.
"So the scoreline, I think, was a bit harsh on us. That's not to say that we deserve to win the game.
"The Netherlands played well, and they deserve to win.
"But we have to take the positives, I think, and we have to learn from the things that we can do better."