The 3D Mario games have always had a 'look' to them -- kind of a house-style of bright colours, simple textures, and smooth geometry. In most of them the worlds are even made almost exclusively out of 'Mario gameplay blocks' -- things which are obviously ramps and conveyers and blocks and so forth, without any real attempt at disguise.
The desire for the player to understand those 'gameplay blocks' clearly informed the classic 3D Mario art style. But those 'worlds' look like levels or stages. Odyssey by contrast is deliberately subverting our expectations of what an environment in a Mario game will look like. This is played for comic effect, but it also makes sense from the perspective of introducing the mechanic of 'discovering' the those 'gameplay blocks' in a more emergent-seeming way. Nintendo take a gameplay-first, art-second approach to design and I would guess this factored in to their decision to introduce a variety of art styles in Odyssey. An open world can't feel like a stage to run through, with all its core gameplay elements signposted. It wouldn't be as fun!
EDIT: I can see that this subversion won't be everyone's cup of tea though. They didn't necessarily have to go 'more realistic' (and I think to be fair they clearly didn't in all the levels!). What they did have to do is go 'more detailed'/'organic', and I think if that's the problem people have, then it's fine to think that doesn't work in a Mario game. I disagree - I really like the vegetable world in particular
- but I can understand why someone might think it.