When the djay/vjay master out is set to the controller, the volume doesn’t change even when I operate the Master Volume control on the djay/vjay’s GUI.

This is due to the specifications of djay/vjay. Adjust the volume on the DDJ-ERGO itself. Set the volume in the djay/vjay application to the maximum setting, then use the MASTER VOLUME control on the DDJ-ERGO to adjust the volume.